Proven Scaling - Group Blog

At Proven Scaling, we’re great fans of using pairs of MySQL servers replicating to each other (commonly known as master-master replication or dual-master replication) as a way of ensuring high availability for MySQL databases.

Deploying servers in this way enables one half of the pair to be taken offline for maintenance work while the other half carries on dealing with queries from clients — meaning that, for instance, lengthy ALTER TABLE operations can be done with no impact on service. This strategy has been in use at many sites for years, and has been very successful at minimizing downtime.

The usual way of implementing this model is to have IP addresses floating between the two MySQL servers. Rather than having the clients use the actual IP addresses or hostnames of the servers themselves, these “floating IPs” (or “virtual IPs”, “VIPs”, “IP aliases”) are used by clients to access MySQL based on a role (typically “writable” and “read-only”). The floating IP addresses can be moved between the servers as required to ensure that each role is always available.

There are some tools already available to manage pairs of MySQL servers, most notably mysql-master-master (MMM) and the High Availability Linux project.

Today, we’re announcing the release of Flipper, a tool for managing access to MySQL servers using master-master replication.

Reinventing the wheel?

Although the existing tools have worked well for some people in some situations, we (and our customers) have been frustrated by the number of situations where they’re not suitable.

Most of the existing tools are specific to Linux, and therefore no good to users of Solaris, FreeBSD and other operating systems. Most are heavy-weight implementations, with monitoring daemons running all the time. Configuration is rarely simple, sometimes because some of the available solutions try to be all things to all men, doing things that would be better handled elsewhere.

A lot of the effort that’s been put into other tools has been aimed at implementing automatic failover. Sometimes this can be very useful (for instance in stateless applications, restartable services, etc.), but very often this is implemented with little consideration for the possible consequences in a stateful, database environment.

Bringing failed servers back into service prematurely (as often happens with hardware load-balancing solutions) can be disastrous, with bad data being returned to clients, or data received from clients and theoretically committed being lost. Likewise, servers may be incorrectly diagnosed as having failed, causing a painful, lengthy, and potentially irreversible failover process to take place for what should have been a barely noticeable event. In some cases, an automatic failover system may change its mind back and forth, causing repeated failover events (known as “flapping”). All in all, we’re not convinced that completely automatic failover is always a good idea¹.

Automated, but manually triggered

Flipper’s design comes from a very pragmatic perspective. It’s a standalone tool that doesn’t require constantly running monitoring daemons — it evaluates the current situation at the moment that it’s executed, and does only what it’s told. It doesn’t attempt to do anything fancy right now; it just manages moving IP addresses between MySQL nodes and reconfiguring a typical master-master setup, in a safe, controlled manner. If one of the MySQL masters fails, it will allow you to move services away from the failed master, enabling you to fix the failure.

Flipper has been designed to be as portable as possible. It’s capable of running on almost any UNIX-like operating system, as it’s written in Perl and uses DBD::mysql to communicate with MySQL servers. Flipper itself doesn’t necessarily require any special privileges, user accounts, or daemons; it uses ssh and sudo to run system commands (and you’d typically want to set up SSH keys, and use ssh-agent to avoid typing your passphrase so many times).

We will add additional features in the future, but the system will always remain modular — you’ll be able to use whichever parts of it you want.

Where can I find out more?

We’ve set up a new (and currently rather minimal) micro-site for Flipper at provenscaling.com/software/flipper with documentation and links to various resources.

We will also post on this blog when there’s a new release, or some other important Flipper-related news.

¹ Of course, we’d be delighted to hear from anyone who wants to try and convince us that any of the current MySQL automatic failover/HA strategies are error-proof, or anyone who’s got new ideas about how this can be achieved.