This release is a MRU release and is recommended if you are being affected by bugs which are fixed in this release. A number of server crashes have been addressed. Details for changes in this release can be found in the MySQL Enterprise 5.0.74 release notes.

For more information about the different MySQL release version numbers, see our previous post on version numbers.

This release is a MRU release and is recommended if you are being affected by bugs which are fixed in this release. A number of server crashes have been addressed. Details for changes in this release can be found in the MySQL Enterprise 5.0.72 release notes.

For more information about the different MySQL release version numbers, see our previous post on version numbers.

The event has been organised by Sun’s Startup Essentials team, and is aimed at young companies looking to tackle their growing pains head on. I will be alongside speakers from Last.fm, Sun and Fav.or.it.

Lots of people have registered for the event, but as far as I know there are some places still available - you can sign up at the Make It Scale site.

This release is a MRU release and is recommended if you are being affected by bugs which are fixed in this release. A number of server crashes have been addressed. Details for changes in this release can be found in the MySQL Enterprise 5.0.70 release notes.

For more information about the different MySQL release version numbers, see our previous post on version numbers.

When using mysqldump for backups with a MySQL replication slave, the master-data option is often required in order to save the slave replication log file name and position in the backup. By default, the master-data option will execute a FLUSH TABLES WITH READ LOCK command—which disallows writes to any table—and will keep the read lock for the entire duration of the dump to ensure that the replication position won’t change during the dump. This can be problematic, because any clients trying to write to tables during the dump process are locked out. With MyISAM, this is necessary, but with InnoDB it isn’t.

With a little help from InnoDB’s transaction and multi-versioning support, it’s possible to keep the read lock for a very short period of time—just long enough to quiesce the system and check the replication status. InnoDB guarantees that a transaction will see a consistent view of the database after a transaction has started. If all of the tables being dumped are InnoDB, using the single-transaction flag to mysqldump will instruct it to only keep a read lock long enough to gather the replication state and start a transaction. After the transaction has been started, the read lock is released so that the dump and other clients can continue without locking each other out.

On the topic of consistency for a dump, there is an important distinction between the lock-all-tables and lock-tables options: With the lock-all-tables option, mysqldump will use the same type of global read lock it uses when gathering the replication position. This can be used to ensure a consistent dump when not using replication. The lock-tables option, however, issues a LOCK TABLES query for each database it’s dumping. If you choose multiple databases to dump, each database will be locked and unlocked in turn. This introduces a race condition: modifications can be made to database A while database B is locked and being dumped. If you’re using InnoDB, single-transaction is a much better option.

Most people worry about the total time it takes to perform a backup, but the most important timing aspect of any backup is not the time it takes to create the backup, but the time it takes to restore it. Nonetheless, Maatkit includes mk-parallel-dump and mk-parallel-restore which offer a nice speed boost over the traditional MySQL tools.

Deploying servers in this way enables one half of the pair to be taken offline for maintenance work while the other half carries on dealing with queries from clients — meaning that, for instance, lengthy ALTER TABLE operations can be done with no impact on service. This strategy has been in use at many sites for years, and has been very successful at minimizing downtime.

The usual way of implementing this model is to have IP addresses floating between the two MySQL servers. Rather than having the clients use the actual IP addresses or hostnames of the servers themselves, these “floating IPs” (or “virtual IPs”, “VIPs”, “IP aliases”) are used by clients to access MySQL based on a role (typically “writable” and “read-only”). The floating IP addresses can be moved between the servers as required to ensure that each role is always available.

There are some tools already available to manage pairs of MySQL servers, most notably mysql-master-master (MMM) and the High Availability Linux project.

Today, we’re announcing the release of Flipper, a tool for managing access to MySQL servers using master-master replication.

Reinventing the wheel?

Although the existing tools have worked well for some people in some situations, we (and our customers) have been frustrated by the number of situations where they’re not suitable.

Most of the existing tools are specific to Linux, and therefore no good to users of Solaris, FreeBSD and other operating systems. Most are heavy-weight implementations, with monitoring daemons running all the time. Configuration is rarely simple, sometimes because some of the available solutions try to be all things to all men, doing things that would be better handled elsewhere.

A lot of the effort that’s been put into other tools has been aimed at implementing automatic failover. Sometimes this can be very useful (for instance in stateless applications, restartable services, etc.), but very often this is implemented with little consideration for the possible consequences in a stateful, database environment.

Bringing failed servers back into service prematurely (as often happens with hardware load-balancing solutions) can be disastrous, with bad data being returned to clients, or data received from clients and theoretically committed being lost. Likewise, servers may be incorrectly diagnosed as having failed, causing a painful, lengthy, and potentially irreversible failover process to take place for what should have been a barely noticeable event. In some cases, an automatic failover system may change its mind back and forth, causing repeated failover events (known as “flapping”). All in all, we’re not convinced that completely automatic failover is always a good idea¹.

Automated, but manually triggered

Flipper’s design comes from a very pragmatic perspective. It’s a standalone tool that doesn’t require constantly running monitoring daemons — it evaluates the current situation at the moment that it’s executed, and does only what it’s told. It doesn’t attempt to do anything fancy right now; it just manages moving IP addresses between MySQL nodes and reconfiguring a typical master-master setup, in a safe, controlled manner. If one of the MySQL masters fails, it will allow you to move services away from the failed master, enabling you to fix the failure.

Flipper has been designed to be as portable as possible. It’s capable of running on almost any UNIX-like operating system, as it’s written in Perl and uses DBD::mysql to communicate with MySQL servers. Flipper itself doesn’t necessarily require any special privileges, user accounts, or daemons; it uses ssh and sudo to run system commands (and you’d typically want to set up SSH keys, and use ssh-agent to avoid typing your passphrase so many times).

We will add additional features in the future, but the system will always remain modular — you’ll be able to use whichever parts of it you want.

Where can I find out more?

We’ve set up a new (and currently rather minimal) micro-site for Flipper at provenscaling.com/software/flipper with documentation and links to various resources.

We will also post on this blog when there’s a new release, or some other important Flipper-related news.

¹ Of course, we’d be delighted to hear from anyone who wants to try and convince us that any of the current MySQL automatic failover/HA strategies are error-proof, or anyone who’s got new ideas about how this can be achieved.

In order to demonstrate this, let’s first create a few stored procedures that call each other in order to make things interesting:

use test;

delimiter ;;
drop procedure if exists s_test;;
create procedure s_test ()
begin
  call s_test_foo();
end;;

drop procedure if exists s_test_foo;;
create procedure s_test_foo ()
begin
  call s_test_bar();
end;;

drop procedure if exists s_test_bar;;
create procedure s_test_bar ()
begin
  select 1 from blarbo;
end;;

delimiter ;

Notice that the final stored procedure tries to select from a table that we haven’t mentioned yet; that’s because that table doesn’t exist. This should generate an error for us reliably. Here’s what happens when we call the stored procedure:

call test.s_test();

ERROR 1146 (42S02): Table 'test.blarbo' doesn't exist

Now that’s not very helpful: the user didn’t try to do anything with test.blarbo so he has no idea what has just happened! Technically it would be much better if the s_test_bar procedure could have generated a custom error message… I’ll leave that for a future post.

In the past I’ve discussed a patch to add a CALLER() function, and this patch is a natural evolution of that code. We’ve written a proof of concept patch to add a SHOW ERROR STACK TRACE command. This works by saving the call stack, which is tracked using the code from the old CALLER() patch, whenever an error occurs. Let’s try it out, after receiving any error from a SQL statement, you may run SHOW ERROR STACK TRACE as follows:

SHOW ERROR STACK TRACE;

+-------+----------------------+
| Depth | Query                |
+-------+----------------------+
|     0 | select 1 from blarbo |
|     1 | call s_test_bar()    |
|     2 | call s_test_foo()    |
|     3 | call test.s_test()   |
+-------+----------------------+
4 rows in set (0.00 sec)

That’s certainly more useful! I’ve still got a few ideas on how to improve it even more:

• Add something like e.g. SHOW ERROR VARIABLES FOR 1 which could return the contents of all local variables from a particular stack level from the stack trace.
• Add something like e.g. SHOW ERROR CONTEXT FOR 1 which could return the few lines of code before and after the error from a particular stack level from the stack trace.

Essentially, it would be great if you could do most of what gdb is capable of when debugging C code. I hope you like it!

It’s often assumed that changing global variables in MySQL means that the value changes for each connection. However, in the case of max_allowed_packet and many others, the local setting of the variable is inherited from the global variable when a connection is first established. This has special implications when using connection pooling.

In a connection pooling environment, connections are pooled on the application server. When the application needs a connection to the MySQL server, it borrows it from the pool, and returns it when finished. Typically when a connection is borrowed from the pool, the connection pool library automatically calls its API’s variant of the “change user” API function (which uses the COM_CHANGE_USER protocol command) to reset some connection specific parameters.

The session-level max_allowed_packet setting for a connection is normally inherited from the global setting when a client first connects, but it isn’t copied when the user is changed. This means that even though the global max_allowed_packet may have been changed, the applications which are using those connections may not pick up the change until the connection pool’s connections are forced to reconnect, usually by restarting the application server.

A good analogy to this cold startup process is that of starting from a dead stop in manual transmission car: While 5th gear is great for cruising at highway speeds, it’s not really suited for getting you up to speed; that’s what 1st through 4th gear are for. In most cars that aren’t too underpowered, with enough finessing and cajoling you can start from a dead stop in any gear… revving the engine up high to keep it from stalling, and gently easing the clutch in to get rolling… but it’s not pleasant. Besides the unpleasantness, it also takes much longer to get up to highway speeds compared to progressing through the gears in the normal way. Additionally, you may start off rolling just fine, but some time before you’ve gotten up to speed, you hit a small hill—which in the correct gear would be no problem. In the wrong gear, the car slows down, starts chugging, and eventually the engine stalls.

Most database systems behave much like cars with only 5th gear. When tuned properly and appropriately sized, they cruise along perfectly well at “highway speeds”; when the caches are “hot” and finely tuned. They are, in the end, largely designed to operate from an in-memory cache; cache misses, and thus disk reads, should be a relative rarity. If you have too many cache misses, performance is terrible, and the system gets backed up and potentially overloaded. We’ve seen all too many times the situation that occurs when a server (say, a read slave from a pool of many) is rebooted with cold caches, and then immediately put into service in a busy environment. All hell breaks loose.

I have written a proof of concept patch (for InnoDB plugin for MySQL 5.1) which adds a facility (admittedly quite hacky in this draft) to save the list of cached pages to a text file on shutdown. Then, on startup, it restores those pages back into the cache. Since it saves the pages by page number, rather than by saving the contents of the pages, it is perfectly safe¹ and fairly efficient². Note that you should probably also use the information_schema patch to be able to see that this patch is actually working.

While this feature isn’t nearly as sexy to show demos of in a static blog, it has a lot of potential in a production environment. I intend to do a lot more development on it, to provide at least:

• A facility for changing the file name/path for the save file
• The ability to skip the restore step, optionally
• The ability to restore only part of the cache before accepting user connections, deferring the remainder of the cache to a background thread which continues running after startup

Some other ideas I’ve had:

• Use multiple threads to read the pages, which should make much better use of RAIDs
• Add runtime commands to save/restore the buffer pool state from a running server, without shutting down
• Add an option to save some of the statistics about the pages so that they can be restored “smarter” rather than just have all pages stuffed back in the cache with equal priority

Does this sound like a useful feature? What other features should it have? Do you have problems with cold startup in production?

¹ In the worst case, if the file is complete gibberish, the requested pages either won’t exist, or will contain irrelevant data. Either case should do no harm. It wouldn’t be a bad idea to do some more niceties around the file format: a checksum, magic at the beginning of the file, etc. The current file format especially is very much a proof of concept.

² There’s still a lot of work to do to make things more efficient: ordering pages, using multiple threads to read from different tablespaces, subdividing reads for a tablespace into multiple threads, the possibilities are endless.