This release is a MRU release and is recommended if you are being affected by bugs which are fixed in this release. A number of server crashes have been addressed. Details for changes in this release can be found in the MySQL Enterprise 5.0.70 release notes.

For more information about the different MySQL release version numbers, see our previous post on version numbers.

When using mysqldump for backups with a MySQL replication slave, the master-data option is often required in order to save the slave replication log file name and position in the backup. By default, the master-data option will execute a FLUSH TABLES WITH READ LOCK command—which disallows writes to any table—and will keep the read lock for the entire duration of the dump to ensure that the replication position won’t change during the dump. This can be problematic, because any clients trying to write to tables during the dump process are locked out. With MyISAM, this is necessary, but with InnoDB it isn’t.

With a little help from InnoDB’s transaction and multi-versioning support, it’s possible to keep the read lock for a very short period of time—just long enough to quiesce the system and check the replication status. InnoDB guarantees that a transaction will see a consistent view of the database after a transaction has started. If all of the tables being dumped are InnoDB, using the single-transaction flag to mysqldump will instruct it to only keep a read lock long enough to gather the replication state and start a transaction. After the transaction has been started, the read lock is released so that the dump and other clients can continue without locking each other out.

On the topic of consistency for a dump, there is an important distinction between the lock-all-tables and lock-tables options: With the lock-all-tables option, mysqldump will use the same type of global read lock it uses when gathering the replication position. This can be used to ensure a consistent dump when not using replication. The lock-tables option, however, issues a LOCK TABLES query for each database it’s dumping. If you choose multiple databases to dump, each database will be locked and unlocked in turn. This introduces a race condition: modifications can be made to database A while database B is locked and being dumped. If you’re using InnoDB, single-transaction is a much better option.

Most people worry about the total time it takes to perform a backup, but the most important timing aspect of any backup is not the time it takes to create the backup, but the time it takes to restore it. Nonetheless, Maatkit includes mk-parallel-dump and mk-parallel-restore which offer a nice speed boost over the traditional MySQL tools.

Deploying servers in this way enables one half of the pair to be taken offline for maintenance work while the other half carries on dealing with queries from clients — meaning that, for instance, lengthy ALTER TABLE operations can be done with no impact on service. This strategy has been in use at many sites for years, and has been very successful at minimizing downtime.

The usual way of implementing this model is to have IP addresses floating between the two MySQL servers. Rather than having the clients use the actual IP addresses or hostnames of the servers themselves, these “floating IPs” (or “virtual IPs”, “VIPs”, “IP aliases”) are used by clients to access MySQL based on a role (typically “writable” and “read-only”). The floating IP addresses can be moved between the servers as required to ensure that each role is always available.

There are some tools already available to manage pairs of MySQL servers, most notably mysql-master-master (MMM) and the High Availability Linux project.

Today, we’re announcing the release of Flipper, a tool for managing access to MySQL servers using master-master replication.

Reinventing the wheel?

Although the existing tools have worked well for some people in some situations, we (and our customers) have been frustrated by the number of situations where they’re not suitable.

Most of the existing tools are specific to Linux, and therefore no good to users of Solaris, FreeBSD and other operating systems. Most are heavy-weight implementations, with monitoring daemons running all the time. Configuration is rarely simple, sometimes because some of the available solutions try to be all things to all men, doing things that would be better handled elsewhere.

A lot of the effort that’s been put into other tools has been aimed at implementing automatic failover. Sometimes this can be very useful (for instance in stateless applications, restartable services, etc.), but very often this is implemented with little consideration for the possible consequences in a stateful, database environment.

Bringing failed servers back into service prematurely (as often happens with hardware load-balancing solutions) can be disastrous, with bad data being returned to clients, or data received from clients and theoretically committed being lost. Likewise, servers may be incorrectly diagnosed as having failed, causing a painful, lengthy, and potentially irreversible failover process to take place for what should have been a barely noticeable event. In some cases, an automatic failover system may change its mind back and forth, causing repeated failover events (known as “flapping”). All in all, we’re not convinced that completely automatic failover is always a good idea¹.

Automated, but manually triggered

Flipper’s design comes from a very pragmatic perspective. It’s a standalone tool that doesn’t require constantly running monitoring daemons — it evaluates the current situation at the moment that it’s executed, and does only what it’s told. It doesn’t attempt to do anything fancy right now; it just manages moving IP addresses between MySQL nodes and reconfiguring a typical master-master setup, in a safe, controlled manner. If one of the MySQL masters fails, it will allow you to move services away from the failed master, enabling you to fix the failure.

Flipper has been designed to be as portable as possible. It’s capable of running on almost any UNIX-like operating system, as it’s written in Perl and uses DBD::mysql to communicate with MySQL servers. Flipper itself doesn’t necessarily require any special privileges, user accounts, or daemons; it uses ssh and sudo to run system commands (and you’d typically want to set up SSH keys, and use ssh-agent to avoid typing your passphrase so many times).

We will add additional features in the future, but the system will always remain modular — you’ll be able to use whichever parts of it you want.

Where can I find out more?

We’ve set up a new (and currently rather minimal) micro-site for Flipper at provenscaling.com/software/flipper with documentation and links to various resources.

We will also post on this blog when there’s a new release, or some other important Flipper-related news.

¹ Of course, we’d be delighted to hear from anyone who wants to try and convince us that any of the current MySQL automatic failover/HA strategies are error-proof, or anyone who’s got new ideas about how this can be achieved.

In order to demonstrate this, let’s first create a few stored procedures that call each other in order to make things interesting:

use test;

delimiter ;;
drop procedure if exists s_test;;
create procedure s_test ()
begin
  call s_test_foo();
end;;

drop procedure if exists s_test_foo;;
create procedure s_test_foo ()
begin
  call s_test_bar();
end;;

drop procedure if exists s_test_bar;;
create procedure s_test_bar ()
begin
  select 1 from blarbo;
end;;

delimiter ;

Notice that the final stored procedure tries to select from a table that we haven’t mentioned yet; that’s because that table doesn’t exist. This should generate an error for us reliably. Here’s what happens when we call the stored procedure:

call test.s_test();

ERROR 1146 (42S02): Table 'test.blarbo' doesn't exist

Now that’s not very helpful: the user didn’t try to do anything with test.blarbo so he has no idea what has just happened! Technically it would be much better if the s_test_bar procedure could have generated a custom error message… I’ll leave that for a future post.

In the past I’ve discussed a patch to add a CALLER() function, and this patch is a natural evolution of that code. We’ve written a proof of concept patch to add a SHOW ERROR STACK TRACE command. This works by saving the call stack, which is tracked using the code from the old CALLER() patch, whenever an error occurs. Let’s try it out, after receiving any error from a SQL statement, you may run SHOW ERROR STACK TRACE as follows:

SHOW ERROR STACK TRACE;

+-------+----------------------+
| Depth | Query                |
+-------+----------------------+
|     0 | select 1 from blarbo |
|     1 | call s_test_bar()    |
|     2 | call s_test_foo()    |
|     3 | call test.s_test()   |
+-------+----------------------+
4 rows in set (0.00 sec)

That’s certainly more useful! I’ve still got a few ideas on how to improve it even more:

• Add something like e.g. SHOW ERROR VARIABLES FOR 1 which could return the contents of all local variables from a particular stack level from the stack trace.
• Add something like e.g. SHOW ERROR CONTEXT FOR 1 which could return the few lines of code before and after the error from a particular stack level from the stack trace.

Essentially, it would be great if you could do most of what gdb is capable of when debugging C code. I hope you like it!

It’s often assumed that changing global variables in MySQL means that the value changes for each connection. However, in the case of max_allowed_packet and many others, the local setting of the variable is inherited from the global variable when a connection is first established. This has special implications when using connection pooling.

In a connection pooling environment, connections are pooled on the application server. When the application needs a connection to the MySQL server, it borrows it from the pool, and returns it when finished. Typically when a connection is borrowed from the pool, the connection pool library automatically calls its API’s variant of the “change user” API function (which uses the COM_CHANGE_USER protocol command) to reset some connection specific parameters.

The session-level max_allowed_packet setting for a connection is normally inherited from the global setting when a client first connects, but it isn’t copied when the user is changed. This means that even though the global max_allowed_packet may have been changed, the applications which are using those connections may not pick up the change until the connection pool’s connections are forced to reconnect, usually by restarting the application server.

A good analogy to this cold startup process is that of starting from a dead stop in manual transmission car: While 5th gear is great for cruising at highway speeds, it’s not really suited for getting you up to speed; that’s what 1st through 4th gear are for. In most cars that aren’t too underpowered, with enough finessing and cajoling you can start from a dead stop in any gear… revving the engine up high to keep it from stalling, and gently easing the clutch in to get rolling… but it’s not pleasant. Besides the unpleasantness, it also takes much longer to get up to highway speeds compared to progressing through the gears in the normal way. Additionally, you may start off rolling just fine, but some time before you’ve gotten up to speed, you hit a small hill—which in the correct gear would be no problem. In the wrong gear, the car slows down, starts chugging, and eventually the engine stalls.

Most database systems behave much like cars with only 5th gear. When tuned properly and appropriately sized, they cruise along perfectly well at “highway speeds”; when the caches are “hot” and finely tuned. They are, in the end, largely designed to operate from an in-memory cache; cache misses, and thus disk reads, should be a relative rarity. If you have too many cache misses, performance is terrible, and the system gets backed up and potentially overloaded. We’ve seen all too many times the situation that occurs when a server (say, a read slave from a pool of many) is rebooted with cold caches, and then immediately put into service in a busy environment. All hell breaks loose.

I have written a proof of concept patch (for InnoDB plugin for MySQL 5.1) which adds a facility (admittedly quite hacky in this draft) to save the list of cached pages to a text file on shutdown. Then, on startup, it restores those pages back into the cache. Since it saves the pages by page number, rather than by saving the contents of the pages, it is perfectly safe¹ and fairly efficient². Note that you should probably also use the information_schema patch to be able to see that this patch is actually working.

While this feature isn’t nearly as sexy to show demos of in a static blog, it has a lot of potential in a production environment. I intend to do a lot more development on it, to provide at least:

• A facility for changing the file name/path for the save file
• The ability to skip the restore step, optionally
• The ability to restore only part of the cache before accepting user connections, deferring the remainder of the cache to a background thread which continues running after startup

Some other ideas I’ve had:

• Use multiple threads to read the pages, which should make much better use of RAIDs
• Add runtime commands to save/restore the buffer pool state from a running server, without shutting down
• Add an option to save some of the statistics about the pages so that they can be restored “smarter” rather than just have all pages stuffed back in the cache with equal priority

Does this sound like a useful feature? What other features should it have? Do you have problems with cold startup in production?

¹ In the worst case, if the file is complete gibberish, the requested pages either won’t exist, or will contain irrelevant data. Either case should do no harm. It wouldn’t be a bad idea to do some more niceties around the file format: a checksum, magic at the beginning of the file, etc. The current file format especially is very much a proof of concept.

² There’s still a lot of work to do to make things more efficient: ordering pages, using multiple threads to read from different tablespaces, subdividing reads for a tablespace into multiple threads, the possibilities are endless.

There are three main branches of MySQL being produced by Sun/MySQL: Community, Enterprise MRU, and Enterprise QSP. All Community and Enterprise releases are distributed under the terms of the GPLv2 license. The changes between each release are tracked in the MySQL manual. The difference between these branches are subtle but important.

MySQL Community is a branch of MySQL that is released less frequently, but also contains contributions from developers outside of Sun/MySQL. So far the only notable feature in the Community release is SHOW PROFILE, a query-profiling feature originally developed by Proven Scaling. Unless you want the SHOW PROFILE feature, there is currently no reason to use this release instead of a newer Enterprise release.

MySQL Enterprise MRU (commonly called just MySQL Enterprise) is a “Monthly Rapid Update” that is regularly updated with bug fixes and changes that were developed by MySQL during the previous month. According to MySQL’s guidance, MRU releases should only be used when a critical fix is needed that hasn’t yet been included in a QSP release. MRU releases don’t have any special naming scheme. At the time of this post the version number of the most recent release is 5.0.66a. You can find the source or binaries of this version on the Proven Scaling mirror.

MySQL Enterprise QSP is a “Quarterly Service Pack” which is released once per quarter. This release contains all the changes in the MRU releases since the previous QSP release. QSP releases can also contain critical bug fixes from MRU releases with higher version numbers than the QSP has. For instance, 5.0.56sp1 contains fixes from 5.0.58 — this can make it somewhat more difficult to understand whether or not you’re affected by a particular bug. Other critical fixes may only be in MRU releases until the next scheduled QSP release. You can find the source or binaries of this version on the Proven Scaling mirror.

The format of the release notes pages in the manual has changed between 5.0.56sp1 and 5.0.60sp1. In version 5.0.56sp1 the release notes page contained all the changes in the previous MRU releases since the last QSP. As of 5.0.60sp1 the release notes page¹ contains only the changes back ported from the next MRU release. It can be confusing when comparing release notes between versions. Be sure to read the paragraph at the top of the release notes page to see what is actually contained within.

¹ Note that 5.0.60sp1 has no changes from 5.0.60. It was merely recompiled with a new version number and released as 5.0.60sp1.

Most patches are considered rough draft, proof of concept quality at best. They are suitable for testing, playing around, using them to gain insight about your data, but not for production use. Use at your own risk!

Using information_schema to view the buffer pool contents

This patch creates several new information_schema tables to provide visibility into the contents of the InnoDB buffer pool. The new tables are:

• innodb_buffer_pool_pages
• innodb_buffer_pool_pages_index
• innodb_buffer_pool_pages_blob

Note that accessing these tables is not cheap, as the buffer pool must be scanned.

innodb_buffer_pool_pages

This table provides some generic information about all pages present in the buffer pool, and is primarily useful for getting statistics about the number and types of pages present at any given point in time. For example:

SELECT
  page_type,
  count(*) AS nr
FROM information_schema.innodb_buffer_pool_pages
GROUP BY page_type;

+-----------+------+
| page_type | nr   |
+-----------+------+
| allocated | 4051 |
| bitmap    |    2 |
| blob      |    5 |
| fsp_hdr   |    2 |
| index     |   14 |
| inode     |    2 |
| sys       |    3 |
| trx_sys   |    1 |
| undo_log  |   15 |
+-----------+------+

innodb_buffer_pool_pages_index

This table gives you detailed statistics on index pages, which for InnoDB means all data (as part of the primary key) and all defined indexes. For example (some columns removed for clarity):

SELECT *
FROM information_schema.innodb_buffer_pool_pages_index;

+-------------+------------------+------------+----------+---------+--------+-----------+--------------+...
| schema_name | table_name       | index_name | space_id | page_no | n_recs | data_size | lru_position |...
+-------------+------------------+------------+----------+---------+--------+-----------+--------------+...
| NULL        | SYS_IBUF_TABLE_0 | CLUST_IND  |        0 |       4 |      0 |         0 |            6 |...
| NULL        | SYS_INDEXES      | CLUST_IND  |        0 |      11 |      8 |       536 |            7 |...
| NULL        | SYS_TABLES       | CLUST_IND  |        0 |       8 |      5 |       351 |           12 |...
| NULL        | SYS_COLUMNS      | CLUST_IND  |        0 |      10 |     24 |      1436 |           14 |...
| NULL        | SYS_TABLES       | ID_IND     |        0 |       9 |      5 |       126 |           15 |...
| NULL        | SYS_FIELDS       | CLUST_IND  |        0 |      12 |      8 |       315 |           16 |...
| NULL        | SYS_FOREIGN      | ID_IND     |        0 |      46 |      0 |         0 |           50 |...
| NULL        | SYS_FOREIGN      | FOR_IND    |        0 |      47 |      0 |         0 |           51 |...
| NULL        | SYS_FOREIGN      | REF_IND    |        0 |      48 |      0 |         0 |           52 |...
| NULL        | SYS_FOREIGN_COLS | ID_IND     |        0 |      49 |      0 |         0 |           53 |...
| test        | bt               | PRIMARY    |       26 |       3 |      3 |        39 |           66 |...
| test        | bt               | PRIMARY    |       26 |       7 |      1 |      8124 |           69 |...
| test        | bt               | PRIMARY    |       26 |       9 |      2 |     16248 |           71 |...
| test        | bt               | PRIMARY    |       26 |       6 |      2 |     16248 |           74 |...
+-------------+------------------+------------+----------+---------+--------+-----------+--------------+...

Besides the obvious value of this, notice that you can see InnoDB’s internal tables (with schema_name as NULL) as well!

innodb_buffer_pool_pages_blob

This table gives you detailed statistics on BLOB-storage pages, which store parts of BLOBs that overflow in-row storage (another post on that later, but suffice it to say, if the BLOB is larger than 8KB… sort of). For example:

SELECT *
FROM information_schema.innodb_buffer_pool_pages_blob;

+----------+---------+------------+----------+--------------+--------------+-----------+------------+
| space_id | page_no | compressed | part_len | next_page_no | lru_position | fix_count | flush_type |
+----------+---------+------------+----------+--------------+--------------+-----------+------------+
|       26 |       5 |          0 |     9232 |            0 |           72 |         0 |          0 |
|       26 |      10 |          0 |     9232 |            0 |           73 |         0 |          0 |
|       26 |       8 |          0 |     9232 |            0 |           75 |         0 |          0 |
|       26 |      11 |          0 |     9232 |            0 |           76 |         0 |          0 |
|       26 |       4 |          0 |     9232 |            0 |           77 |         0 |          0 |
+----------+---------+------------+----------+--------------+--------------+-----------+------------+

We’d love to get the table name and PRIMARY KEY value associated with each BLOB-storage page, but I’m not sure if that’s possible.

An example of the power of these tables

Have you ever wondered what your page fill rate is? Probably not, because you didn’t realize it was either interesting or important, but we worry about these things (and often justifiably so). In short, page fill rate is the percentage of each page that is filled with data. Since InnoDB only caches whole pages, if your average page fill rate is 75%, you would be wasting 25% of the space in each page. That would mean that your table is spread across more pages than necessary, which will waste both disk space and memory you’ve allocated to the buffer pool.

Here’s a simple way to figure out the average page fill rate of pages currently in cache:

SELECT
  schema_name,
  table_name,
  index_name,
  COUNT(*) AS n_pages,
  ROUND(AVG(n_recs), 2) AS recs_per_page,
  ROUND(SUM(data_size)/1048576, 2) AS total_data_size_M,
  ROUND(AVG(data_size/(
    SELECT variable_value
    FROM information_schema.global_status
    WHERE variable_name
    LIKE 'innodb_page_size'
  )) * 100, 2) AS page_fill_pct
FROM
  information_schema.innodb_buffer_pool_pages_index
WHERE 1
AND schema_name IS NOT NULL
GROUP BY
  schema_name,
  table_name,
  index_name
ORDER BY
  total_data_size_M DESC;

+-------------+------------+-----------------+---------+---------------+-------------------+---------------+
| schema_name | table_name | index_name      | n_pages | recs_per_page | total_data_size_M | page_fill_pct |
+-------------+------------+-----------------+---------+---------------+-------------------+---------------+
| test        | n          | GEN_CLUST_INDEX |    4079 |        252.55 |             58.56 |         91.88 |
| test        | bt         | PRIMARY         |       2 |          2.50 |              0.02 |         49.70 |
| test        | p          | PRIMARY         |       1 |          2.00 |              0.00 |          0.52 |
| test        | p          | c               |       1 |          2.00 |              0.00 |          0.37 |
+-------------+------------+-----------------+---------+---------------+-------------------+---------------+

This is showing some statistics for each index from each table in the cache: the number of pages in the cache, the average number of records per page for those cached pages, the total number of megabytes of cache used, and the page fill rate in percent. There is currently no other way to calculate this information from any statistics or data that InnoDB provides.

What else?

What ideas do you have? What do you think about these features? Leave a comment to let us know!

A little about Proven Scaling and our history

Proven Scaling was founded by Jeremy Cole and Eric Bergen in May, 2006 in order to provide our skills in MySQL architecture, performance, reliability, and availability to a larger audience. Currently Proven Scaling consists of:

• Jeremy Cole — Jeremy left Yahoo! after 1.5 years, a few months ahead of Eric, to kickstart Proven Scaling in July, 2006. Previous to Yahoo!, Jeremy worked for MySQL AB for 4.5 years as tech writer, trainer, consultant, and occasional developer. Jeremy has a personal blog at jcole.us.
• Eric Bergen — Eric left his 1.5 years at Yahoo! to join Jeremy in September, 2006. Previous to Yahoo!, Eric was responsible for MySQL and related systems at a major domain registrar. Eric has a personal blog at ebergen.net
• Mike Griffiths — Based in London, UK, Mike joined Proven Scaling in November, 2007. Before joining Proven Scaling, Mike worked at Yahoo! for 7 years, responsible for MySQL in Europe, Asia and beyond. Mike has so far steadfastly refused to join the blogosphere, arguing that no-one’s really interested in his thoughts on sport, DIY, perl hacking, warm English beer and the other things which take up his leisure time.

A lot more than consulting…

At Proven Scaling we spend a lot time working on various software projects. Jeremy and Eric have been publishing their own patches to MySQL and other projects for many years. One of the primary reasons for forming a company together in the structure that we have is to ensure that we maintain the hardware resources, ability, and time available to work on these projects and provide them to the public in as quick and efficient manner as possible. In addition to continuing our previous work, Proven Scaling has been working on several new independent projects to solve problems in the MySQL space.

We needed to place to distribute that software along with other widely and not-so-widely available software packages, so we host a software mirror at mirror.provenscaling.com. This site hosts hard to find copies of various software packages we use every day, in addition to RedHat-friendly (and CentOS friendly) yum repositories for many packages. In fact, one of the reasons we are starting this blog is so that we have an official place to keep you updated as to the contents of our mirror site. As we update the mirror with new releases of existing software and entirely new software, we will post entries describing the updates.

What will we blog about?

You can get some idea of the topics you might see here from Jeremy and Eric’s MySQL posts on their personal blogs: MySQL on jcole.us and MySQL on ebergen.net.

In addition to the updates described above, some of the topics we are especially interested in writing about are:

• Announcements of new Proven Scaling software and new releases of our software
• New patches or ideas for patches to MySQL and other software
• Deep dissections of various parts of MySQL and InnoDB
• Knowledge gained from working with our customers, such as advice for real-world scenarios or unexpected situations
• Many more exciting ideas we’re working on, which we will share when they are ready!

Sounds good?

Go ahead and subscribe to our RSS feed, or bookmark this page. Keep an eye on the author names above, so that you know who’s writing.