automaticable

Ubuntu 9.04 “Jaunty Jackelope” is set for release in less than two months, and is currently in the “Alpha 6″ testing stage.  Arguably the hottest new feature is stable support for Linux’s new filesystem, ext4.  The new filesystem has been shown to have significant performance gains over any other Linux filesystem to date.  But think twice before you upgrade to ext4–many users have been reporting considerable data loss.  From a user’s comment:

This has happened to me twice, the first time erasing Firefox settings, and the second time erasing gnome-terminal settings. Both cases were caused by a kernel panic locking up the system. Also, both times the program whose settings were affected was in use during the kernel panic.

This has been a wide-spread problem, particularly for Ubuntu users testing the new Jaunty alpha releases.  It manifests itself after a system crash, where open files will simply be emptied and erased.  It is particularly prevalent in users’ settings files, such as those used for GNOME or KDE.  Fortunately, there cause of the bug is relatively well-known, as kernel developer Theodore Ts’o explains in his detailed comment:

OK, so enter ext4 and delayed allocation. With delayed allocation, we don’t allocate a location on disk for the data block right away. Since there is no location on disk, there is no place to write the data on a commit; but it also means that there is no security problem. It also results in a massive performance improvements; for example, if you create a scratch file, and then delete it 20 seconds later, it will probably never hit the disk. Unfortunately, the default VM tuning parameters, which can be controlled by /proc/sys/vm/dirty_expire_centiseconds and /proc/sys/vm/dirty_writeback_centiseconds, means that in practice, a newly created file won’t hit disk until about 45-150 seconds later, depending on how many dirty pages are in the page cache at the time. (This isn’t unique to ext4, by the way — any advanced filesystem which does delayed allocation, which includes xfs and the in the future, btrfs, will have the same issue.)

Essentially, the data for truncated and overwritten files is often cached in memory before actually hitting the disk, for performance reasons.  However, the file truncation doesn’t get cached, and if there is a system failure between the two operations, your files will simply be wiped clean with all previous and new data gone.  This happens frequently for desktop settings because hundreds of small hidden files are updated to maintain user preferences.  As a result, there is a relatively high probability that a truncation and rewrite operation could be happened at any time– including during a system failure.

By POSIX definition, this isn’t really a bug in ext4.  The state of a file after this type of crash isn’t clearly defined.  According to Theodore, it’s really a bug in the way desktop environments are writing their settings files:

[...] If your application is stupid, and has hundreds of dot files in your home directory, each one taking up a 4k disk block even though it is only storing 4 to 12 bytes of data in each singleton dot file, and you have to [write to disk] for each of your one hundred dot files — and worse yet, your application for some stupid, unknown reason is writing all of these hundred+ dot files every few seconds, then [writing to disk] will be very painful. But it is painful because the application is stupidly written — not for any fundamental filesystem fault. It’s like if you had a robot which was delivering mail to mail box numbers 1, 2, 3, 4, 5, and crossing the street for each mail box; on a busy road, this is unsafe, and the robot was getting run over when it kept on jaywalking — so you can tell the robot to only cross at crosswalks, when the “walk” light is on, which is safe, but slow — OR, you could rewrite the robot’s algorithsm so it delieveres the mail more intelligently (i.e., one side of the street, and then cross, safely at the crosswalk, and then do the other side of the street).

Regardless of what a permanant solution should include, there are a series of patches which are set to be included in the final Jaunty kernel to address this issue.  The patches are set to handle the crash situation similar to ext3– keeping the original version in-tact.  However, it comes at a price of performance.  Ext4 won’t be any slower than ext3, but the patch will negate many of the performance gains that ext4 previously achieved.

Once again, ext4 will likely be patched and fixed by the official Ubuntu Jaunty release.  However, it isn’t present in Jaunty Alpha 6, and hasn’t been pushed as an update yet (as of 3/12/09).  There is lengthy and ongoing discussion in the original bug report about possible long-term solutions.  In conclusion, be very careful if you choose to test out the new ext4 filesystem until it is patched.

If you're new here, you can subscribe to automaticable's RSS feed by clicking here. Or, you can get post updates through your email. Thanks for visiting!