Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 304727

Summary: Serious problem: By default, fdisk creates partitions not aligned to 4K boundaries
Product: Gentoo Linux Reporter: Timothy Miller <theosib>
Component: [OLD] Core systemAssignee: Gentoo's Team for Core System packages <base-system>
Status: RESOLVED FIXED    
Severity: major CC: bugzilla, dogshu, gentoo-bugzilla, gregorcy, kfm, notordoktor, pablog.ubuntu, steeeeeveee, tobias.pal, virtuousfox, web
Priority: High    
Version: unspecified   
Hardware: All   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---

Description Timothy Miller 2010-02-12 14:37:38 UTC
As you know, drives with physical 4KB sectors have come onto the market.  Drives like the WD Caviar Green advertise 512-byte sectors but implement them as 4096-byte sectors physically.  Any 4K cluster write that is not aligned to an 8-logical-sector boundary will incur a significant performance penalty.

I performed an experiment where I wrote random 4K blocks with O_SYNC to the disk at various 512-byte offsets.  Consistently, those aligned on proper 4K boundaries are much faster than those misaligned.

When I got one of these drives, I partitioned the disk with fdisk and used defaults to create the partition.  fdisk positions the first primary partition at an offset of LBA 63.  This is misaligned and will therefore result in all writes to the partition being misaligned.  The real performance hit will not be as severe as the one I demonstrated, but it would be measurable.

Considering how these 4K-sector drives have been appearing on the market and will grow in numbers, the fact that fdisk does the wrong thing is a serious issue that needs to be fixed very quickly.  

This issue has been known about and discussed for well over a year.  It's unfortunate that it's been ignored.  Note that WD claims that Linux is "unaffected" by this issue.  This is likely due to Linux being pretty good about bundling large contiguous writes, but random access is definitely going to be affected.  Also, I read somewhere someone speculating that Linux may do some alignment magic within the partition, but this would violate the partition abstraction, breaking volumes that were migrated from disk to disk at different alignments.  So, no.  Linux is definitely going to suffer on misaligned partitions.

Reproducible: Always
Comment 1 Timothy Miller 2010-02-15 00:36:26 UTC
In this article, I detail some tests I performed to demonstrate the problem:

http://www.osnews.com/story/22872/Linux_Not_Fully_Prepared_for_4096-Byte_Sector_Hard_Drives
Comment 2 Sergey Kondakov 2010-03-02 18:37:00 UTC
i've got WD15EARS HDD and have experienced this issue.

looks like there is no way to go around this issue unless default behaviour for any partitioning software (fdisk\parted\gdisk and their front-ends) is to align beginnings and ends of all partitions by divided-by-8 sector numbers and ignore cylinders for HDDs which reporting 512-byte sectors and manufactured in 2009 and later.

right now problem can be avoided for any quantity of partitons on such devices with util-linux-2.17.1's fdisk: running "fdisk -c -S 56 -u <device>" is a way to go.

there should be no issues with parted-2.2 too but cfdisk and gparted will still screw up stuff.

http://www.osnews.com/thread?409410 <- this an interesting comment on issue from "Martin K. Petersen: Kernel Developer, Oracle Linux Engineering"

hardware vendors really should not make hardware lie about its basic parameters because that way we're all screwed, not only poor bastards with "Win 5.x family"

summary for this bug and many articles are misleading though: fdisk not aligning to 4K is not a problem. problem is that hardware lies to fdisk, system and a user altogether. so, technically, bug is fixed in newer fdisk and should be closed since what making default for it is decision for upstream, and tough one.
what we can do is:

1) persuade upstream of partitioning tools, like fdisk, parted and gdisk, to make such workaround (not a scary thing since most (all?) filesystems use 4096 blocks anyway and some even support tail packing)

2) convince upstream of graphical front-ends, such as gparted and partitionmanager, to provide more aligning options and drop cylinders restriction (gparted still whine about inappropriate partitioning with auto-aligning by cylinders off and refuse to create a parttiion)

3) tell WD and all guys at www.bigsector.org that this ("emulating") was a BAD idea...
Comment 3 Wormo (RETIRED) gentoo-dev 2010-03-14 23:57:12 UTC
Thanks for publicizing this issue, Timothy, I think it has reminded relevant maintainers that it's becoming common now with latest hardware. I already had run across your article via the LWN coverage of this topic BTW.

I'll go ahead and assign this bug to base-system in case somebody there isn't already aware of the issue, but don't be discouraged if they then close it as "upstream" since that's where the action is taking place. They'll bump ebuilds as packages get updated to help users deal better with this partitioning mess.
Comment 4 Paul Tobias 2010-03-30 15:26:50 UTC
A good read in this subject: http://ata.wiki.kernel.org/index.php/ATA_4_KiB_sector_issues
Comment 5 SpanKY gentoo-dev 2010-03-30 21:53:20 UTC
i dont think there's anything that needs to be done in Gentoo specifically
Comment 6 kfm 2011-02-27 15:09:09 UTC
I'd like for this situation to be re-evaluated. Please consider the following points:

*  Now a *year* on, =util-linux-2.17* remains the stable release in portage.
*  util-linux-2.17.2 does not disable DOS compatibility mode by default.
*  Thus, to this day, anyone with a 4K sector hard drive who uses the gentoo 
   media and follows the handbook may be affected (and, worse, may not
   realise it).
*  The handbook does nothing to mitigate this and general public awareness
   of the issue appears to be limited.
*  Several high-profile distros have treated this issue with the seriousness
   it deserves and taken remedial steps with respect to their installation
   processes.

I filed bug 344753 asking for the simplest thing: to change the handbook so as to instruct the user to use the "-c" parameter. Why? Because it turns off DOS 'compatibility' mode which solves the issue in the 2.17 branch and is a no-op in the 2.18 branch (because it's already a default there). In other words, by adding these two characters, the whole issue could be put to bed and nobody would get hurt. However, I was essentially given three reasons as to why this should not be done:

1) A claim was made that the newer version would be stabilised soon anyway.
   well, I checked for a stabilisation bug at the time and saw nothing and -
   sure enough - it hasn't happened.

2) There was talk of not wanting to have to add -c then "have" to remove it
   again later. This is a strawman as specifying -c does no harm under any
   circumstances. In the newer branches, it's a default ... that's fine.

3) It is stated not to "impede" the installation process. Sure, except that
   the *result* of the installation process could yield a system with up to
   a 230% loss in filesystem performance and no way to fix it without
   blowing away the existing partitions and starting again.

Microsoft had this fixed as of Vista SP1 [1]. Fedora had this fixed as of 13 [2]. Ubuntu had this fixed as of Lucid Lynx [3]. I'm sure there are plenty of counter-examples and, frankly, it's sad to see how slow on the uptake some distros are. Just because Gentoo doesn't have an installer per se does not change the fact that the steps which are prescribed by the documentation effectively constitute the installation process and that the ramifications for the user should not be borne in mind.

Now, getting util-linux-2.18 stabilised and pushed out to the installation media would be great but can we not just prescribe the -c parameter and be done with it? Even upon the installation media being fixed (whenever that will be), it's not necessarily the case that everyone performing an install will be using said media. Why not play it safe?

[1] http://www.seagate.com/docs/pdf/whitepaper/tp613_transition_to_4k_sectors.pdf
[2] https://bugzilla.redhat.com/show_bug.cgi?id=574220
[3] https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/530071
Comment 7 SpanKY gentoo-dev 2011-03-01 07:59:21 UTC
ive filed a bug to stabilize 2.18