Every release, there's at least one bug that affects a significant number of users. Most of the time, we get lucky, and the users can get the system installed, and we can fix the problem in an update. The worse-case scenario however is the case where a bug prevents installation, or prevents booting post-install.

F10 had a bug just like this.

It manifested in several different ways.
For some people, it result in a 10 second pause during boot up. For others, it failed to find the root filesystem, even after waiting.

What happened here?

The initrd has code to wait for the kernel to finish enumerating devices. This is an important thing to do so we don't try anything silly like trying to mount the root filesystem before the driver for the disk controller has found the disks plugged into it.

In F9 and earlier, we would emit this code when the initrd included the scsi_mod module.

Late in the F10 development, we moved a bunch of things to be built-ins, to tweak the boot-up speed.
This turned out to be a really bad idea, as it meant the 'stabilization' code didn't get emitted. Ouch.

How come it didn't affect everyone? Different controllers behave differently. Some initialise so quickly that there's no reason to wait any longer.

Why didn't it get caught during testing? We did have a large number of reports, but the correlation wasn't made until post-release.

Should it have held up the release? Possibly, if we knew it would affect so many people.

FTR: None of the hardware I personally tested was affected by this problem, which made it a real pain to track down.

The good news is that in the hopefully not to distant future, all this code is going the way of the dodo anyway.

Hacking on the new initrd stuff with Jeremy..

"Yeah, you probably don't need those sound/ modules in the initrd"
"Dude, music whilst you boot!"
"We could have a contest to have people submit a 30 second boot-time song"

Yeah. Maybe this idea won't make the cut.

Some of the more sensible ideas in a post later. In the meantime, some rough notes.

After reading some of the comments at lwn about my sessions last week at kernel summit/plumbers conf about writing a 'make initrd' target for the kernel that would work on every distro, I feel some clarifications are in order.

What I am not doing.

• Dropping the Fedora mkinitrd script into the kernel.org tree and hoping other distros will make it work for them.
  
• Obsoleting the existing mkinitrd from Fedora for a considerable amount of time. (Think RHEL7 timeframe).
  

What I am doing.

• Right now, nothing but decompressing. Last week was pretty intense, with a lot of feedback to take in.

What I've done

• I've looked at other distros initrd's & creation tools. They all suck equally. Really. If you disagree, you either haven't looked hard enough, or you have emotional attachment issues. There is really nothing amazing about your distros tools over any others. (Which is why I'm of the opinion that "do it right, once, upstream" is the right answer.
  
• Written very little code. A few dozen lines of shell on the planeride home. I'm not posting anything until it at least is limping[1] along.
  

What I'll do next

• Yes, I'm starting afresh. I'll be borrowing some bits from various distros initrd's as I stumble into "hmm, how _does_ this currently work" territory, but this is going to be done with baby steps, first of all just booting off simple setups with /dev/[hs]da, no root on nfs/iscsi/nbd etc.
  
• At least in part, I'm starting with something completely new for political reasons. It's the only way to get anyone to agree on how to make progress. With everyone so invested in their current working solutions, "just use the debian one" or "just use the fedora one" isn't going to get us anywhere.
  

How is this going to work?

• For Fedora systems, we'll be building an additional initrd. Currently we only build it at kernel RPM install time, tailored for that system. Going forward, a one-size-fits-all initrd will be made during the package build process. Which gets used will depend on an /etc/sysconfig/kernel/ setting. This way, we can continue to use the crufty mkinitrd until this thing is ready, and the bleeding edge lunatics and people actually wanting to work on this can set the variable and have the new hotness.
  
• The very first step however is recovering from some of the fallout of the boot/init session of plumbersconf. Reviewing our CONFIG options. It's likely that use of 'make initrd' will imply that certain options should be set certain ways.
  

Once I'm feeling a little more on my feet[2], I'll make some more posts on what I intend to do, and how. Until then, continue to speculate wildly.


[1] Trapped sciatic nerves are the _worst_.
[2] Seriously. Screw flying anywhere for a while. I'm staying home.