I decided to move my blog again. It's now a self-hosted wordpress instance at my homepage. Wordpress was deceptively easy to set up. I had imagined it to be a lot more involved. There are many knobs to tweak though, so I'll likely be fiddling with it for months to come. I've still not found a 'theme' I'm happy with, but what's there now will do for the time being. (anyone sensible will read it with an rss reader anyway).

At the same time, I've decided to start blogging about my noise-making habit on a separate site. The separation of Linux nerdery from my modular synth nerdery should appease those who couldn't care less about one or the other. (And for those oddballs that are interested in both, there's this neat thing called RSS).

Why the move? I've been toying with the idea of doing this for a long time, and procrastination was the only real reason I didn't get to it. My web site was looking a little neglected, so it felt like the right time for an overhaul.
Additionally, my livejournal account runs out sometime next year, and I don't think I'll bother renewing it. (It's not like I ever used any of the paidmembers features anyway).

I'll let it revert to a free account, and will still use it to comment on other peoples blogs etc, but I'll likely not post here again after this.

I've been thinking about just how much a nightmare video codecs are lately. The usual case that everyone complains about is the inability for Linux distros to ship various patented codecs. But when I thought about it, the problem is much bigger than Linux.

There's not a single operating system, or media playback device that will reliably play every single format that could be thrown at it. Not even close.

My OSX box gets kinda close thanks to perian, but still fails with many of the wmv variants amongst others. Windows also requires a bunch of apps & codecs installed. It's not just a Linux problem, and in honesty, never has been.

A few nights ago, I rigged up ushare, to see how the PS3 coped with its built-in media player. It failed pretty spectacularly. Other than MPEGs, almost nothing worked.

I've been toying with the idea of getting a PMP for a while, but it's the same story. I'm not particularly interested in re-encoding all my music again, and even more disillusioned at the thought of not being able to actually play most of the videos I have.

Free, open standards like vorbis should make all of this a non-issue. Yet hardware manufacturers seem reluctant to implement them. The workarounds of taking codecs from other OS's and running them in Linux seems to have become 'the norm' for playing back a lot of formats in Linux. This of course works great for x86, but all the embedded variants of Linux I've seen so far in media playback devices are running ARM, or MIPS, which means of course.. no playback of those formats.

I'd like to believe it's a problem that will eventually go away, but each seems to get worse, with yet another format we're unable to play back universally.

initramfs@vger dot kernel dot org now exists. As usual 'subscribe initramfs' in the body to majordomo @ vger.

This list will be for discussing the cross-distro initramfs support I've been suggesting at for the past few months.

Every release, there's at least one bug that affects a significant number of users. Most of the time, we get lucky, and the users can get the system installed, and we can fix the problem in an update. The worse-case scenario however is the case where a bug prevents installation, or prevents booting post-install.

F10 had a bug just like this.

It manifested in several different ways.
For some people, it result in a 10 second pause during boot up. For others, it failed to find the root filesystem, even after waiting.

What happened here?

The initrd has code to wait for the kernel to finish enumerating devices. This is an important thing to do so we don't try anything silly like trying to mount the root filesystem before the driver for the disk controller has found the disks plugged into it.

In F9 and earlier, we would emit this code when the initrd included the scsi_mod module.

Late in the F10 development, we moved a bunch of things to be built-ins, to tweak the boot-up speed.
This turned out to be a really bad idea, as it meant the 'stabilization' code didn't get emitted. Ouch.

How come it didn't affect everyone? Different controllers behave differently. Some initialise so quickly that there's no reason to wait any longer.

Why didn't it get caught during testing? We did have a large number of reports, but the correlation wasn't made until post-release.

Should it have held up the release? Possibly, if we knew it would affect so many people.

FTR: None of the hardware I personally tested was affected by this problem, which made it a real pain to track down.

The good news is that in the hopefully not to distant future, all this code is going the way of the dodo anyway.

Saving money by implementing better power management isn't the fast path to richness it seems. Writing about it for $279 a pop seems to be.

I wonder how many people actually buy these 'reports'. Probably a ridiculously high number.

I really like spicy food. I eat a fair amount of spicy indian & mexican. But none of this prepared me for what I tasted a few nights ago. This stuff is quite possibly the most brutal hot sauce I've ever tasted.

I had a tiny amount and tasted it. Within seconds I had to chug down a load of milk to try and make the burning stop. Even with that, it continued burning for a while. If I had any appreciable quantity of this stuff I think I'd have some kind of religious experience. It's not so much spicy, it's just ridiculous amounts of searing pain.

Habañero peppers, Scotch Bonnet peppers and African olesoresin.

I'm curious how many scoville's this stuff is, but the internet seems not to know. The closest guess is somewhere in the 300-500,000 range.

On a scale of 1 to 10, I'd score this one "oh my god I think I'm going to die".

Thanks Erinn.

Hacking on the new initrd stuff with Jeremy..

"Yeah, you probably don't need those sound/ modules in the initrd"
"Dude, music whilst you boot!"
"We could have a contest to have people submit a 30 second boot-time song"

Yeah. Maybe this idea won't make the cut.

Some of the more sensible ideas in a post later. In the meantime, some rough notes.

Some questions related to virtual memory never seem to go away.

Here are some of my favorites that I just love getting asked at least once every couple weeks. (Despite not being a VM hacker).

• My computer is using N MB of swap space, and there's memory free! This sucks!
  
  To understand this, think about what happens to infrequently used pages of a process when there's memory pressure. The OS decided to use those pages of RAM for something more useful.
  
• My computer has N MB of swap space free, and my processes still get oom killed. the kernel is buggy!
  
  When something got oom-killed, you ran out of _memory_, not swap space, and there was nothing else that could be swapped out to free up memory.
  
• My computer had N MB of RAM free, and they still got oom killed! the kernel sucks!
  
  There are different zones of memory. Having hundreds of MB of HIGHMEM available is irrelevant if something really needs memory it can use for DMA (Which is a very tight resource).
  

And my all-time favorite..

• I run without any swapspace configured, and once my app that allocates hundreds of gigs of memory keeps getting oom-killed. How do I stop the kernel doing this.
  

Answers on a postcard for that one ...

Dilemma. I need some portable storage device, but all the USB/Firedire ones seem to be made of fail. But whilst searching, I found this thing which raised some questions.

• The obvious one - what's up with the bamboo?
  
  
• What the hell is Turbo USB?
  
  Asides from "25% faster than a USB 2.0 connection", it doesn't really say. So I poked around the interwebs a bit.
  The best I've been able to dig up so far, is buffalo claiming a 60% speed increase over USB2 (wait, 60% ? hmm), and this speculation
  that it's just a cache that sits between the host and the USB->SATA bridge. From a little searching, it seems that a whole
  bunch of vendors now implement this same gimmick, whatever it really is.
  
  
• From reading the comments left by some customers on the amazon page linked above, the "auto-sensing power feature that turns the drive on and off with your computer" doesn't seem to actually work (and that's in those other OS's, good luck getting it work in Linux if it doesn't even work there). This doesn't really surprise me much. I've had many USB/Firewire drive enclosures over the years, and every single one of them has screwed this up. (Most of them make it impossible to use features like SMART too).
  The amusing thing is that of all the enclosures I've had, the electronic parts are nearly always identical.
  So they suffer all the same firmware bugs across vendors, just in different cases, with different flashing lights.

All of this just makes me wonder. Isn't it about time that USB storage died a long overdue death and we all just moved to eSATA?

• It would be faster. No pointless protocol conversion. Transfers running the speed of the SATA link rather than the USB link.
  
• It wouldn't suffer unfixable firmware dumb bugs in protocol conversion.
  
• It would allow access to native features the drive is capable of like ALPM, SMART etc..
  
• Given the addition of ALPM & spindown that works, it would end up using less power.
  

The only thing missing seems to be that hardly any laptops feature eSATA connectors yet.

update:Immediately after posting this, I stumbled across this which is as Kyle McMartin said "a lot of data to accidentally put through the wash". I had no idea they were making usb sticks that big now. I think a handful of these is probably a better option than any usb->ata convertor + rotating media right now.

Recently I've read about or spoken with a few people using Fedora on eeepc's who have been making some fairly big blunders without realising it. I've played with a few of these now, in their various incarnations.
The current one I've been carrying around is the 900 model
with a whopping 20GB of flash. It's quite deceptive, because there are actually two SSDs in there (one 4GB, and one 16GB)
These ssds are also pretty damn awful performance-wise compared to the newer generation of SSDs, but short of opening it up and retrofitting something, there's not much that can be done. The tips below should at least make it more bearable.

• First off, don't use the default partitioning scheme.
  By default, anaconda will choose to use lvm, and make a contiguous volume out of the two SSDs. This idea is fail, because the two disks aren't the same, and run at different speeds.
  
  # hdparm -t /dev/sda
  
  /dev/sda:
  Timing buffered disk reads: 108 MB in 3.04 seconds = 35.57 MB/sec
  
  # hdparm -t /dev/sdb
  
  /dev/sdb:
  Timing buffered disk reads: 86 MB in 3.05 seconds = 28.20 MB/sec
  
  
  So, don't do that. Just create regular partitions, and make sure you put / on the faster of the two disks (the 4GB one), and leave the 16GB one for /home
  
  
• Next, the default filesystem will be ext3. You really don't want this.
  Given the journal is in a fixed location on disk, scribbling to it every time a file gets written is a great way to wear out the flash. Go with ext2. (Given that you've only got a few GB of flash anyway, a fsck doesn't take that long should you need to). Additionally, not having to write to the journal means that you're doing less IO, which is obviously a win when it's on such slow media.
  
  
• This should go without saying - no swap.
  Not only for the flash wear problem in the previous bullet, but also, because it's slow as all hell. If you find you run out of ram and get stuff oom-killed in this setup, well, you probably need to add more ram, or consider a real laptop.
  
  
• After installing, change the fstab so that everything gets mounted with noatime. Writes to the disk are just painful, so minimising them is the path to success here. It's doubtful that you'll be running anything on an eee that would actually care about atimes anyway.
  

With these points taken into consideration, the eee isn't a half-bad machine.
I still wouldn't want to be building kernels and such on it, but it's perfectly usable for email and such whilst travelling.

The cheap memory sticks I blogged about recently finally arrived today.
As one of the comments in that entry mentioned, Pavel Machek had bad experiences with usb sticks of the same brand, so I was curious to see how well mine would stand up. I plugged it in, and mounted it, no problems. Straight to my favorite file system stress test.. (fsx). It survived for a while. So I control-C'd it. Then I umounted it, and noticed dmesg got flooded with a few hundred..

FAT: Invalid FSINFO signature: 0x00000000, 0x00000000 (sector = 1)

Hmm. Looking at the partition table in fdisk was enlightening.

Disk /dev/sdj: 4043 MB, 4043309056 bytes
255 heads, 63 sectors/track, 491 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x04dd5721

Device Boot Start End Blocks Id System
/dev/sdj1 * 1 492 3948512+ b W95 FAT32
Partition 1 has different physical/logical endings:
phys=(490, 254, 63) logical=(491, 145, 38)

Umm. What?

I blew it away, and created a new partition table (with 'o'), added a new primary partition the size of the whole disk (accepting all defaults).

Device Boot Start End Blocks Id System
/dev/sdj1 1 491 3943926 83 Linux

Interesting. A block shorter.

I ran mkfs.ext2 with -cc option, and went out for a while. When I got back, all looked well..

Testing with pattern 0xaa: done
Reading and comparing: done
Testing with pattern 0x55: done
Reading and comparing: done
Testing with pattern 0xff: done
Reading and comparing: done
Testing with pattern 0x00: done
Reading and comparing: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done

Well, it does better than the iram ever did.
I created a vfat filesystem on there again, and reran the same stress tests as initially, occasionally syncing/umounting/unplugging, and I've not managed to break it since I created a non-bogus partition table. More destruction testing as it happens..

This week has really flown by. Friday already?
On Sunday I traveled down to New York by train for the Linux Foundation End User summit. Traveling by train is seriously the way forward. The actual time in motion may have been longer, but there was no queuing and waiting around at airports, no security hassles, at least twice as much legroom, and power at my seat. Unfortunately the on-train food was equally as bad as airplane food.

In New York, I spent two days in the Desmond Tutu center along with a bunch of other Linux developers, and Linux users from assorted backgrounds (though the financial/trading markets dominated unsurprisingly given our location).
The content was mostly a show-and-tell from both sides. From the developer side for eg, Ted Ts'o spoke about ext4, followed by Chris Mason showing us the future of btrfs. There were a few panel format sessions of end-users answering assorted questions about their Linux deployments. Some of this was interesting information, and I would have liked to have heard more, but due to the nature of their businesses, I got the impression some of the users felt somewhat uncomfortable revealing too much information with their competitors in the room. This is a situation we've encountered many times before. Sometimes Red Hat gets a bug report from one of our customers, and in getting a fix for that bug upstream, we get asked questions about the environment in which it was seen (or more directly: a test case for reproducing the bug, which most of the time not even we get).

I sat in on the file-system/storage sessions, which were well attended. It was interesting to hear some horror stories from users, and also to see the keen interest in Linux's next-gen file-systems. Chris Mason hopes to get btrfs upstream for 2.6.29, which could make for an interesting F11 feature. (At that point in time however it'll probably be at least a little rough around the edges, and may require a bootflag like we did for ext4 in Fedora "iamanext4developer"). No doubt though, that once this gets included in Linus' tree there will be a lot more people jumping on it.

I had a number of interesting 'hallway track' conversations with some end-users, where I found several of them actually regularly test Fedora to see "what's coming in future RHEL releases". Sadly, when asked if they participate in Fedora development (even if only by filing bug reports), there was mostly a negative response, with assorted reasons.

I lost Wednesday to traveling back to Boston. Thursday was 'catch up with email day', and today is pretty much more of the same. (Along with a bunch of other equally boring things like dealing with Visa paperwork). Looking forward to getting back to 'real work' next week.

With the LF summit out of the way, I'm done traveling for quite a while. I'm bailing on LCA this year, and I've canceled a trip to Portland,OR that was supposed to happen next month. I'm still not 100% recovered from my back/leg/whatever injury, and decided to play it safe. It's actually nice to not have to worry about going anywhere. It's probably going to be summer again before I'm back in an airplane.

This seems ridiculously cheap.
A number of times in the last week I've ordered stuff from amazon where the shipping has cost more than the item. (I have amazon prime, so get free shipping, but this doesn't work for 3rd party sellers using Amazon as a storefront). There's also an 8gb model which seems remarkably well priced.

Curiosity got the better of me, and even with the shipping making it ~$10, it still seemed like a great deal for 4GB, so I ordered one. The Fedora livecd-iso-to-disk script makes it pretty simple to create a bootable usb stick preloaded with Fedora. 4GB is a pretty usable amount of space. (It's that same that I had on my first eeepc, which proved more than adequate). Hopefully it's not something substandard with a low number of write-cycles, or slow speed or the like. I guess I'll know in a week. If this works out though, I think I'll never have to bother carrying a rescue CD with me every time I travel.

Some things I'd really love to see to improve the usefulness of bugzilla (and other bugtrackers)

• Comment nesting.
  Basically multiple "Reply to this comment" buttons, one after each post, instead of one "post a new comment" button.
  
• A "collapse this thread in *my* view of this bug".
  Useful for ignoring inconsequential stuff that people may post to the bug. "HEY! I SAW A KERNEL OOPS ONCE TOO!"
  
• (going a bit crazy now). Comments Points/Moderation/Scoring.
  Vote down a comment so it gets auto-collapsed in everyones view of the bug.
  

(This is beginning to sound like the bastard child of bugzilla & digg/slashdot, so I'll stop here).

I've had these thoughts a few times over the last few years triaging kernel bugs for Fedora. I think they would be worth additions especially for the instances of a single bug that gets many people jumping on it. Especially high profile issues like the current e1000e scare that has a bunch of people worried.
(Though the Fedora bug tracking that problem is quite tame in comparison to the ubuntu one right now. I actually pity those guys having to wade through all those comments, with so much not-directly-useful-or-relevant commentary).

But we've had issues in the past in Fedora where we've easily had over 100 comments. Trying to keep up with what's going on when there are 16 different people all with slightly different (or even completely different in some cases) problems is nearly impossible after a point. On a number of occasions, I've just closed the bug with a comment along the lines of "guys, this is madness, please file individual bugs, and we'll work through them".

Part of the problem is context. If all I had to do was look at that one bug, and work on it, it wouldn't be such a big deal. Due to there being more bugs than developers (in any distro), time-slicing occurs, and regaining 'state' when loading up a bug again takes a while. When there's pages and pages of comments, it's sometimes impossible to regain the full picture just due to the amount of noise.

After reading some of the comments at lwn about my sessions last week at kernel summit/plumbers conf about writing a 'make initrd' target for the kernel that would work on every distro, I feel some clarifications are in order.

What I am not doing.

• Dropping the Fedora mkinitrd script into the kernel.org tree and hoping other distros will make it work for them.
  
• Obsoleting the existing mkinitrd from Fedora for a considerable amount of time. (Think RHEL7 timeframe).
  

What I am doing.

• Right now, nothing but decompressing. Last week was pretty intense, with a lot of feedback to take in.

What I've done

• I've looked at other distros initrd's & creation tools. They all suck equally. Really. If you disagree, you either haven't looked hard enough, or you have emotional attachment issues. There is really nothing amazing about your distros tools over any others. (Which is why I'm of the opinion that "do it right, once, upstream" is the right answer.
  
• Written very little code. A few dozen lines of shell on the planeride home. I'm not posting anything until it at least is limping[1] along.
  

What I'll do next

• Yes, I'm starting afresh. I'll be borrowing some bits from various distros initrd's as I stumble into "hmm, how _does_ this currently work" territory, but this is going to be done with baby steps, first of all just booting off simple setups with /dev/[hs]da, no root on nfs/iscsi/nbd etc.
  
• At least in part, I'm starting with something completely new for political reasons. It's the only way to get anyone to agree on how to make progress. With everyone so invested in their current working solutions, "just use the debian one" or "just use the fedora one" isn't going to get us anywhere.
  

How is this going to work?

• For Fedora systems, we'll be building an additional initrd. Currently we only build it at kernel RPM install time, tailored for that system. Going forward, a one-size-fits-all initrd will be made during the package build process. Which gets used will depend on an /etc/sysconfig/kernel/ setting. This way, we can continue to use the crufty mkinitrd until this thing is ready, and the bleeding edge lunatics and people actually wanting to work on this can set the variable and have the new hotness.
  
• The very first step however is recovering from some of the fallout of the boot/init session of plumbersconf. Reviewing our CONFIG options. It's likely that use of 'make initrd' will imply that certain options should be set certain ways.
  

Once I'm feeling a little more on my feet[2], I'll make some more posts on what I intend to do, and how. Until then, continue to speculate wildly.


[1] Trapped sciatic nerves are the _worst_.
[2] Seriously. Screw flying anywhere for a while. I'm staying home.

Just got this mail from comcast..


On October 1, 2008, we will post an updated AUP that will go into
effect at that time.

In the updated AUP, we clarify that monthly data (or bandwidth) usage
of more than 250 Gigabytes (GB) is the specific threshold that
defines excessive use of our service. We have an excessive use policy
because a fraction of one percent of our customers use such a
disproportionate amount of bandwidth every month that they may
degrade the online experience of other customers.

250 GB/month is an extremely large amount of bandwidth and it's very
likely that your monthly data usage doesn't even come close to that
amount. In fact, the threshold is approximately 100 times greater
than the typical or median residential customer usage, which is 2 to
3 GB/month. To put it in perspective, to reach 250 GB of data usage
in one month a customer would have to do any one of the following:

* Send more than 50 million plain text emails (at 5 KB/email);
* Download 62,500 songs (at 4 MB/song); or
* Download 125 standard definition movies (at 2 GB/movie).


I guess I won't be syncing rawhide packages on a daily basis any more.
Or doing complete package tree checkouts.

I wonder what other bandwidth excesses I take for granted I'll have to cut back on.

'all the onscreen video is rendered by Moment Factory's custom rig, a trio of Linux-based devices collectively known as "the brain.".'

And for the record, that light show kicks so much ass. Some truly impressive effects. Some of which are on youtube, but cellphone captures really don't do them justice. If you get the chance to see this tour, even if you aren't a huge NIN fan, I really recommend it. I remember 20 years ago, people were saying similar things about Pink Floyds lightshows. This is all that, and so much more. (And apparently still being refined/enhanced, making me eager to see them again. Vegas at xmas maybe?)

Also, that wired article has confirmation that the 'nine inch fail bsod was intentional. Well duh.

I'm curious to find other stuff the moment factory guys have been responsible for. They're based in Montreal. Maybe we can get them to keynote next years OLS :)

Was on vacation last week in San Francisco, which was pretty awesome. A really fun city with a lot to see and do. Good to catch up with a few people, and also good to play tourist and go do touristy things (like visiting Alcatraz). Highlight of the trip by far though, was seeing Nine Inch Nails live at Oakland colliseum.

The low point of it all, was I seemed to somehow do myself an injury. When I arrived, I was aching a lot, but I put it down to just the usual cramped airplane flight, and then walked around SF, a lot. Including (not one my brighter moments) up some pretty harsh hills.
By the time Saturday came around, I couldn't stand up for too long without back/leg ache. Then I flew home yesterday in complete agony. The over the counter painkillers I got did pretty much nothing. By the time I got back to Boston, I was hobbling along with all my luggage at some ridiculously slow pace.

I hardly slept last night due to the discomfort. First thing this morning, I went to the ER to get checked out. Got poked and prodded a lot, and the best they could come up with was "probably a trapped sciatic nerve". So they loaded me up with percocet,valium and ibuprofen and sent me home. I took some, and headed back on the T. About halfway into my journey, I think they kicked in, because I spaced out, and completely missed my stop. They make me feel kind of drunk.
Got off, and had some lunch, and resumed my trip home.

So, great start to my first day back at work. I'm ridiculously backlogged from email right now, so will spend most of this week catching up, and taking it easy in the hope that I'm over this injury before the weekend when I'm due to fly to Portland,OR for kernel summit/plumbers.

[apologies if any of the above sounds somewhat more incoherent than usual, seriously, these drugs are making my brain do funny things]

My entry for the "most extensive debug output from a single event" world record.

.. and with that, I'm off on vacation for a week.

this is pretty cool.
I'm an avid collector of effect units (I lost count a while back just how many I have). Having something tweakable in this manner is kind of neat. Though I don't think I'll be getting one for a few reasons.

First, I use my noise-making as an escape from open source. Sometimes, even I need to switch off, and moving to a different room to run gdb in a different use-case isn't really escaping.

Secondly, since the beginning of the year I've been building a modular analog synth. The beauty of this thing is it's pretty much entirely open. Want more oscillators? fine, buy some, screw them in. Extra filters? same deal. Once you have a collection of mounted modules, you then get a large number of possible patch combinations between the various modules. For many of the modules, people have even made various 'hacks' to improve them in some manner or other. The manufacturers have in some cases even adopted those changes in later revisions of modules which is really awesome.

Whilst as a synth, it's primarily for the creation of sound, it also works really well as a sound-mangler of any input source, making it a supe r guitar effects pedal on steroids. And with cool sounding modules like "malgorithm", I'm easily persuaded.
The nice advantage of analog gear is that it rarely goes wrong in the sense that computers do. It frequently does completely unpredictable things, but it's usually a 'happy accident' than a complete disaster.

The only downside is these things get to be habit forming, and start to take over the house. Mine has already
grown to twice the size I was initially expecting, and will probably increase some more before I'm "done".

I keep meaning to record some of the output of this and other devices. One of these days I'll get to it. Until then, just imagine mains hum modulated by howling feedback with sub-bass that makes the walls shake, and you're probably pretty close to the sorts of sonic mayhem this thing puts out. (It's not /all/ it can do, it's just what I tend to make it do a lot).