The Linux Kernel column #90 – the state of the kernel

Linux User & Developer's resident Linux kernel expert, Jon Masters, gave the keynote speech at this year's Linux Symposium. In this column he mirror's his keynote with a look back at the past year of kernel developments…

Working with upstream
This time last year, Microsoft came out of the woodwork with a set of patches for its Hyper‑V virtualisation technology (known as ‘hv’ within the kernel community). There followed many months of to-and-fro as Microsoft learned some painful lessons about the kernel community – for example, that you can’t just throw code over the wall and walk away. There were a number of times when the community made threats to just pull Microsoft’s code from the ‘staging’ tree where it lives now (in preparation for possible official acceptance).

Those same community members did actually pull various Android code pieces from the official kernel as a growing tension exists trying to get out-of-tree Google Android pieces merged in a fashion acceptable to the conventions of the upstream kernel. More recently, the ‘suspend blockers’ debate (over Android’s aggressive tendency to go into low power states unless told not to – the opposite of more conventional power management wisdom) pitted a number of Android folks – who had spent many weeks working on upstream patches – against the likes of Ingo Molnar, who pointed out that we can’t measure a feature’s worthiness in terms of how much time has been spent by engineers, but only that it is the right solution to support for the longer term.
It’s easy to sympathise with the Google (and even Microsoft) folks. But even relatively mundane updates can cause a stir. One developer learned this lesson the hard way in attempting to post seemingly innocuous updates to the PC BIOS e820 table support (code that reads BIOS tables about memory region uses) only to find the whole of e820 deemed dead and a push to switch to the LMB (Logical Memory Block) code used on SPARC and PowerPC. Suddenly a lot more work was required than a merge update, and many patch cycles needed.


Over the past year, we’ve seen a growing trend for Linus to push back on developers who don’t play by the rules with respect to the merge window. This is the period of time during which new features are allowed to be proposed for inclusion into the official kernel source. In some cases, Linus has refused to take patches from even well known developers, just to prove his point that a two week merge window doesn’t mean “a one day merge window after 13 days of silence”. Linus has let his displeasure be known in other areas too, for example the addition of nouveau support to certain distribution kernels prior to being officially accepted upstream, and the fallout subsequent incompatible fixes and changes to upstream caused to users of nouveau.

Perhaps one of our biggest enemies is regressions. These happen when a new feature or other change caused kernel performance or function to regress in comparison to where it was before. Rafael J Wysocki does a good job in providing summaries of outstanding regressions – and the rate of addressing them, in cases where it is known – but there is still a worrying trend in some cases to focus on new features over existing issues. This is especially true when it comes to performance: it really matters that some benchmarks have shown a decrease in performance over the past ten kernel releases.

The decrease may not be much, but over time this becomes problematic. Going forward, there are likely to be more issues arising in the embedded space as big corporate interests continue to drive a market segment that is so fast-moving as to see devices obsolete even before support for them is fully merged into the official kernel (and not in some vendor kernel). Just this month, the debate over proprietary graphics drivers for embedded systems has heated up with the upstream maintainer essentially issuing an ultimatum that embedded folks get with the programme and not mess around.

Linux 2.6.35
The merge window finally closed on new features going into the 2.6.35 kernel, this month saw the first of the RCs (release candidates) being made available. It includes new features such as cpu_stop (an ability to selectively tie up specific CPUs – as in the case of module loading in existing kernels), Memory Compaction (a kind of ‘defragment’ for memory), and performance events support for virtual machines wherein guest machines share their data with the host Linux system using modifications to an existing interface.

Debate has continued on implementing a full hardware error reporting subsystem that can merge existing mcelog, edac and other functionality into a single means to report and (perhaps) handle failing hardware gracefully. A louder debate took place as one engineer provided a somewhat scathing summary of his opinions on the design of Btrfs, which he considers broken. Chris Mason, the developer of Btrfs, has thus far been very gracious in responding carefully to the legitimate technical concerns, skipping any flamebait.

Things were a little quieter than usual as Linus Torvalds took a couple of weeks out for his longest vacation in many years, to go diving with his family – which was well deserved, in my opinion.

Click here for more opinion pieces from Linux User & Developer, or see highlights from issue 90 here.

Jon Masters is a Linux kernel hacker who has been working on Linux for almost 14 years, since he first attended university at the age of 13. Jon lives in Cambridge, Massachusetts, and works for a large enterprise Linux vendor. He publishes a daily Linux kernel mailing list summary at