2011 was a very interesting year in the world of Linux kernel development.
There were a total of five kernel releases (one of which changed the major version – from 2.x to 3.x – for the first time in the better part of a decade), long-term core kernel hindrances to scalability such as the Big Kernel Lock were finally removed, and exciting features such as transparent huge pages (that had been shipped by the Enterprise vendors) were finally merged into the official kernel. Along the way, we lost several great visionaries who had defined our industry (and UNIX in particular), had a massive security breach that disrupted kernel.org activities for months, and celebrated 20 years of working together on the most successful open source operating system the world has ever seen. Next year promises to bring many more surprises, but first let’s take a look back on the year that was.
The year began with the release of Linux 2.6.37. Its principal feature was an ability to build (in certain configurations) without the Big Kernel Lock, a massively coarse-grained software-locking mechanism that dated back to the earliest days of Linux support for multiprocessor systems, and the biggest remaining roadblock to overall system scalability. This combined nicely with other interesting scalability work, such as support for I/O throttling of groups of processes between defined bandwidth limits. A secondary feature of 2.6.37 was the final enabling of Fanotify, which can be used by anti-malware (and anti-virus) software to intercept certain file system activity and screen files prior to their access, in ways similar to that seen on other operating systems on the market today.
Linux 2.6.38 (released in March), added an extension to processor scheduling cgroups that became known as ‘the patch that does wonders’. This was in reality a small patch that would group all of the processes having the same session ID (ie from the same login, running in the same terminal for example) in the same scheduling entity. Therefore, this allowed an entire kernel build and all of its attached processes to be automatically handled as a unit, preserving sufficient remaining CPU time to retain an interactive desktop session that does not become sluggish simply because one builds a kernel.
Another exciting feature of 2.6.38 was the merging of transparent huge pages, allowing the kernel to automatically use and manipulate CPU support for huge pages (managing memory in much larger chunks and better utilising limited CPU TLB entries) without needing to modify applications. This is particularly useful for data-intensive workloads, such as virtual machines.
A slightly shorter kernel release cycle led to Linux 2.6.39 (May), which finally killed off the Big Kernel Lock in all configurations (with a patch entitled ‘That’s all, folks’), and added support for what is known as Transcendent Memory. Transcendent Memory is special in that it is highly volatile, of an unknown size, and writes to it may fail at any time. The underlying idea is, for example, to be able to provide running virtual machines with extra unutilised host system memory for use as a cache or for similar purposes. Since the memory may be yanked away at any time, the host is under no obligation and can freely reclaim the memory if it needs to provide it to another virtual machine or has more pressing commitments.
The 20th anniversary of Linux was celebrated slightly early with the switch to a new major version in Linux 3.0 (July). Unlike earlier major version changes, 3.0 intentionally did not coincide with a radically fundamental overhaul. 3.0 added support for running Xen Dom0 domains (ie running as a host system when providing Xen-based virtualisation) rather than requiring a set of patches, which had long been the case (the actual Xen hypervisor remains separate from the kernel). Most recently, Linux 3.1 added support for dynamic writeback throttling (to more precisely control when processes will write out data to disk) and also support for OpenRISC, a fully open source architecture that can be implemented using FPGAs or real silicon ASICs and should allow anyone interested in studying computer design to really examine all of the stages of new architecture implementation.
As the year comes to a close, it is clear that the state of kernel development is strong. Linus continues to provide good overall leadership and he is (just about) continuing to scale in line with the overall number of patches flowing into the kernel by way of his various subsystem maintainers. There are some challenges ahead as the growth in popularity of Android and continuing use of Linux in less traditional contexts pulls development in new directions, but the community should be positioned to handle whatever comes along.