Something to be aware of as Linux LTS kernel support is changing to 2 years and there will be fewer LTS kernels supported.
The Open Source Summit provides an update on what’s new in the Linux kernel and where it’s going from here.
Written by Steven Vaughan-Nichols
BILBAO, Spain: At the Open Source Summit Europe, Jonathan Corbet, Linux kernel developer and executive editor of Linux Weekly News, caught everyone up with what’s new in the Linux kernel and where it’s going from here.
Here’s one major change coming down the road: Long-term support (LTS) for Linux kernels is being reduced from six to two years.
Currently, there are six LTS Linux kernels — 6.1, 5.15, 5.10, 5.4, 4.19, and 4.14. Under the process to date, 4.14 would roll off in January 2024, and another kernel would be added. Going forward, though, when the 4.14 kernel and the next two drop off, they won’t be replaced.
Why? Simple, Corbet explained: “There’s really no point to maintaining it for that long because people are not using them.” I agree. While I’m sure someone out there is still running 4.14 in a production Linux system, there can’t be many of them.
Another reason, and a far bigger problem than simply maintaining LTS, according to Corbet, is that Linux code maintainers are burning out. It’s not that developers are a problem. The last few Linux releases have involved an average of more than 2,000 programmers — including about 200 new developers coming on board — working on each release. However, the maintainers — the people who check the code to see if it fits and works properly — are another matter.
Maintainers face numerous obstacles to doing their jobs. Obstacle one: Many maintainers aren’t paid to maintain. They maintain code in addition to their day jobs. On top of that, they face increasing demands on their time — because of understaffing and because of the use of fuzzers to find bugs. While fuzzers are helpful, they also uncover way too many minor bugs, each of which must be examined and then dismissed by maintainers.
The result? To quote Josef Bacik, Linux kernel file system developer and maintainer: “Maintainers are burning out [because] maintainers don’t scale.” Added Darrick Wong, another senior Linux kernel maintainer: “This cannot stand. We need help.”
How can they get help? Well, for one thing, Corbet suggests maintainers talk to their employers about paying them for their maintainer work. As Wong observed, “Most of my friends work for small companies, nonprofits, and local governments. They report the same problems with overwork, pervasive fear, and anger, and struggle to understand and adapt to new ideas that I observe here. They see the direct connection between their org’s lack of revenue and resources. They don’t understand why the hell the same happens to me and my workplace proximity associates when we all work for companies that clear hundreds of billions of dollars.”
That’s a good question. Companies must realize they need to give back to Linux if they want to continue to reap its benefits.
A related issue: Linux is now embracing Rust as an experiment. While that’s good news in many ways — Rust removes entire classes of errors that Linux’s main language C is vulnerable to — it also poses problems for maintainers. After all, if a maintainer has spent 30 years working in C, asking them to become a Rust expert is a big ask.
In addition, Rust is still evolving. Many Rust patches are needed to get the language to work properly at a deep level in Linux. That also means you need lots of glue code to get Rust and Linux working well together.
Then there are some Linux kernel developers who don’t like Rust. As one said, “There are possibly some well-designed and written [Linux] parts which have not suffered a memory safety issue in many years. It’s insulting to present this as an improvement over what was achieved by those doing all this hard work.”
Even so, Corbet believes that the decision point — whether Rust becomes a mainstream part of the kernel — is coming soon. That day will come, he noted, “when we merge the first feature that users depend on.”
That day is near: Three important new Rust-based additions to Linux kernel code are on the way, Corbet said. These are an implementation of the PuzzleFS, a read/write Plan9 filesystem server; and — the one that will make the biggest headlines — the Apple M1 GPU driver. Indeed, the first conformant Linux OpenGL ES 3.1 drivers are now available for Apple’s M1- and M2-family GPUs, arrived in late August 2023. With work like this well in train, Corbett would be very surprised if Rust doesn’t make it permanently into Linux.
Another subject in the news lately is how Red Hat’s tweaking of its Red Hat Enterprise Linux (RHEL) license has caused Oracle, SUSE, and the CIQ to fork RHEL with the Open Enterprise Linux Association (OpenELA). Leaving aside the business and licensing complications that led to this fight, there are also Linux kernel concerns.
These concerns orbit around the question: Which kernel should you use for your Linux distribution? There are two real choices: 1) Run the latest stable kernel or 2) Run an old kernel plus backported fixes. The latter is what Red Hat, and the other enterprise Linux distributors, tend to do.
The latter also results in vendor-specific kernels. And while this offers stability, it distances these distros from community support and makes them reliant on specific vendors. It’s this last consequence — which first caused AlmaLinux and Rocky Linux to start their own takes on CentOS (Red Hat’s free RHEL clone) after Red Hat shut CentOS down in favor of CentOS Stream — that sparked the fire between Red Hat and OpenELA. What OpenELA wants is a RHEL clone, which uses the RHEL older patched kernel. Stay tuned for more developments as this conflict continues to burn.
On the other hand, Corbet noted, Android “has been pushing very hard towards this generic kernel image and has been basing this on stable updates. That’s because they find that this helps improve Android’s security. They’ve found that the vast majority of security problems are disclosed in the kernel and or even fixed in the Android kernels before they are disclosed because they were already incorporated before anybody knew that they were actually security-related bugs.”
That’s yet another issue of which Linux kernel developers are painfully aware. As Corbet explained:
“One of the interesting aspects of kernel development is that almost anything can be a security bug. And you don’t really know that it is until somebody finds a way to exploit it somehow. So an awful lot of fixes go in, and they’re not marked as security fixes. Not because the kernel community is trying to hide security fixes. I mean, sometimes there’s a little bit of sneakiness that goes on there that I personally don’t like. But most of the time, there really is just that nobody knows that this bug is a security bug. It’s only later that somebody figures this out. And so the only way to protect yourself against these sorts of bugs is to put in all of the fixes”
This is why Corbet, and anyone who really knows Linux, recommends that if you’re building a Linux distro, you always include all the patches. For older kernels, such as 4.14, that can be up to 26,799 commits. But, if you try to pick and choose what patches to use, you’ll certainly open your doors to security holes.
Finally, Corbet noted that Scott McNealy, Sun’s former long-time CEO, once said, “Open source is free like a puppy is free.” McNealy had a point. Using open-source and Linux is easy. Paying for the training it needs not to make messes on the kitchen floor, that’s harder.