A very good overview by Jesse Smith on the Red Hat saga, featured on Distrowatch.
https://distrowatch.com/weekly.php?issue=20230703#qa
By Jesse Smith
A little over a week ago we reportedRed Hat had announced the company would be changing how it shared the source code for Red Hat Enterprise Linux. Now that the dust has settled somewhat, this seems like a good opportunity to discuss what the change is, how this affects people who use members of the Red Hat Enterprise Linux family, and what it will mean for distributions based on Red Hat Enterprise Linux (RHEL).
First, a little context is in order. It’s important to understand the flow of code from one place to another. Typically source code originates with an “upstream” project. Upstream projects are open source organizations like KDE, GNOME, LibreOffice, Mozilla, and the Linux kernel. These organizations publish the source code for their applications, services, and desktop environments publicly and usually for free. Then distributions, such as Fedora, take the source code and package it. Distributions like Fedora usually create two types of packages – source packages which contain the source code, images, and data files required to build the software; and binary packages which contain the executable programs and resources required to run the software.
In the case of the Fedora and Red Hat ecosystem, the source code passes from the upstream projects to Fedora and then to CentOS Stream. CentOS serves as a sort of slow-moving testing ground for the community and Red Hat. From there, the source code flows downstream to Red Hat Enterprise Linux where it is used to build binary packages which are provided to Red Hat’s customers.
For most of the life of RHEL, Red Hat has published its source packages publicly while reserving the binary packages for Red Hat customers and developers with Red Hat Enterprise Linux subscriptions. The publicly available source packages meant developers could take the source packages, strip out anything bearing Red Hat’s trademarks, and use the source code to build their own equivalents of RHEL. These equivalents are called RHEL clones.
Red Hat’s approach to publishing its source code has meant it has been possible for other organizations to make clones of RHEL which use the same source code and therefore are considered 1:1 compatible (or “bug for bug” compatible) with RHEL. This has given rise to distributions such as Rocky Linux, AlmaLinux OS, Oracle Linux, EuroLinux, and a few others.
In theory, this situation has led to a mutually beneficial relationship. The Linux community gets free clones of RHEL for small deployments and testing purposes. Meanwhile, small organizations who start out with a free RHEL clone can “upgrade” seamlessly to RHEL’s commercially supported distribution when they get larger and more successful. The clones act like free samples, something which might appear to take away from Red Hat’s business while also promoting the company’s software stack and encouraging people who don’t need commercial support to stick around in Red Hat ecosystem until they do.
Red Hat’s recent announcement changes this relationship a bit. What the company has done is effectively say they will no longer publicly publish the source packages for RHEL. They will only provide the source code used to build RHEL to Red Hat’s customers (and subscription holders) and those customers need to abide by a license which prevents them (in essence) from creating Red Hat clones. Meanwhile, the source code for Fedora and CentOS Stream will remain publicly available.
The company’s announcement led to a lot of people questioning whether the company can do this under the terms of the GNU General Public License (GPL), which governs the distribution of much of the company’s software. The answer is: they can. The GPL does not require organizations to provide public access to their source code, it only requires that companies offer to provide their source code to people to whom they distribute binary copies of their software. In other words, Red Hat only needs to provide their source code to Red Hat subscription holders who request it, according to the GPL. The GNU organization explains this in its GPL FAQ document:
The GPL does not require you to release your modified version, or any part of it. You are free to make modifications and use them privately, without ever releasing them. This applies to organizations (including companies), too; an organization can make a modified version and use it internally without ever releasing it outside the organization.
But if you release the modified version to the public in some way, the GPL requires you to make the modified source code available to the program’s users, under the GPL.
In short, what Red Hat is doing might be considered outside the “spirit” of open source software development, but it is within the “letter” of the appropriate licenses.
Some people have pointed out that section 6 of the GPLv2 states, in part: “You may not impose any further restrictions on the recipients’ exercise of the rights granted herein.” Which some have suggested contradicts Red Hat’s stance that their clients may not use Red Hat’s source code to create new clones. However, while the GPL’s clause likely prevents Red Hat from legally blocking the creation of RHEL clones, it does not require Red Hat to continue to do business with clients who assist in the building of RHEL clones. In other words, Red Hat likely can’t sue customers for making a clone, but they can deactivate the accounts of people they suspect are assisting the development of clones.
What will this mean for clones of RHEL? That is a bit of an open-ended question. In a blog post titled Impact of RHEL changes to AlmaLinux, Benny Vasquez outlines the issues they face, how it came as a surprise even before Red Hat’s announcement, and what this is going to mean for AlmaLinux OS and other clones:
Late last week one of our build SIG members noticed that some updates for Red Hat 8 hadn’t been published on git.centos.org like they were supposed to be. They assumed it was a bug and opened a report appropriately, but as the days went on with no resolution, we knew something was up. This morning we got our answer:
Red Hat has decided to continue to use the Customer Portal to share source code with our partners and customers, while treating CentOS Stream as the venue for collaboration with the community.
This change means that we, as builders of a RHEL clone, will now be responsible for following the licensing and agreements that are in place around Red Hat’s interfaces, in addition to following the licenses included in the software sources. Unfortunately the way we understand it today, Red Hat’s user interface agreements indicate that re-publishing sources acquired through the customer portal would be a violation of those agreements.
A similar news post was published by Rocky Linux, expressing surprise over the restriction to sharing source code that was introduced in the middle of the life cycles of RHEL versions 8 and 9:
Last week we had identified we were about ten updates behind. Wednesday’s announcement confirmed these missing updates had not been a simple oversight. So we have been solving immediate concerns while simultaneously developing mid- and longer-term responses. After tireless efforts by team members, we have completed update composes for Rocky Linux 8 and 9, including all the updates we were missing, also including all errata information.
In other words, clones of RHEL are now cut off from using Red Hat’s source code and security patches. The AlmaLinux blog post goes on to say that, in the short-term, AlmaLinux will be working with other clones and the CentOS Stream source repositories to try to keep up with security fixes. In the long-term, well, that is more of an open question and one for which nobody seems to have an answer. Without access to Red Hat’s source code it becomes difficult to (legally) make a 1:1 clone of the company’s distribution.
The Rocky Linux team published a statement, indicating plans to keep the project going and continue their work despite Red Hat’s embargo.
In response to the backlash from people who were upset about Red Hat cutting off access to their source code in the middle of a support cycle after having used open source to build their business model, Mike McGrath, the Vice President of Core Platforms Engineering at Red Hat, wrote:
Ultimately, we do not find value in a RHEL rebuild and we are not under any obligation to make things easier for rebuilders; this is our call to make.
The AlmaLinux team fired back, listing several of the ways free clones of RHEL contribute to the Red Hat community:
AlmaLinux community members have submitted PRs to projects such as RPM, AWX, and VirtualBox. Our community has sent over 50 PRs to GlusterFS and also extended openQA. A Red Hat employee even thanked us for enabling Fedora tests to run on ELN and RHEL. An AlmaLinux contributor was so fired up by our community that he now maintains over 600 Fedora and Extra Packages for Enterprise Linux (EPEL) packages, including some widely-used ones like certbot, brotli, iperf3, imapsync, and countless Python libraries, many of them as the primary contributor maintaining them for the greater Fedora and Enterprise Linux ecosystem. EPEL is tremendously important to both Red Hat and RHEL users.
As it stands, clones of Red Hat’s distribution likely either need to switch to a new base (such as CentOS Stream), convince Red Hat to change direction on this decision, find a way around Red Hat’s restrictions, or shut down their derivative projects. Time will tell which way each clone decides to go. This issue is likely to be especially of interest to Oracle as the company has long presented itself as an inexpensive, binary-compatible alternative to Red Hat with optimizations for Oracle’s database software. At the moment, it’s too soon to tell what the various RHEL clones will do to protect their existence, however there are some ideas now in circulation.
The Rocky developers have started exploring ways to work around Red Hat’s blockade by using paid-for-containers and cloud instances:
One option is through the usage of UBI container images which are based on RHEL and available from multiple online sources (including Docker Hub). Using the UBI image, it is easily possible to obtain Red Hat sources reliably and unencumbered. We have validated this through OCI (Open Container Initiative) containers and it works exactly as expected.
Another method that we will leverage is pay-per-use public cloud instances. With this, anyone can spin up RHEL images in the cloud and thus obtain the source code for all packages and errata. This is the easiest for us to scale as we can do all of this through CI pipelines, spinning up cloud images to obtain the sources via DNF, and post to our Git repositories automatically.
An interesting side-effect of this situation is seeing how other open source projects are reacting. Red Hat has, over the years, gradually been isolating its RHEL distribution, distancing itself from community efforts, and making it more difficult for clones to exist and users to function in their extended ecosystem. We’ve seen these efforts in the taking over of the CentOS project and then terminating CentOS Linux, replacing it with the moving platform of CentOS Stream. We’ve seen efforts to make it harder for other distributions to use Red Hat’s kernel. More recently, Red Hat has discontinued the position of Project Manager at Fedora and dropped the maintenance of LibreOffice packages for their ecosystem. Red Hat appears to be trying to take control of its ecosystem and weeding out community contributions, and it’s not making developers in the community happy.
For example, Jeff Gerrling announced plans to drop support for RHEL in Ansible roles – Ansible is a popular way to manage multiple machines on a network:
Support will be ‘best effort’, and if you mention you are using my work on Red Hat Enterprise Linux, I will close your bug/feature/support request as ‘not reproducible’, since doing so would require I jump through artificial barriers Red Hat has erected to prevent the use of their Linux distribution by the wider community.
Other developers, particularly those in the Fedora community (such as Philip Wyett), are now questioning why they should bother contributing to Fedora when their efforts are going to disappear behind Red Hat’s paywall:
Let us face it, our efforts with the Fedora project are not valued and it means nothing to the new corporate IBM/Red Hat enterprise systems that we have to struggle to get access to SRPMS to make a community. What is community now to Red Hat? I see an impasse here. Why contribute to Fedora when Red Hat will lock it down in other products?
The Software Freedom Conservancy meanwhile has a look back at the history of Red Hat’s business, their relationship with community clones of RHEL, and concerns over software licensing. Even SUSE, another commercial enterprise Linux company, took issue with Red Hat’s move, writing:
RHEL’s existence owes much to the collaborative efforts of many upstream projects, including the Linux kernel developed by many different contributors, among them SUSE. At the center of our world is innovating together. We are all working to build something greater than the sum of all our parts. We are all interdependent.
It will be interesting to see how this plays out. Red Hat seems to be trying to slowly gather up the pieces of its scattered ecosystem and squeeze them under the umbrella of the company’s control. However, many developers are not keen to have a more centralized, locked-down environment in which to work. For now it’s too soon to tell how clones of RHEL will adjust in an attempt to stay alive or what effect this change will have on other community efforts like RPM Fusion which builds add-on packages for Fedora and RHEL. We’re almost certainly going to see ripples coming off of Red Hat’s decision to restrict access to its source code in the months to come.
What Red Hat is doing appears to be legal and within the bounds of its licenses, but this surprise move has left a bad taste in the mouths of many community members and developers.