If you’ve ever seen a bullfight, you’ll know that toreadors weaken the bull by continually spearing it with short, blood-letting darts before the matador ever steps into the ring. The wounds ensure the bull never has a chance against a well-trained matador.
Red Hat Linux must sometimes feel the same way about the Red Hat Enterprise Linux 3 (RHEL) source packages that the GPL forces Red Hat to make available to the public.
Several Web sites sketch out how their owners build Red Hat-like enterprise distributions from those sources – inspiring shows of geek dexterity. But most of these are individual’s variants of Michael Redinger’s RHEL Rebuild mini-howto. More significantly, three projects are using Red Hat’s source to create complete, ready-to-load enterprise-ready distributions.

Whiter shade of Red
Highest profile is Whitebox Enterprise Linux which posted Whiteboxlinux Release Candidate 1 in November 2003. Now at RC 3, WBEL 3 is based on Red Hat Linux source that its developer John Morris has stripped of Red Hat-owned logos and trademarks. The project started off as an attempt to buy five years breathing space in which to continue supporting the Red Hat servers he maintained for a Los Angeles public library after Red Hat announced its strategy change last year.
Says Morris: “Being based off of RHEL3 means that a machine should be able to avoid the upgrade treadmill until Oct 2008 since RHEL promises Errata availability for five years from date of initial release and RHEL3 shipped in Oct 2003.”
Morris calculates that RC 1 saw over 18,000 downloads in November and continued cleaning the distribution until he shipped the final release in February. The site’s acknowledgements of independent developers who have found and fixed bugs suggests that around ten people are going through the code closely.
While lacking Red Hat’s support; in the best Linux tradition, WBEL 3 is supported by Morris, by users and independent developers via a mailing list instead.
“Even if I drop the ball, self-hosting means any user can rebuild errata themselves,” points out Morris.
Self-hosting is key to any RHEL-based alternative distribution. It means that new source RPMs can be built on the system itself without having to use Red Hat’s original source. That means the packages are built without Red Hat logos or EULAs that leave enterprises in breach of Red Hat’s licensing terms.
Morris plans to stick another dart in Red Hat’s wound when it releases Enterprise Linux 4. He plans to offer it as WBEL 4 and continue releasing WBEL 3 errata alongside it.

Creating cAos
Releasing RHEL 2 and 3-based enterprise distributions was also just the first rung for the cAos project. Led by Greg Kurtzer, cAos is the umbrella name for two separate enterprise Linux distributions, a new installer called Cinch and a community of builders capable of supporting it all.
For cAos, their self-hosting CentOS 2 and 3 – fully compatible alternatives to Red Hat Enterprise Linux 2 and 3 based on Red Hat SRPMs – is just the starting point. CAos has also built a parallel community-supported enterprise distribution, itself called cAos, that is free to develop as community wishes dictate.
“CAos makes a general attempt at binary compatibility with RHEL,” explains project leader Greg Kurtzer. “While the core is similar to CentOS, the goal is to be a crisp, clean distribution using many pristine, unpatched, sources from the maintainers themselves.” That’s a reference to a controversial practice of Red Hat’s three-man kernel team, which patches the kernel according to requests from commercial customers and vendors.
CAos aims to be a very different enterprise distribution, one that visibly nods to the controversies that have dogged Red Hat’s build practices. It ships with an older glibc and native gcc but more recent kernels, Gnome, and Mozilla web browser. And by combining the expertise of both cAos and CentOS developers, cAos – the community, that is – hopes to combine the advantages of both.
“Once we become a formal non-profit,” says Kurtzer, “we are hoping to build our paid developer base so that may of the developers can formally focus on cAos as a primary source of income.”
CAos is also setting up a commercial organisation to provide paid-for support alongside the non-profit cAos foundation. However successful cAos is, the cAos team will continue, and extend, the CentOS 2 and 3 RHEL-line distro by releasing “extras” as RPM packages for them.

Teaching Tao
Circling Red Hat at a greater distance is Tao Linux – the only distribution that supports s390 and a thin client version, though it is built only for i686 in Intel’s architectures and lacks non-English language support.
While Tao Linux works with cAos, its author – David Parsley – maintains a philosophical boundary: he recommends using Red Hat’s own Enterprise Linux distribution for mission-critical servers. He does so himself. His reasoning is simple: the money paid for Red Hat’s support is worth it – even though he is capable of supporting either distribution himself.
“Are you comfortable supporting it yourself?” he asks. “Are you comfortable mirroring and rebuilding the RHEL 3 SRPMs and troubleshooting any problems that arise?”
Comfort goes to the heart of the cost-benefit balance for these alternative distributions. Red Hat spokesperson Leigh Day commented, “Anybody can do this – it just won’t be supported with maintenance and services.”

Cost of alternatives
Alternative distributions come with their own costs for enterprises that want to check the packages’ integrity and security. Said Morris, “Any competent software engineer can, in a day or so, examine the small set of SRPMs signed with our key, compare the md5sum on the main tarball, examine the differences in the .spec and look at the .patch and assure themselves no backdoors, etc, exist.”
For many adopters of the Red Hat-created Enterprise Linux, working through task list would involve a consultant’s visit. And that would cost more than buying from Red Hat. Morris’ Web site also includes his notes on alleged problems he found in RHEL’s packaging – along with terse criticisms of the way Red Hat built RHEL 3.
If they achieve nothing other than domination of a few independent-minded sysadmin’s headspace, these rebel distributions deserve their place in history for another reason. They are the first time that software vendors selling mass-market software have found not only their product but much of their preparation process under the microscope and publicly criticised.