Rodrick Brown wrote:
> There have been many debates to date about using custom kernels vs tbe stock
> kernel from vendors like Redhat and Suse. I know many of the enterprise
> vendors like Redhat and Suse, have put extensive amount of work into making
> their kernels very streamline and adaptive for various hardware options and
> configurations however has anyone really come up with a really optimized
> build for their production infrastructure and have seen good success in this
> work? Also can you care to comment about maintenance and support issues
> especially with 3rd party applications?

I've done it professionally, and I've got to tell you, it sucks to have to do.
The reason for the custom kernel is often quite foolish (such as a refusal to
upgrade to a new operating system release), and often results in having to
backport features from the newer kernels to the older kernels to get hardware
support that some smart-aleck decides is necessary to make a sale or do a new
service. The infrastructure necessary to keep backporting is painful, and if
your developers are not carefully organized in their source code handling,
it's deadly to your kernel stability.

While some developers believe strongly in heavily optimizing their kernels,
the resulting divergence from the main code lines makes maintaining them very
painful. If your patch is that good, push it upstream to get it into the main
codeline: that's what it's *for*.

> I want to know if the optimizations worth the effort? My applications are
> multithreaded with most of the i/o being cpu and network bound not many
> vendor tools a bulk of it is written in house.

Well, I'm thinking back to the Wacom graphics tablets, and the "select" patch
to optimize for Squid servers. I did a bunch of work getting them both into
packaged kernels, instead of the hand-built "make install" people had been
using previously, for lots of stability reasons. As long as your developers
are willing to follow good practices to integrate and source control their
changes, and as long as you have a reasonable test rig to build new kernels
quickly and test them as part of a whole OS, not just on the developer's box,
and to remotely salvage the system when the kernel fails, you can make it work.

If you have a kernel developer who says "Oh, we can backport anything we
need", first slap them with a rotten codfish, then send them back to NetApp
where they learned that bad habit. (I have a few people in mind!)

But my ghod, the SuSE kernels were a pain in the ass to customize. They relied
on components in their SRPM's to compile that SuSE has never published
(they're simply source control widgets, near as I can tell, and can be
commented out), they stuff their patches inside tarballs managed by shell
scripts called from the .spec file, which adds layers of indirection that are
nightmarish to source control and for which there is *NO* excuse, etc. To
replace most patches, you have to rebuild and rebundle the tarballs, which
just breaks source control in so many ways! And installing two kernels in
parallel, so you can boot with the old one to debug the new one? YaST doesn't
support that, and flushes your old grub configuration. You have to keep around
your own tool to rebuild the boot setups. Stupid, stupid, stupid!

RedHat's kernels are much cleaner to manage, thank you ghod, and make testing
such environments far easier. I'd like to buy the people who wrote that beer.

> I would love to hear from others who are using a custom kernel build in
> their production environment especially for those in the electronic trading
> sector.

In RPM based systems, you need to assure that your kernel never gets
accidentally replaced by the package management system. You can usually do
this the easy way, by providing a "kernel-company" sepcific and setting the
..spec file to say "Provides: kernel-%{release}", "Provides:
kernel-smp-%{release}", "Provides: kernel-PAE", etc. It can get a bit tricky
with add-on modules like NVidia drivers or NTFS, but those usually need to be
re-compiled anyway.