Over the last couple years, I've had problems where servers running
RedHat would be running fine for months, then suddenly we'd start
getting all sorts of errors (socket errors). When we log into the
machine we notice that at least one of the filesystems had changed
from Read/Write to Read Only.

At first this was thought to be a software RAID issue, but now we've
confirmed it on machines from 7.3 to 9.0 on machines that do not have
any RAID running. We've also had reports from other companies we work
with that they have experienced the same random filesystem changes on
some of their RedHat machines.

In some cases, it's not the / partition and things can be remounted
manually without restarting. A couple times though the / partition
has become read only and every attempt to shut down everything and
force a remount comes back without an error, but the system doesn't
change out of read only mode.

In the cases with the / partition - we've always been forced to reboot
and pray. So far it's a 50/50 split. Some machines reboot and
remount / fine and come back up, but some do not - and we've ended up
reinstalling or replacing the machine.

Some machines are under light loads, some heavy loads, and all
machines are webserver machines with slightly different configurations

We don't get any reports from chkrootkit etc. to lead us to believe
it's a hacking issue, and for the machines that are repaired live or
reboot and come up normally, they seem to run fine again and all loads
and logs look normal.

Any ideas or suggestions in what this could be would be wonderful. I
wouldn't have posted this, but now that I've talked to other companies
that have had the problem, I'd like to find out if anyone has figured
out why this happens.