My site is getting joe-jobbed, fending it off ok.
Problem is that my RH 7.3 linux mail server is taking some of the
incoming SMTP traffic (gateways are really busy) and seems to be
struggling with providing TCP sockets for HTTPD (Webmail), POP3 or
SSHD.

Any advice? It takes 2-6 times for HTTPD to actually answer a
request. The HTTPD log confirms that the daemon isn't getting the
request initially. SSHD takes 3-4 attempts to connect.

Box is running at 0.45 usage, plenty of mem, daemons are running
properly, just cannot reach them. Any configuration ideas? I've
tried many IPtables rules, but they haven't helped so far. The
incoming flood is from unique IP addresses, real or spoofed seems to
be mostly real as the victims bounce the message.