Quoting Thomas Jacob :

> That didn't seem to do it for us, the bind instance in question ran
> for about 38 hours and then it refused to accept tcp connections again.


Did it actually start building up a lot of TCP connections in SYN_RECV
state again, or did it just crash?

The reason I ask is because my busiest caches will crash after running
for a day or two with the following entries in the logs:

Jul 27 13:15:07 s_all@cachens-3/69.20.2.254 named[24864]:
socket.c:1736: INSIST(!sock->pending_recv) failed
Jul 27 13:15:07 s_all@cachens-3/69.20.2.254 named[24864]: exiting (due
to assertion failure)

If anybody would benefit from seeing the gdb output, let me know. I
have *plenty* of core files laying around now.

> I found the following error message in the logs at about the
> time of the outage:
>
> 27-Jul-2008 15:35:12.234 resolver: notice: clients-per-query decreased to 17
> 27-Jul-2008 15:35:34.440 general: error: socket.c:1996: unexpected error:
> 27-Jul-2008 15:35:34.440 general: error: internal_accept: fcntl()
> failed: Too many open files
> 27-Jul-2008 15:35:34.452 general: error: socket.c:1996: unexpected error:
> 27-Jul-2008 15:35:34.452 general: error: internal_accept: fcntl()
> failed: Too many open files


You can fix that with ulimit. Check out ulimit -n to see how many
open files you currently allow, and try increasing it. I keep it set
to 16384 on my busier caches without any issues. Note that setting
the limit with ulimit won't be persistent, you will want to change
/etc/security/limits.conf and add something like the following:

* soft nofile 16384
* hard nofile 16384

Of course, this changes the limits for everything, but since mine are
dedicated nameservers with almost nothing else running on them, this
isn't an issue.

Hope this helps.

-- Jason


Confidentiality Notice: This e-mail message (including any attached or
embedded documents) is intended for the exclusive and confidential use of the
individual or entity to which this message is addressed, and unless otherwise
expressly indicated, is confidential and privileged information of Rackspace.
Any dissemination, distribution or copying of the enclosed material is prohibited.
If you receive this transmission in error, please notify us immediately by e-mail
at abuse@rackspace.com, and delete the original message.
Your cooperation is appreciated.