In article <1179909344.731327.163870@u30g2000hsc.googlegroups. com>
theredmini@gmail.com writes:
>
>I have a problem with an SSH server we have here, where any SSH
>connections to it will take around 30 seconds before prompting the
>user for a password. The most common cause of such pauses from reading
>other posts would appear to be reverse lookups failing, but that
>doesn't seem to be the case here.

Why don't you think so? It seems to match the symptoms perfectly.

>Everything seems to go ok until the client requests the server's
>protocol and SSH version. At that point, there is a pause for around
>20 - 25 secs before it responds with "Remote protocol version 1.99,
>remote software version OpenSSH_4.1".

The client doesn't "request" that, it's the first thing the server
sends, and it does so spontaneously. If it's waiting for a reverse DNS
lookup of the client's IP address, it won't send it until the lookup
completes or times out. You can simply telnet to port 22 on the server
and see the delay - on a properly working server you get the version
string immediately.

If not DNS, it could also be a problem with IDENT lookup - I don't think
OpenSSH's sshd has that builtin, but it's frequently built to use
libwrap, which may or may not do IDENT lookups depending on compile-
time settings and/or config (hosts.{allow,deny}). If you're simply
dropping incoming IDENT connections on the client due to firewall
config, the lookups will take a potentially long time to fail. It's
generally better to have the firewall respond with RST, or let the
connections through and the host stack will respond with RST due to
nothing listening on port 113.

--Per Hedeland
per@hedeland.org

Thanks for your response Per, please see my comments inserted below...

On May 23, 10:39 pm, p...@hedeland.org (Per Hedeland) wrote:
> In article <1179909344.731327.163...@u30g2000hsc.googlegroups. com>
>
> theredm...@gmail.com writes:
>
> >I have a problem with an SSH server we have here, where any SSH
> >connections to it will take around 30 seconds before prompting the
> >user for a password. The most common cause of such pauses from reading
> >other posts would appear to be reverse lookups failing, but that
> >doesn't seem to be the case here.
>
> Why don't you think so? It seems to match the symptoms perfectly.

Several reasons. DNS was my first port of call when I noticed this
problem, I spent a while checking that everything was setup correctly
and confirmed that I can do DNS lookups and reverse lookups either way
from client to server or vice-versa. Also, from other posts I've read
about reverse lookups causing the connection to hang, the hang appears
to occur at a different point in the connection, according to the
debug output (though I concede that it could potentialy occur at
various points). I tried an ssh connection from the server to
127.0.0.1, which also resulted in the connection delay.

>
> >Everything seems to go ok until the client requests the server's
> >protocol and SSH version. At that point, there is a pause for around
> >20 - 25 secs before it responds with "Remote protocol version 1.99,
> >remote software version OpenSSH_4.1".
>
> The client doesn't "request" that, it's the first thing the server
> sends, and it does so spontaneously. If it's waiting for a reverse DNS
> lookup of the client's IP address, it won't send it until the lookup
> completes or times out. You can simply telnet to port 22 on the server
> and see the delay - on a properly working server you get the version
> string immediately.

Interesting. I'd thought that the string was requested by the client
due to the kread in the truss output, which hangs for a while,
followed by the response from the server. I tried your suggestion of
telneting on port 22, which produced the following output (server
names changed):

[root@myclient]:/# telnet mysshserver 22
Trying...
Connected to mysshserver.
Escape character is '^]'.
SSH-2.0-OpenSSH_4.1

This confirms your suggestion that the protocol/version string is sent
by the server un-prompted. The SSH version string does take a similar
amount of time to appear (around 20-25 secs), suggesting that your
reverse lookup diagnosis is correct. This puzzles me, as I've checked
and confirmed that the reverse lookup is working.

>
> If not DNS, it could also be a problem with IDENT lookup - I don't think
> OpenSSH's sshd has that builtin, but it's frequently built to use
> libwrap, which may or may not do IDENT lookups depending on compile-
> time settings and/or config (hosts.{allow,deny}). If you're simply
> dropping incoming IDENT connections on the client due to firewall
> config, the lookups will take a potentially long time to fail. It's
> generally better to have the firewall respond with RST, or let the
> connections through and the host stack will respond with RST due to
> nothing listening on port 113.
>

This isn't something that I've yet considered. The OpenSSH install on
this server is an out-of-the-box pre-compiled version from IBM and is
identical to the version that is running without the same problem on
an apparently identical server. I've no idea whether IDENT would be
enabled in this build, but it's something for me to look into, so
thanks for the suggestion. For info, both the client and server are
behind the firewall so there is no firewall in between the two servers
and hence nothing should be being blocked. Also, we are not using
hosts.allow/deny files. For info, a telnet to the server on port 113
results in a failed connection and netstat shows nothing listening on
port 113 on the server. The same is the case for the other server that
is working properly though.

> --Per Hedeland
> p...@hedeland.org


Cheers,
Neil

In article <1179994341.906284.171180@w5g2000hsg.googlegroups.c om>
theredmini@gmail.com writes:
>Several reasons. DNS was my first port of call when I noticed this
>problem, I spent a while checking that everything was setup correctly
>and confirmed that I can do DNS lookups and reverse lookups either way
>from client to server or vice-versa. Also, from other posts I've read
>about reverse lookups causing the connection to hang, the hang appears
>to occur at a different point in the connection, according to the
>debug output (though I concede that it could potentialy occur at
>various points). I tried an ssh connection from the server to
>127.0.0.1, which also resulted in the connection delay.

Well, that last item rules out a lot of things I'd say - neither DNS or
IDENT are likely causes then.

>Interesting. I'd thought that the string was requested by the client
>due to the kread in the truss output, which hangs for a while,
>followed by the response from the server.

Yes, the client tries to read from its local socket, but the server
doesn't get told about that. The client blocks in the read until the
server decides to send something.

> For info, a telnet to the server on port 113
>results in a failed connection and netstat shows nothing listening on
>port 113 on the server. The same is the case for the other server that
>is working properly though.

The IDENT connections go from server to client, which is one reason that
they occasionally cause "strange" problems. But per above and everything
else you wrote, this isn't a likely cause. Try running the *server* in
debug mode, you may get some clues from where the output pauses. And/or
if there is something like 'strace' or 'truss' on AIX, it can be very a
useful tool to figure out what is going on.

--Per Hedeland
per@hedeland.org

On May 24, 7:51 pm, p...@hedeland.org (Per Hedeland) wrote:



> The IDENT connections go from server to client, which is one reason that
> they occasionally cause "strange" problems. But per above and everything
> else you wrote, this isn't a likely cause. Try running the *server* in
> debug mode, you may get some clues from where the output pauses. And/or
> if there is something like 'strace' or 'truss' on AIX, it can be very a
> useful tool to figure out what is going on.
>
> --Per Hedeland
> p...@hedeland.org

Per, you suggested I try running the *server* in debug mode
(previously I've only been running the client in debug mode). This
sounds like a sensible course of action, so I gave it a go, and got
some pretty strange results.

I started the sshd in debug mode, after stopping the currently running
sshd server. I then tried connecting from the client that I've been
using all along and.... it connected with no delay. This I found very
odd, as it appeared that restarting the sshd had fixed the problem,
but I'd tried this before several times. Further investigation shows
that the sshd works fine until I attempt to connect to it from a
client that cannot be resolved via a reverse lookup. In this case, I
see output like "debug3: Trying to reverse map address "
and it hangs like previously, which I guess is to be expected.

Now for the interesting bit ;-) If I then attempt a connection to the
server from the client who's address *can* be resolved via reverse
lookup.... it hangs again!?! As far as I can tell, the sshd works fine
until I attempt a connection from a source who cannot be reverse
looked up. Any subsequent connections, whether or not their addresses
can be resolved successfully will experience the pause during the
lookup. I don't understand why this is happening, but I intend to test
various scenarios to see if I can find a cause. This will be a little
difficult, as I can't just stop/start the sshd at any time. I might
look into starting the sshd in debug mode on a different port, so I
don't have to stop the currently running daemon.

Thanks for your help so far, I'll report back with what I find...

Cheers,
Neil

In article <1180093707.455457.325910@k79g2000hse.googlegroups. com>
theredmini@gmail.com writes:
>
>Now for the interesting bit ;-) If I then attempt a connection to the
>server from the client who's address *can* be resolved via reverse
>lookup.... it hangs again!?! As far as I can tell, the sshd works fine
>until I attempt a connection from a source who cannot be reverse
>looked up. Any subsequent connections, whether or not their addresses
>can be resolved successfully will experience the pause during the
>lookup. I don't understand why this is happening, but I intend to test
>various scenarios to see if I can find a cause.

Well, sounds like something "breaks" in your name service functionality
when it hits a failure, then. I can't imagine how this could happen with
"standard" DNS resolver libraries or name servers, but maybe you have
some evil proprietary proxy/cache in front of it on AIX? I know that at
least in the past, Solaris had something called the "Name Service Cache
Daemon" (nscd) that sat in front of all types of lookups (DNS, NIS,
files...) and caused all kinds of weird problems. At least it was
configurable as to what things it should "help" with, and you pretty
quickly learned to disable it at least for "hosts" if you wanted to stay
sane...

> This will be a little
>difficult, as I can't just stop/start the sshd at any time. I might
>look into starting the sshd in debug mode on a different port, so I
>don't have to stop the currently running daemon.

Yes, that's "standard operating procedure" for this kind of debugging -
no need to stop the regular daemon or even use a separate config file,
just give the debug instance a different port with -p.

--Per Hedeland
per@hedeland.org

On 2007-05-25, Per Hedeland wrote:

> Well, sounds like something "breaks" in your name service functionality
> when it hits a failure, then. I can't imagine how this could happen with
> "standard" DNS resolver libraries or name servers, but maybe you have
> some evil proprietary proxy/cache in front of it on AIX? I know that at
> least in the past, Solaris had something called the "Name Service Cache
> Daemon" (nscd) that sat in front of all types of lookups (DNS, NIS,

Can this funny resolver behaviour be produced outside of sshd?

Is a process only impaired by its own earlier lookups (which would make
it sound more like a problem in the resolver library and less like nscd) ?

Do you get a problem like this with both forward and reverse lookups?
Are you doing both in sshd (in the style of -DPARANOID tcp-wrqppers)?

Are you patched up to date?

Can truss and tcpdump tell you where these resolution attempts are going wrong?
(e.g. silly use of a search path, some load-balancer pushed into a an unhelpful
state by a failure, a rapid series of lookups interfered with by a firewall?)

--
Elvis Notargiacomo master AT barefaced DOT cheek
http://www.notatla.org.uk/goen/