Thank you for the -vvv switch, at least I can see what's happening now,
and I think I have repaired it.

I can see from the debugs that even though the machine lets me jump from
the desktop to the first server, when I jump to the second server it
checks the agent for a key and finds none, then fails over to checking
for the user's nonexistant local private key and then to
keyboard-interactive,password.

I noticed it checks for specific filenames when it looks at the agent.
It wanted files named either identity, id_dsa, or id_rsa. I had made my
key "id_rsa_jpowers" to differentiate it and then loaded it into ssh-add
right from the filesystem. Therefore on the second hop when the
ssh-agent went back to the desktop and looked for jpowers/.ssh/identity,
it found none.

So I used cat id_rsa_jpowers >> identity on the desktop and set the
chmod. Then I ran ssh-add -D to clear the other key, and just ssh-add
alone to let the software pick a key. It grabbed `identity`, let me hop
to one server, then to the second. Perfect.

I can now do exactly what I needed: happily ssh and scp data back and
forth across different servers and users.

Thanks John for -vvv and to all who replied.

Jason Powers

John Paul Heaton wrote:
> Jason,
>
> I understood what you were trying to do when you first mailed the list.
> I provided a suggestion (invoking ssh with -vvv) as to how to further
> troubleshoot the problem. You should do something like this:
>
> Test 1:
> (with ssh_config ForwardAgent=yes)
> desktop$ ssh -vvv server1
> server1$ ssh server2
>
> Test 2:
> (with ssh_config ForwardAgent=yes)
> desktop$ ssh server1
> server1$ ssh -vvv server2
>
> The debug output can be overwhelming, so it is good to isolate the
> output to individual hops. The debug info should tell you what was
> sent, who got it, and where there is a hold-up.
>
> As for my question on the public keys, I wanted to know if the key for
> one user was the same for another. Forwarding the Agent will allow for
> added private-keys to be used via the forward. If the users on the
> second hop do not have a valid public-key, then the forwarding will not
> work.
>
> I tested that setting ForwardAgent=yes does work from my desktop system
> to a intermediate server and then the final destination. The Desktop
> had the valid private key and the intermediate and final destination had
> the corresponding public-key. It worked.
>
> Then again, I am using non-red-hat systems. I hope my email gives you
> hope and another avenue for troubleshooting your problem.
>
> John H.
>
> On Thu, 5 Oct 2006, Jason Powers wrote:
>
>> Date: Thu, 05 Oct 2006 13:28:34 -0400
>> From: Jason Powers
>> To: John Paul Heaton
>> Cc: secureshell@securityfocus.com
>> Subject: Re: Agent Forwarding Question for the list
>>
>> I'm not making keys for these service users. I'm going to use them to
>> run scripts and monitors in the way our present (RH7.2-based) setup
>> does. I put my public key in the .ssh/authorized_keys file for each of
>> these users. I have to run ssh-agent/ssh-add on my local machine, then
>> I can jump right into the first server as whichever service user
>> account I've set up this way. That part already works. It's hopping to
>> the next server that asks for a password.
>>
>> Most of the documentation I found suggests it's possible to do this,
>> and I can already do it with ssh-3.2.9-1 on our old setup. I wonder if
>> there is something specific about redhat's build of openssh or pam
>> that will intercept these requests, or if there is a directive I must
>> set in ssh_config|sshd_config or /etc/pam.d/sshd to allow it. Our old
>> setup also allows jumping to/through root users, but there may be
>> different rules for that in a Fedora 5 configuration as well.
>>
>> When I'm running tails on all the logs, the ssh connection reqest
>> never shows up on the second server, so I strongly suspect that PAM is
>> intercepting the request and asking for a password. However, I was
>> determined to ask the experts in case it was a common mistake or
>> something that simply is not possible under openssh.
>>
>> By setting my public key in system users I should be able to jump from
>> one machine to the next, or scp files around. Say in the ideal setup
>> for development servers I'd have a cronuser, scriptuser, monitoruser,
>> cvsuser, and root (I know it's poor security) all configured with my
>> public key and that I could jump in and out of each not only from my
>> own Linux Desktop, but through each user to each user on other servers
>> in the development chain. After reading all the documentation and FAQs
>> I could find, I had assumed ssh-agent on the desktop and agent
>> forwarding on the servers would be sufficient, but something is
>> blocking the forwarding, or I'm way off and this isn't how it's meant
>> to work.
>>
>> Thanks
>>
>> Jason Powers
>>
>> John Paul Heaton wrote:
>>
>>> You can get a detailed idea of what ssh is doing by using the -v
>>> flag. You can get more detail by using more v's, up to three, like
>>> -vvv. It is a a good way to see what ssh is doing.
>>>
>>> As for your problem, does the "otheruser" have the same public-key as
>>> "someuser" in the authorized_keys file?
>>>
>>> John
>>>
>>> On Wed, 4 Oct 2006, Jason Powers wrote:
>>>
>>>> Date: Wed, 04 Oct 2006 18:18:02 -0400
>>>> From: Jason Powers
>>>> To: secureshell@securityfocus.com
>>>> Subject: Agent Forwarding Question for the list
>>>> Resent-Date: Thu, 5 Oct 2006 08:53:26 -0600 (MDT)
>>>> Resent-From: secureshell-return-8989@securityfocus.com
>>>>
>>>> I have looked through the archives and googled this pretty
>>>> thoroughly, I'm having a tough time finding someone else who has
>>>> asked the same question previously. There's a lot of information
>>>> about openssh, but surprisingly little detail about port forwarding.
>>>> Either it works for everyone all the time, or my configuration is a
>>>> little bit particular compared to others.
>>>>
>>>> We would like to change from ssh2 to openssh for all of our linux
>>>> servers. I am testing new equipment with Fedora Core 5 with openssh
>>>> configured out of the box. I have no need to forward X11 windows, I
>>>> just want to be able to jump from machine to machine with a
>>>> terminal, ssh and scp, and use different accounts without having to
>>>> type a password. A lot of our production process revolves around
>>>> this, so it pretty much has to work for me to convert us.
>>>>
>>>> I made users and keys with openssh instead of using the old ones,
>>>> put them in the accounts I wanted to jump to on multiple servers. I
>>>> set the perms on the authorized_keys files to 600. I set the
>>>> ssh_config file in /etc/ to say ForwardAgent yes.
>>>>
>>>> Now let's say that I have a linux desktop and two linux servers,
>>>> assuming I've configured things correctly, then from the desktop box
>>>> I should be able to:
>>>>
>>>> me@desktop> ssh-add
>>>> (type pass for key)
>>>> me@desktop> ssh someuser@server1
>>>>
>>>> now from that terminal
>>>> someuser@server1> ssh otheruser@server2
>>>>
>>>> It asks me for a password when I try to jump to the second server. I
>>>> can put the password in and it works, but I think at this point it
>>>> should be forwarding the key.
>>>>
>>>> I have tail -f running on the secure log on each machine in question
>>>> so I can see if there's anything happening.
>>>>
>>>> It does not enter into the log on the target machine that I am
>>>> attempting to open a connection while it waits for a password, so I
>>>> was thinking that pam may be intercepting the request and demanding
>>>> one.
>>>>
>>>> Has anyone known pam to do such a thing?
>>>> Am I seeing a common non-error?
>>>> Is this a situation where ssh-agent on the servers may be
>>>> interfering with the one from the desktop?
>>>> Do I have to turn on X11forwarding to get agent forwarding on these
>>>> servers, which don't even have x installed?
>>>> Does this have something to do with xauth on the servers, or is that
>>>> only for x11 forwarding?
>>>>
>>>> Thanks
>>>>
>>>> Jason Powers
>>>>
>>>
>>> --------------------------------------------
>>> -- John Heaton - Computer System Engineer --
>>> -- George Mason University --
>>> -- Information Technology Unit --
>>> -- Systems Engineering (ESM) --
>>> -- * email: jheaton@gmu.edu --
>>> -- * phone: 703.993.3558 --
>>> --------------------------------------------

>>
>>

>
> --------------------------------------------
> -- John Heaton - Computer System Engineer --
> -- George Mason University --
> -- Information Technology Unit --
> -- Systems Engineering (ESM) --
> -- * email: jheaton@gmu.edu --
> -- * phone: 703.993.3558 --
> --------------------------------------------