ssh service broken on server - Suse

This is a discussion on ssh service broken on server - Suse ; (OpenSUSE 10.2) We had a power blip the other day which locked up the server forcing a reboot and now the ssh service refuses to work on the server. I can still log in on workstations using LDAP, run sessions ...

+ Reply to Thread
Results 1 to 8 of 8

Thread: ssh service broken on server

  1. ssh service broken on server

    (OpenSUSE 10.2)
    We had a power blip the other day which locked up the server forcing a
    reboot and now the ssh service refuses to work on the server.
    I can still log in on workstations using LDAP, run sessions on it remotely
    using vnc but the ssh service refuses to start.

    # rcsshd start
    Starting SSh daemon/usr/sbin/sshd: error while loading shared libraries:
    libkrb5.so.3: cannot open shared object file: no such file or directory
    startproc: exit status of parent of /usr/sbin/sshd:127

    My first thought was a file corruption and to find the package providing
    libkrb5.so.3 and reinstall it - krb5-1.5.1-23.14.rpm

    YaST failed work so from the CLI ...

    # rpm --force -U krb5-1.5.1-23.14.i586.rpm
    error: unpacking of archive failed on file /usr/lib/libkdb5.so.4.0: cpio:
    rename failed - Permission denied

    # ls -al /usr/lib/kdb5.so.4*
    ls: cannot access /usr/lib/libkdb5.so.4.0: Permission denied
    lrwxrwxrwx 1 root root 14 May 3 09:15 /usr/lib/libkdb5.so.4 ->
    libkdb5.so.4.0
    -rw-r--r-- 1 root root 31048 May 1 18:12 /usr/lib/libkdb5.so.4.0;4819fa00
    -rw-r--r-- 1 root root 31048 May 2 17:43 /usr/lib/libkdb5.so.4.0;481b44b0
    -rw-r--r-- 1 root root 31048 May 2 17:45 /usr/lib/libkdb5.so.4.0;481b4510
    -rw-r--r-- 1 root root 31048 May 2 17:46 /usr/lib/libkdb5.so.4.0;481b4575
    -rw-r--r-- 1 root root 31048 May 2 17:47 /usr/lib/libkdb5.so.4.0;481b459f
    -rw-r--r-- 1 root root 31048 May 2 17:47 /usr/lib/libkdb5.so.4.0;481b45a0
    -rw-r--r-- 1 root root 31048 May 2 18:05 /usr/lib/libkdb5.so.4.0;481b49f3
    -rw-r--r-- 1 root root 31048 May 2 18:08 /usr/lib/libkdb5.so.4.0;481b4a74
    -rw-r--r-- 1 root root 31048 May 3 08:45 /usr/lib/libkdb5.so.4.0;481c1817
    -rw-r--r-- 1 root root 31048 May 3 09:15 /usr/lib/libkdb5.so.4.0;481c1f1a

    Ok so now I'm lost. What do the values after the ';' mean?
    I tried to rename one to just libkdb5.so.4.0 to fix the link but I
    get 'permission denied' (even as root).

    Is there any way I can repair this without a rebuild?

    Alan
    --
    email =~ s/nospam/fudokai/

  2. Re: ssh service broken on server

    ajp wrote:
    > (OpenSUSE 10.2)
    > We had a power blip the other day which locked up the server forcing a
    > reboot and now the ssh service refuses to work on the server.
    > I can still log in on workstations using LDAP, run sessions on it remotely
    > using vnc but the ssh service refuses to start.


    > Ok so now I'm lost. What do the values after the ';' mean?
    > I tried to rename one to just libkdb5.so.4.0 to fix the link but I
    > get 'permission denied' (even as root).
    >
    > Is there any way I can repair this without a rebuild?


    Most likely some sort of file coruption. What I do in those cases is
    first a simpe reboot. The shutdown and startup process often helps and
    is faster then doing a fsck.

    If that does not help, the next thing to try is fsck and the step after
    that is to do a repair by booting from the 10.2 disk and selecting
    repair instead of install.

    houghi
    --
    This was written under the influence of the following:
    | Artist : Acda en De Munnik
    | Song : Als je bij me weggaat
    | Album : Bootleg

  3. Re: ssh service broken on server

    houghi wrote:

    > ajp wrote:
    >> (OpenSUSE 10.2)
    >> We had a power blip the other day which locked up the server forcing a
    >> reboot and now the ssh service refuses to work on the server.
    >> I can still log in on workstations using LDAP, run sessions on it
    >> remotely using vnc but the ssh service refuses to start.

    >
    >> Ok so now I'm lost. What do the values after the ';' mean?
    >> I tried to rename one to just libkdb5.so.4.0 to fix the link but I
    >> get 'permission denied' (even as root).
    >>
    >> Is there any way I can repair this without a rebuild?

    >
    > Most likely some sort of file coruption. What I do in those cases is
    > first a simpe reboot. The shutdown and startup process often helps and
    > is faster then doing a fsck.
    >
    > If that does not help, the next thing to try is fsck and the step after
    > that is to do a repair by booting from the 10.2 disk and selecting
    > repair instead of install.
    >
    > houghi


    So far I've tried all the above apart from the repair - I was hoping to
    avoid this in case it f**ks up some of the configurations (LDAP, SMB
    NFS ....). They're all backed up of course, it's just a PITA trying to
    remember which config file to restore some are obvious but others not. Plus
    I've got non-standard packages (from packman etc) on there which I'm not
    sure repair will honour.

    The other reason in trying to avoid the repair boot is that the server
    usually runs headless so it means finding and connecting VDU, kbd etc. -
    guess I'll just have to bite the bullet :/

    Alan

    --
    email =~ s/nospam/fudokai/

  4. Re: ssh service broken on server

    ajp wrote:
    > So far I've tried all the above apart from the repair - I was hoping to
    > avoid this in case it f**ks up some of the configurations (LDAP, SMB
    > NFS ....). They're all backed up of course, it's just a PITA trying to
    > remember which config file to restore some are obvious but others not. Plus
    > I've got non-standard packages (from packman etc) on there which I'm not
    > sure repair will honour.


    That is why I do a reboot and a fsck myself first always. Onthe plus
    side a repair repairs only those things that are broken So it will only
    repair those files that are broken.

    You can even decide what to look for and what not. It will probably also
    tell you what it has found and what it has changed (not 100% sure of how
    and where it does this)

    I have done a chown or chmod on /etc once and a repair just put that
    back and did not overwrite anything.

    So now will be the ideal moment to look at battery backup and automatic
    shutdown and restart in case of powerfailure. ;-)

    houghi
    --
    This was written under the influence of the following:
    | Artist : Evanescence
    | Song : Lies
    | Album : Origin

  5. Re: ssh service broken on server

    houghi wrote:

    > ajp wrote:
    >> So far I've tried all the above apart from the repair - I was hoping to
    >> avoid this in case it f**ks up some of the configurations (LDAP, SMB
    >> NFS ....). They're all backed up of course, it's just a PITA trying to
    >> remember which config file to restore some are obvious but others not.
    >> Plus I've got non-standard packages (from packman etc) on there which I'm
    >> not sure repair will honour.

    >
    > That is why I do a reboot and a fsck myself first always. Onthe plus
    > side a repair repairs only those things that are broken So it will only
    > repair those files that are broken.
    >
    > You can even decide what to look for and what not. It will probably also
    > tell you what it has found and what it has changed (not 100% sure of how
    > and where it does this)
    >
    > I have done a chown or chmod on /etc once and a repair just put that
    > back and did not overwrite anything.
    >
    > So now will be the ideal moment to look at battery backup and automatic
    > shutdown and restart in case of powerfailure. ;-)
    >
    > houghi


    I've tried the repair option and it finds a problem with krb5 but it can't
    fix it - the attempt to reinstall fails with the same error as when I tried
    to update from the CLI.

    The thing is that root cannot access /usr/lib/libkdb5.so.4.0 - any attempts
    to delete, link, rename produces
    cannot access /usr/lib/libkdb5.so.4.0: Permission denied

    I did manage to create a temp directory in /usr/lib and mv all the
    libkdb5.so.4.0* files into it but even then if I try
    # touch /usr/lib/libkdb5.so.4.0
    I get permission denied.

    Alan
    --
    email =~ s/nospam/fudokai/

  6. Re: ssh service broken on server

    On Sun, 04 May 2008 12:10:59 +0100, ajp wrote:

    > houghi wrote:
    >
    >> ajp wrote:
    >>> So far I've tried all the above apart from the repair - I was hoping
    >>> to avoid this in case it f**ks up some of the configurations (LDAP,
    >>> SMB NFS ....). They're all backed up of course, it's just a PITA
    >>> trying to remember which config file to restore some are obvious but
    >>> others not. Plus I've got non-standard packages (from packman etc) on
    >>> there which I'm not sure repair will honour.




    >
    > I've tried the repair option and it finds a problem with krb5 but it
    > can't fix it - the attempt to reinstall fails with the same error as
    > when I tried to update from the CLI.
    >


    If you have already done a _full_ manual fsck on this filesystem and you
    still have a permission error as root then I would consider the
    possibility of a disk hardware problem. I think you maybe need to
    replace the disk.

    JohnK

  7. Re: ssh service broken on server

    JohnK wrote:

    > On Sun, 04 May 2008 12:10:59 +0100, ajp wrote:
    >
    >> houghi wrote:
    >>
    >>> ajp wrote:
    >>>> So far I've tried all the above apart from the repair - I was hoping
    >>>> to avoid this in case it f**ks up some of the configurations (LDAP,
    >>>> SMB NFS ....). They're all backed up of course, it's just a PITA
    >>>> trying to remember which config file to restore some are obvious but
    >>>> others not. Plus I've got non-standard packages (from packman etc) on
    >>>> there which I'm not sure repair will honour.

    >
    >
    >
    >>
    >> I've tried the repair option and it finds a problem with krb5 but it
    >> can't fix it - the attempt to reinstall fails with the same error as
    >> when I tried to update from the CLI.
    >>

    >
    > If you have already done a _full_ manual fsck on this filesystem and you
    > still have a permission error as root then I would consider the
    > possibility of a disk hardware problem. I think you maybe need to
    > replace the disk.
    >
    > JohnK


    B***cks - I was **sure** I did a full reiserfsck by hand on /usr but I've
    just done one again and this time it's thrown up errors
    requiring --rebuild-tree. I've done that, re-installed krb5 and we're
    pretty well back up and running

    Alan
    --
    email =~ s/nospam/fudokai/

  8. Re: ssh service broken on server

    On 2008-05-05 11:42, ajp wrote:

    > B***cks - I was **sure** I did a full reiserfsck by hand on /usr but I've
    > just done one again and this time it's thrown up errors
    > requiring --rebuild-tree. I've done that, re-installed krb5 and we're
    > pretty well back up and running
    >
    > Alan


    I had the same problems that you had, where reiserfs isolated some broken
    files by deny access to them.

    The only permanent way to fix it is to replace the disk, but I made a backup,
    and was running badblocks , and then created a new filesystem with mkfs.reiserfs
    -- badblocks and it's still running fine after recover of some files
    from older backups.

    You can use rpm --verify to detect broken files, or even for all
    packages with rpm --verify $(rpm -qa) but it will complain about everything
    and give a long list files you changed, or just touched.

    /bb

+ Reply to Thread