LVM crash - Linux

This is a discussion on LVM crash - Linux ; During installation of a new scsi disk in a ProLiant 370 server with RHEL4, LVM manager crashed (hung) and the configuration was corrupted. The machine cannot boot, and gets kernel panic. Configuration: /dev/sda1: mounted as /boot non lvm /dev/sda2: included ...

+ Reply to Thread
Results 1 to 9 of 9

Thread: LVM crash

  1. LVM crash

    During installation of a new scsi disk in a ProLiant 370 server with
    RHEL4, LVM manager crashed (hung) and the configuration was corrupted.
    The machine cannot boot, and gets kernel panic.

    Configuration:

    /dev/sda1: mounted as /boot non lvm

    /dev/sda2: included in VolGroup00 (ext3),

    /dev/sdb1: included in VolGroup00 (ext3)

    /dev/sdc1: should have been included in VolGroup00 (the new disk)
    /dev/sdc3: util partition on the disk or something from the vendor.

    / (root) was mountet on VolGroup00, except /boot

    Printouts fra lvm (bootet fra RHEL4 rescue cd)

    Output from lvscan and vgscan:

    lvm>lvscan
    lvm>vgscan

    Couldn't find device with uid m51fn4 .... oWH4t1
    [repeat a couple of times]
    Volume group "VolGroup00" not found

    lvm>pvscan

    Couldn't find device with uid m51fn4 .... oWH4t1
    PV /dev/sda2 VG VolGroup00 lvm2 [33,81 GB / 128MB free]
    PV /dev/sdb1 VG VolGroup00 lvm2 [33,91 GB / 0 free]
    PV unknown device VG VolGroup00 lvm2 [0 / 0 free]

    lvm>

    If only the lvm is corrupt, and the partitions and data are ok, I
    suppose it's possible to recover from this. But how? I do not want to
    experiment with this server, and I'm not very fluent in lvm configuration...

    --
    jon martin solaas

  2. Re: LVM crash

    Jon Martin Solaas wrote:
    > During installation of a new scsi disk in a ProLiant 370 server with
    > RHEL4, LVM manager crashed (hung) and the configuration was corrupted.
    > The machine cannot boot, and gets kernel panic.
    >
    > Configuration:
    >
    > /dev/sda1: mounted as /boot non lvm
    >
    > /dev/sda2: included in VolGroup00 (ext3),
    >
    > /dev/sdb1: included in VolGroup00 (ext3)
    >
    > /dev/sdc1: should have been included in VolGroup00 (the new disk)
    > /dev/sdc3: util partition on the disk or something from the vendor.
    >
    > / (root) was mountet on VolGroup00, except /boot
    >
    > Printouts fra lvm (bootet fra RHEL4 rescue cd)
    >
    > Output from lvscan and vgscan:
    >
    > lvm>lvscan
    > lvm>vgscan
    >
    > Couldn't find device with uid m51fn4 .... oWH4t1
    > [repeat a couple of times]
    > Volume group "VolGroup00" not found
    >
    > lvm>pvscan
    >
    > Couldn't find device with uid m51fn4 .... oWH4t1
    > PV /dev/sda2 VG VolGroup00 lvm2 [33,81 GB / 128MB free]
    > PV /dev/sdb1 VG VolGroup00 lvm2 [33,91 GB / 0 free]
    > PV unknown device VG VolGroup00 lvm2 [0 / 0 free]
    >
    > lvm>
    >
    > If only the lvm is corrupt, and the partitions and data are ok, I
    > suppose it's possible to recover from this. But how? I do not want to
    > experiment with this server, and I'm not very fluent in lvm
    > configuration...



    If the system boots up, it is probable that /dev/sda is accessible.

    If the UID is for /dev/sdb, it seems that your system does not
    find the disk at all. Is the disk still in the system?

    If yes, is it possible that the new disk (/dev/sdc?) is attempting
    to use the same SCSI bus ID as the previous /dev/sdb. Check the
    jumpering in the new disk.

    --

    Tauno Voipio
    tauno voipio (at) iki fi

  3. Re: LVM crash

    Tauno Voipio wrote:
    > Jon Martin Solaas wrote:
    >> During installation of a new scsi disk in a ProLiant 370 server with
    >> RHEL4, LVM manager crashed (hung) and the configuration was corrupted.
    >> The machine cannot boot, and gets kernel panic.
    >>
    >> Configuration:
    >>
    >> /dev/sda1: mounted as /boot non lvm
    >>
    >> /dev/sda2: included in VolGroup00 (ext3),
    >>
    >> /dev/sdb1: included in VolGroup00 (ext3)
    >>
    >> /dev/sdc1: should have been included in VolGroup00 (the new disk)
    >> /dev/sdc3: util partition on the disk or something from the vendor.
    >>
    >> / (root) was mountet on VolGroup00, except /boot
    >>
    >> Printouts fra lvm (bootet fra RHEL4 rescue cd)
    >>
    >> Output from lvscan and vgscan:
    >>
    >> lvm>lvscan
    >> lvm>vgscan
    >>
    >> Couldn't find device with uid m51fn4 .... oWH4t1
    >> [repeat a couple of times]
    >> Volume group "VolGroup00" not found
    >>
    >> lvm>pvscan
    >>
    >> Couldn't find device with uid m51fn4 .... oWH4t1
    >> PV /dev/sda2 VG VolGroup00 lvm2 [33,81 GB / 128MB free]
    >> PV /dev/sdb1 VG VolGroup00 lvm2 [33,91 GB / 0 free]
    >> PV unknown device VG VolGroup00 lvm2 [0 / 0 free]
    >>
    >> lvm>
    >>
    >> If only the lvm is corrupt, and the partitions and data are ok, I
    >> suppose it's possible to recover from this. But how? I do not want to
    >> experiment with this server, and I'm not very fluent in lvm
    >> configuration...

    >
    >
    > If the system boots up, it is probable that /dev/sda is accessible.


    It does not boot, ref the kernel panic mentioned above.

    >
    > If the UID is for /dev/sdb, it seems that your system does not
    > find the disk at all. Is the disk still in the system?


    To me it seems sdb is just fine, ref the pvscan output. And, yes, the
    disk is very much in the system.

    >
    > If yes, is it possible that the new disk (/dev/sdc?) is attempting
    > to use the same SCSI bus ID as the previous /dev/sdb. Check the
    > jumpering in the new disk.


    I think the disks are basically ok, before the **** hit the fan, the
    system was up and running off /dev/sda and /dev/sdb, and I could use
    fdisk to make a partition on /dev/sdc.

    --
    jon martin solaas

  4. Re: LVM crash

    On Sat, 18 Feb 2006 19:40:55 +0100, Jon Martin Solaas wrote:


    >Tauno Voipio wrote:
    >> Jon Martin Solaas wrote:
    >>> During installation of a new scsi disk in a ProLiant 370 server with
    >>> RHEL4, LVM manager crashed (hung) and the configuration was corrupted.
    >>> The machine cannot boot, and gets kernel panic.
    >>>
    >>> Configuration:
    >>>
    >>> /dev/sda1: mounted as /boot non lvm
    >>>
    >>> /dev/sda2: included in VolGroup00 (ext3),
    >>>
    >>> /dev/sdb1: included in VolGroup00 (ext3)
    >>>
    >>> /dev/sdc1: should have been included in VolGroup00 (the new disk)
    >>> /dev/sdc3: util partition on the disk or something from the vendor.
    >>>
    >>> / (root) was mountet on VolGroup00, except /boot
    >>>
    >>> Printouts fra lvm (bootet fra RHEL4 rescue cd)
    >>>
    >>> Output from lvscan and vgscan:
    >>>
    >>> lvm>lvscan
    >>> lvm>vgscan
    >>>
    >>> Couldn't find device with uid m51fn4 .... oWH4t1
    >>> [repeat a couple of times]
    >>> Volume group "VolGroup00" not found
    >>>
    >>> lvm>pvscan
    >>>
    >>> Couldn't find device with uid m51fn4 .... oWH4t1
    >>> PV /dev/sda2 VG VolGroup00 lvm2 [33,81 GB / 128MB free]
    >>> PV /dev/sdb1 VG VolGroup00 lvm2 [33,91 GB / 0 free]
    >>> PV unknown device VG VolGroup00 lvm2 [0 / 0 free]
    >>>
    >>> lvm>
    >>>
    >>> If only the lvm is corrupt, and the partitions and data are ok, I
    >>> suppose it's possible to recover from this. But how? I do not want to
    >>> experiment with this server, and I'm not very fluent in lvm
    >>> configuration...

    >>
    >>
    >> If the system boots up, it is probable that /dev/sda is accessible.


    >It does not boot, ref the kernel panic mentioned above.


    You can't do anything until you can boot the system. Have you tried
    booting from your distro's install CD?

    Do a google search for "rebuild linux LVM". I've done similar searches for
    rebuilding raid volumes. You should find everything you need in the various
    LVM how-to out there.

  5. Re: LVM crash


    >
    > You can't do anything until you can boot the system. Have you tried
    > booting from your distro's install CD?


    Yes, the printouts from pvscan, lvscan and vgscan are run off the rescue
    cd. So my hypothesis is that I can recreate the configuration for lvm
    using the tools on the rescue cd and get the machine up again.

    >
    > Do a google search for "rebuild linux LVM". I've done similar searches for
    > rebuilding raid volumes. You should find everything you need in the various
    > LVM how-to out there.


    Didn't really find what I needed in the lvm-howto, but I guess the
    answer is out there. I've tried a bunch of goole-searchdes, thankyou,
    but not the exact one you've suggested. Maybe it's the magic one :-)

    Thanx,

    --
    jon martin solaas

  6. Re: LVM crash

    Jon Martin Solaas wrote:

    >
    > Didn't really find what I needed in the lvm-howto, but I guess the
    > answer is out there. I've tried a bunch of goole-searchdes, thankyou,
    > but not the exact one you've suggested. Maybe it's the magic one :-)


    One of the more interesting things revealed by searching for "recover
    linux LVM" was the suggestion to use lvimport:

    --------------
    http://groups.google.com/group/comp....a378b4765d6ce5



    Try booting from CD-ROM and use vgimport with the -f flag to get
    your LVM back:


    vgimport -f -v data /dev/hda1 /dev/hda5 /dev/hda7


    When booting from disk next time, save the configuration with
    vgcfgbackup.
    --------------

    In my case it'd be

    vgimport -f -v VolGroup00 /dev/sda2 /dev/sdb1

    Does anyone know if this is likely to work? What does this command
    actually do when run off the rescue cd? Where is actually the
    configuration stored when I shutdown from the rescue disk and boots off
    the harddisks again? Will the fact that the root filesystem is on the
    VolGroup00 cause any problems?

    --
    jon martin solaas

  7. Re: LVM crash

    On Sun, 19 Feb 2006 21:07:01 +0100, Jon Martin Solaas wrote:

    > Jon Martin Solaas wrote:
    >
    >>
    >> Didn't really find what I needed in the lvm-howto, but I guess the
    >> answer is out there. I've tried a bunch of goole-searchdes, thankyou,
    >> but not the exact one you've suggested. Maybe it's the magic one :-)

    >
    > One of the more interesting things revealed by searching for "recover
    > linux LVM" was the suggestion to use lvimport:
    >
    > --------------
    > http://groups.google.com/group/comp....a378b4765d6ce5
    >
    >
    >
    > Try booting from CD-ROM and use vgimport with the -f flag to get
    > your LVM back:
    >
    >
    > vgimport -f -v data /dev/hda1 /dev/hda5 /dev/hda7
    >
    >
    > When booting from disk next time, save the configuration with
    > vgcfgbackup.
    > --------------
    >
    > In my case it'd be
    >
    > vgimport -f -v VolGroup00 /dev/sda2 /dev/sdb1
    >
    > Does anyone know if this is likely to work? What does this command
    > actually do when run off the rescue cd? Where is actually the
    > configuration stored when I shutdown from the rescue disk and boots off
    > the harddisks again? Will the fact that the root filesystem is on the
    > VolGroup00 cause any problems?


    Good luck Jon, a knotty problem indeed. LVM is wonderful for its dynamic
    configuration capability but definitely introduces an additional layer of
    abstraction into your system.

    Look in /etc/lvm for the configuration settings if you're able to access
    that, and preserve/back them up before performing the
    import function.

    Keep us posted on your progress and hoped for success.

    Frank


  8. Re: LVM crash

    frank wrote:

    >
    > Good luck Jon, a knotty problem indeed. LVM is wonderful for its dynamic
    > configuration capability but definitely introduces an additional layer of
    > abstraction into your system.
    >
    > Look in /etc/lvm for the configuration settings if you're able to access
    > that, and preserve/back them up before performing the
    > import function.
    >
    > Keep us posted on your progress and hoped for success.


    Well, I'll keep you posted, not much posted on this topic until now ...

    I was also considering using vgexport, but I'm not really sure what that
    command does, except "make volume groups unknown to the system" (manpage)

    I have a brand spanking new 146gb disk which could hold everything
    exported from the volumegroup on /dev/sda2 + /dev/sdb1, but I'm not
    capable of extracting the outcome of using this command from the man-page.

    --
    martin s.

  9. Re: LVM crash

    Jon Martin Solaas wrote:
    >
    > I was also considering using vgexport, but I'm not really sure what that
    > command does, except "make volume groups unknown to the system" (manpage)
    >
    > I have a brand spanking new 146gb disk which could hold everything
    > exported from the volumegroup on /dev/sda2 + /dev/sdb1, but I'm not
    > capable of extracting the outcome of using this command from the man-page.
    >


    vgreduce --removemissing VolGroup00

    did the trick.

    Just before I had partitioned and formatted the new disk, and changed
    partition type from Linux LVM to linux, just in case ...

    Now everything is up and running again.

    --
    martin s.

+ Reply to Thread