Cannot mirrorvg after replacing defective hdisk - Aix

This is a discussion on Cannot mirrorvg after replacing defective hdisk - Aix ; The scenario: 1) One of our servers had mirroring of the rootvg (hdisk1 & hdisk3). 2) hdisk1 died with messages in errpt - LVs listed as open/stale 3) Ran rmlvcopy and removed all copies on the failed hdisk1 4) Was ...

+ Reply to Thread
Results 1 to 5 of 5

Thread: Cannot mirrorvg after replacing defective hdisk

  1. Cannot mirrorvg after replacing defective hdisk

    The scenario:

    1) One of our servers had mirroring of the rootvg (hdisk1 & hdisk3).
    2) hdisk1 died with messages in errpt - LVs listed as open/stale
    3) Ran rmlvcopy and removed all copies on the failed hdisk1
    4) Was able to reducevg and remove the affected disk.
    5) rmdev -l hdisk1 -d
    6) lsvg -l rootvg shows open/syncd for all LVs with 1 LP = 1 PP

    IBM sent a hot-swappable replacement, and we took out the old disk and
    inserted the new one. Ran cfgmgr and new disk shows up (new PVID). So
    we go ahead and try to add it back into the rootvg.

    We extendvg for the new hdisk1 in rootvg, but when we mirrorvg it
    tells us that it could not synchronize the data. Checking the system
    shows the mirroring in place but most of the LVs remained stale.

    Ran rmlvcopy on hd1 (/home) and goes back to syncd. Tried mklvcopy and
    then syncvg -l hd1, and goes back to open/stale. Here is the error:

    0516-934 /etc/syncvg: Unable to synchronize logical volume hd1.

    oslevel -s
    5300-07-01-0748

    A reboot of the server had no effect.

    Also to be noted: My first thought was the new disk was DOA, but it
    was certified, and if I add it to another VG and do mklvcopy/syncvg on
    those LVs, it works fine.



    We expect to call IBM AIX Support as our next step. Just wondered if
    anyone has seen this before and wanted to be sure it's not something
    silly before we call them.

  2. Re: Cannot mirrorvg after replacing defective hdisk

    On Nov 6, 10:05 pm, "steven_nospam at Yahoo! Canada"
    wrote:
    > The scenario:
    >
    > 1) One of our servers had mirroring of the rootvg (hdisk1 & hdisk3).
    > 2) hdisk1 died with messages in errpt - LVs listed as open/stale
    > 3) Ran rmlvcopy and removed all copies on the failed hdisk1
    > 4) Was able to reducevg and remove the affected disk.
    > 5) rmdev -l hdisk1 -d
    > 6) lsvg -l rootvg shows open/syncd for all LVs with 1 LP = 1 PP
    >
    > IBM sent a hot-swappable replacement, and we took out the old disk and
    > inserted the new one. Ran cfgmgr and new disk shows up (new PVID). So
    > we go ahead and try to add it back into the rootvg.
    >
    > We extendvg for the new hdisk1 in rootvg, but when we mirrorvg it
    > tells us that it could not synchronize the data. Checking the system
    > shows the mirroring in place but most of the LVs remained stale.
    >
    > Ran rmlvcopy on hd1 (/home) and goes back to syncd. Tried mklvcopy and
    > then syncvg -l hd1, and goes back to open/stale. Here is the error:
    >
    > 0516-934 /etc/syncvg: Unable to synchronize logical volume hd1.
    >
    > oslevel -s
    > 5300-07-01-0748
    >
    > A reboot of the server had no effect.
    >
    > Also to be noted: My first thought was the new disk was DOA, but it
    > was certified, and if I add it to another VG and do mklvcopy/syncvg on
    > those LVs, it works fine.
    >
    > We expect to call IBM AIX Support as our next step. Just wondered if
    > anyone has seen this before

    So far no.
    but
    - Is the errpt saying anythink ?
    - do mirroring one LV at a time - beginning with hd5

    Search the IBM support side seaching for ' mwc , syncvg ' ;-(

    cheers
    Hajo

  3. Re: Cannot mirrorvg after replacing defective hdisk

    On 6 Nov, 21:05, "steven_nospam at Yahoo! Canada"
    wrote:
    > The scenario:
    >
    > 1) One of our servers had mirroring of the rootvg (hdisk1 & hdisk3).
    > 2) hdisk1 died with messages in errpt - LVs listed as open/stale
    > 3) Ran rmlvcopy and removed all copies on the failed hdisk1
    > 4) Was able to reducevg and remove the affected disk.
    > 5) rmdev -l hdisk1 -d
    > 6) lsvg -l rootvg shows open/syncd for all LVs with 1 LP = 1 PP
    >
    > IBM sent a hot-swappable replacement, and we took out the old disk and
    > inserted the new one. Ran cfgmgr and new disk shows up (new PVID). So
    > we go ahead and try to add it back into the rootvg.
    >
    > We extendvg for the new hdisk1 in rootvg, but when we mirrorvg it
    > tells us that it could not synchronize the data. Checking the system
    > shows the mirroring in place but most of the LVs remained stale.
    >
    > Ran rmlvcopy on hd1 (/home) and goes back to syncd. Tried mklvcopy and
    > then syncvg -l hd1, and goes back to open/stale. *Here is the error:
    >
    > 0516-934 /etc/syncvg: Unable to synchronize logical volume hd1.
    >
    > oslevel -s
    > 5300-07-01-0748
    >
    > A reboot of the server had no effect.
    >
    > Also to be noted: *My first thought was the new disk was DOA, but it
    > was certified, and if I add it to another VG and do mklvcopy/syncvg on
    > those LVs, it works fine.
    >
    > We expect to call IBM AIX Support as our next step. Just wondered if
    > anyone has seen this before and wanted to be sure it's not something
    > silly before we call them.


    Does the logical volume has bad-block relocation policy turned off?
    This may have caused the command to fail.

    HTH
    Duncan

  4. Re: Cannot mirrorvg after replacing defective hdisk

    On Nov 6, 4:57*pm, Hajo Ehlers wrote:
    > On Nov 6, 10:05 pm, "steven_nospam at Yahoo! Canada"
    >
    >
    >
    > wrote:
    > > The scenario:

    >
    > > 1) One of our servers had mirroring of the rootvg (hdisk1 & hdisk3).
    > > 2) hdisk1 died with messages in errpt - LVs listed as open/stale
    > > 3) Ran rmlvcopy and removed all copies on the failed hdisk1
    > > 4) Was able to reducevg and remove the affected disk.
    > > 5) rmdev -l hdisk1 -d
    > > 6) lsvg -l rootvg shows open/syncd for all LVs with 1 LP = 1 PP

    >
    > > IBM sent a hot-swappable replacement, and we took out the old disk and
    > > inserted the new one. Ran cfgmgr and new disk shows up (new PVID). So
    > > we go ahead and try to add it back into the rootvg.

    >
    > > We extendvg for the new hdisk1 in rootvg, but when we mirrorvg it
    > > tells us that it could not synchronize the data. Checking the system
    > > shows the mirroring in place but most of the LVs remained stale.

    >
    > > Ran rmlvcopy on hd1 (/home) and goes back to syncd. Tried mklvcopy and
    > > then syncvg -l hd1, and goes back to open/stale. *Here is the error:

    >
    > > 0516-934 /etc/syncvg: Unable to synchronize logical volume hd1.

    >
    > > oslevel -s
    > > 5300-07-01-0748

    >
    > > A reboot of the server had no effect.

    >
    > > Also to be noted: *My first thought was the new disk was DOA, but it
    > > was certified, and if I add it to another VG and do mklvcopy/syncvg on
    > > those LVs, it works fine.

    >
    > > We expect to call IBM AIX Support as our next step. Just wondered if
    > > anyone has seen this before

    >
    > So far no.
    > but
    > *- Is the errpt saying anythink ?
    > *- do mirroring one LV at a time - beginning with hd5
    >
    > Search the IBM support side seaching for ' mwc , syncvg ' ;-(
    >
    > cheers
    > Hajo- Hide quoted text -
    >
    > - Show quoted text -


    Hi Hajo,

    When we were first testing the new disk, we had no errors showing up,
    but I just worked with an IBM rep this morning to run a series of
    tests, and now we are getting these errors when we try the syncvg:

    LVM_IO_FAIL
    DISK_ERR2
    DISK_ERR4

    Both the AIX support person and I agreed that it looks like the new
    disk installed is operational, but the LVM_IO_FAIL is pointing to a
    possible SCSI cable or port issue, so we are calling IBM back in to
    take another look at the hardware.

    More news next week when the IBM SE shows up to review the situation.

    Another thing I noticed was that the failing disk (hdisk1) was a 72GB
    and the working one is 144GB (hdisk3). There are sufficient FREE PPs
    on the small disk to handle the USED PPs on the working disk so it is
    not a space issue, and bad block relocation is on and there are no
    signs the issue was a space problem, but I pointed it out to the IBM
    engineer just in case. They said we normally would get a different
    error if insufficient PPs were available on the other disk for
    mirroring.

    Anyway, they will be checking it out and I will update this post when
    I find out more.

  5. Re: Cannot mirrorvg after replacing defective hdisk

    did you format the disk? NOT certify only but format, diag menu item

    diag --> task selection ---> Format media

    This is the real test. If you can format both th epath to the disk and the disk are good

+ Reply to Thread