GPFS detection of Disk Array Loss - Aix

This is a discussion on GPFS detection of Disk Array Loss - Aix ; Hi Hajo My optimism was a bit premature! In fact, the instant failover has not repeated since. I've cut down the environment and I belive the problem must lie in the fibre network area. If I yank a disk out ...

+ Reply to Thread
Page 2 of 2 FirstFirst 1 2
Results 21 to 28 of 28

Thread: GPFS detection of Disk Array Loss

  1. Re: GPFS detection of Disk Array Loss

    Hi Hajo

    My optimism was a bit premature!

    In fact, the instant failover has not repeated since. I've cut down
    the environment and I belive the problem must lie in the fibre network
    area. If I yank a disk out of the array, the detection is instant by
    GPFS. The controller must be signalling back to aix, the disk is
    removed. However, when the controller is takendown, no "disk loss"
    signal is given. I repeated the test by unplugging the fibre switches.
    GPFS again took 15 minutes to detect the drop before failing over.
    I'm not sure where to look. Something is telling aix that the router
    has gone, as I can see the error log entries immediately. What is the
    link to GPFS to tell it to abandon access to the unreachable disk?

    any clues? I know this post has gone cold, but I need any help.

    Thanks

    Rob

  2. Re: GPFS detection of Disk Array Loss

    > In fact, the instant failover has not repeated since. I've cut down
    > the environment and I belive the problem must lie in the fibre network
    > area. If I yank a disk out of the array, the detection is instant by
    > GPFS.

    I assume that you are using multipath access to your disk. Thus i my
    understanding you disabled a LUN since a simple disk failure should
    not lead to a GPFS failure.


    > The controller must be signalling back to aix, the disk is
    > removed. However, when the controller is takendown, no "disk loss"
    > signal is given.

    What do mean with controller down ? On our CX600 we have 2
    controllers. Even in case one controller fails the other one will take
    over.

    I would recreate the GPFS environment by implementing a single node
    GPFS cluster. Thus using only one node. No NSD server or what so ever.
    Also set unmountOnDiskFail to yes.

    Then for testing you should simply remove one fc connection after
    another on that test node and check how the node behaves. As soon as
    the last path to your SAN disk is gone the GPFS fs should be umounted.

    After that do the same procedure for your DISK storage. Meaning to
    remove one fc connections at a time from the storage and see now how
    the node behaves.

    Verify your SAN setup. As an example:
    It is allowed that on a two controller/port SAN array a given node can
    connect from any hba to any port on the storage or not ?

    hth
    Hajo

  3. Re: GPFS detection of Disk Array Loss

    I'm setting it up for single node. I'll let you know later.

    Yes I have multipath. Two hba's per server. One goes to each switch.
    DS4300 does have two controllers.
    To simulate disk failure I pulled out the spindle with the GPFS FS on.
    (the other half of the fail group is on the other Array on site B).
    GPFS on all nodes on both sites caried on fine. GPFS took down the
    removed nsd. Two controllers doesn;t help in this case, as there's no
    data disk.

    All I'm trying to do is simulate total site failure. So powering down
    the rack or the array takes both controllers in the downed array out.
    In this case, GPFS on the surviving site doesn't failover.

    SAN setup is:

    Server has 2 hbas. Each connected to a separate switch. Each switch
    connects to a single controller on each of the two Arrays.
    As this is spread over two sites, each "single switch" is actually two
    switches, one in each site, joined in the same zone, so to act as one
    switch.

    I'll simplify and feedback

    Cheers

    Rob

  4. Re: GPFS detection of Disk Array Loss

    Hi

    I've cleared down the GPFS to 1 node. This still has a filesystem,sfs,
    split into 2 failgroups, with half on each disk array. sfs1 and sfs2.
    (+fs quorum)
    The NSDs still have a primary server specified, but it's the main, and
    only, node.

    It mounts up fine. Access is normal.

    I can unplug the first hba from the node and all is fine. Both sfs1
    and sfs2 are up, ready. the filesystem access is normal.
    I unplug the second and, access to filesystem is blocked and it's
    difficult to see the output from mmlsdisk sfs, but I suspect it still
    "thinks" the NSDs are up and ready.

    After putting the wires back in and resetting the env, I repeated from
    the other end.

    I have 4 fibres in each array.
    I can remove 3 with no effect on GPFS. The fabric resilience gives
    access to both arrays still, so no problem.
    When I remove the final fibre, Access to the SFS is blocked and all IO
    is left hanging. (After some intial cache access on read only basis).
    mmlsdisk sfs, reports that sfs1 and sfs2 are up ready!! After 2
    minutes, sfs2 is now reported as "unavailable, down" and IO is allowed
    access via sfs1 to SFS filesystem.

    When I plug the fibres back in, it takes 15 seconds before "mmchdisk
    sfs start -d sfs2" allows sfs2 to be brought "up" again. prior to that
    IO error is returned. Aftert a quick restripe, everything works fine
    again.

    FYI I did get the times down to 2 minutes approximately, by setting
    the dar0 and dar1 attributes "switch_retries" to 0

    What is aen_frequency and hlthchk_freq ? They are both set at 600
    seconds by default.

    Many thanks for helping

    Rob




  5. Re: GPFS detection of Disk Array Loss

    ....
    In case you have Gigabit PCI-E Single or Dual Port Fibre Channel
    adapters (feature codes 5773 or 5774) please read and apply latest
    fix.

    http://www14.software.ibm.com/webapp...D=4012#IY99608

    hth
    Hajo

  6. Re: GPFS detection of Disk Array Loss

    Hi Hajo

    I've checked the devices, we have 5773/4's.

    I've raised a PMR with IBM and they've come back to tell me that "AIX
    and the DS4000 are working as designed."

    This is a bit dissapointing as the documentation says it provides
    transparent failover on site failure. Although failover does occur,
    eventually, I would say that this is not entirely transparent. Their
    case is that GPFS is relying on the underlying SAN and OS to respond
    to the site loss. They say that LVM suffers the same issues, it's not
    related to GPFS.

    I have now got our team reworking our software to workaround GPFS,
    hopefully this will be successful, although there will be extended
    impacts to analyse.

    Thanks for your help, again and again.
    Have a good holiday time

    Rob

  7. Re: GPFS detection of Disk Array Loss

    Why not using GPFS replication of data and metadata ? In this case a
    site failure does no harm at all since it is using the mirrored path ?

    Hajo

  8. Re: GPFS detection of Disk Array Loss

    On 21 Dec, 14:15, Hajo Ehlers wrote:
    > Why not using GPFS replication of data and metadata ? In this case a
    > site failure does no harm at all since it is using the mirrored path ?
    >
    > Hajo


    Hi Hajo

    This scenario is using replication. There is a good copy on each site.
    GPFS freezes while the replication fails, as the site has dropped.
    (mmlsdisk sfs, still shows both mirrors as ready,up) When the disk is
    recognised as not available any more(1-2 mins) then GPFS is happy to
    continue on the surviving copy.

    I had feedback from IBM who are happy this is working within design.
    I'm not convinced this is "transparent failover" myself, with such a
    delay. The view from Blue is that the OS(aix) and the SAN are
    responsible for the delay. This seems odd as GPFS is going to be
    mostly working on top off a SAN.

    We've re-integrated that application over the holiday,and now don't
    rely so heavily on GPFS. Our main problem is now worked around,
    although we will have to live with the delay is certain cases.

    Thanks for your help

    Rob

+ Reply to Thread
Page 2 of 2 FirstFirst 1 2