"correct" way to set up NFS for graceful shutdown with cross-mountedvolumes? - NFS

This is a discussion on "correct" way to set up NFS for graceful shutdown with cross-mountedvolumes? - NFS ; I've got a bunch of (mostly Linux) servers that each share data to the others via NFSv3. If I need to shut down the systems they will hang when trying to unmount NFS volumes of a machine that has already ...

+ Reply to Thread
Results 1 to 5 of 5

Thread: "correct" way to set up NFS for graceful shutdown with cross-mountedvolumes?

  1. "correct" way to set up NFS for graceful shutdown with cross-mountedvolumes?

    I've got a bunch of (mostly Linux) servers that each share data to the
    others via NFSv3. If I need to shut down the systems they will hang
    when trying to unmount NFS volumes of a machine that has already shut
    down. What is the best way to set up these servers so this doesn't
    happen? Maybe I should have scripts to try to unmount all nfs volumes
    before actually shutting down the servers.

    I'm guessing that this is somewhat common and has been dealt with before
    so I'd like to hear what people do.

    Thanks,

    Steve


  2. Re: "correct" way to set up NFS for graceful shutdown with cross-mounted volumes?


    Steve Cousins wrote:
    > I've got a bunch of (mostly Linux) servers that each share data to the
    > others via NFSv3. If I need to shut down the systems they will hang
    > when trying to unmount NFS volumes of a machine that has already shut
    > down. What is the best way to set up these servers so this doesn't
    > happen? Maybe I should have scripts to try to unmount all nfs volumes
    > before actually shutting down the servers.
    >
    > I'm guessing that this is somewhat common and has been dealt with before
    > so I'd like to hear what people do.
    >
    > Thanks,
    >
    > Steve


    Solution is easy but maybe not what you want to hear. Don't crossmount
    !

    A few reasons why crossmounts are wrong :
    - doesn't scale
    - difficult to administer
    - easily deadlocks for lot's of reasons
    - backup _may be _ missed as data is spread on lots of places
    - inefficient use of available disc's

    I'm shure other reasons exists.

    If for some strange reason one has to crossmount use an automounter.


  3. Re: "correct" way to set up NFS for graceful shutdown with cross-mountedvolumes?

    phn wrote:

    >Steve Cousins wrote:
    >
    >
    >>I've got a bunch of (mostly Linux) servers that each share data to the
    >>others via NFSv3. If I need to shut down the systems they will hang
    >>when trying to unmount NFS volumes of a machine that has already shut
    >>down. What is the best way to set up these servers so this doesn't
    >>happen? Maybe I should have scripts to try to unmount all nfs volumes
    >>before actually shutting down the servers.
    >>
    >>I'm guessing that this is somewhat common and has been dealt with before
    >>so I'd like to hear what people do.
    >>
    >>Thanks,
    >>
    >>Steve
    >>
    >>

    >
    >Solution is easy but maybe not what you want to hear. Don't crossmount
    >!
    >

    Not really a "solution".

    >A few reasons why crossmounts are wrong :
    >- doesn't scale
    >

    Not sure what you mean

    >- difficult to administer
    >

    I agree

    >- easily deadlocks for lot's of reasons
    >
    >

    Not sure what you mean.

    >- backup _may be _ missed as data is spread on lots of places
    >
    >

    I agree. This is part of "difficult to administer"

    >- inefficient use of available disc's
    >
    >

    Actually, I'm doing it because it is a more efficient use of available
    discs rather than using a monolithic server. We have a small set of
    servers that we run ocean models on and some of the servers have disk
    arrays in them too so they serve data and can run models. We have
    assigned particular users to certain of these servers and have put their
    home directories on the local arrays. However, they can also login to
    other servers and run their models (same home directory) on other
    compute servers if needed. One server also has our primary storage
    attached to it and all servers need access to this too.

    When I say it is efficient, I mean that it is a way for us to add
    storage without a lot of cost. Rather than buying another $12K RAID DAS
    box we can use existing servers to add capacity. It definitely isn't
    efficient to administer when we need to do maintenance and bring the
    servers down cleanly.

    >I'm shure other reasons exists.
    >
    >If for some strange reason one has to crossmount use an automounter.
    >

    Doesn't really help much (as I understand it) since the volumes are
    almost always being used.

    Possibly we are using a different definition of cross-mount (I may not
    be using the correct term). What we have in its simplest form is that
    Server A serves /usr2 to all servers and Server B serves /usr3 to all
    servers (including Server A). So if Server A shuts down and then Server
    B tries to shut down, it hangs when it tries to umount /usr2.

    Should I be using "soft" mounts or be setting timeo and retry?

    Thanks,

    Steve


  4. Re: "correct" way to set up NFS for graceful shutdown with cross-mounted volumes?

    On Tue, 22 Aug 2006 15:25:06 -0400, Steve Cousins
    wrote:

    >phn wrote:
    >
    >>Steve Cousins wrote:
    >>
    >>
    >>>I've got a bunch of (mostly Linux) servers that each share data to the
    >>>others via NFSv3. If I need to shut down the systems they will hang
    >>>when trying to unmount NFS volumes of a machine that has already shut
    >>>down. What is the best way to set up these servers so this doesn't
    >>>happen? Maybe I should have scripts to try to unmount all nfs volumes
    >>>before actually shutting down the servers.
    >>>
    >>>I'm guessing that this is somewhat common and has been dealt with before
    >>>so I'd like to hear what people do.

    >
    >>A few reasons why crossmounts are wrong :
    >>- doesn't scale
    >>

    >Not sure what you mean


    The more hosts you do this with the worse your environment becomes.
    Worse being unstable.


    >>- easily deadlocks for lot's of reasons
    >>
    >>

    >Not sure what you mean.


    If one of those servers eats it for some reason, you just got hung
    mounts all over your environment. If this is Solaris 8 or above you
    can force the umounts (usually), if it's Linux of almost any flavor
    you're likely hosed and have to reboot your entire environment to
    clear it.

    If you want to take a server down cleanly you'll have to unmount that
    export on every host in the environment. That circles back to the
    no-scale aspect mentioned.


    >>

    >Actually, I'm doing it because it is a more efficient use of available
    >discs rather than using a monolithic server. We have a small set of
    >servers that we run ocean models on and some of the servers have disk
    >arrays in them too so they serve data and can run models. We have
    >assigned particular users to certain of these servers and have put their
    >home directories on the local arrays. However, they can also login to
    >other servers and run their models (same home directory) on other
    >compute servers if needed. One server also has our primary storage
    >attached to it and all servers need access to this too.


    We have been doing crossmounts in our environment and it's now killing
    us with 1k+ nodes. The stability of the environment is a constant
    issue and we are working fast and hard to remove them. Problem is,
    once users get accustomed to a certain way of doing things it's hard
    to change. Even though they complain, and rightly so, about the lack
    of stability, they are extremely reluctant to change their processes
    and scripts so that we can remove crossmounts.
    Something to keep in mind if you think you're going to grow.

    Start down the scaleable path now, don't wait. Even if it costs you
    more money or you lose some performance by buying lower cost disk, go
    down the path of scaleability if at all possible.


    >
    >Possibly we are using a different definition of cross-mount (I may not
    >be using the correct term). What we have in its simplest form is that
    >Server A serves /usr2 to all servers and Server B serves /usr3 to all
    >servers (including Server A). So if Server A shuts down and then Server
    >B tries to shut down, it hangs when it tries to umount /usr2.


    Nope, you're using the term correctly. I;'m not aware of any real way
    to accomplish what you're asking. I suppose it could be done via
    massive scripting and a foolproof push mechanism but see above.

    >
    >Should I be using "soft" mounts or be setting timeo and retry?


    You'll still have hung mounts, they just won't hang the system. But
    if you want to use that mount again you'll have to muck with it as
    above.

    ~F

  5. Re: "correct" way to set up NFS for graceful shutdown with cross-mountedvolumes?



    Faeandar wrote:
    > We have been doing crossmounts in our environment and it's now killing
    > us with 1k+ nodes. The stability of the environment is a constant
    > issue and we are working fast and hard to remove them. Problem is,
    > once users get accustomed to a certain way of doing things it's hard
    > to change. Even though they complain, and rightly so, about the lack
    > of stability, they are extremely reluctant to change their processes
    > and scripts so that we can remove crossmounts.
    > Something to keep in mind if you think you're going to grow.
    >
    > Start down the scaleable path now, don't wait. Even if it costs you
    > more money or you lose some performance by buying lower cost disk, go
    > down the path of scaleability if at all possible.


    Thanks Faeandar and Peter,

    I'll see if we can rework this. I always thought it was just that I
    didn't have it set up correctly but it looks like it is just not
    something we should expect to be able to do gracefully.

    We currently have a couple of DAS boxes totalling about 15 TB (usable)
    on one server which is our primary storage. We then have another 10 TB
    mixed between four of our compute servers. It is transparent to our
    users but when I have to do maintenance it is always painful. It is 90%
    Linux and I might try scripting it by going through the /proc file
    system to see which process are using nfs mounts (more accurate than
    lsof it seems) and kicking those processes off before umounting. It
    seems like it is prone to error though so probably some shift in how we
    do things is in order.
    Oh well I guess those "cheap" disk systems come at a cost.

    Thanks again,

    Steve

+ Reply to Thread