Problem mounting r/w (r/o works) - NFS

This is a discussion on Problem mounting r/w (r/o works) - NFS ; I am having trouble mounting a directory read/write from one machine on two other machines. I can mount directories read-only with no problmes. I can mount at least one directory read/write from one machine to another. 10.0.0.2 M2Hub.localdomain Red Hat ...

+ Reply to Thread
Results 1 to 6 of 6

Thread: Problem mounting r/w (r/o works)

  1. Problem mounting r/w (r/o works)

    I am having trouble mounting a directory read/write from one machine on
    two other machines. I can mount directories read-only with no problmes.
    I can mount at least one directory read/write from one machine to
    another.

    10.0.0.2 M2Hub.localdomain Red Hat Linux 9

    fstab: 10.0.0.3:/home /home3 nfs rsize=8192,hard,intr,bg,rw 0 0
    10.0.0.3:/pub3 /pub3 nfs rsize=8192,hard,intr,bg,ro 0 0
    ...
    10.0.0.4:/home /home4 nfs rsize=8192,hard,intr,bg,rw 0 0
    10.0.0.4:/pub4 /pub4 nfs rsize=8192,hard,intr,bg,ro 0 0

    exports:/home *(rw,no_root_squash,async)
    /pub2 M2Gate(rw,noroot_squash,async)
    10.0.0.3(rw,no_root_squash,async) *(ro,async)

    hosts: 10.0.0.2 M2Hub.localdomain M2Hub
    10.0.0.3 M2Dual.localdomain M2Dual
    10.0.0.4 VMWare.localdomain VMWare

    # mount -a
    mount: 10.0.0.3:/home failed, reason given by server: Permission denied
    #ls -ld /home
    drwxr-xr-x 13 root root 4096 Aug 5 20:57 /home

    tail of /var/log/messages on 10.0.0.3:
    Aug 9 23:31:44 M2Dual rpc.mountd: authenticated mount request from
    M2Hub.localdomain;923 for /home (/home)
    Aug 9 23:31:44 M2Dual rpc.mountd: getfh failed: Operation not premitted


    10.0.0.3 M2Dual.localdomain Red Hat Linux 9

    fstab: 10.0.0.2:/home /home2 nfs rw,bg,hard,intr,nosuid 0 0
    10.0.0.2:/pub2 /pub2 nfs ro,bg,hard,intr,nosuid 0 0
    ...
    10.0.0.4:/home /home4 nfs rw,bg,hard,intr,nosuid 0 0
    10.0.0.4:/pub4 /pub4 nfs ro,bg,hard,intr,nosuid 0 0

    exports:/pub3 m2hub(ro,async) vmware(ro,async) *.localdomain(ro,async)
    10/8(ro,async)
    /home m2hub(rw,async,no_root_squash)
    vmware(rw,async,no_root_squash) 10/8(rw,async) *.localdomain(rw,async)

    hosts: 10.0.0.2 M2Hub.localdomain M2Hub
    10.0.0.3 M2Dual.localdomain M2Dual
    10.0.0.4 VMWare.localdomain VMWare

    # mount -a
    mount: 10.0.0.2:/home failed, reason given by server: Permission denied
    # ls -ld /home

    tail of /var/log/messages from 10.0.0.2:
    Aug 9 23:46:15 M2Hub rpc.mountd: authenticated mount request from
    M2Dual.localdomain:860 for /home (/home)
    Aug 0 23:46:15 M2Hub rpc.mountd: getfh failed: Operation not permitted


    10.0.0.3 VMWare.localdomain Red Hat Linux 7.2

    fstab: 10.0.0.2:/pub2 /pub2 nfs bg,ro 0 0
    10.0.0.2:/home /home2 nfs bg,rw 0 0
    ...
    10.0.0.3:/pub3 /pub3 nfs bg,ro 0 0
    10.0.0.3:/home /home3 nfs bg,ro 0 0

    exports:/home *(rw,nohide,async)
    /pub4 10.0.0.2(rw,nohide,sync) 10.0.0.3(rw,nohide,sync)
    10.0.0.0/8(ro,nohide)

    hosts: 127.0.0.1 VMWare localhost.localdomain localhost

    # mount -a
    mount: 10.0.0.2:/home failed, reason given by server: Permission denied
    mount: 10.0.0.3:/home failed, reason given by server: Permission denied
    # ls -ld /home
    drwxr-xr-x 11 root root 4096 Jul 29 23:23 /home

    tail of /var/log/messages form 10.0.0.2 and 10.0.0.3:
    Aug 10 00:05:00 M2Hub rpc.mountd: authenticted mount request form
    VMWare.localdomain:749 for /home (/home)
    Aug 10 00:05:00 M2Hub rpc.mountd: getfh failed: Operation not permitted
    Aug 10 00:05:00 M2Dual rpc.mountd: authenticated mount request from
    VMWare.localdomain:751 for /home (/home)
    Aug 10 00:05:00 M2Dual rpc.mountd: getfh failed: Operation not premitted


    I think this is related to a problem with exporting the file.
    # Exportfs -av on 10.0.0.3 produced the following:
    ....
    exporting M2Hub.localdomain:/ to kernel
    VMWare.localdomain:/: Invalid argument

    Why it has dropped the 'home' from '/home' is probably the answer to
    this puzzle.

    Max.TenEyck.Woodbury@verizon.net

  2. Re: Problem mounting r/w (r/o works)

    Does anybody have an idea on this. It's coming up on 48 hours since I
    posted this and there is no evidence that anybody beside myself has
    looked at the problem.

    "Max T.E. Woodbury" wrote:
    >
    > I am having trouble mounting a directory read/write from one machine on
    > two other machines. I can mount directories read-only with no problmes.
    > I can mount at least one directory read/write from one machine to
    > another.
    >
    > 10.0.0.2 M2Hub.localdomain Red Hat Linux 9
    >
    > fstab: 10.0.0.3:/home /home3 nfs rsize=8192,hard,intr,bg,rw 0 0
    > 10.0.0.3:/pub3 /pub3 nfs rsize=8192,hard,intr,bg,ro 0 0
    > ...
    > 10.0.0.4:/home /home4 nfs rsize=8192,hard,intr,bg,rw 0 0
    > 10.0.0.4:/pub4 /pub4 nfs rsize=8192,hard,intr,bg,ro 0 0
    >
    > exports:/home *(rw,no_root_squash,async)
    > /pub2 M2Gate(rw,noroot_squash,async)
    > 10.0.0.3(rw,no_root_squash,async) *(ro,async)
    >
    > hosts: 10.0.0.2 M2Hub.localdomain M2Hub
    > 10.0.0.3 M2Dual.localdomain M2Dual
    > 10.0.0.4 VMWare.localdomain VMWare
    >
    > # mount -a
    > mount: 10.0.0.3:/home failed, reason given by server: Permission denied
    > #ls -ld /home
    > drwxr-xr-x 13 root root 4096 Aug 5 20:57 /home
    >
    > tail of /var/log/messages on 10.0.0.3:
    > Aug 9 23:31:44 M2Dual rpc.mountd: authenticated mount request from
    > M2Hub.localdomain;923 for /home (/home)
    > Aug 9 23:31:44 M2Dual rpc.mountd: getfh failed: Operation not premitted
    >
    > 10.0.0.3 M2Dual.localdomain Red Hat Linux 9
    >
    > fstab: 10.0.0.2:/home /home2 nfs rw,bg,hard,intr,nosuid 0 0
    > 10.0.0.2:/pub2 /pub2 nfs ro,bg,hard,intr,nosuid 0 0
    > ...
    > 10.0.0.4:/home /home4 nfs rw,bg,hard,intr,nosuid 0 0
    > 10.0.0.4:/pub4 /pub4 nfs ro,bg,hard,intr,nosuid 0 0
    >
    > exports:/pub3 m2hub(ro,async) vmware(ro,async) *.localdomain(ro,async)
    > 10/8(ro,async)
    > /home m2hub(rw,async,no_root_squash)
    > vmware(rw,async,no_root_squash) 10/8(rw,async) *.localdomain(rw,async)
    >
    > hosts: 10.0.0.2 M2Hub.localdomain M2Hub
    > 10.0.0.3 M2Dual.localdomain M2Dual
    > 10.0.0.4 VMWare.localdomain VMWare
    >
    > # mount -a
    > mount: 10.0.0.2:/home failed, reason given by server: Permission denied
    > # ls -ld /home
    >
    > tail of /var/log/messages from 10.0.0.2:
    > Aug 9 23:46:15 M2Hub rpc.mountd: authenticated mount request from
    > M2Dual.localdomain:860 for /home (/home)
    > Aug 0 23:46:15 M2Hub rpc.mountd: getfh failed: Operation not permitted
    >
    > 10.0.0.3 VMWare.localdomain Red Hat Linux 7.2
    >
    > fstab: 10.0.0.2:/pub2 /pub2 nfs bg,ro 0 0
    > 10.0.0.2:/home /home2 nfs bg,rw 0 0
    > ...
    > 10.0.0.3:/pub3 /pub3 nfs bg,ro 0 0
    > 10.0.0.3:/home /home3 nfs bg,ro 0 0
    >
    > exports:/home *(rw,nohide,async)
    > /pub4 10.0.0.2(rw,nohide,sync) 10.0.0.3(rw,nohide,sync)
    > 10.0.0.0/8(ro,nohide)
    >
    > hosts: 127.0.0.1 VMWare localhost.localdomain localhost
    >
    > # mount -a
    > mount: 10.0.0.2:/home failed, reason given by server: Permission denied
    > mount: 10.0.0.3:/home failed, reason given by server: Permission denied
    > # ls -ld /home
    > drwxr-xr-x 11 root root 4096 Jul 29 23:23 /home
    >
    > tail of /var/log/messages form 10.0.0.2 and 10.0.0.3:
    > Aug 10 00:05:00 M2Hub rpc.mountd: authenticted mount request form
    > VMWare.localdomain:749 for /home (/home)
    > Aug 10 00:05:00 M2Hub rpc.mountd: getfh failed: Operation not permitted
    > Aug 10 00:05:00 M2Dual rpc.mountd: authenticated mount request from
    > VMWare.localdomain:751 for /home (/home)
    > Aug 10 00:05:00 M2Dual rpc.mountd: getfh failed: Operation not premitted
    >
    > I think this is related to a problem with exporting the file.
    > # Exportfs -av on 10.0.0.3 produced the following:
    > ...
    > exporting M2Hub.localdomain:/ to kernel
    > VMWare.localdomain:/: Invalid argument
    >
    > Why it has dropped the 'home' from '/home' is probably the answer to
    > this puzzle.
    >
    > Max.TenEyck.Woodbury@verizon.net


  3. Re: Problem mounting r/w (r/o works)

    so are you trying to export an nfs mounted filesystem again via nfs
    (sure looks that way).

    Max T.E. Woodbury wrote:
    > Does anybody have an idea on this. It's coming up on 48 hours since I
    > posted this and there is no evidence that anybody beside myself has
    > looked at the problem.
    >
    > "Max T.E. Woodbury" wrote:
    >
    >>I am having trouble mounting a directory read/write from one machine on
    >>two other machines. I can mount directories read-only with no problmes.
    >>I can mount at least one directory read/write from one machine to
    >>another.
    >>
    >>10.0.0.2 M2Hub.localdomain Red Hat Linux 9
    >>
    >>fstab: 10.0.0.3:/home /home3 nfs rsize=8192,hard,intr,bg,rw 0 0
    >> 10.0.0.3:/pub3 /pub3 nfs rsize=8192,hard,intr,bg,ro 0 0
    >> ...
    >> 10.0.0.4:/home /home4 nfs rsize=8192,hard,intr,bg,rw 0 0
    >> 10.0.0.4:/pub4 /pub4 nfs rsize=8192,hard,intr,bg,ro 0 0
    >>
    >>exports:/home *(rw,no_root_squash,async)
    >> /pub2 M2Gate(rw,noroot_squash,async)
    >>10.0.0.3(rw,no_root_squash,async) *(ro,async)
    >>
    >>hosts: 10.0.0.2 M2Hub.localdomain M2Hub
    >> 10.0.0.3 M2Dual.localdomain M2Dual
    >> 10.0.0.4 VMWare.localdomain VMWare
    >>
    >># mount -a
    >>mount: 10.0.0.3:/home failed, reason given by server: Permission denied
    >>#ls -ld /home
    >>drwxr-xr-x 13 root root 4096 Aug 5 20:57 /home
    >>
    >>tail of /var/log/messages on 10.0.0.3:
    >>Aug 9 23:31:44 M2Dual rpc.mountd: authenticated mount request from
    >>M2Hub.localdomain;923 for /home (/home)
    >>Aug 9 23:31:44 M2Dual rpc.mountd: getfh failed: Operation not premitted
    >>
    >>10.0.0.3 M2Dual.localdomain Red Hat Linux 9
    >>
    >>fstab: 10.0.0.2:/home /home2 nfs rw,bg,hard,intr,nosuid 0 0
    >> 10.0.0.2:/pub2 /pub2 nfs ro,bg,hard,intr,nosuid 0 0
    >> ...
    >> 10.0.0.4:/home /home4 nfs rw,bg,hard,intr,nosuid 0 0
    >> 10.0.0.4:/pub4 /pub4 nfs ro,bg,hard,intr,nosuid 0 0
    >>
    >>exports:/pub3 m2hub(ro,async) vmware(ro,async) *.localdomain(ro,async)
    >>10/8(ro,async)
    >> /home m2hub(rw,async,no_root_squash)
    >>vmware(rw,async,no_root_squash) 10/8(rw,async) *.localdomain(rw,async)
    >>
    >>hosts: 10.0.0.2 M2Hub.localdomain M2Hub
    >> 10.0.0.3 M2Dual.localdomain M2Dual
    >> 10.0.0.4 VMWare.localdomain VMWare
    >>
    >># mount -a
    >>mount: 10.0.0.2:/home failed, reason given by server: Permission denied
    >># ls -ld /home
    >>
    >>tail of /var/log/messages from 10.0.0.2:
    >>Aug 9 23:46:15 M2Hub rpc.mountd: authenticated mount request from
    >>M2Dual.localdomain:860 for /home (/home)
    >>Aug 0 23:46:15 M2Hub rpc.mountd: getfh failed: Operation not permitted
    >>
    >>10.0.0.3 VMWare.localdomain Red Hat Linux 7.2
    >>
    >>fstab: 10.0.0.2:/pub2 /pub2 nfs bg,ro 0 0
    >> 10.0.0.2:/home /home2 nfs bg,rw 0 0
    >> ...
    >> 10.0.0.3:/pub3 /pub3 nfs bg,ro 0 0
    >> 10.0.0.3:/home /home3 nfs bg,ro 0 0
    >>
    >>exports:/home *(rw,nohide,async)
    >> /pub4 10.0.0.2(rw,nohide,sync) 10.0.0.3(rw,nohide,sync)
    >>10.0.0.0/8(ro,nohide)
    >>
    >>hosts: 127.0.0.1 VMWare localhost.localdomain localhost
    >>
    >># mount -a
    >>mount: 10.0.0.2:/home failed, reason given by server: Permission denied
    >>mount: 10.0.0.3:/home failed, reason given by server: Permission denied
    >># ls -ld /home
    >>drwxr-xr-x 11 root root 4096 Jul 29 23:23 /home
    >>
    >>tail of /var/log/messages form 10.0.0.2 and 10.0.0.3:
    >>Aug 10 00:05:00 M2Hub rpc.mountd: authenticted mount request form
    >>VMWare.localdomain:749 for /home (/home)
    >>Aug 10 00:05:00 M2Hub rpc.mountd: getfh failed: Operation not permitted
    >>Aug 10 00:05:00 M2Dual rpc.mountd: authenticated mount request from
    >>VMWare.localdomain:751 for /home (/home)
    >>Aug 10 00:05:00 M2Dual rpc.mountd: getfh failed: Operation not premitted
    >>
    >>I think this is related to a problem with exporting the file.
    >># Exportfs -av on 10.0.0.3 produced the following:
    >>...
    >>exporting M2Hub.localdomain:/ to kernel
    >>VMWare.localdomain:/: Invalid argument
    >>
    >>Why it has dropped the 'home' from '/home' is probably the answer to
    >>this puzzle.
    >>
    >>Max.TenEyck.Woodbury@verizon.net



  4. Re: Problem mounting r/w (r/o works)

    "Marc D. Behr" wrote:
    >
    > so are you trying to export an nfs mounted filesystem again via nfs
    > (sure looks that way).


    I've found the problem. It is an arrogant implementation of an ambiguous
    specification and a very poor selection of error codes.

    The ambiguous term is 'reexport'. If you read the documentation of NFS,
    it is stated plainly that you can NOT import a file system using NFS
    and export it again using NFS. (I.E. the exporting of an nfs mounted
    file system Marc is referring to.) For clarity I will call this
    import/reexport.

    The problem I have is that one part of a local file system is being
    exported R/W and a subset of that file system is being exported R/O.
    For clarity I will call this activity double export.

    Import/reexport can cause extra network traffic. Arguably, it can be
    useful at bridge and routing points, but the protocol expressly forbids
    it.

    Double export is another matter. While I have not dug up the RFC,
    several otherwise competent books do not mention this as being a
    problem. It can be mildly confusing since you may be able to change
    a file using one path name and have the changes appear magically
    when accessed using another path name. The access permissions may also
    be different using the different names. Cache entries may be stale as
    a result of this kind of thing, however the problem is no worse than
    the problems associated with any local changes. Other than this
    confusion, there is no insuperable reason for forbidding double
    exports.

    However, the Linux implementation does exactly that. What makes this
    confusing is that it labels the attempt with an 'invalid argument'
    error. This error is usually reserved for badly formed arguments,
    with missing components, bad pointers or syntax errors. The error code
    that should be returned should mean 'request refused' since
    the refusal to export the sub-tree is a policy decision. However
    this is not the worst part of the problem. It might be possible to
    work around the problem by ordering the entries in the /etc/exports
    file from less restrictive to more restrictive, much as the ordering
    of ACLs can be used for precise control of file permissions. However
    the /etc/exports parsing code arbitrarily reorders the requests,
    making this impossible.

    I got around this problem by only specifying the most liberal access
    on the server and used the client side specifications to restrict the
    access modes. This solution has some very obvious security problems
    associated with it. I can do this because I can trust all the people
    who use my sub-net, but I would have a very serious problem if I had
    to meet a serious security audit. (NFS has a poor security reputation
    precisely because of problems like this.)

    Max

  5. Re: Problem mounting r/w (r/o works)

    Now I see what is happening a bit better. This all has to do with
    FSID's (file system ID's). During the mount operation, the NFS client
    asks mountd on the server for the filehandle for an exported fileystem.
    The mountd then returns a FSID for this filesystem, which is a unique
    handle for every mounted filesystem on the sever. So, when you ask for
    the handle for /export/home, let's say you get back 0x12345 as the
    filehandle. now, according to the exports file, /export/home is rw.

    You also have /export/home/allmine exported as ro (which is technically
    illegal, because you have already exported the filesystem with the
    handle 0x12345). Depending on the server, either it ignores the new
    entry, or it replaces it. The client then sends a mount request to the
    server for /export/home/allmine. The mountd on the server looks at the
    request, and it finds the one (and only) entry for that particular
    filesystem (with the unique id of 0x12345) and returns that entry.
    The client then sees that he got back the entry for the parent, but that
    you asked for the child, so he simply mounts the child based on the
    parents permissions.

    My explanation may be a bit 'fuzzy', but I hope that this helps.

    Marc

    Max T.E. Woodbury wrote:
    > "Marc D. Behr" wrote:
    >
    >>so are you trying to export an nfs mounted filesystem again via nfs
    >>(sure looks that way).

    >
    >
    > I've found the problem. It is an arrogant implementation of an ambiguous
    > specification and a very poor selection of error codes.
    >
    > The ambiguous term is 'reexport'. If you read the documentation of NFS,
    > it is stated plainly that you can NOT import a file system using NFS
    > and export it again using NFS. (I.E. the exporting of an nfs mounted
    > file system Marc is referring to.) For clarity I will call this
    > import/reexport.
    >
    > The problem I have is that one part of a local file system is being
    > exported R/W and a subset of that file system is being exported R/O.
    > For clarity I will call this activity double export.
    >
    > Import/reexport can cause extra network traffic. Arguably, it can be
    > useful at bridge and routing points, but the protocol expressly forbids
    > it.
    >
    > Double export is another matter. While I have not dug up the RFC,
    > several otherwise competent books do not mention this as being a
    > problem. It can be mildly confusing since you may be able to change
    > a file using one path name and have the changes appear magically
    > when accessed using another path name. The access permissions may also
    > be different using the different names. Cache entries may be stale as
    > a result of this kind of thing, however the problem is no worse than
    > the problems associated with any local changes. Other than this
    > confusion, there is no insuperable reason for forbidding double
    > exports.
    >
    > However, the Linux implementation does exactly that. What makes this
    > confusing is that it labels the attempt with an 'invalid argument'
    > error. This error is usually reserved for badly formed arguments,
    > with missing components, bad pointers or syntax errors. The error code
    > that should be returned should mean 'request refused' since
    > the refusal to export the sub-tree is a policy decision. However
    > this is not the worst part of the problem. It might be possible to
    > work around the problem by ordering the entries in the /etc/exports
    > file from less restrictive to more restrictive, much as the ordering
    > of ACLs can be used for precise control of file permissions. However
    > the /etc/exports parsing code arbitrarily reorders the requests,
    > making this impossible.
    >
    > I got around this problem by only specifying the most liberal access
    > on the server and used the client side specifications to restrict the
    > access modes. This solution has some very obvious security problems
    > associated with it. I can do this because I can trust all the people
    > who use my sub-net, but I would have a very serious problem if I had
    > to meet a serious security audit. (NFS has a poor security reputation
    > precisely because of problems like this.)
    >
    > Max



  6. Re: Problem mounting r/w (r/o works)

    "Marc D. Behr" wrote:
    >
    > Now I see what is happening a bit better. This all has to do with
    > FSID's (file system ID's). During the mount operation, the NFS client
    > asks mountd on the server for the filehandle for an exported fileystem.
    > The mountd then returns a FSID for this filesystem, which is a unique
    > handle for every mounted filesystem on the sever. So, when you ask for
    > the handle for /export/home, let's say you get back 0x12345 as the
    > filehandle. now, according to the exports file, /export/home is rw.
    >
    > You also have /export/home/allmine exported as ro (which is technically
    > illegal, because you have already exported the filesystem with the
    > handle 0x12345).


    1) Why is it illegal? Besides it ends up doing the equivalent of
    exporting /export/home/allmine and refusing to export /export/home.
    Not exporting /export/home/allmine is my work-around.

    2) In the real case /home was the root of the file system, not
    /home/pub but /home/pub gets exported and /home does not.

    3) There are various 'export' points involved. /var/lib/nfs/etab,
    /var/lib/nfs/xtab and the kernel tables which can be checked using
    /proc/fs/nfs/exports in addition to /etc/exports. The export of
    /pub (a.k.a /home/pub) and /home both make it into /var/lib/nfs/etab
    but only /pub makes it into /var/lib/nfs/xtab and
    /proc/fs/nfs/exports. The entries in the various files appear in
    different orders.

    > Depending on the server, either it ignores the new entry, or it
    > replaces it.


    In fact neither happens. On Linux it returns a bogus 'Invalid Argument'
    error code. However that is a bit of a nit. I will NOT quibble that it
    in effect ignores the request.

    > The client then sends a mount request to the
    > server for /export/home/allmine. The mountd on the server looks at
    > the request, and it finds the one (and only) entry for that
    > particular filesystem (with the unique id of 0x12345) and returns
    > that entry. The client then sees that he got back the entry for the
    > parent, but that you asked for the child, so he simply mounts the
    > child based on the parents permissions.


    Once I was able to figure out what was going on, this is almost what is
    going on. In fact that may be exactly what is going on since I have not
    tried to write to the 'ro' directory. However, I would not be surprised
    if the client refused to forward a write request since as far as it is
    concerned the fs is 'ro'.

    > My explanation may be a bit 'fuzzy', but I hope that this helps.


    I can handle the fuzz. What I am having trouble with is the lousy
    error messages and the fact the the server reorders my export requests.
    Subtract EITHER one of those and my debug time would have gone way down.

    Max

+ Reply to Thread