NFS v4 and pNFS - NFS

This is a discussion on NFS v4 and pNFS - NFS ; In article , Rick Jones writes: |> Nick Maclaren wrote: |> > Well, yes, if you correct it to say that it is only a problem if the |> > network is VERY lossy. |> |> What would be your ...

+ Reply to Thread
Page 2 of 3 FirstFirst 1 2 3 LastLast
Results 21 to 40 of 50

Thread: NFS v4 and pNFS

  1. Re: NFS v4 and pNFS


    In article ,
    Rick Jones writes:
    |> Nick Maclaren wrote:
    |> > Well, yes, if you correct it to say that it is only a problem if the
    |> > network is VERY lossy.
    |>
    |> What would be your definition of VERY here?

    Enough to make the overall packet loss cause a significant performance
    degradation - for a 22-fragment packet, that means above 1% if the
    protocol and implementation are good, above 0.1% if they are poor,
    or whatever if they are complete crocks.


    Regards,
    Nick Maclaren.

  2. Re: NFS v4 and pNFS

    Nick Maclaren wrote:
    > Enough to make the overall packet loss cause a significant
    > performance degradation - for a 22-fragment packet, that means above
    > 1% if the protocol and implementation are good, above 0.1% if they
    > are poor, or whatever if they are complete crocks.


    OK, those sound reasonable. Thanks.

    rick jones
    --
    oxymoron n, commuter in a gas-guzzling luxury SUV with an American flag
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  3. Re: NFS v4 and pNFS


    Faeandar wrote:
    > I've seen the spec's for both but can someone describe the features in
    > layman's terms? I understand, for the most part, how NFS v3 works.
    > I'm not a total novice, but by no means a coder.


    You may find
    http://nfsworld.blogspot.com/2005/12...t-lisa-05.html
    and http://www.snia.org/events/past/deve...utureOfNFS.pdf
    of interest.


  4. Re: NFS v4 and pNFS


    Brane2 wrote:
    > Nick Maclaren wrote:
    >
    > > A quick glance at pNFS makes me think that it is yet another fiasco in
    > > the making.


    Yeah but Nick says that about everything involving NFS.
    It's a wonder he uses it.

    >
    > Does this mean that NFSv4 is a fiasco ?
    > I wondered why I can't find anyone using it despite all the hype
    > around...
    >
    > I have tried to use it myself, but can't make it to work reliably, or
    > at all (most cases)...


    If you are using Solaris 10, you are probably using NFSv4.

    If you are using Linux one should expect when combining a free
    operating system with complex protocol that there will be years
    of pain before the free operating system gets it right. This was
    the experience with NFSv2 and Linux, and NFSv3 and Linux, and
    NFSv[23] aren't even that complex. That said, rather than waiting ten
    years after the RFC was published, as was the case with NFSv3,
    the Linux NFSv4 effort got going from the get go. So there's a glimmer
    of light at the end of the tunnel.


  5. Re: NFS v4 and pNFS


    In article <1151420450.592689.165070@c74g2000cwc.googlegroups. com>,
    Mike Eisler" writes:
    |> Brane2 wrote:
    |> >
    |> > > A quick glance at pNFS makes me think that it is yet another fiasco in
    |> > > the making.
    |>
    |> Yeah but Nick says that about everything involving NFS.
    |> It's a wonder he uses it.

    Er, no. Please correct your prejudices. NFS may be pretty ghastly
    in many respects, but it hasn't been a complete fiasco since the very
    early days.

    |> If you are using Solaris 10, you are probably using NFSv4.

    Hmm. Maybe, but a hell of a lot of the Solaris 10 administrators I
    know of configured it to run NFS v3.

    |> If you are using Linux one should expect when combining a free
    |> operating system with complex protocol that there will be years
    |> of pain before the free operating system gets it right. ....

    In what way is that different from the experiences with paid-for
    operating systems?

    Solaris (and, previously, SunOS) did better than most systems with
    NFS because both they and NFS were developed by Sun.


    Regards,
    Nick Maclaren.

  6. Re: NFS v4 and pNFS

    Nick Maclaren wrote:
    > In article <1151420450.592689.165070@c74g2000cwc.googlegroups. com>,
    > Mike Eisler" writes:
    > |> Brane2 wrote:
    > |> >
    > |> > > A quick glance at pNFS makes me think that it is yet another fiasco in
    > |> > > the making.
    > |>
    > |> Yeah but Nick says that about everything involving NFS.
    > |> It's a wonder he uses it.
    >
    > Er, no. Please correct your prejudices. NFS may be pretty ghastly


    Nothing pre-judged at all Nick.
    I'm just going by past performance.

    > in many respects, but it hasn't been a complete fiasco since the very
    > early days.
    >
    > |> If you are using Solaris 10, you are probably using NFSv4.
    >
    > Hmm. Maybe, but a hell of a lot of the Solaris 10 administrators I
    > know of configured it to run NFS v3.


    Really? Why?

    >
    > |> If you are using Linux one should expect when combining a free
    > |> operating system with complex protocol that there will be years
    > |> of pain before the free operating system gets it right. ....
    >
    > In what way is that different from the experiences with paid-for
    > operating systems?


    Paid for operating systems have paid full time programmers on staff.

    >
    > Solaris (and, previously, SunOS) did better than most systems with
    > NFS because both they and NFS were developed by Sun.


    Well Sun wasn't even first out the door with NFSv3; Digital Equipment
    was. 10 years before Linux. Past performance of open source continues
    to be a predicter of its future performance.


  7. Re: NFS v4 and pNFS


    In article <1151425482.620267.88370@x69g2000cwx.googlegroups.c om>,
    "Mike Eisler" writes:
    |> >
    |> > |> If you are using Solaris 10, you are probably using NFSv4.
    |> >
    |> > Hmm. Maybe, but a hell of a lot of the Solaris 10 administrators I
    |> > know of configured it to run NFS v3.
    |>
    |> Really? Why?

    Many reasons, including:

    Use of Solaris 10 as a server for other systems (including Solaris 9
    and Linux) - a VERY common reason. Possibly also the other way round.

    Bugs in the Cassini driver, showing apparent failure of NFS v4 and
    not v3 - probably due to the TCP/UDP differences and not the NFS version.

    Problems with getting rid of all traces of Kerberos, especially when
    in a hurry. Just saying "sod it" and setting NFS v3 is easier!


    Regards,
    Nick Maclaren.

  8. Re: NFS v4 and pNFS

    Mike Eisler wrote:
    > If you are using Solaris 10, you are probably using NFSv4.


    > If you are using Linux one should expect when combining a free
    > operating system with complex protocol that there will be years of
    > pain before the free operating system gets it right.


    Isn't Solaris "free" too?-)

    rick jones
    --
    The computing industry isn't as much a game of "Follow The Leader" as
    it is one of "Ring Around the Rosy" or perhaps "Duck Duck Goose."
    - Rick Jones
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  9. Re: NFS v4 and pNFS


    "Mike Eisler" wrote in message
    news:1151420450.592689.165070@c74g2000cwc.googlegr oups.com...
    >
    >
    > If you are using Linux one should expect when combining a free
    > operating system with complex protocol that there will be years
    > of pain before the free operating system gets it right. This was
    > the experience with NFSv2 and Linux, and NFSv3 and Linux, and
    > NFSv[23] aren't even that complex. That said, rather than waiting ten
    > years after the RFC was published, as was the case with NFSv3,
    > the Linux NFSv4 effort got going from the get go. So there's a glimmer
    > of light at the end of the tunnel.
    >


    Mike,

    Oh Mike, so little confidence. Trond M. and Charles L. are
    both working on this at UMICH, paid, full time employees
    of $vendor. Also the Bull Group, as well as OSDL are
    involved. Funding coming from EMC, NetApp, IBM,
    and others. It's a bit pessimistic to say that it will be years.
    It's moving pretty quickly these days... and will most likely
    see much more rapid acceptance once the Linux reference
    platforms are readily available and have a complete
    implementation. I believe the file delegations are currently
    being worked on.....

    Enjoy,
    Postmaster.



  10. Re: NFS v4 and pNFS


    Nick Maclaren wrote:
    > In article <1151425482.620267.88370@x69g2000cwx.googlegroups.c om>,
    > "Mike Eisler" writes:
    > |> >
    > |> > |> If you are using Solaris 10, you are probably using NFSv4.
    > |> >
    > |> > Hmm. Maybe, but a hell of a lot of the Solaris 10 administrators I
    > |> > know of configured it to run NFS v3.
    > |>
    > |> Really? Why?
    >
    > Many reasons, including:
    >
    > Use of Solaris 10 as a server for other systems (including Solaris 9
    > and Linux) - a VERY common reason. Possibly also the other way round.


    So why turn NFSv4 off in that case.

    >
    > Bugs in the Cassini driver, showing apparent failure of NFS v4 and
    > not v3 - probably due to the TCP/UDP differences and not the NFS version.


    So they are tunring TCP off in NFSv3 too? Wow. Welcome to
    data corruption.

    >
    > Problems with getting rid of all traces of Kerberos, especially when
    > in a hurry. Just saying "sod it" and setting NFS v3 is easier!


    NFSv3 supports Kerberos just fine. Turning off NFSv4
    is futile.


  11. Re: NFS v4 and pNFS


    Rick Jones wrote:
    > Mike Eisler wrote:
    > > If you are using Solaris 10, you are probably using NFSv4.

    >
    > > If you are using Linux one should expect when combining a free
    > > operating system with complex protocol that there will be years of
    > > pain before the free operating system gets it right.

    >
    > Isn't Solaris "free" too?-)


    At the time the Solaris NFSv4 implemention was developed, Solaris
    wasn't free, and so there was a business model that kept a dozen
    or more well paid NFS developers busy on NFSv4. A model
    which doesn't exist for Linux. What Sun's business model for
    Solaris 11+ and NFSv4.1 will be and whether it will support
    a substantial team of developers remains to be seen. But
    Sun (along with several others, including Hummingbird, IBM, and NetApp)
    can
    claim to have produced a shippable NFSv4 feature, and those companies
    share(d) a model that takes profits on binaries built from non-free
    source
    code. The companies with other models can't make that claim yet.


  12. Re: NFS v4 and pNFS


    Postmaster wrote:
    > "Mike Eisler" wrote in message
    > news:1151420450.592689.165070@c74g2000cwc.googlegr oups.com...
    > >
    > >
    > > If you are using Linux one should expect when combining a free
    > > operating system with complex protocol that there will be years
    > > of pain before the free operating system gets it right. This was
    > > the experience with NFSv2 and Linux, and NFSv3 and Linux, and
    > > NFSv[23] aren't even that complex. That said, rather than waiting ten
    > > years after the RFC was published, as was the case with NFSv3,
    > > the Linux NFSv4 effort got going from the get go. So there's a glimmer
    > > of light at the end of the tunnel.
    > >

    >
    > Mike,
    >
    > Oh Mike, so little confidence. Trond M. and Charles L. are
    > both working on this at UMICH, paid, full time employees
    > of $vendor. Also the Bull Group, as well as OSDL are
    > involved. Funding coming from EMC, NetApp, IBM,
    > and others. It's a bit pessimistic to say that it will be years.


    Relative to when Hummingbird, NetApp, IBM, and Sun shipped
    it has been shipped. It isn't pessimism, it is stating a fact.

    > It's moving pretty quickly these days... and will most likely
    > see much more rapid acceptance once the Linux reference
    > platforms are readily available and have a complete
    > implementation. I believe the file delegations are currently
    > being worked on.....


    I agree that once Linux is ready, NFSv4 will take off. I'm
    recommending patience for those that want to use Linux.


  13. Re: NFS v4 and pNFS


    In article <1151512316.433940.302220@m73g2000cwd.googlegroups. com>,
    "Mike Eisler" writes:
    |>
    |> > Use of Solaris 10 as a server for other systems (including Solaris 9
    |> > and Linux) - a VERY common reason. Possibly also the other way round.
    |>
    |> So why turn NFSv4 off in that case.

    Because the other systems don't support NFS v4, and you don't want to
    provoke bugs by confusing the issue?

    |> > Bugs in the Cassini driver, showing apparent failure of NFS v4 and
    |> > not v3 - probably due to the TCP/UDP differences and not the NFS version.
    |>
    |> So they are tunring TCP off in NFSv3 too? Wow. Welcome to
    |> data corruption.

    Well, NFS didn't support TCP at all until v3, and many vendors strongly
    discouraged it in v3, so are you saying that all older versions of NFS
    caused data corruption?

    Anyway, if the situation is that using TCP causes your systems to have
    an unacceptably low MTBF, and the failure symptoms are such that they
    need an operator to log into the system controller to power cycle the
    domain, and that using UDP cures that, would you continue to use TCP?

    |> > Problems with getting rid of all traces of Kerberos, especially when
    |> > in a hurry. Just saying "sod it" and setting NFS v3 is easier!
    |>
    |> NFSv3 supports Kerberos just fine. Turning off NFSv4
    |> is futile.

    Not at all. NFS v4's handling of Kerberos is very different from v3's,
    and that trick can work, especially when talking to other systems.
    Yes, it's a totally insane hack, but you need lots of those when you
    are inflicted with Kerberos.


    You really don't seem to understand the position of a hassled
    administrator. The symptom is often that NFS fails to start correctly,
    or fails during use, the log files on both symptoms are unhelpful,
    each vendor blames the other, and the users and management are demanding
    miracles yesterday. He quite reasonably will change the configuration
    in ways that MIGHT help, and use the first one that gets the systems
    running again. It might well not be the cause - so what?


    Regards,
    Nick Maclaren.

  14. Re: NFS v4 and pNFS

    Mike Eisler wrote:
    > So they are tunring TCP off in NFSv3 too? Wow. Welcome to data
    > corruption.


    Based on?

    rick jones
    --
    oxymoron n, commuter in a gas-guzzling luxury SUV with an American flag
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  15. Re: NFS v4 and pNFS


    Nick Maclaren wrote:
    > In article <1151512316.433940.302220@m73g2000cwd.googlegroups. com>,
    > "Mike Eisler" writes:
    > |>
    > |> > Use of Solaris 10 as a server for other systems (including Solaris 9
    > |> > and Linux) - a VERY common reason. Possibly also the other way round.
    > |>
    > |> So why turn NFSv4 off in that case.
    >
    > Because the other systems don't support NFS v4, and you don't want to
    > provoke bugs by confusing the issue?


    Did they turn NFSv3 off too until a 2 years ago?

    >
    > |> > Bugs in the Cassini driver, showing apparent failure of NFS v4 and
    > |> > not v3 - probably due to the TCP/UDP differences and not the NFS version.
    > |>
    > |> So they are tunring TCP off in NFSv3 too? Wow. Welcome to
    > |> data corruption.
    >
    > Well, NFS didn't support TCP at all until v3, and many vendors strongly
    > discouraged it in v3, so are you saying that all older versions of NFS
    > caused data corruption?


    No, NFSv2 over TCP predates v3, albeit not on Solaris.

    >
    > Anyway, if the situation is that using TCP causes your systems to have
    > an unacceptably low MTBF, and the failure symptoms are such that they
    > need an operator to log into the system controller to power cycle the
    > domain, and that using UDP cures that, would you continue to use TCP?


    No, I'd switch to a client that works.

    >
    > |> > Problems with getting rid of all traces of Kerberos, especially when
    > |> > in a hurry. Just saying "sod it" and setting NFS v3 is easier!
    > |>
    > |> NFSv3 supports Kerberos just fine. Turning off NFSv4
    > |> is futile.
    >
    > Not at all. NFS v4's handling of Kerberos is very different from v3's,


    I'm a glutton for punishment: what is difference?


  16. Re: NFS v4 and pNFS


    In article <1151527622.210001.307920@j72g2000cwa.googlegroups. com>,
    "Mike Eisler" writes:
    |>
    |> > |> > Bugs in the Cassini driver, showing apparent failure of NFS v4 and
    |> > |> > not v3 - probably due to the TCP/UDP differences and not the NFS version.
    |> > |>
    |> > Anyway, if the situation is that using TCP causes your systems to have
    |> > an unacceptably low MTBF, and the failure symptoms are such that they
    |> > need an operator to log into the system controller to power cycle the
    |> > domain, and that using UDP cures that, would you continue to use TCP?
    |>
    |> No, I'd switch to a client that works.

    Fascinating. Do you know what I was referring to by the Cassini driver,
    and what it implies?


    Regards,
    Nick Maclaren.

  17. Re: NFS v4 and pNFS


    Rick Jones wrote:
    > Mike Eisler wrote:
    > > So they are tunring TCP off in NFSv3 too? Wow. Welcome to data
    > > corruption.

    >
    > Based on?


    http://www.ietf.org/internet-drafts/...harmful-00.txt

    http://nfsworld.blogspot.com/2006/03...thon-2006.html

    If the UDP datagram fits in the MTU, the corruption rates for
    NFS/UDP are no worse than NFS/TCP.


  18. Re: NFS v4 and pNFS


    Ah, so "frankengrams"

    > http://www.ietf.org/internet-drafts/...harmful-00.txt


    I get "The page you are looking for cannot be found." trying to follow
    the link above.

    > http://nfsworld.blogspot.com/2006/03...thon-2006.html


    Since I nitpit for fun and food:

    UDP does this by breaking the NFS message into IP fragments that
    each fit into the MTU.

    That should be saying that it is IP doing the fragmentation not UDP

    Meanwhile, the client is busy doing other NFS/UDP things, and the
    datagram identifier gets re-used. The identifier is just 16 bits;
    assuming 32 kilo byte writes, giga bit/sec transmission speeds,
    then 2^16 * 32 * 1024 * 8 / 1000^3 is just 17.2 seconds. If the
    TTL is greater than 17 seconds, then the re-use of the identifier
    for another 32 Kbyte NFS WRITE will result in the first fragment
    of the new NFS request being used as the first fragment of the old
    NFS request. That first fragment has some interesting stuff in it,
    such as the file handle, and the offset into the file. If the file
    handles are different, then we are writing data for one file into
    another file. That's a security hole and a data corruption. If the
    file handles are the same, and the offsets are different, then we
    get data corruption. If the file handles and offsets are the same,
    we can still get data corruption, because 17 seconds ago, a retry
    of the first NFS WRITE might have succeeded with no transmission
    loss, and this new NFS WRITE request is an intentional over write
    (say a database record update).

    Um, _only_ if the UDP checksum happens to match... Now, it may still
    not be as "safe" as TCP, but it is quite a bit different from what is
    written above.

    I admit to never encountering the above myself, but I'd long since
    given up on NFS/UDP back when ethernet was a lot slower.

    I've not see the corruption aspect (the UDP checksum covering things)
    but I saw the frankengram aspect ~15 years ago in the HP-UX 8.07
    timeframe with both netperf UDP_STREAM tests and some NFS stuff.

    As for high latency, Brent Welch pointed out that long distance
    WAN links aren't going to exceed a few hundred milliseconds.

    So long as there aren't any satelite hops involved.

    IIRC the Linux folks have added some IP fragment reassembly heuristics
    intended to reduce albeit not elimitate Frankengrams - if more than N
    other fragments have arrived without a particular datagram
    successfully reassembling, it is ass-u-me-d that one or more fragments
    of that datagram were lost and the whole thing is tossed.

    rick jones
    --
    denial, anger, bargaining, depression, acceptance, rebirth...
    where do you want to be today?
    these opinions are mine, all mine; HP might not want them anyway...
    feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

  19. Re: NFS v4 and pNFS


    In article ,
    Rick Jones writes:
    |>
    |> Ah, so "frankengrams"

    Well, back in the days when I was dealing with FTP transfers between
    different systems (MVS, MacOS, Unix etc.), I got quite a lot of
    "frankenfiles". I never did track down exactly why, but the TCP
    layer was clearly involved, given that the transfer code in FTP itself
    would have a major job achieving that effect, and the blocks were not
    network-sized. Most of the time, it was zeroed blocks, swapped blocks
    and duplicated blocks.

    That was one of the main reasons that I wrote a portable, decent
    quality, checksum program :-)

    I haven't seen it as commonly since, so I suspect it was because the
    implementations had some seriously substandard error recovery, but
    I have seen such corruption on a fair number of Unix-Unix transfers
    since, albeit rarely. I am not convinced that TCP is as reliable as
    many people claim, but can't say whether the problem is in the design,
    the Berkeley reference implementation, the operating systems, POSIX,
    the C standard (sic) or what.

    My experience is that such corruption is actually LESS common with
    NFS over UDP, but the issue is such that I don't read anything into
    that. It might be merely that I have not tracked down such events
    with NFS.


    Regards,
    Nick Maclaren.

  20. Re: NFS v4 and pNFS


    Rick Jones wrote:
    > Ah, so "frankengrams"
    >
    > > http://www.ietf.org/internet-drafts/...harmful-00.txt

    >
    > I get "The page you are looking for cannot be found." trying to follow
    > the link above.
    >
    > > http://nfsworld.blogspot.com/2006/03...thon-2006.html

    >
    > Since I nitpit for fun and food:


    You are both nit picky and apparently helpless.

    i-ds get updated all the time. Try:


    http://www.ietf.org/internet-drafts/...harmful-02.txt

    and no promises that it won't be -03 or higher by the time
    you click your browser (remember the days when we didn't have
    browsers and we'd actually look for things?) ... this is
    after all i-d season a week or two before an IETF meeting.

    > Meanwhile, the client is busy doing other NFS/UDP things, and the
    > datagram identifier gets re-used. The identifier is just 16 bits;
    > assuming 32 kilo byte writes, giga bit/sec transmission speeds,
    > then 2^16 * 32 * 1024 * 8 / 1000^3 is just 17.2 seconds. If the


    > Um, _only_ if the UDP checksum happens to match... Now, it may still
    > not be as "safe" as TCP, but it is quite a bit different from what is
    > written above.


    So you think it is 17.2 seconds * 2^16 between corruptions? The
    Internet Draft says that the experiment with 10 TB dataset there were
    121 corruptions.
    So at 1 giga bit/sec
    10 * 1024 ^ 4 * 8 / 1000^3 / 121 = 726 seconds between corruptions.

    I believe my result may be correct
    even if the math that got me there is not quite right.
    Keep the birthday paradox in mind. The birthday paradox says
    that for an N bit checksum, instead of requiring 2^N messages
    to produce a collision, only square root of 2^N messages are
    need to produce a collision. N is 32, so square root of 2^32 is 2^16,
    the same coefficient that kicked off my calculation.

    Anyway whether your result is right, my result is right, or the i-d
    is right, UDP is gonna really suck when 10 giga/sec ethernet is used.

    > IIRC the Linux folks have added some IP fragment reassembly heuristics
    > intended to reduce albeit not elimitate Frankengrams - if more than N
    > other fragments have arrived without a particular datagram
    > successfully reassembling, it is ass-u-me-d that one or more fragments
    > of that datagram were lost and the whole thing is tossed.


    Nothing like beating a dead horse, and UDP is a dead as they come.
    Energy that would be better invested in making NFS, sftp, iscsi work
    better over TCP.


+ Reply to Thread
Page 2 of 3 FirstFirst 1 2 3 LastLast