aio_read/write versus O_NONBLOCK - Unix

This is a discussion on aio_read/write versus O_NONBLOCK - Unix ; On May 31, 2:30*am, phil-news-nos...@ipal.net wrote: > I believe that the logic I described is completely sensible. *If it had > been included in the original design, and left in place to this day, > would you be continually complaining ...

+ Reply to Thread
Page 5 of 5 FirstFirst ... 3 4 5
Results 81 to 97 of 97

Thread: aio_read/write versus O_NONBLOCK

  1. Re: aio_read/write versus O_NONBLOCK

    On May 31, 2:30*am, phil-news-nos...@ipal.net wrote:

    > I believe that the logic I described is completely sensible. *If it had
    > been included in the original design, and left in place to this day,
    > would you be continually complaining about it?


    You have still not described the logic. Is "a page the process tried
    to read" meaning any of the set of pages the process has tried to read
    at any point in the past? When is a page removed from the set?

    There are so many bugs with your suggestion I hardly know where to
    start. Here's another:

    1) The process tries to read block X, it would block.

    2) Block X loads in.

    3) By complete coincidence, the process reads block X.

    4) How does the implementation know the read in step 3 was by
    coincidence and it still needs to indicate readability because block X
    is loaded?

    You haven't come anywhere close to defining semantics.

    DS

  2. Re: aio_read/write versus O_NONBLOCK

    phil-news-nospam@ipal.net writes:
    > On Thu, 29 May 2008 13:31:06 +0200 Rainer Weikusat wrote:
    > | phil-news-nospam@ipal.net writes:
    > |> On Wed, 28 May 2008 08:41:11 +0200 Rainer Weikusat wrote:
    > |
    > | [...]
    > |
    > |> | This 'examples' that the usual semantics of the call are unsuitable
    > |> | for file I/O. This does not mean that you, Mr Ahern, the tooth fairy
    > |> | or any other anonymous terrorist cannot 'silently' change the
    > |> | multiplexing call to do something different which would make it
    > |> | possible to use it with files, but that someone would have to.
    > |> | Which is the single point I have been re-iterating for quite some time
    > |> | now.
    > |>
    > |
    > | [...]
    > |
    > |> Either show where the logic I described cannot work, or be silent
    > |> regarding my logic giving the appearance that you do not
    > |> see why it cannot work.
    > |
    > | The basic is question is not wether something could conceivably be
    > | implemented (everything can be implemented, subject to the laws of
    > | physics) but wether implementing it would be sensible. IMO,
    > | overloading the existing synchronous I/O-multiplexing primitves to
    > | accomplish a loosely-related 'other task' is not sensible, especially
    > | taking into account that this 'other task' would only be useful in
    > | fringe cases on computer systems using particular hardware and
    > | particular filesystem layouts.
    >
    > "sensible" is subjective.


    That's why it is called 'an opinion'.

    [...]

    > Instead, I believe that what it does is restores the part that was removed
    > by the arbitrary decision that non-blocking would not be allowed for file
    > or disk I/O.


    Something which has never existed cannot have been removed.

    > I believe that the logic I described is completely sensible.
    > If it had been included in the original design, and left in place to
    > this day, would you be continually complaining about it?


    It is not possible to answer such a question.

  3. Re: aio_read/write versus O_NONBLOCK

    David Schwartz writes:
    > On May 30, 9:57*am, Rainer Weikusat wrote:
    >> This remark was intended to refer to the recurring discussions
    >> regarding wether 'multiplexing call returns "readable"' means 'the
    >> next read will not block', and consequently, wether using of
    >> O_NONBLOCK is 'necessary'.

    >

    [...]

    > This is no different from an application that calls 'statfs' to
    > check the free space and then assumes a subsequent write won't run
    > out of space because the space was available in the past.


    It is different, although this is apparently hard to understand: A
    data buffer in the kernel associated with a particular 'readable' file
    description is only accessible by processes which have a descriptor
    refering it. In absence of specific actions with the (usually
    intended) side effect of creating 'shared file descriptions', this will
    be only the process which created the descriptor. OTOH, the
    filesystem is designed to be a ressource shared among all applications
    running on a particular system.

    But in the given context, this is besides the point.

    [...]

    >> The means that especially 're-purposing code' which was written to
    >> deal with other types of things file descriptors can be associated
    >> with, will not necessarily be a trivial task.

    >
    > I agree that this is true, but this argument doesn't support that
    > claim.


    Time to modify believed-to-be-working code is always in short
    supply.

    [...]

    >> The last (and only) version of Windows I have been in closer contact
    >> with than occasionally helping someone with something was 3.1. I don't
    >> use it and I don't develop anything for it. I don't even care if code
    >> is portable to Windows, because unless this is specifically requested,
    >> I am not willing to deal with any Microsoft weirdnesses.

    >
    > I think this is a mistake. As a result, you won't learn another way of
    > doing things, in what way it's better and in what ways it's worse.


    Let me put it this way: I am a happy, fourth-generation UNIX(*)-bigot.
    I would rather avoid discussing this topic in more detail, because
    that would IMHO resemble the way a light-bulb works a lot: While that
    is conceived to be enlightening, it actually generates mostly (waste)
    heat.

  4. Re: aio_read/write versus O_NONBLOCK

    On Jun 1, 10:11*am, Rainer Weikusat wrote:

    > It is different, although this is apparently hard to understand: A
    > data buffer in the kernel associated with a particular 'readable' file
    > description is only accessible by processes which have a descriptor
    > refering it. In absence of specific actions with the (usually
    > intended) side effect of creating 'shared file descriptions', this will
    > be only the process which created the descriptor. OTOH, the
    > filesystem is designed to be a ressource shared among all applications
    > running on a particular system.


    In the case of a socket, there is always something at the other end of
    that socket. The connection is in fact a shared resource with both
    ends of the connection accessing it asynchronously. It is a serious
    mistake for a programmer to think that he is the only thing accessing
    a connection, because there is always something on the other end that
    can perform all kinds of operations and over which he has no control.

    For example, 'select' and 'poll' are protocol-neutral. Their semantics
    are precisely the same whether the socket is TCP, UDP, or XXX (a
    protocol I just made up). Suppose XXX permits one side to "cancel"
    data it has sent to the other side provided the other side had not yet
    read it. In this case, a read hit from 'select' clearly can't ensure a
    future 'read' won't block because the other side might cancel it.

    Because the semantics of readability are protocol-neutral, they cannot
    provide any guarantees that are not also protocol-neutral.

    Lest you think that XXX doesn't correspond to any real protocol, in
    fact a UDP datagram can be dropped at any time. Real-world programs
    suffered from denial of service attacks in an obscure case where Linux
    dropped a datagram after it had signaled readability based on it. No
    rule or standard prohibits an implementation from dropping a UDP
    datagram after triggering readability but before an actual read call.

    Code that assumes that past readability implies future readability is
    just as broken as code that makes similar assumptions about calls like
    'statfs' and 'access'. This is a well-known and well-understood
    programming error. No system call provides future guarantees of this
    kind, and it is well-understood that 'select' and 'poll' are status
    reporting functions.

    DS

  5. Re: aio_read/write versus O_NONBLOCK

    David Schwartz writes:
    > On Jun 1, 10:11*am, Rainer Weikusat wrote:
    >> It is different, although this is apparently hard to understand: A
    >> data buffer in the kernel associated with a particular 'readable' file
    >> description is only accessible by processes which have a descriptor
    >> refering it. In absence of specific actions with the (usually
    >> intended) side effect of creating 'shared file descriptions', this will
    >> be only the process which created the descriptor. OTOH, the
    >> filesystem is designed to be a ressource shared among all applications
    >> running on a particular system.

    >
    > In the case of a socket, there is always something at the other end of
    > that socket. The connection is in fact a shared resource with both
    > ends of the connection accessing it asynchronously.


    The 'connection' (insofar there is any) is entirely virtual (that's
    why it is called a 'virtual circuit') and nobody can access it.

    [...]

    > Code that assumes that past readability implies future readability is
    > just as broken as code that makes similar assumptions about calls like
    > 'statfs' and 'access'. This is a well-known and well-understood
    > programming error.


    This is called a TOCTOU-race and you (repeatedly, as I may add) taken
    some pains to demonstrate that it certainly isn't "well understood".
    But I have no intention to get into this particular discussion.

  6. Re: aio_read/write versus O_NONBLOCK

    David Schwartz writes:
    > On Jun 1, 10:11*am, Rainer Weikusat wrote:
    >> It is different, although this is apparently hard to understand: A
    >> data buffer in the kernel associated with a particular 'readable' file
    >> description is only accessible by processes which have a descriptor
    >> refering it. In absence of specific actions with the (usually
    >> intended) side effect of creating 'shared file descriptions', this will
    >> be only the process which created the descriptor. OTOH, the
    >> filesystem is designed to be a ressource shared among all applications
    >> running on a particular system.

    >
    > In the case of a socket, there is always something at the other end of
    > that socket. The connection is in fact a shared resource with both
    > ends of the connection accessing it asynchronously.


    The 'connection' (insofar there is any) is entirely virtual (that's
    why it is called a 'virtual circuit') and nobody can access it.

    [...]

    > Code that assumes that past readability implies future readability is
    > just as broken as code that makes similar assumptions about calls like
    > 'statfs' and 'access'. This is a well-known and well-understood
    > programming error.


    This is called a TOCTOU-race and the problem is not that there is a time
    interval between 'check' and 'use' but that there is a time interval
    between checking the state of a shared ressource and using this
    ressource. This means (free translation from German) that existence of
    this time interval is a necessary but not a sufficient condition for
    the problem. Which, in turn, leads to the conclusion that
    demonstrating existence of such a time interval does not imply that
    the possibility for such a race exists, eg it is completely safe to
    assume that the following code snippet

    int i, j;

    i = 1;
    j = 1000;
    do --j; while (j);
    if (i) puts("it works!");

    will print 'it works', despite the fact the it would certainly
    possible to implement a compiler which cause it 'to send missiles
    directed at the wrong targets instead', not to mention that the kernel
    could change the value at any time if it was programmed to do so.








  7. Re: aio_read/write versus O_NONBLOCK

    On Jun 1, 10:42*pm, Rainer Weikusat wrote:

    > This is called a TOCTOU-race and you (repeatedly, as I may add) taken
    > some pains to demonstrate that it certainly isn't "well understood".
    > But I have no intention to get into this particular discussion.


    It most certainly is well understood. The mistake is one of the
    problems frequently repeated by inexperienced programmers, and almost
    never made by experienced programmers. It's no more mysterious that
    forgetting to reset the bits of interest in a 'select' loop or calling
    'access' and then assuming that the file is still the same file when
    you access it later.

    DS

  8. Re: aio_read/write versus O_NONBLOCK

    David Schwartz writes:
    > On Jun 1, 10:42*pm, Rainer Weikusat wrote:
    >> This is called a TOCTOU-race and you (repeatedly, as I may add) taken
    >> some pains to demonstrate that it certainly isn't "well understood".
    >> But I have no intention to get into this particular discussion.

    >
    > It most certainly is well understood.


    It wouldn't be too difficult to locate at least one 'embedded
    Linux-engineer' who added such a race to busybox a couple of years ago
    by searching the mailing list archives and the 'surroundings' of this
    particular microdiscussion would give a strong hint that not only did
    this particular person not understand the problem, but that none of
    the other subscribers, "principal developers" included, did, too (at
    that time).

    > The mistake is one of theproblems frequently repeated by
    > inexperienced programmers, and almost never made by experienced
    > programmers. It's no more mysterious that forgetting to reset the
    > bits of interest in a 'select' loop or calling 'access' and then
    > assuming that the file is still the same file when
    > you access it later.


    As I wrote in the supersede posting: The problem is not created by the
    time interval and state inquiries of the filesystem are not analogous
    to I/O-readiness notifications for pollable descriptors.


  9. Re: aio_read/write versus O_NONBLOCK

    On Jun 2, 2:24*am, Rainer Weikusat wrote:

    > It wouldn't be too difficult to locate at least one 'embedded
    > Linux-engineer' who added such a race to busybox a couple of years ago
    > by searching the mailing list archives and the 'surroundings' of this
    > particular microdiscussion would give a strong hint that not only did
    > this particular person not understand the problem, but that none of
    > the other subscribers, "principal developers" included, did, too (at
    > that time).


    That's sad, and IMO inexcusable. These kinds of problems have been
    known for a very long time. Sadly, the problem is no so much that
    people aren't aware of the problem but that they refuse to acknowledge
    it.

    For example, check out this Debian bug report:
    http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=284501

    "Linux select(2) is not POSIX compliant; select can report that a
    socket
    has data to read when it does not. This is so in kernel series 2.6.x
    and
    possibly earlier versions. If an application wants non-blocking
    behaviour on Linux, it must set O_NONBLOCK on the file handle. See the
    thread on LKML .

    select can return true but there is no data available if e.g. a packet
    with a bad checksum is received from the network. So cupsd can be
    easily
    hung by sending a UDP packet with a bad checksum to UDP port 631"

    This is complete and utter nonsense. POSIX does not say that an
    implementation can't choose to drop a packet after a 'select' hit for
    readability but before a call to 'recvmsg'. Their code was broken --
    if an implementation chose to drop a packet in the TOCTTOU interval,
    their code would hang. This is *precisely* what Linux did.

    That some silly people continue to argue that black is white doesn't
    make the difference between these two not well-understood.

    DS

  10. Re: aio_read/write versus O_NONBLOCK

    David Schwartz writes:
    > On Jun 2, 2:24*am, Rainer Weikusat wrote:


    [...]

    > For example, check out this Debian bug report:
    > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=284501
    >
    > "Linux select(2) is not POSIX compliant; select can report that a
    > socket has data to read when it does not. This is so in kernel
    > series 2.6.x and possibly earlier versions. If an application wants
    > non-blocking behaviour on Linux, it must set O_NONBLOCK on the file
    > handle. See the thread on LKML
    > .
    >
    > select can return true but there is no data available if e.g. a packet
    > with a bad checksum is received from the network. So cupsd can be
    > easily hung by sending a UDP packet with a bad checksum to UDP port 631"
    >
    > This is complete and utter nonsense. POSIX does not say that an
    > implementation can't choose to drop a packet after a 'select' hit for
    > readability but before a call to 'recvmsg'.


    That's your own votum separatum in the referenced thread, while
    everybody else (minus lurkers feeling the desire to AOL! something) is
    of the opinion that

    A descriptor shall be considered ready for writing when a call
    to an output function with O_NONBLOCK clear would not block,
    whether or not the function would transfer data successfully.

    means 'a call would not block' (hence doing the call will not block),
    and not 'a call would not have blocked' (which is how the sentence
    should have been written to be consistent with your interpretation of
    it). And this is the more interpretation-friendly version, because the
    poll-specification says

    POLLIN
    Data other than high-priority data may be read without
    blocking.

    Putting this into the subjunctive mode you assume here would result in
    "[...] might have been read without blocking.", which is a lot less
    similar to the actual text than it was for select.

    But by-and-large, this is a phantom discussion because code not using
    O_NONBLOCK will not work correctly on the overwhelming majority of
    existing Linux-installation.

    As sort-of a soundbite: You should really be using O_NONBLOCK in these
    cases, unless you are willing to assume that so-called 'hard
    scientists', often even taking pride in their ignorance regarding
    language issues, are not only aware of the existence of different
    subjunctive modes but actually understand them.

    And that is something I would certainly not be willing to do :->.

  11. Re: aio_read/write versus O_NONBLOCK

    On Jun 2, 4:01*am, Rainer Weikusat wrote:

    > That's your own votum separatum in the referenced thread, while
    > everybody else (minus lurkers feeling the desire to AOL! something) is
    > of the opinion that


    > * * * * A descriptor shall be considered ready for writing when a call
    > * * * * to an output function with O_NONBLOCK clear would not block,
    > * * * * whether or not the function would transfer data successfully.


    I agree with that.

    > means 'a call would not block' (hence doing the call will not block),
    > and not 'a call would not have blocked' (which is how the sentence
    > should have been written to be consistent with your interpretation of
    > it).


    Umm, no. The phrase "would not block" does not mean "will not block"
    nor does it mean "would not have blocked".

    > And this is the more interpretation-friendly version, because the
    > poll-specification says
    >
    > * * * * POLLIN
    > * * * * * * Data other than high-priority data may be read without
    > * * * * * * blocking.
    >
    > Putting this into the subjunctive mode you assume here would result in
    > "[...] might have been read without blocking.", which is a lot less
    > similar to the actual text than it was for select.


    You lost me there.

    It is perfectly sensible to say that "access" for example, indicates
    whether an access "would succeed" or that it indicates that the file
    "may be accessed without generating a permissions error". But we all
    know about TOCTTOU. None of these phrasing guarantee a future result.
    Things can change.

    > But by-and-large, this is a phantom discussion because code not using
    > O_NONBLOCK will not work correctly on the overwhelming majority of
    > existing Linux-installation.
    >
    > As sort-of a soundbite: You should really be using O_NONBLOCK in these
    > cases, unless you are willing to assume that so-called 'hard
    > scientists', often even taking pride in their ignorance regarding
    > language issues, are not only aware of the existence of different
    > subjunctive modes but actually understand them.
    >
    > And that is something I would certainly not be willing to do :->.


    It has nothing to do with the mode. It is simply TOCTTOU. None of
    these imply a future guarantee.

    One could quite sensibly say that "access" indicates that a read
    "would not fail due to permissions" and no sane person would read that
    to imply that some other future read will not fail due to permissions.
    For one thing, there's the "would not" to "will not". For another
    thing, they are two different reads. The one that you are making now
    is clearly not the one that "would not fail" back in the past when you
    called "access".

    Sorry, the standard is clear. Because some people can be insistently
    stupid doesn't mean that the truth is not totally clear.

    DS

  12. Re: aio_read/write versus O_NONBLOCK

    David Schwartz writes:
    > On Jun 2, 4:01*am, Rainer Weikusat wrote:
    >> That's your own votum separatum in the referenced thread, while
    >> everybody else (minus lurkers feeling the desire to AOL! something) is
    >> of the opinion that

    >
    >> * * * * A descriptor shall be considered ready for writing when a call
    >> * * * * to an output function with O_NONBLOCK clear would not block,
    >> * * * * whether or not the function would transfer data successfully.

    >
    > I agree with that.
    >
    >> means 'a call would not block' (hence doing the call will not block),
    >> and not 'a call would not have blocked' (which is how the sentence
    >> should have been written to be consistent with your interpretation of
    >> it).

    >
    > Umm, no. The phrase "would not block" does not mean "will not block"
    > nor does it mean "would not have blocked".


    Something 'would not block' means 'will not block if it was done' (but
    maybe it won't be done) and 'would not have blocked' means 'would not
    have blocked if it had been done'. The latter is exactly consistent
    with your interpretation of the text, despite that actually uses the
    former.

    [...]

    > Sorry, the standard is clear.


    It indeed is: After poll returned POLLIN, normal data may be read
    without blocking. There is zero room anywhere in this statement to
    interpret it as 'had a read been done instead of a poll, that read had
    not blocked'. That's simply wrong.

    But thank you for the nice demonstration.


  13. Re: aio_read/write versus O_NONBLOCK

    On Jun 2, 6:19*am, Rainer Weikusat wrote:

    > Something 'would not block' means 'will not block if it was done' (but
    > maybe it won't be done) and 'would not have blocked' means 'would not
    > have blocked if it had been done'. The latter is exactly consistent
    > with your interpretation of the text, despite that actually uses the
    > former.


    Right, neither mean 'will not block if it is done'.

    > It indeed is: After poll returned POLLIN, normal data may be read
    > without blocking. There is zero room anywhere in this statement to
    > interpret it as 'had a read been done instead of a poll, that read had
    > not blocked'. That's simply wrong.


    Since when is "may" a guarantee? It would be perfectly reasonable to
    describe "access" as saying an open for write "may be done without
    getting a permissions error". But only a crazy person would interpret
    this to mean that a future write will be done without an error.

    There is simply no way to interpret the standard to mean what you say.
    If I say, "you may use my car if it's here and you want to use it",
    that doesn't mean that if at some instant it's here and you want to
    use it, you can use it at every future instant whether or not the car
    is still here.

    Your reading is not just strained, it's outright absurd.

    DS

  14. Re: aio_read/write versus O_NONBLOCK

    David Schwartz writes:
    > On Jun 2, 6:19*am, Rainer Weikusat wrote:


    [...]

    >> It indeed is: After poll returned POLLIN, normal data may be read
    >> without blocking. There is zero room anywhere in this statement to
    >> interpret it as 'had a read been done instead of a poll, that read had
    >> not blocked'. That's simply wrong.

    >
    > Since when is "may" a guarantee?


    Used in this form, since ever, cf 'You may go no', 'You may not
    enter the premises without permission' or 'European community
    residents may proceed to counter XY now'.

    But this has really degraded into a discussion of 'English grammer' at
    9th or 10th form (German) level, which is quite absurd in itself.

  15. Re: aio_read/write versus O_NONBLOCK


    Rainer Weikusat wrote:

    > > Since when is "may" a guarantee?


    > Used in this form, since ever, cf 'You may go no',


    This is not a guarantee.

    > 'You may not
    > enter the premises without permission'


    This is not a guarantee.

    >or 'European community
    > residents may proceed to counter XY now'.


    This is not a guarantee.

    DS

  16. Re: aio_read/write versus O_NONBLOCK

    David Schwartz writes:
    > Rainer Weikusat wrote:
    >
    >> > Since when is "may" a guarantee?

    >
    >> Used in this form, since ever, cf 'You may go no',

    >
    > This is not a guarantee.


    Instead of discussing this to death with me, you should really
    consider the opinion of someone whose field of expertise this is, eg
    any high-school English teacher you may be capable of meeting
    somewhere, because unless you manage to demonstrate that you hold a
    comparable qualification to, say, my mother, which has studied this
    topic and taught it for more than thirty years, you are certainly not
    going to convince me.


  17. Re: aio_read/write versus O_NONBLOCK


    Rainer Weikusat wrote:

    > Instead of discussing this to death with me, you should really
    > consider the opinion of someone whose field of expertise this is, eg
    > any high-school English teacher you may be capable of meeting
    > somewhere, because unless you manage to demonstrate that you hold a
    > comparable qualification to, say, my mother, which has studied this
    > topic and taught it for more than thirty years, you are certainly not
    > going to convince me.


    Ahh, I see. It doesn't matter how good my arguments are and how much
    evidence I can muster to support them, it only matters what
    qualifications I have. I will point out, for the record, that
    interpreting standards is nothing like understanding English grammar
    -- English grammar doesn't have an as-if rule.

    DS

+ Reply to Thread
Page 5 of 5 FirstFirst ... 3 4 5