aio_read/write versus O_NONBLOCK - Unix

This is a discussion on aio_read/write versus O_NONBLOCK - Unix ; phil-news-nospam@ipal.net writes: > On Fri, 23 May 2008 16:29:15 +0200 Rainer Weikusat wrote: > | The first thing to remark is would be that a read which cannot make > | progress immediatly is the 'rare special case the application ...

+ Reply to Thread
Page 3 of 5 FirstFirst 1 2 3 4 5 LastLast
Results 41 to 60 of 97

Thread: aio_read/write versus O_NONBLOCK

  1. Re: aio_read/write versus O_NONBLOCK

    phil-news-nospam@ipal.net writes:
    > On Fri, 23 May 2008 16:29:15 +0200 Rainer Weikusat wrote:
    > | The first thing to remark is would be that a read which cannot make
    > | progress immediatly is the 'rare special case the application has no
    > | way of really knowing about' and that the same is true for a write
    > | which cannot immediatly make progess, ie copy the data into 'buffer
    > | memory' and then return.
    >
    > You do realize that if O_NONBLOCK is not used, nothing is different about
    > read/write. A program that chooses to use O_NONBLOCK on a file would be
    > expected to handle how it all works, just as it is expected to handle it
    > for other things like serial ports and sockets.


    This is not what I was writing about. A UNIX(*)-application working
    with files does not work with actual devices, but accesses the
    filesystem, which provides a hierarchical namespace to access storage
    areas on (theoretically) any number of different devices linked
    together in a way unknown to an application and potentially with
    wildly different physical properties, eg disks, NAND-flash
    masquerading as IDE-device, NOR-flash, virtual-memory-based
    filesystems, network-mounted filesystems, network block devices and so
    forth. If an I/O-operation on a file can result in a delay depends on
    past I/O-operations of other applications on the same system, ie is
    the accessed page already in memory, and on the properties of the
    device which is actually accessed, eg the appliance filesystem I work
    with resides on a NOR-flash being accessed through the Linux
    'read-only' MTD block device driver and I have fairly recently removed
    any support for queued, asychronous reads from this driver, because a
    NOR-flash has practically no seek times and this support makes
    absolutely no sense there (the reason it was in there is because
    Mr 'Linux-MTD' [David Woodhouse] either didn't understand this
    at the time he wrote the code or just didn't care).

    A program which would use O_NONBLOCK (with semantics which would still
    need to be defined sensibly and consistently) on files would basically
    have certain assumption of the hardware which is going to be
    available on computers the program will run on and certain assumptions
    about all the individual filesystem layouts of said systems coded
    in. And UNIX(*) is simply designed to not work this way, while, for
    instance, certain single-tasking program loaders for outdated
    PCs were.

    > | at this point, such functionality can be expected to be only
    > | rarely useful because 'I/O-cache' (page cache, block buffer cache)
    > | should really satisfy most I/O requests going to block devices,
    > | because if it doesn't, the system is going to perform really
    > | bad. There is ample empirical evidence that this caching actually
    > | works as intended, going back more than twenty-five years.
    >
    > If what is intended does not include being able to overlap I/O in a consistent
    > manner (e.g. works the same for files as for all other things), then
    > maybe so.


    'Overlapping I/O to devices being part of the filesystem' (or any such
    'lapping') is the business of the program which controls the hardware.
    It is commonly called 'kernel'. It may be debatable if 'having a
    kernel' is actually a design error, but it is certainly there on
    UNIX(*), so the discussion is moot in the context of it.

    > | The next problem would be that the existing sychronous I/O
    > | multiplexing primitives are not designed for random access files but
    > | for (implicitly) time-ordered streams, ie it is completely ok to
    > | create a socket and then call poll to wait until data to read is
    > | available. This cannot possibly work on a file. It would be possible
    > | to extend the semantics to work with files, but this would add lots of
    > | ugly special cases and 'weird' interface restrictions for (see above)
    > | very little benefit. Creating a new interface designed to support
    > | 'asychronous file I/O' is a much more straight forward solution.
    >
    > Why do you say it cannot work on a file?


    Without doing a read first, no data is ever going to be readable
    from the file, because there is no active partner which could provide
    it unilaterally.

    > Creating a new interface _just_ for files is what is really goofy.


    'Shoehorning' all kinds of different behaviour into identically
    named calls is nothing but 'creating new interfaces' via multiplexing
    routines (like ioctl). It's just a more weird appearing way of doing so.
    Some people consider naming different things differently a useless
    source of confusion. I believe they are wrong.

  2. Re: aio_read/write versus O_NONBLOCK

    Alex Fraser writes:
    > Rainer Weikusat wrote:
    >> Alex Fraser writes:

    > [snip]
    >>> So would you say that offering non-blocking semantics for files would
    >>> impose an implementation (eg one with a per-file write buffer) that
    >>> is, in other respects, less desirable?

    >> The 'livelock' statement is kind-of a red herring because anything
    >> accessible via file descriptor can be shared among multiple processes
    >> (the filesystem interface can be regarded as design for shared
    >> access, though).

    >
    > Sockets and pipes have the same livelock problem (actually, it's more
    > like "spurious wakeup" since some progress will usually be made, just
    > very inefficiently);


    That's what I was trying to express.

    >>> That would be a valid point, but is quite different to saying such
    >>> semantics would be "useless".

    >> The simple fact that actual I/O to such devices is already performed
    >> asynchronously would make them 'rarely useful', with 'rarely' being a
    >> situation which should be prevented from ever occuring.

    >
    > So AIO is "rarely useful" too?


    Why do you believe that a statement about non-blocking synchronous I/O
    should apply to asynchronous I/O?

    > [snip]
    >> In the end, a suggestion may be 'download one of the kernels with
    >> source code available and implement whatever you consider to be useful
    >> as part of it'. A lot of people (me included) apparently believe this
    >> would not be worth the effort. They could be wrong. Other people
    >> believe something different could be wrong, too.

    >
    > I think you misunderstand my position - I am not trying to say
    > non-blocking I/O on files should be implemented, I am trying to
    > understand why it hasn't been, ignoring the existence of AIO (or to
    > put it another way, why AIO was created instead).


    I already wrote that: The purpose of 'aio' is to get rid of
    synchronously operating, explicitly written 'I/O-code'. Instead, an
    application starts asychronous I/O requests which complete 'by magic'
    while the program is doing something different. A more detailed
    explanation is in the 'asychronous I/O'-section of the SUS rationale.

    > So far, the only sound (IMO) reasons I can see are that it would be
    > complicated to implement


    A grottily designed hack to overload a set of existing primitives to
    do something nearly completely different from their intended purpose
    in order to solve a non-existant problem isn't necessarily
    'complicated to implement' (more precisely, everything is 'complicated
    to implement' for someone and not 'complicated to implement' for
    someone else).

    What would be 'complicated' here is to defined a consistent and
    sensible semantic extension. IMO, it is actually impossible.

  3. Re: aio_read/write versus O_NONBLOCK

    On Sun, 25 May 2008 09:49:07 +0200 Rainer Weikusat wrote:

    |> Why do you say it cannot work on a file?
    |
    | Without doing a read first, no data is ever going to be readable
    | from the file, because there is no active partner which could provide
    | it unilaterally.

    So why not read it first.

    --
    |WARNING: Due to extreme spam, googlegroups.com is blocked. Due to ignorance |
    | by the abuse department, bellsouth.net is blocked. If you post to |
    | Usenet from these places, find another Usenet provider ASAP. |
    | Phil Howard KA9WGN (email for humans: first name in lower case at ipal.net) |

  4. Re: aio_read/write versus O_NONBLOCK

    phil-news-nospam@ipal.net writes:
    > On Sun, 25 May 2008 09:49:07 +0200 Rainer Weikusat wrote:
    > |> Why do you say it cannot work on a file?
    > |
    > | Without doing a read first, no data is ever going to be readable
    > | from the file, because there is no active partner which could provide
    > | it unilaterally.
    >
    > So why not read it first.


    ,----
    | The next problem would be that the existing sychronous I/O
    | multiplexing primitives are not designed for random access files but
    | for (implicitly) time-ordered streams, ie it is completely ok to
    | ^^^^^^^^^^^^^^^^^^^^^^
    | create a socket and then call poll to wait until data to read is
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^
    | available. This cannot possibly work on a file.
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    `----

  5. Re: aio_read/write versus O_NONBLOCK

    On May 24, 2:19*am, Alex Fraser wrote:

    > So would you say that offering non-blocking semantics for files would
    > impose an implementation (eg one with a per-file write buffer) that is,
    > in other respects, less desirable? That would be a valid point, but is
    > quite different to saying such semantics would be "useless".


    Actually, I'm not sure that's true. It might be true, but that's not
    precisely the point I'm making.

    The point I'm making is that non-blocking semantics is reasonable for
    sockets. It's not great, but it's reasonable. When you're going to
    read from a socket, there's no question how much you're going to read
    or what you're going to read. You're going to read all the unread
    bytes. It's pretty much your only option. When you're going to write,
    there's no question at least where you're going to write. It is a bit
    irritating that there's no good way to specify how much you're going
    to write, but you generally are happy to write as much as possible or
    all you need to. So it works out.

    For files, there are just too many variables. First, programs that aim
    for concurrency generally don't want to have to deal with the file
    position at all. They can use 'pread' and 'pwrite'. However, to use a
    normal non-blocking operation, they would have to position the offset
    where they wanted to read and, presumably, keep it there the whole
    time they were waiting. That just doesn't make any sense at all. The
    semantics for a change in the file offset would be very hard to
    implement sensibly.

    On the write end, there's generally no reason to wait to start to the
    write. And if you're waiting for space in the virtual memory and
    caching system, it's really just not a per-file thing, and there's
    nothing to keep something else from grabbing it first.

    On the other hand, an AIO mechanism solves all these problems and
    brings a few advantages of its own. The OS can complete the AIO
    directly, copying the data into user space, saving a copy. This works
    for both write and read. In addition, the file position won't be a
    problem, since the AIO operations can have an offset specified.

    For sockets, there's no logical reason you might want to wait to be
    able to do two "different" reads or writes, since there's only one
    place to read and once place to write. But for files, there might well
    be.

    So an AIO mechanism basically *is* the equivalent of the socket non-
    blocking functionality for files, with a few additional advantages.

    DS

  6. Re: aio_read/write versus O_NONBLOCK

    On May 24, 4:02*pm, phil-news-nos...@ipal.net wrote:

    > | You seem to have trouble understanding that we collectively don't believe
    > | such a change would be useful.


    > You seem to have trouble understanding TWO things:
    > 1. *I really would find it useful.
    > 2. *It wouldn't break anything to have allowed it.


    Both of these statements are made with no supporting evidence and
    ignoring all the contrary evidence already presented in the thread.
    First, arguments were already made that it would break things to allow
    it, due to livelock between competing writers. Second, strong
    arguments were made that the semantics would not be possible to
    define, so the "it" you claim would be useful has no referent.

    DS

  7. Re: aio_read/write versus O_NONBLOCK

    On Sun, 25 May 2008 19:18:57 +0200 Rainer Weikusat wrote:
    | phil-news-nospam@ipal.net writes:
    |> On Sun, 25 May 2008 09:49:07 +0200 Rainer Weikusat wrote:
    |> |> Why do you say it cannot work on a file?
    |> |
    |> | Without doing a read first, no data is ever going to be readable
    |> | from the file, because there is no active partner which could provide
    |> | it unilaterally.
    |>
    |> So why not read it first.
    |
    | ,----
    | | The next problem would be that the existing sychronous I/O
    | | multiplexing primitives are not designed for random access files but
    | | for (implicitly) time-ordered streams, ie it is completely ok to
    | | ^^^^^^^^^^^^^^^^^^^^^^
    | | create a socket and then call poll to wait until data to read is
    | | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^
    | | available. This cannot possibly work on a file.
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    | `----

    So why not read it first.
    ^^^^^^^^^^^^^^^^^^^^^^^^^

    --
    |WARNING: Due to extreme spam, googlegroups.com is blocked. Due to ignorance |
    | by the abuse department, bellsouth.net is blocked. If you post to |
    | Usenet from these places, find another Usenet provider ASAP. |
    | Phil Howard KA9WGN (email for humans: first name in lower case at ipal.net) |

  8. Re: aio_read/write versus O_NONBLOCK

    phil-news-nospam@ipal.net writes:
    > On Sun, 25 May 2008 19:18:57 +0200 Rainer Weikusat wrote:
    > | phil-news-nospam@ipal.net writes:
    > |> On Sun, 25 May 2008 09:49:07 +0200 Rainer Weikusat wrote:
    > |> |> Why do you say it cannot work on a file?
    > |> |
    > |> | Without doing a read first, no data is ever going to be readable
    > |> | from the file, because there is no active partner which could provide
    > |> | it unilaterally.
    > |>
    > |> So why not read it first.
    > |
    > | ,----
    > | | The next problem would be that the existing sychronous I/O
    > | | multiplexing primitives are not designed for random access files but
    > | | for (implicitly) time-ordered streams, ie it is completely ok to
    > | | ^^^^^^^^^^^^^^^^^^^^^^
    > | | create a socket and then call poll to wait until data to read is
    > | | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^
    > | | available. This cannot possibly work on a file.
    > | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    > | `----
    >
    > So why not read it first.
    > ^^^^^^^^^^^^^^^^^^^^^^^^^


    Which part of 'create, then poll without read not worky-worky on disk
    file' is too complicated for you?

  9. Re: aio_read/write versus O_NONBLOCK

    David Schwartz wrote:
    > On May 24, 2:19?am, Alex Fraser wrote:


    > > So would you say that offering non-blocking semantics for files would
    > > impose an implementation (eg one with a per-file write buffer) that is,
    > > in other respects, less desirable? That would be a valid point, but is
    > > quite different to saying such semantics would be "useless".


    > Actually, I'm not sure that's true. It might be true, but that's not
    > precisely the point I'm making.


    > The point I'm making is that non-blocking semantics is reasonable for
    > sockets. It's not great, but it's reasonable. When you're going to
    > read from a socket, there's no question how much you're going to read
    > or what you're going to read. You're going to read all the unread
    > bytes. It's pretty much your only option.


    This isn't true at all. If it were, people wouldn't have so much trouble
    handling edge-trigged interfaces, like available w/ epoll or kqueue. Level
    triggered is almost universally understood to be both easier and more costly
    to implement. This is analagous to polling on files: it's easier for the
    programmer, more difficult for the implementator.

    The fact is that most I/O protocols--whether reading or writing from a file,
    pipe, or socket--are line based, or otherwise based on content, not buffer
    details.


    > For files, there are just too many variables. First, programs that aim
    > for concurrency generally don't want to have to deal with the file
    > position at all. They can use 'pread' and 'pwrite'. However, to use a
    > normal non-blocking operation, they would have to position the offset
    > where they wanted to read and, presumably, keep it there the whole
    > time they were waiting. That just doesn't make any sense at all. The
    > semantics for a change in the file offset would be very hard to
    > implement sensibly.


    This doesn't fly, either.

    First, if most programmers were as interested in optimizing file access as
    your argument suggests (in your implicit cost/benefit analysis), then Unix
    pipes, I/O redirection, and the "file" descriptor paradigm wouldn't be so
    useful.

    Second, the problem with two threads interfering with each other applies
    nearly as much to sockets. The solution to two threads reading or writing
    from the same socket, interleaving I/O, is almost universally solved by
    *NOT* having two threads interleave I/O like that, unless the programmer is
    willing to carefully handle all the problems.

    Unix is the poster child of solving problems by dumping them on the
    developer. Oddly, this has worked out extremely well.

    > On the write end, there's generally no reason to wait to start to the
    > write. And if you're waiting for space in the virtual memory and
    > caching system, it's really just not a per-file thing, and there's
    > nothing to keep something else from grabbing it first.


    As has been pointed out else-thread, this applies to sockets as well.
    Another thread can invalidate readiness by exhausting a buffer.

    > On the other hand, an AIO mechanism solves all these problems and
    > brings a few advantages of its own. The OS can complete the AIO
    > directly, copying the data into user space, saving a copy. This works
    > for both write and read. In addition, the file position won't be a
    > problem, since the AIO operations can have an offset specified.


    AIO solves very little, because it prohibits common types of abstraction.
    Sockets and pipes are streams of data. Streams are more limited than a
    random access file, yet oddly they tend to be easier to work with. The
    tradeoffs tend toward simpler and/or more appropriate solutions.

    > For sockets, there's no logical reason you might want to wait to be
    > able to do two "different" reads or writes, since there's only one
    > place to read and once place to write. But for files, there might well
    > be.


    For database developers, and a handful of other influential squeaky wheels.
    The majority of C programmers probably need to lookup the man page for fseek
    every time they use it--which is rarely.

    > So an AIO mechanism basically *is* the equivalent of the socket non-
    > blocking functionality for files, with a few additional advantages.


    The problem with AIO is that is solves _two_ problems at the same time,
    without allowing people to realize the first benefit on its own.

    Any worthwhile implementation of AIO actually implements readiness
    notification at the block layer, *and* modifies the buffer/page cache
    accordingly. We all agree that this is difficult to achieve (certainly the
    latter part), because it invalidates 30 years of assumptions built into most
    filesystem stacks. It's why Linux _still_ doesn't have kernel AIO for
    buffered files.

    So, when AIO is implemented per the spirit of the specification, the kernel
    provides the sort of readiness notification that would allow polling. The
    very same sort that is claimed to be useless, (And if you want to argue on
    this point, practice by telling the I/O scheduler implementors why they're
    useless, and how everybody should be using sychronous AIO; afterall,
    certainly a sufficiently talented engineer could do a better job given a
    particular set of tasks.)

    Yes, there are caveats regarding seeking. So what? People who want to poll()
    on a file descriptor _clearly_ aren't interested in seeking all over the
    file. It's quite obvious that they want to treat it as-if it were a socket.
    And the majority of programs already treat a file as a pipe or stream,
    either intentionally or implicitly.

    At a minimum the AIO spec writers should have provided a way to poll(2) on
    an AIO request. I'd like to see somebody defend _that_ decision....


  10. Re: aio_read/write versus O_NONBLOCK



    William Ahern wrote:
    [snip]

    I don't disagree with anything you say, I just don't see how it
    contradicts anything I said.

    DS

  11. Re: aio_read/write versus O_NONBLOCK

    On Mon, 26 May 2008 13:33:35 +0200 Rainer Weikusat wrote:
    | phil-news-nospam@ipal.net writes:
    |> On Sun, 25 May 2008 19:18:57 +0200 Rainer Weikusat wrote:
    |> | phil-news-nospam@ipal.net writes:
    |> |> On Sun, 25 May 2008 09:49:07 +0200 Rainer Weikusat wrote:
    |> |> |> Why do you say it cannot work on a file?
    |> |> |
    |> |> | Without doing a read first, no data is ever going to be readable
    |> |> | from the file, because there is no active partner which could provide
    |> |> | it unilaterally.
    |> |>
    |> |> So why not read it first.
    |> |
    |> | ,----
    |> | | The next problem would be that the existing sychronous I/O
    |> | | multiplexing primitives are not designed for random access files but
    |> | | for (implicitly) time-ordered streams, ie it is completely ok to
    |> | | ^^^^^^^^^^^^^^^^^^^^^^
    |> | | create a socket and then call poll to wait until data to read is
    |> | | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^
    |> | | available. This cannot possibly work on a file.
    |> | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |> | `----
    |>
    |> So why not read it first.
    |> ^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    | Which part of 'create, then poll without read not worky-worky on disk
    | file' is too complicated for you?

    Why do you insist on not doing it the way that would work?

    --
    |WARNING: Due to extreme spam, googlegroups.com is blocked. Due to ignorance |
    | by the abuse department, bellsouth.net is blocked. If you post to |
    | Usenet from these places, find another Usenet provider ASAP. |
    | Phil Howard KA9WGN (email for humans: first name in lower case at ipal.net) |

  12. Re: aio_read/write versus O_NONBLOCK

    phil-news-nospam@ipal.net writes:
    > On Mon, 26 May 2008 13:33:35 +0200 Rainer Weikusat wrote:
    > | phil-news-nospam@ipal.net writes:
    > |> On Sun, 25 May 2008 19:18:57 +0200 Rainer Weikusat wrote:
    > |> | phil-news-nospam@ipal.net writes:
    > |> |> On Sun, 25 May 2008 09:49:07 +0200 Rainer Weikusat wrote:
    > |> |> |> Why do you say it cannot work on a file?
    > |> |> |
    > |> |> | Without doing a read first, no data is ever going to be readable
    > |> |> | from the file, because there is no active partner which could provide
    > |> |> | it unilaterally.
    > |> |>
    > |> |> So why not read it first.
    > |> |
    > |> | ,----
    > |> | | The next problem would be that the existing sychronous I/O
    > |> | | multiplexing primitives are not designed for random access files but
    > |> | | for (implicitly) time-ordered streams, ie it is completely ok to
    > |> | | ^^^^^^^^^^^^^^^^^^^^^^
    > |> | | create a socket and then call poll to wait until data to read is
    > |> | | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^
    > |> | | available. This cannot possibly work on a file.
    > |> | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    > |> | `----
    > |>
    > |> So why not read it first.
    > |> ^^^^^^^^^^^^^^^^^^^^^^^^^
    > |
    > | Which part of 'create, then poll without read not worky-worky on disk
    > | file' is too complicated for you?
    >
    > Why do you insist on not doing it the way that would work?


    Because I intended to give an example where synchronous
    I/O-multiplexing would need to work differently than it usually does
    when files were to be supported.

  13. Re: aio_read/write versus O_NONBLOCK

    On May 27, 3:25*am, Rainer Weikusat wrote:

    > Because I intended to give an example where synchronous
    > I/O-multiplexing would need to work differently than it usually does
    > when files were to be supported.


    Spot on. Imagine this: you open a file, you the wait for readability.
    What are you waiting for exactly? Suppose the operating system does
    not normally read ahead when a file is opened (which I believe most do
    not). After all, the OS has no idea what part of the file you're going
    to read, you might be about to seek to the end. So at this point there
    is nothing to wait for.

    If you then wait for readability, should the OS be required to start a
    read ahead? In every other case, waiting doesn't ever require the OS
    to do anything. It's a *major* semantic point that waiting for or
    testing readability or writability does not ever change a descriptor's
    state.

    So what should waiting do in this case? Should it be required to start
    a read ahead, changing the state of the socket so that there is an
    asynchronous operation?

    (And if you say, "yes it should", then it should start to be pretty
    clear that what you asking for is really AIO and not like what non-
    blocking sockets do.)

    DS

  14. Re: aio_read/write versus O_NONBLOCK

    David Schwartz wrote:
    > On May 27, 3:25?am, Rainer Weikusat wrote:


    > > Because I intended to give an example where synchronous
    > > I/O-multiplexing would need to work differently than it usually does
    > > when files were to be supported.


    > Spot on. Imagine this: you open a file, you the wait for readability.
    > What are you waiting for exactly? Suppose the operating system does
    > not normally read ahead when a file is opened (which I believe most do
    > not). After all, the OS has no idea what part of the file you're going
    > to read, you might be about to seek to the end. So at this point there
    > is nothing to wait for.


    > If you then wait for readability, should the OS be required to start a
    > read ahead? In every other case, waiting doesn't ever require the OS
    > to do anything. It's a *major* semantic point that waiting for or
    > testing readability or writability does not ever change a descriptor's
    > state.
    >
    > So what should waiting do in this case? Should it be required to start
    > a read ahead, changing the state of the socket so that there is an
    > asynchronous operation?


    The same thing it has always done, return readiness. When an actual I/O
    request blocks, the kernel then has a sufficient hint to queue a request;
    it's the prior request that then gets polled on. From then on, the kernel
    follows in lock-step; we've swapped the producer-consumer ordering--so that
    an open doesn't necessarily load a page on a whim--and alls well w/ the
    world.


  15. Re: aio_read/write versus O_NONBLOCK

    William Ahern writes:
    > David Schwartz wrote:


    [...]

    >> So what should waiting do in this case? Should it be required to start
    >> a read ahead, changing the state of the socket so that there is an
    >> asynchronous operation?

    >
    > The same thing it has always done, return readiness. When an actual I/O
    > request blocks, the kernel then has a sufficient hint to queue a
    > request;


    That's the more straight-forward 'obvious hack-around': Make the first
    call return a fictional result with an unknown probability of being
    wrong and assume that the application will then call read, harvest
    EAGAIN and go back to waiting, which will actually take place if
    necessary on the second call. This has the 'nice' side effect of
    breaking each and every piece of code which naively assumed that the
    return values of system calls were actually meant in earnest and still
    requires the behavious of the subroutines implementing the interface
    to be modified because an operation the interface is not suitable for
    shall (for some weird reason) be put under its umbrella, aka "all of
    my fifteen children are called 'Hey you!!' and I just throw stones at the
    one I actually meant to address".

    Implement it.



  16. Re: aio_read/write versus O_NONBLOCK

    On May 27, 11:49*am, William Ahern
    wrote:

    > > So what should waiting do in this case? Should it be required to start
    > > a read ahead, changing the state of the socket so that there is an
    > > asynchronous operation?


    > The same thing it has always done, return readiness.


    Which is what? Are you suggesting it should say that the file is not
    ready for reading (since the data is not available) and that it never
    should return ready no matter how long you wait, perhaps unless some
    other process coincidentally happens to cause the first few bytes of
    data to be resident in memory?

    > When an actual I/O
    > request blocks, the kernel then has a sufficient hint to queue a request;
    > it's the prior request that then gets polled on.


    Exactly. You want AIO, not non-blocking semantics. You want to issue a
    request and then know what happens to it, not wait for something
    external to happen.

    > From then on, the kernel
    > follows in lock-step; we've swapped the producer-consumer ordering--so that
    > an open doesn't necessarily load a page on a whim--and alls well w/ the
    > world.


    Exactly. What you really want is AIO semantics, and you could possibly
    fake it in ugly and unsatisfying ways with non-blocking semantics.

    DS

  17. Re: aio_read/write versus O_NONBLOCK

    On Tue, 27 May 2008 12:25:10 +0200 Rainer Weikusat wrote:
    | phil-news-nospam@ipal.net writes:
    |> On Mon, 26 May 2008 13:33:35 +0200 Rainer Weikusat wrote:
    |> | phil-news-nospam@ipal.net writes:
    |> |> On Sun, 25 May 2008 19:18:57 +0200 Rainer Weikusat wrote:
    |> |> | phil-news-nospam@ipal.net writes:
    |> |> |> On Sun, 25 May 2008 09:49:07 +0200 Rainer Weikusat wrote:
    |> |> |> |> Why do you say it cannot work on a file?
    |> |> |> |
    |> |> |> | Without doing a read first, no data is ever going to be readable
    |> |> |> | from the file, because there is no active partner which could provide
    |> |> |> | it unilaterally.
    |> |> |>
    |> |> |> So why not read it first.
    |> |> |
    |> |> | ,----
    |> |> | | The next problem would be that the existing sychronous I/O
    |> |> | | multiplexing primitives are not designed for random access files but
    |> |> | | for (implicitly) time-ordered streams, ie it is completely ok to
    |> |> | | ^^^^^^^^^^^^^^^^^^^^^^
    |> |> | | create a socket and then call poll to wait until data to read is
    |> |> | | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^
    |> |> | | available. This cannot possibly work on a file.
    |> |> | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |> |> | `----
    |> |>
    |> |> So why not read it first.
    |> |> ^^^^^^^^^^^^^^^^^^^^^^^^^
    |> |
    |> | Which part of 'create, then poll without read not worky-worky on disk
    |> | file' is too complicated for you?
    |>
    |> Why do you insist on not doing it the way that would work?
    |
    | Because I intended to give an example where synchronous
    | I/O-multiplexing would need to work differently than it usually does
    | when files were to be supported.

    You're just giving an example of a way to do things that does not work and
    ignore how to make things work. This examples nothing.

    --
    |WARNING: Due to extreme spam, googlegroups.com is blocked. Due to ignorance |
    | by the abuse department, bellsouth.net is blocked. If you post to |
    | Usenet from these places, find another Usenet provider ASAP. |
    | Phil Howard KA9WGN (email for humans: first name in lower case at ipal.net) |

  18. Re: aio_read/write versus O_NONBLOCK

    On Tue, 27 May 2008 06:42:23 -0700 (PDT) David Schwartz wrote:
    | On May 27, 3:25?am, Rainer Weikusat wrote:
    |
    |> Because I intended to give an example where synchronous
    |> I/O-multiplexing would need to work differently than it usually does
    |> when files were to be supported.
    |
    | Spot on. Imagine this: you open a file, you the wait for readability.
    | What are you waiting for exactly? Suppose the operating system does
    | not normally read ahead when a file is opened (which I believe most do
    | not). After all, the OS has no idea what part of the file you're going
    | to read, you might be about to seek to the end. So at this point there
    | is nothing to wait for.
    |
    | If you then wait for readability, should the OS be required to start a
    | read ahead? In every other case, waiting doesn't ever require the OS
    | to do anything. It's a *major* semantic point that waiting for or
    | testing readability or writability does not ever change a descriptor's
    | state.
    |
    | So what should waiting do in this case? Should it be required to start
    | a read ahead, changing the state of the socket so that there is an
    | asynchronous operation?
    |
    | (And if you say, "yes it should", then it should start to be pretty
    | clear that what you asking for is really AIO and not like what non-
    | blocking sockets do.)

    The box you think in is a small box.

    --
    |WARNING: Due to extreme spam, googlegroups.com is blocked. Due to ignorance |
    | by the abuse department, bellsouth.net is blocked. If you post to |
    | Usenet from these places, find another Usenet provider ASAP. |
    | Phil Howard KA9WGN (email for humans: first name in lower case at ipal.net) |

  19. Re: aio_read/write versus O_NONBLOCK

    On Tue, 27 May 2008 21:25:27 +0200 Rainer Weikusat wrote:
    | William Ahern writes:
    |> David Schwartz wrote:
    |
    | [...]
    |
    |>> So what should waiting do in this case? Should it be required to start
    |>> a read ahead, changing the state of the socket so that there is an
    |>> asynchronous operation?
    |>
    |> The same thing it has always done, return readiness. When an actual I/O
    |> request blocks, the kernel then has a sufficient hint to queue a
    |> request;
    |
    | That's the more straight-forward 'obvious hack-around': Make the first
    | call return a fictional result with an unknown probability of being
    | wrong and assume that the application will then call read, harvest
    | EAGAIN and go back to waiting, which will actually take place if
    | necessary on the second call. This has the 'nice' side effect of
    | breaking each and every piece of code which naively assumed that the
    | return values of system calls were actually meant in earnest and still
    | requires the behavious of the subroutines implementing the interface
    | to be modified because an operation the interface is not suitable for
    | shall (for some weird reason) be put under its umbrella, aka "all of
    | my fifteen children are called 'Hey you!!' and I just throw stones at the
    | one I actually meant to address".

    So your logic is that existing code ... existing code that will do a wait
    before any attempt at I/O ... will break if we make O_NONBLOCK work for
    files. So do you have an example of code that is doing this wait-then-read
    sequence for files?

    --
    |WARNING: Due to extreme spam, googlegroups.com is blocked. Due to ignorance |
    | by the abuse department, bellsouth.net is blocked. If you post to |
    | Usenet from these places, find another Usenet provider ASAP. |
    | Phil Howard KA9WGN (email for humans: first name in lower case at ipal.net) |

  20. Re: aio_read/write versus O_NONBLOCK

    On Tue, 27 May 2008 15:55:07 -0700 (PDT) David Schwartz wrote:
    | On May 27, 11:49?am, William Ahern
    | wrote:
    |
    |> > So what should waiting do in this case? Should it be required to start
    |> > a read ahead, changing the state of the socket so that there is an
    |> > asynchronous operation?
    |
    |> The same thing it has always done, return readiness.
    |
    | Which is what? Are you suggesting it should say that the file is not
    | ready for reading (since the data is not available) and that it never
    | should return ready no matter how long you wait, perhaps unless some
    | other process coincidentally happens to cause the first few bytes of
    | data to be resident in memory?

    Being "ready to read" does not mean the actual data is in RAM. It just means
    you can proceed to do the read call if your program has an interest in
    reading something (which presumably it does if you included that descriptor
    for reading in poll or select).


    |> When an actual I/O
    |> request blocks, the kernel then has a sufficient hint to queue a request;
    |> it's the prior request that then gets polled on.
    |
    | Exactly. You want AIO, not non-blocking semantics. You want to issue a
    | request and then know what happens to it, not wait for something
    | external to happen.

    AIO is not necessary outside of the small thinking box you use.


    |> From then on, the kernel
    |> follows in lock-step; we've swapped the producer-consumer ordering--so that
    |> an open doesn't necessarily load a page on a whim--and alls well w/ the
    |> world.
    |
    | Exactly. What you really want is AIO semantics, and you could possibly
    | fake it in ugly and unsatisfying ways with non-blocking semantics.

    Faked? Just follow the logic and it can work.

    --
    |WARNING: Due to extreme spam, googlegroups.com is blocked. Due to ignorance |
    | by the abuse department, bellsouth.net is blocked. If you post to |
    | Usenet from these places, find another Usenet provider ASAP. |
    | Phil Howard KA9WGN (email for humans: first name in lower case at ipal.net) |

+ Reply to Thread
Page 3 of 5 FirstFirst 1 2 3 4 5 LastLast