[9fans] Pipes on UNIX - Plan9

This is a discussion on [9fans] Pipes on UNIX - Plan9 ; Hello, I understand from [1] that named pipes are suposed to use the filesystem as an "unlimited" buffer for the pipe. This would be an advantage against a pipe deadlock I'm experiencing using unnamed pipes (a CGI communicating wit Apache), ...

+ Reply to Thread
Results 1 to 13 of 13

Thread: [9fans] Pipes on UNIX

  1. [9fans] Pipes on UNIX

    Hello,

    I understand from [1] that named pipes are suposed to use the
    filesystem as an "unlimited" buffer for the pipe. This would be an
    advantage against a pipe deadlock I'm experiencing using unnamed pipes
    (a CGI communicating wit Apache), but I cannot reproduce the
    "unlimited buffer" in Linux. Maybe I should fcntl the named pipe or
    something similar setting a buffer limit, and the filesystem will be
    used? Or maybe the named pipes were never supposed to use the disk as
    the pipe buffer?

    I don't know any other place where I may ask this question and get
    answered. My google search didn't bring me useful results since now.

    Otherwise I will probably write a simple program which acts as an
    unlimited pipe, using a disk backend as data queue.

    [1] http://www.cs.cmu.edu/~UniCon/refere...Manual_41.html

    (Sorry for a non-plan9 question...)

  2. Re: [9fans] Pipes on UNIX

    On 9/6/07, LluĂ*s Batlle wrote:
    > Hello,
    >
    > I understand from [1] that named pipes are suposed to use the
    > filesystem as an "unlimited" buffer for the pipe. This would be an
    > advantage against a pipe deadlock I'm experiencing using unnamed pipes
    > (a CGI communicating wit Apache), but I cannot reproduce the
    > "unlimited buffer" in Linux.


    On linux you want fifo(7), which says "When processes are exchanging
    data via the FIFO, the kernel passes all data internally without
    writing it to the file system.". Should answer the rest of your
    questions.
    -sqweek

  3. Re: [9fans] Pipes on UNIX

    2007/9/6, sqweek :
    > On linux you want fifo(7), which says "When processes are exchanging
    > data via the FIFO, the kernel passes all data internally without
    > writing it to the file system.". Should answer the rest of your
    > questions.

    Oh, thank you. I expected that mkfifo(1) would point to that page if
    it existed, and in my installation it doesn't point.

  4. Re: [9fans] Pipes on UNIX

    On 9/6/07, LluĂ*s Batlle wrote:
    > 2007/9/6, sqweek :
    > > On linux you want fifo(7),

    > Oh, thank you. I expected that mkfifo(1) would point to that page if
    > it existed, and in my installation it doesn't point.


    You expected a GNU man page to be useful? Are you kidding me??
    -sqweek

  5. Re: [9fans] Pipes on UNIX

    I was more used to blame GNU's code than man pages

    To Enrico, in fact I want to use the disk-buffered pipe program in a shell.
    Something like:
    generate_data | diskpipe | data_eater
    I don't think a 9p server would do anything here.

    Thank you,
    Lluís.

    2007/9/6, sqweek :
    > On 9/6/07, Lluís Batlle wrote:
    > > 2007/9/6, sqweek :
    > > > On linux you want fifo(7),

    > > Oh, thank you. I expected that mkfifo(1) would point to that page if
    > > it existed, and in my installation it doesn't point.

    >
    > You expected a GNU man page to be useful? Are you kidding me??
    > -sqweek
    >


  6. Re: [9fans] Pipes on UNIX

    On 9/6/07, Lluís Batlle wrote:
    > I was more used to blame GNU's code than man pages
    >
    > To Enrico, in fact I want to use the disk-buffered pipe program in a shell.
    > Something like:
    > generate_data | diskpipe | data_eater
    > I don't think a 9p server would do anything here.
    >


    generate_data > input
    cat input > output
    data_eater < output

    with output and input being files from a file server.

    --
    - curiosity sKilled the cat

  7. Re: [9fans] Pipes on UNIX

    I was thinking in a general solution for having pipes with very big buffers....
    but your much simpler proposal fixes perfectly my problem

    I'm having too much problems on over-engineering these last days...

    Thank you,
    Lluís.

    2007/9/6, Gorka Guardiola :
    > On 9/6/07, Lluís Batlle wrote:
    > > I was more used to blame GNU's code than man pages
    > >
    > > To Enrico, in fact I want to use the disk-buffered pipe program in a shell.
    > > Something like:
    > > generate_data | diskpipe | data_eater
    > > I don't think a 9p server would do anything here.
    > >

    >
    > generate_data > input
    > cat input > output
    > data_eater < output
    >
    > with output and input being files from a file server.
    >
    > --
    > - curiosity sKilled the cat
    >


  8. Re: [9fans] Pipes on UNIX

    On 9/6/07, Gorka Guardiola wrote:

    > generate_data > input
    > cat input > output
    > data_eater < output


    This in the general case is a bad idea, from many angles :-)

    ron

  9. Re: [9fans] Pipes on UNIX

    I wrote it without coffee... would be more like:

    generate_data > input &
    data_eater < output

    :-).

    And input and output can come from a stupid fileserver like pipefs...
    or pipebackedbyaharddiskfs :-).
    --
    - curiosity sKilled the cat

  10. Re: [9fans] Pipes on UNIX

    I better explain my problem.
    I have a CGI for Apache. I wrote it in C, and it simply parses text
    from a html textarea, giving another html in output. I made it as a
    state machine which outputs and outputs according to the input
    characters.

    Apache communicates with the CGI process with two pipes linked to the
    process' stdin and stdout.
    But I think that apache code is like this:
    while()
    {
    read(socket,buffer)
    write(pipe_process1, buffer)
    }
    while()
    {
    read(pipe_process2, buffer)
    write(socket,buffer)
    }

    Therefore, if I had a CGI similar to 'cat' (and I have), it would
    deadlock due to the pipe_process2 not being read by apache if it gets
    filled, the process blocks, and then pipe_process1 gets filled.

    As I want the CGI to accept input from almost any length, I need a
    "pipe buffer" big enough.

    Then, based on Gorka suggestion (which I didn't understand as a
    9pserver , I finally agreed on the easiest solution wrapping the CGI
    with a sh script:
    #!/bin/sh
    head -n 10000000 > inputfile # limiting the disk ussage to 10MB, raw cutting.
    mycgi < inputfile
    #EOF

    That's all. I will not write any "superpipe" program, or any 9pserver,
    because at the end I need the disk. The sh wrapper works well enough.

    Thank you for your help in this non-plan9-related problem! Simply the
    unicon documentation pointed to something strange in named pipes which
    I couldn't check anywhere.

    Regards,
    Lluís

    2007/9/6, Gorka Guardiola :
    > I wrote it without coffee... would be more like:
    >
    > generate_data > input &
    > data_eater < output
    >
    > :-).
    >
    > And input and output can come from a stupid fileserver like pipefs...
    > or pipebackedbyaharddiskfs :-).
    > --
    > - curiosity sKilled the cat
    >


  11. Re: [9fans] Pipes on UNIX

    > I better explain my problem.
    > I have a CGI for Apache. I wrote it in C, and it simply parses text
    > from a html textarea, giving another html in output. I made it as a
    > state machine which outputs and outputs according to the input
    > characters.
    >
    > Apache communicates with the CGI process with two pipes linked to the
    > process' stdin and stdout.
    > But I think that apache code is like this:
    > while()
    > {
    > read(socket,buffer)
    > write(pipe_process1, buffer)
    > }
    > while()
    > {
    > read(pipe_process2, buffer)
    > write(socket,buffer)
    > }
    >
    > Therefore, if I had a CGI similar to 'cat' (and I have), it would
    > deadlock due to the pipe_process2 not being read by apache if it gets
    > filled, the process blocks, and then pipe_process1 gets filled.


    no, you wouldn't.

    i believe the problem is the cgi and apache disagree on the termination
    condition. make sure you close the fd 0 when you are done reading.

    - erik


  12. Re: [9fans] Pipes on UNIX

    2007/9/6, erik quanstrom :
    > > Therefore, if I had a CGI similar to 'cat' (and I have), it would
    > > deadlock due to the pipe_process2 not being read by apache if it gets
    > > filled, the process blocks, and then pipe_process1 gets filled.

    >
    > no, you wouldn't.
    >
    > i believe the problem is the cgi and apache disagree on the termination
    > condition. make sure you close the fd 0 when you are done reading.

    I used gdb, and my process doesn't reach at all the end of reading. It
    gets blocked writting to stdout. The system input TCP queue to apache
    is still filled with 70KB of data in the test I did. And the browser
    didn't receive any byte. And I know my program processed an amount of
    data (it makes the hard disk pain a bit, so I can 'hear' when it does
    something).

  13. Re: [9fans] Pipes on UNIX

    > Thank you for your help in this non-plan9-related problem! Simply the
    > unicon documentation pointed to something strange in named pipes which
    > I couldn't check anywhere.


    i couldn't work out where they got that impression or information.
    i'm sure i've never seen that in any of the unix-y implementations i've read.


+ Reply to Thread