Two-way pipes to a child process - Unix

This is a discussion on Two-way pipes to a child process - Unix ; Hi All, I am trying to get a parent-child process relationship working with a pair of pipes that pass data bidirectionally between the two processes. The idea (as part of a larger project) is to implement filtering via standard shell ...

+ Reply to Thread
Results 1 to 9 of 9

Thread: Two-way pipes to a child process

  1. Two-way pipes to a child process

    Hi All,

    I am trying to get a parent-child process relationship working with a
    pair of pipes that pass data bidirectionally between the two
    processes. The idea (as part of a larger project) is to implement
    filtering via standard shell commands prior to output.

    When I run /usr/bin/rev through my program (source code below), I see
    that both the parent and child processes are stuck in read().

    Terminal:

    [binand@kalki]$ ./a.out /usr/bin/rev
    Running in execl() mode
    Child PID: 5045, My PID: 5044
    DW: 59

    [sits there]
    [binand@kalki]$ strace -p 5044
    Process 5044 attached - interrupt to quit
    read(5,

    [sits there]
    Child:

    [binand@kalki]$ strace -p 5045
    Process 5045 attached - interrupt to quit
    read(0,

    [sits there]
    When I run ./reverse (which is my own implementation of /usr/bin/rev,
    source code below), I get the correct behaviour.

    [binand@kalki]$ ./a.out ./reverse
    Running in execl() mode
    Child PID: 5586, My PID: 5585
    DW: 59
    DR: 59
    D|llew derebmemer emit a emac xof eht fo raey eht ni yad enO|
    DW: 84
    DR: 84
    D|lleb kcalb gib eht fo gnillot eht draeh nus gnisir eht fo nam gnuoy
    gnorts eht nehW|

    Can anyone tell me what causes this behaviour? My best guess is that
    stdio's buffering is the reason, but whatever I do with setvbuf(), I
    still can't get the program to work with /usr/bin/rev (or for that
    matter, any other filter in /usr/bin). As you can see, my reverse.c
    has an fflush(stdout); at the end of a line of output (the same code,
    when inline in ptest.c, works just fine without the fflush() call.
    This behaviour can be seen by compiling with -DUSE_INLINE).

    I am on Redhat Enterprise Linux 3 Update 7, using gcc 3.2.3.

    Thanks in advance,

    Binand

    Source code:

    [binand@kalki]$ cat ptest.c
    #include
    #include
    #include
    #include

    #define PIPE_READEND 0
    #define PIPE_WRITEEND 1

    int pipe_init(char *, int *, int *);

    int pipe_init(char *prog, int *rfd, int *wfd) {
    int p2c[2], c2p[2];
    int tfd;
    pid_t pid;
    char *s, *argv0;

    s = strrchr(prog, '/');
    argv0 = s ? s + 1 : prog;
    pipe(p2c);
    pipe(c2p);
    pid = fork();

    switch (pid) {
    case -1:
    return -1;
    case 0:
    close(p2c[PIPE_WRITEEND]);
    close(c2p[PIPE_READEND]);
    tfd = fileno(stdin);
    if (tfd >= 0 && p2c[PIPE_READEND] != tfd) {
    dup2(p2c[PIPE_READEND], tfd);
    }
    tfd = fileno(stdout);
    if (tfd >= 0 && c2p[PIPE_WRITEEND] != tfd) {
    dup2(c2p[PIPE_WRITEEND], tfd);
    }
    close(p2c[PIPE_READEND]);
    close(c2p[PIPE_WRITEEND]);
    #ifndef USE_INLINE
    fprintf(stderr, "Running in execl() mode\n");
    execl(prog, argv0, NULL);
    exit(1);
    #else
    fprintf(stderr, "Running in inline mode\n");
    {
    char b[BUFSIZ];
    while (fgets(b, sizeof b, stdin) != NULL) {
    size_t l, m, i;
    char c;
    l = strlen(b);
    if (b[l-1] == '\n') {
    b[l-1] = '\0';
    l--;
    }
    m = l%2 ? (l-1)/2 : l/2;
    for (i = 0; i < m; i++) {
    c = b[i];
    b[i] = b[l-i-1];
    b[l-i-1] = c;
    }
    fputs(b, stdout);
    putc('\n', stdout);
    }
    }
    exit(0);
    #endif
    default:
    fprintf(stderr, "Child PID: %d, My PID: %d\n", pid,
    getpid());
    close(p2c[PIPE_READEND]);
    close(c2p[PIPE_WRITEEND]);
    *rfd = c2p[PIPE_READEND];
    *wfd = p2c[PIPE_WRITEEND];
    }
    return 0;
    }

    #define TESTDATA1 "One day in the year of the fox came a time
    remembered well\n"
    #define TESTCOUNT1 strlen(TESTDATA1)
    #define TESTDATA2 "When the strong young man of the rising sun heard
    the tolling of the big black bell\n"
    #define TESTCOUNT2 strlen(TESTDATA2)

    int main (int ac, char **av) {
    char b[BUFSIZ], *c;
    int r, w;
    if (ac < 2) {
    fprintf(stderr, "Usage: ptest \n");
    exit(1);
    }
    if (pipe_init(av[1], &r, &w) != 0) {
    fprintf(stderr, "pipe_init failed\n");
    exit(1);
    }
    fprintf(stderr, "DW: %d\n", write(w, TESTDATA1, TESTCOUNT1));
    fprintf(stderr, "DR: %d\n", read(r, b, TESTCOUNT1));
    c = strrchr (b, '\n');
    if (c) {
    *c = '\0';
    }
    fprintf(stderr, "D|%s|\n", b);
    fprintf(stderr, "DW: %d\n", write(w, TESTDATA2, TESTCOUNT2));
    fprintf(stderr, "DR: %d\n", read(r, b, TESTCOUNT2));
    c = strrchr (b, '\n');
    if (c) {
    *c = '\0';
    }
    fprintf(stderr, "D|%s|\n", b);
    close(w);
    close(r);
    exit(0);
    }

    [binand@kalki]$ cat reverse.c
    #include
    #include

    int main(void) {
    char b[BUFSIZ];
    while (fgets(b, sizeof b, stdin) != NULL) {
    size_t l, m, i;
    char c;
    l = strlen(b);
    if (b[l-1] == '\n') {
    b[l-1] = '\0';
    l--;
    }
    m = l%2 ? (l-1)/2 : l/2;
    for (i = 0; i < m; i++) {
    c = b[i];
    b[i] = b[l-i-1];
    b[l-i-1] = c;
    }
    fputs(b, stdout);
    putc('\n', stdout);
    fflush(stdout);
    }
    }

  2. Re: Two-way pipes to a child process

    "binand@gmail.com" writes:

    >Hi All,


    >I am trying to get a parent-child process relationship working with a
    >pair of pipes that pass data bidirectionally between the two
    >processes.


    Only read to here, so far.
    Instead of two pipes, why not consider a socketpair?
    You then only need a single communication channel between them
    (particularly in systems where pipes are really neutered socketpairs).

    --
    Chris.

  3. Re: Two-way pipes to a child process

    On Wed, 24 Sep 2008 23:24:28 -0700, binand@gmail.com wrote:

    > When I run /usr/bin/rev through my program (source code below), I see
    > that both the parent and child processes are stuck in read().


    You don't check the return value of many library calls. pipe can fail,
    dup2 can fail, close can fail, etc. etc.

    How about using select() or poll() to test whether the read or write end
    of the child is ready?

    read() returns ssize_t, which isn't the same as int, and can't be passed
    to printf( "%d". use (int) or "%zd".

  4. Re: Two-way pipes to a child process

    >I am trying to get a parent-child process relationship working with a
    >pair of pipes that pass data bidirectionally between the two
    >processes.


    DEADLOCK ALERT! It's possible to make this work with stdio buffering
    but it takes care to avoid deadlock. It appears you have a filter
    which requires all the input before outputting any, which does
    simplify things.

    >The idea (as part of a larger project) is to implement
    >filtering via standard shell commands prior to output.
    >
    >When I run /usr/bin/rev through my program (source code below), I see
    >that both the parent and child processes are stuck in read().


    Things to watch for:

    (1) A pipe gets EOF on read only when *all* of the write ends of the
    pipe have been closed. Be careful of extra pipe ends in the parent
    and the child.

    (2) Watch out for stdio buffering. Call fflush() on your output
    pipes before reading input pipes (or use unbuffered read() and
    write()). This, I think, is your problem.

    My theory: /usr/bin/rev does *not* fflush() the output on your
    system for each line of output.

    - Parent writes line into pipe
    - Child reads line from pipe
    - Child outputs reversed line, no fflush, so no write.
    - Child loops back to read another line, and blocks in read.
    - Parent waits on read of answer from child.
    - Both processes are now stuck in reads.

    If you cannot modify the source code of the child, you've got
    a problem. Possible solutions are:

    (1) suck() on the end of the pipe to force the other end to flush it.
    This has the problem that no known system has implemented it. Similar
    tactics like threats of signals don't work either.
    (2) Use a pty, not a pair of pipes.
    (3) Don't use blocking reads and writes, use select() and non-blocking
    reads and writes.

    >
    >Terminal:
    >
    >[binand@kalki]$ ./a.out /usr/bin/rev
    >Running in execl() mode
    >Child PID: 5045, My PID: 5044
    >DW: 59
    >
    >[sits there]
    >[binand@kalki]$ strace -p 5044
    >Process 5044 attached - interrupt to quit
    >read(5,
    >
    >[sits there]
    >Child:
    >
    >[binand@kalki]$ strace -p 5045
    >Process 5045 attached - interrupt to quit
    >read(0,
    >
    >[sits there]
    >When I run ./reverse (which is my own implementation of /usr/bin/rev,
    >source code below), I get the correct behaviour.
    >
    >[binand@kalki]$ ./a.out ./reverse
    >Running in execl() mode
    >Child PID: 5586, My PID: 5585
    >DW: 59
    >DR: 59
    >D|llew derebmemer emit a emac xof eht fo raey eht ni yad enO|
    >DW: 84
    >DR: 84
    >D|lleb kcalb gib eht fo gnillot eht draeh nus gnisir eht fo nam gnuoy
    >gnorts eht nehW|
    >
    >Can anyone tell me what causes this behaviour? My best guess is that
    >stdio's buffering is the reason, but whatever I do with setvbuf(), I
    >still can't get the program to work with /usr/bin/rev (or for that
    >matter, any other filter in /usr/bin). As you can see, my reverse.c
    >has an fflush(stdout); at the end of a line of output (the same code,
    >when inline in ptest.c, works just fine without the fflush() call.
    >This behaviour can be seen by compiling with -DUSE_INLINE).
    >
    >I am on Redhat Enterprise Linux 3 Update 7, using gcc 3.2.3.
    >
    >Thanks in advance,
    >
    >Binand
    >
    >Source code:
    >
    >[binand@kalki]$ cat ptest.c
    >#include
    >#include
    >#include
    >#include
    >
    >#define PIPE_READEND 0
    >#define PIPE_WRITEEND 1
    >
    >int pipe_init(char *, int *, int *);
    >
    >int pipe_init(char *prog, int *rfd, int *wfd) {
    > int p2c[2], c2p[2];
    > int tfd;
    > pid_t pid;
    > char *s, *argv0;
    >
    > s = strrchr(prog, '/');
    > argv0 = s ? s + 1 : prog;
    > pipe(p2c);
    > pipe(c2p);
    > pid = fork();
    >
    > switch (pid) {
    > case -1:
    > return -1;
    > case 0:
    > close(p2c[PIPE_WRITEEND]);
    > close(c2p[PIPE_READEND]);
    > tfd = fileno(stdin);
    > if (tfd >= 0 && p2c[PIPE_READEND] != tfd) {
    > dup2(p2c[PIPE_READEND], tfd);
    > }
    > tfd = fileno(stdout);
    > if (tfd >= 0 && c2p[PIPE_WRITEEND] != tfd) {
    > dup2(c2p[PIPE_WRITEEND], tfd);
    > }
    > close(p2c[PIPE_READEND]);
    > close(c2p[PIPE_WRITEEND]);
    >#ifndef USE_INLINE
    > fprintf(stderr, "Running in execl() mode\n");
    > execl(prog, argv0, NULL);
    > exit(1);
    >#else
    > fprintf(stderr, "Running in inline mode\n");
    > {
    > char b[BUFSIZ];
    > while (fgets(b, sizeof b, stdin) != NULL) {
    > size_t l, m, i;
    > char c;
    > l = strlen(b);
    > if (b[l-1] == '\n') {
    > b[l-1] = '\0';
    > l--;
    > }
    > m = l%2 ? (l-1)/2 : l/2;
    > for (i = 0; i < m; i++) {
    > c = b[i];
    > b[i] = b[l-i-1];
    > b[l-i-1] = c;
    > }
    > fputs(b, stdout);
    > putc('\n', stdout);
    > }
    > }
    > exit(0);
    >#endif
    > default:
    > fprintf(stderr, "Child PID: %d, My PID: %d\n", pid,
    >getpid());
    > close(p2c[PIPE_READEND]);
    > close(c2p[PIPE_WRITEEND]);
    > *rfd = c2p[PIPE_READEND];
    > *wfd = p2c[PIPE_WRITEEND];
    > }
    > return 0;
    >}
    >
    >#define TESTDATA1 "One day in the year of the fox came a time
    >remembered well\n"
    >#define TESTCOUNT1 strlen(TESTDATA1)
    >#define TESTDATA2 "When the strong young man of the rising sun heard
    >the tolling of the big black bell\n"
    >#define TESTCOUNT2 strlen(TESTDATA2)
    >
    >int main (int ac, char **av) {
    > char b[BUFSIZ], *c;
    > int r, w;
    > if (ac < 2) {
    > fprintf(stderr, "Usage: ptest \n");
    > exit(1);
    > }
    > if (pipe_init(av[1], &r, &w) != 0) {
    > fprintf(stderr, "pipe_init failed\n");
    > exit(1);
    > }
    > fprintf(stderr, "DW: %d\n", write(w, TESTDATA1, TESTCOUNT1));
    > fprintf(stderr, "DR: %d\n", read(r, b, TESTCOUNT1));
    > c = strrchr (b, '\n');
    > if (c) {
    > *c = '\0';
    > }
    > fprintf(stderr, "D|%s|\n", b);
    > fprintf(stderr, "DW: %d\n", write(w, TESTDATA2, TESTCOUNT2));
    > fprintf(stderr, "DR: %d\n", read(r, b, TESTCOUNT2));
    > c = strrchr (b, '\n');
    > if (c) {
    > *c = '\0';
    > }
    > fprintf(stderr, "D|%s|\n", b);
    > close(w);
    > close(r);
    > exit(0);
    >}
    >
    >[binand@kalki]$ cat reverse.c
    >#include
    >#include
    >
    >int main(void) {
    > char b[BUFSIZ];
    > while (fgets(b, sizeof b, stdin) != NULL) {
    > size_t l, m, i;
    > char c;
    > l = strlen(b);
    > if (b[l-1] == '\n') {
    > b[l-1] = '\0';
    > l--;
    > }
    > m = l%2 ? (l-1)/2 : l/2;
    > for (i = 0; i < m; i++) {
    > c = b[i];
    > b[i] = b[l-i-1];
    > b[l-i-1] = c;
    > }
    > fputs(b, stdout);
    > putc('\n', stdout);
    > fflush(stdout);
    > }
    >}




  5. Re: Two-way pipes to a child process

    On Sep 25, 1:42*pm, gordonb.lk...@burditt.org (Gordon Burditt) wrote:
    > (2) Watch out for stdio buffering. *Call fflush() on your output
    > pipes before reading input pipes (or use unbuffered read() and
    > write()). *This, I think, is your problem.


    Yes, I believe so too. My reverse.c behaves identically to /usr/bin/
    rev when I remove the fflush(stdout) in it; and with that function
    call in it behaves exactly the way I want.

    I have tried all combinations of setvbuf() [unbuffered and line-
    buffered] after dup2() but before execl() in the child, but none of
    them seem to work.

    > (3) Don't use blocking reads and writes, use select() and non-blocking
    > reads and writes.


    Not sure how I can force a program that appears to be doing block-
    buffered stdio to be nonblocking, though.

    Binand

  6. Re: Two-way pipes to a child process

    In article
    ,
    "binand@gmail.com" wrote:

    > On Sep 25, 1:42*pm, gordonb.lk...@burditt.org (Gordon Burditt) wrote:
    > > (2) Watch out for stdio buffering. *Call fflush() on your output
    > > pipes before reading input pipes (or use unbuffered read() and
    > > write()). *This, I think, is your problem.

    >
    > Yes, I believe so too. My reverse.c behaves identically to /usr/bin/
    > rev when I remove the fflush(stdout) in it; and with that function
    > call in it behaves exactly the way I want.
    >
    > I have tried all combinations of setvbuf() [unbuffered and line-
    > buffered] after dup2() but before execl() in the child, but none of
    > them seem to work.


    That's because exec replaces the process with that of the new program,
    and this reinitializes stdio.

    If the program you're exec'ing doesn't provide an option to turn off its
    stdio buffering (like GNU grep's --line-buffering), then you'll need to
    use a pty instead of a pipe.

    --
    Barry Margolin, barmar@alum.mit.edu
    Arlington, MA
    *** PLEASE post questions in newsgroups, not directly to me ***
    *** PLEASE don't copy me on replies, I'll read them in the group ***

  7. Re: Two-way pipes to a child process

    >> (2) Watch out for stdio buffering. *Call fflush() on your output
    >> pipes before reading input pipes (or use unbuffered read() and
    >> write()). *This, I think, is your problem.

    >
    >Yes, I believe so too. My reverse.c behaves identically to /usr/bin/
    >rev when I remove the fflush(stdout) in it; and with that function
    >call in it behaves exactly the way I want.
    >
    >I have tried all combinations of setvbuf() [unbuffered and line-
    >buffered] after dup2() but before execl() in the child, but none of
    >them seem to work.


    You need to do these *IN THE CHILD* as well as in the parent.

    If you can't change the source code of the child, I offered 3
    alternatives, of which the one involving suck() won't work because
    of the Jihad threat against anyone daring to implement it (none
    yet), the suggestion to use ptys instead of pipes probably will
    work (this issue is largely why ptys exist at all), and the use of
    non-blocking reads and writes will work but uncoupling submitting
    a question and getting an answer may be very awkward.

    >> (3) Don't use blocking reads and writes, use select() and non-blocking
    >> reads and writes.

    >
    >Not sure how I can force a program that appears to be doing block-
    >buffered stdio to be nonblocking, though.


    If you put in a line for it to process, and nothing comes out, keep
    feeding more lines or EOF in until it does (or use ptys). Remember
    that some processes don't do one-line-in, one-line-out, they (for
    example, sort) do all-lines-in, all-lines-out.


  8. Re: Two-way pipes to a child process

    On Sep 26, 6:25 am, gordonb.sk...@burditt.org (Gordon Burditt) wrote:
    > >> (2) Watch out for stdio buffering. Call fflush() on your output
    > >> pipes before reading input pipes (or use unbuffered read() and
    > >> write()). This, I think, is your problem.


    > >Yes, I believe so too. My reverse.c behaves identically to /usr/bin/
    > >rev when I remove the fflush(stdout) in it; and with that function
    > >call in it behaves exactly the way I want.


    > >I have tried all combinations of setvbuf() [unbuffered and line-
    > >buffered] after dup2() but before execl() in the child, but none of
    > >them seem to work.


    > You need to do these *IN THE CHILD* as well as in the parent.


    > If you can't change the source code of the child, I offered 3
    > alternatives, of which the one involving suck() won't work because
    > of the Jihad threat against anyone daring to implement it (none
    > yet), the suggestion to use ptys instead of pipes probably will
    > work (this issue is largely why ptys exist at all), and the use of
    > non-blocking reads and writes will work but uncoupling submitting
    > a question and getting an answer may be very awkward.


    > >> (3) Don't use blocking reads and writes, use select() and
    > >> non-blocking reads and writes.


    > >Not sure how I can force a program that appears to be doing
    > >block- buffered stdio to be nonblocking, though.


    > If you put in a line for it to process, and nothing comes out,
    > keep feeding more lines or EOF in until it does (or use ptys).
    > Remember that some processes don't do one-line-in,
    > one-line-out, they (for example, sort) do all-lines-in,
    > all-lines-out.


    The original poster mentionned wanting to use classical Unix
    filters. IMHO, the best solution for this is to have them
    output to a temporary file, then read that, but otherwise: if
    you can be sure that one of the processes in the pipeline will
    not read until it sees EOF, you can first write all of the data,
    then read it. The only traditional filter that I know that
    guarantees this is sort (not an explicit guarantee, but for
    obvious reasons, sort can't write anything until it's read
    everything), but it's trivial to write a small filter yourself
    which does this, and add it to the end of your pipeline, if
    necessary.

    --
    James Kanze (GABI Software) email:james.kanze@gmail.com
    Conseils en informatique orientée objet/
    Beratung in objektorientierter Datenverarbeitung
    9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34


  9. Re: Two-way pipes to a child process

    On Sep 26, 12:21*am, Barry Margolin wrote:
    > If the program you're exec'ing doesn't provide an option to turn off its
    > stdio buffering (like GNU grep's --line-buffering), then you'll need to
    > use a pty instead of a pipe.


    Thanks are due to everyone who responded. I tried all four suggestions
    (pipe(), socketpair(), ptys and even a temporary file!); finally I got
    it working the way I want with openpty() and some tcsetattr() magic
    (APUE helped here). Now I can use standard Unix filters in a one line
    in, one line out fashion.

    Binand

+ Reply to Thread