Buffering Client data in a Server Application (select() usage) - Unix

This is a discussion on Buffering Client data in a Server Application (select() usage) - Unix ; Hello, I am implementing a server in C using the select function and I have problems implementing a buffering system for holding client data until the client socket is available for reading/writing (sendq and receivq). What I am trying to ...

+ Reply to Thread
Results 1 to 7 of 7

Thread: Buffering Client data in a Server Application (select() usage)

  1. Buffering Client data in a Server Application (select() usage)

    Hello,

    I am implementing a server in C using the select function and I have
    problems implementing a buffering system for holding client data until
    the client socket is available for reading/writing (sendq and
    receivq). What I am trying to do is "save" the data in the client's
    recvq right after data is availiable for the socket and write buffered
    data (sendq) to the socket when it's ready. The prog skeleton is like
    this:

    (The following is not complete C code, it's just an example)

    unsigned char buffer[1024];
    select(nfds, &readset, &writeset, NULL);
    // select returns

    for (i=0; i if (FD_ISSET(i, &readset)) {
    r = read(i,buffer,...);
    // cli = client data structure
    cli = (struct cli *) getclient(i);
    //add data to the client's recvq
    add_data(&cli->recvq, buffer, r);
    if (has_complete_message(&cli->recvq)) {
    parseline(cli);
    // add results to the client's buffer (sendq) for writing them
    later
    }
    }
    if (FD_ISSET(i, &writeset)) {
    // write queued data
    }
    ....

    }

    Is there any code/lib for this purpose with functions like
    add_data(..), read_data(...), get_message(..), get_len(..),
    has_complete_message(..), probably using double linked list or
    something similar

    I have found some str alloc packages but they are not very good for
    this purpose.

    PS. If anyone have played with ircd's, I'm searching for something
    similar to sbuf.c (bahamut) or linebuf.c (ratbox) with more "clear"
    code;

    Thanks for your time and sorry for my bad english.


  2. Re: Buffering Client data in a Server Application (select() usage)

    On 2007-10-10, charpour@yahoo.com wrote:
    [...]
    > I am implementing a server in C using the select function and I have
    > problems implementing a buffering system for holding client data until
    > the client socket is available for reading/writing (sendq and

    [...]
    > Is there any code/lib for this purpose with functions like
    > add_data(..), read_data(...), get_message(..), get_len(..),
    > has_complete_message(..), probably using double linked list or
    > something similar

    [...]

    Usually it does not make sense to use generic libraries for this
    purpose. Efficient buffering for different protocols would be done
    differently. Usually, the protocol processing is much more complex than
    the data buffering. So if you can't write data buffering how can you
    write the protocol processing? Really, what is so hard about data
    buffering?

    --
    Minds, like parachutes, function best when open

  3. Re: Buffering Client data in a Server Application (select() usage)

    On Oct 11, 5:15 pm, Andrei Voropaev wrote:
    > On 2007-10-10, charp...@yahoo.com wrote:
    > [...]> I am implementing a server in C using the select function and I have
    > > problems implementing a buffering system for holding client data until
    > > the client socket is available for reading/writing (sendq and

    > [...]
    > > Is there any code/lib for this purpose with functions like
    > > add_data(..), read_data(...), get_message(..), get_len(..),
    > > has_complete_message(..), probably using double linked list or
    > > something similar

    >
    > [...]
    >
    > Usually it does not make sense to use generic libraries for this
    > purpose. Efficient buffering for different protocols would be done
    > differently. Usually, the protocol processing is much more complex than
    > the data buffering. So if you can't write data buffering how can you
    > write the protocol processing? Really, what is so hard about data
    > buffering?
    >
    > --
    > Minds, like parachutes, function best when open


    The protocol consits of simple messages with len no longer than 1024
    bytes, with an \r\n for each message (much like the ircd proto).
    The problem is that adding and removing messages from buffers (and
    dynamic allocating if needed) is not that easy...

    Btw these days I implemented a buffer system (with my little C
    knowledge) which seems to work, but I am not sure if the logic is ok.
    If anyone has the time to check the code, it would be great.

    (The mempool is not in use)

    Here is the code: http://pastebin.com/m6a3702ad

    Thanks for your time



  4. Re: Buffering Client data in a Server Application (select() usage)

    "DiAvOl" wrote in message
    news:1192185940.176866.304890@i13g2000prf.googlegr oups.com...
    > The protocol consits of simple messages with len no longer than 1024
    > bytes, with an \r\n for each message (much like the ircd proto).
    > The problem is that adding and removing messages from buffers (and
    > dynamic allocating if needed) is not that easy...
    >
    > Btw these days I implemented a buffer system (with my little C
    > knowledge) which seems to work, but I am not sure if the logic is ok.
    > If anyone has the time to check the code, it would be great.
    >
    > (The mempool is not in use)
    >
    > Here is the code: http://pastebin.com/m6a3702ad


    I had a quick look; almost all the complexity comes from the multiple
    buffers, which as far as I can see are fairly pointless. More importantly,
    you don't seem to handle the case that the \r\n straddles a buffer boundary.
    Instead, only look for \n, and then check if the already-copied previous
    character was \r.

    With a single, fixed-size buffer (at least as large as the largest message),
    appending data to the buffer is straightforward. The "get message" function
    can return a pointer into the buffer, usually avoiding any additional
    copying. A simple algorithm is to return complete messages from the buffer
    until the end is reached; then, if there is a partial message, move it to
    the start of the buffer.

    Alex



  5. Re: Buffering Client data in a Server Application (select() usage)

    > I had a quick look; almost all the complexity comes from the multiple
    > buffers, which as far as I can see are fairly pointless. More importantly,
    > you don't seem to handle the case that the \r\n straddles a buffer boundary.
    > Instead, only look for \n, and then check if the already-copied previous
    > character was \r.
    >
    > With a single, fixed-size buffer (at least as large as the largest message),
    > appending data to the buffer is straightforward. The "get message" function
    > can return a pointer into the buffer, usually avoiding any additional
    > copying. A simple algorithm is to return complete messages from the buffer
    > until the end is reached; then, if there is a partial message, move it to
    > the start of the buffer.
    >
    > Alex


    The problem with returning a pointer is that a message may exist in
    more than 1 buffers which makes impossible that approach thus making
    copying mandatory.

    (If you have in mind any url's with code examples implementing a
    similar algorithm please let me know)

    Thanks for your time & help


  6. Re: Buffering Client data in a Server Application (select() usage)

    "DiAvOl" wrote in message
    news:1192237899.096928.295550@v29g2000prd.googlegr oups.com...
    [snip]
    >> With a single, fixed-size buffer (at least as large as the largest
    >> message), appending data to the buffer is straightforward. The "get
    >> message" function can return a pointer into the buffer, usually avoiding
    >> any additional copying.

    [snip]
    > The problem with returning a pointer is that a message may exist in
    > more than 1 buffers which makes impossible that approach thus making
    > copying mandatory.


    That was part of my point: I can't see the benefit of the multiple buffer
    scheme you used, and compared to using a single, fixed-size buffer, the code
    is significantly more complex and requires an extra copy.

    > (If you have in mind any url's with code examples implementing a
    > similar algorithm please let me know)


    Here is some code implementing a fixed-size buffer scheme for
    CRLF-terminated messages:

    struct receive_buffer {
    char *start;
    size_t used;
    char buf[BUFLEN];
    };

    ssize_t fill_buffer(int fd, struct receive_buffer *rb) {
    ssize_t n = read(fd, rb->buf + rb->used, BUFLEN - rb->used);
    if (n != -1) rb->used += n;
    return n;
    }

    char *get_message(struct receive_buffer *rb, int *too_long) {
    /* MUST be called repeatedly until it returns NULL to make more buffer
    * space available
    */
    char *start, *p, *end;

    p = start = rb->start;
    end = rb->buf + rb->used;
    while (p < end) {
    if (*p == '\n') {
    if (p != rb->buf && p[-1] == '\r') {
    p[-1] = 0;
    } else {
    /* No CR before LF: ignored for simplicity */
    *p = 0;
    }
    rb->start = p + 1;
    return start;
    }
    ++p;
    }

    /* Buffer empty or partial message */
    if (rb->used == BUFLEN && start == rb->buf) {
    *too_long = 1;
    } else {
    rb->used = p - start; /* usually zero */
    if (rb->used) memmove(rb->buf, start, rb->used);
    rb->start = rb->buf;
    }
    return 0;
    }

    Typical usage would be along these lines (in response to readable
    notification for a non-blocking descriptor):
    n = fill_buffer(fd, rb);
    if (n == -1) /* handle read() error */;
    too_long = 0;
    while ((msg = get_message(rb, &too_long)))
    process_message(msg);
    if (too_long) /* error */;

    Alex



  7. Re: Buffering Client data in a Server Application (select() usage)

    On Wed, 10 Oct 2007 14:58:48 -0700, charpour wrote:

    > Hello,
    >
    > I am implementing a server in C using the select function and I have
    > problems implementing a buffering system for holding client data until
    > the client socket is available for reading/writing (sendq and receivq).
    > What I am trying to do is "save" the data in the client's recvq right
    > after data is availiable for the socket and write buffered data (sendq)
    > to the socket when it's ready. The prog skeleton is like this:
    >
    > (The following is not complete C code, it's just an example)
    >

    [...]
    >
    > Is there any code/lib for this purpose with functions like add_data(..),
    > read_data(...), get_message(..), get_len(..), has_complete_message(..),
    > probably using double linked list or something similar
    >
    > I have found some str alloc packages but they are not very good for this
    > purpose.
    >
    > PS. If anyone have played with ircd's, I'm searching for something
    > similar to sbuf.c (bahamut) or linebuf.c (ratbox) with more "clear"
    > code;
    >
    > Thanks for your time and sorry for my bad english.


    See:

    http://www.and.org/texts/network_io

    --
    James Antill -- james@and.org
    C String APIs use too much memory? ustr: length, ref count, size and
    read-only/fixed. Ave. 44% overhead over strdup(), for 0-20B strings
    http://www.and.org/ustr/

+ Reply to Thread