append to the recv socket buffer - Unix

This is a discussion on append to the recv socket buffer - Unix ; Hi all, I have a problem. I am writing a server code that has n number of clients (less than 10) connected to it. I am using blocking, so i listen to one client and wait until I receive a ...

+ Reply to Thread
Results 1 to 12 of 12

Thread: append to the recv socket buffer

  1. append to the recv socket buffer

    Hi all,
    I have a problem.
    I am writing a server code that has n number of clients (less than 10)
    connected to it. I am using blocking, so i listen to one client and
    wait until I receive a packet from it and then go the next and do the
    same.

    What I do is I go through a loop and get one packet from every client
    saying how much data they will be sending. Then I allocate a buffer
    and read in that much data. My question is, is it possible to read how
    much data I will be receiving from every client, add up the size and
    then define one big buffer and append data to it as i am looping
    through the clients.

    So first client says i am sending you 2 bytes next time. the second
    client says, I am sending you 3 bytes. then I define a buffer with
    size 5 and read 2 bytes from client 1 and read 3 bytes from client 2
    and append it to the same buffer.

    Thanks


  2. Re: append to the recv socket buffer

    Jack wrote:
    > Hi all,
    > I have a problem.
    > I am writing a server code that has n number of clients (less than 10)
    > connected to it. I am using blocking, so i listen to one client and
    > wait until I receive a packet from it and then go the next and do the
    > same.
    >
    > What I do is I go through a loop and get one packet from every client
    > saying how much data they will be sending. Then I allocate a buffer
    > and read in that much data. My question is, is it possible to read how
    > much data I will be receiving from every client, add up the size and
    > then define one big buffer and append data to it as i am looping
    > through the clients.
    >
    > So first client says i am sending you 2 bytes next time. the second
    > client says, I am sending you 3 bytes. then I define a buffer with
    > size 5 and read 2 bytes from client 1 and read 3 bytes from client 2
    > and append it to the same buffer.
    >

    It's possible, but probably much harder to manage than just giving them
    a buffer each. What problem are you attempting to solve by doing this?

    Would it not be easier to define a maximum message size and dispense
    with the first packet? Failing that, you could prepend the size to the
    data and read it first. Before too long, you will end up defining an on
    the wire protocol, which you probably should do first.

    --
    Ian Collins.

  3. Re: append to the recv socket buffer

    On Apr 1, 5:43 pm, Ian Collins wrote:
    > Jack wrote:
    > > Hi all,
    > > I have a problem.
    > > I am writing a server code that has n number of clients (less than 10)
    > > connected to it. I am using blocking, so i listen to one client and
    > > wait until I receive a packet from it and then go the next and do the
    > > same.

    >
    > > What I do is I go through a loop and get one packet from every client
    > > saying how much data they will be sending. Then I allocate a buffer
    > > and read in that much data. My question is, is it possible to read how
    > > much data I will be receiving from every client, add up the size and
    > > then define one big buffer and append data to it as i am looping
    > > through the clients.

    >
    > > So first client says i am sending you 2 bytes next time. the second
    > > client says, I am sending you 3 bytes. then I define a buffer with
    > > size 5 and read 2 bytes from client 1 and read 3 bytes from client 2
    > > and append it to the same buffer.

    >
    > It's possible, but probably much harder to manage than just giving them
    > a buffer each. What problem are you attempting to solve by doing this?
    >
    > Would it not be easier to define a maximum message size and dispense
    > with the first packet? Failing that, you could prepend the size to the
    > data and read it first. Before too long, you will end up defining an on
    > the wire protocol, which you probably should do first.
    >
    > --
    > Ian Collins.


    well I can not predict the size of data so I can't have a set size. I
    am also trying to minimize the amount of times I call malloc or
    realloc. Another thing is that even if I have a buffer for each
    client, i still have to put them in to one buffer to be able to
    manipulate them etc. so I thought I can skip a couple of steps and
    have one buffer which has everything.



  4. Re: append to the recv socket buffer

    sdastouri@gmail.com wrote:
    > On Apr 1, 5:43 pm, Ian Collins wrote:
    >
    >>Jack wrote:
    >>
    >>>Hi all,
    >>>I have a problem.
    >>>I am writing a server code that has n number of clients (less than 10)
    >>>connected to it. I am using blocking, so i listen to one client and
    >>>wait until I receive a packet from it and then go the next and do the
    >>>same.

    >>
    >>>What I do is I go through a loop and get one packet from every client
    >>>saying how much data they will be sending. Then I allocate a buffer
    >>>and read in that much data. My question is, is it possible to read how
    >>>much data I will be receiving from every client, add up the size and
    >>>then define one big buffer and append data to it as i am looping
    >>>through the clients.

    >>
    >>>So first client says i am sending you 2 bytes next time. the second
    >>>client says, I am sending you 3 bytes. then I define a buffer with
    >>>size 5 and read 2 bytes from client 1 and read 3 bytes from client 2
    >>>and append it to the same buffer.

    >>
    >>It's possible, but probably much harder to manage than just giving them
    >>a buffer each. What problem are you attempting to solve by doing this?
    >>
    >>Would it not be easier to define a maximum message size and dispense
    >>with the first packet? Failing that, you could prepend the size to the
    >>data and read it first. Before too long, you will end up defining an on
    >>the wire protocol, which you probably should do first.
    >>

    *Please* don't quote signatures.
    >
    > well I can not predict the size of data so I can't have a set size. I
    > am also trying to minimize the amount of times I call malloc or
    > realloc. Another thing is that even if I have a buffer for each
    > client, i still have to put them in to one buffer to be able to
    > manipulate them etc. so I thought I can skip a couple of steps and
    > have one buffer which has everything.
    >

    Why is calling malloc a problem? There's nothing to stop you using one
    buffer for the reads, you just have to manage it.

    --
    Ian Collins.

  5. Re: append to the recv socket buffer

    On Apr 1, 6:18 pm, Ian Collins wrote:
    > sdasto...@gmail.com wrote:
    > > On Apr 1, 5:43 pm, Ian Collins wrote:

    >
    > >>Jack wrote:

    >
    > >>>Hi all,
    > >>>I have a problem.
    > >>>I am writing a server code that has n number of clients (less than 10)
    > >>>connected to it. I am using blocking, so i listen to one client and
    > >>>wait until I receive a packet from it and then go the next and do the
    > >>>same.

    >
    > >>>What I do is I go through a loop and get one packet from every client
    > >>>saying how much data they will be sending. Then I allocate a buffer
    > >>>and read in that much data. My question is, is it possible to read how
    > >>>much data I will be receiving from every client, add up the size and
    > >>>then define one big buffer and append data to it as i am looping
    > >>>through the clients.

    >
    > >>>So first client says i am sending you 2 bytes next time. the second
    > >>>client says, I am sending you 3 bytes. then I define a buffer with
    > >>>size 5 and read 2 bytes from client 1 and read 3 bytes from client 2
    > >>>and append it to the same buffer.

    >
    > >>It's possible, but probably much harder to manage than just giving them
    > >>a buffer each. What problem are you attempting to solve by doing this?

    >
    > >>Would it not be easier to define a maximum message size and dispense
    > >>with the first packet? Failing that, you could prepend the size to the
    > >>data and read it first. Before too long, you will end up defining an on
    > >>the wire protocol, which you probably should do first.

    >
    > *Please* don't quote signatures.
    >
    > > well I can not predict the size of data so I can't have a set size. I
    > > am also trying to minimize the amount of times I call malloc or
    > > realloc. Another thing is that even if I have a buffer for each
    > > client, i still have to put them in to one buffer to be able to
    > > manipulate them etc. so I thought I can skip a couple of steps and
    > > have one buffer which has everything.

    >
    > Why is calling malloc a problem? There's nothing to stop you using one
    > buffer for the reads, you just have to manage it.
    >


    Given what i am doing malloc takes too long (being a system call and
    everything). So I rather use it as seldom as possible.

    ok, So here is a sample code

    int sizeOfNextMsg[noMachines];
    int temp;
    double *mainBuffer;
    int i;
    for (i = 0; i < noMachines; i++){
    recv(socket[i], &sizeOfNextMsg[i],sizeof(int), 0);
    temp += (int)ntohl(sizeOfNextMsg[i]);
    }
    mainBuffer = (double*) malloc ((temp) * sizeof(double));
    for (i = 0; i < machines; i++){
    recv(socket[i], sampleBuffer......................,
    sizeof(double), 0);
    //Then I somehow go to read all the values in the sample Buffer in
    to another buffer but converted using ntohl.

    I don't know how to append I tell recv to put the incoming data into
    sampleBuffer.




  6. Re: append to the recv socket buffer

    Jack wrote:
    > On Apr 1, 6:18 pm, Ian Collins wrote:
    >
    >>sdasto...@gmail.com wrote:
    >>
    >>>well I can not predict the size of data so I can't have a set size. I
    >>>am also trying to minimize the amount of times I call malloc or
    >>>realloc. Another thing is that even if I have a buffer for each
    >>>client, i still have to put them in to one buffer to be able to
    >>>manipulate them etc. so I thought I can skip a couple of steps and
    >>>have one buffer which has everything.

    >>
    >>Why is calling malloc a problem? There's nothing to stop you using one
    >>buffer for the reads, you just have to manage it.
    >>

    > Given what i am doing malloc takes too long (being a system call and
    > everything). So I rather use it as seldom as possible.
    >

    Are you sure? Have you profiled it?

    > ok, So here is a sample code
    >
    > int sizeOfNextMsg[noMachines];
    > int temp;


    Why not use long and avoid the later cast?

    > double *mainBuffer;
    > int i;
    > for (i = 0; i < noMachines; i++){
    > recv(socket[i], &sizeOfNextMsg[i],sizeof(int), 0);
    > temp += (int)ntohl(sizeOfNextMsg[i]);
    > }
    > mainBuffer = (double*) malloc ((temp) * sizeof(double));


    You don't have to (and probably shouldn't) cast the return of malloc.

    > for (i = 0; i < machines; i++){
    > recv(socket[i], sampleBuffer......................,
    > sizeof(double), 0);
    > //Then I somehow go to read all the values in the sample Buffer in
    > to another buffer but converted using ntohl.
    >

    Not a good idea if sizeof(double) != sizeof(long).

    > I don't know how to append I tell recv to put the incoming data into
    > sampleBuffer.
    >

    You have to keep track of the start address of each packet in the
    buffer, probably a start and possibly end pointer for each client.
    That's the extra work I mentioned.

    --
    Ian Collins.

  7. Re: append to the recv socket buffer

    Jack wrote:
    > Given what i am doing malloc takes too long (being a system call and
    > everything). So I rather use it as seldom as possible.


    Um. Malloc is not a system call. It does use a system call once in a
    while but that's pretty much independent of how many times you call it.
    The situation is analogous to fwrite(), which will use the write()
    system call once in a while, but the number of write() calls is a
    function of how much data you end up writing, not of how many times you
    call fwrite().

    Bottom line, there's no technical reason you can't do what you propose.
    People here are not convinced that it's the right place to apply
    brainpower[*] or that your reasoning makes sense, but if you're sure it
    needs to be done, go for it.
    [*] The old aphorism that premature optimization is the root of all evil
    springs to mind.

    HT

  8. Re: append to the recv socket buffer

    On Apr 1, 7:39 pm, Henry Townsend wrote:
    > Jack wrote:
    > > Given what i am doing malloc takes too long (being a system call and
    > > everything). So I rather use it as seldom as possible.

    >
    > Um. Malloc is not a system call. It does use a system call once in a
    > while but that's pretty much independent of how many times you call it.
    > The situation is analogous to fwrite(), which will use the write()
    > system call once in a while, but the number of write() calls is a
    > function of how much data you end up writing, not of how many times you
    > call fwrite().
    >
    > Bottom line, there's no technical reason you can't do what you propose.
    > People here are not convinced that it's the right place to apply
    > brainpower[*] or that your reasoning makes sense, but if you're sure it
    > needs to be done, go for it.
    >
    >[*] The old aphorism that premature optimization is the root of all evil
    > springs to mind.


    Thanks for the info. Just one more question. If I am sending an int or
    a double across the wire, should I use ntohl or ntohs since these two
    functions are used for converting network byte order to long and
    short. Is there another function that is to be used for variables of
    type int or double.

    Thanks



  9. Re: append to the recv socket buffer

    Jack wrote:
    > On Apr 1, 7:39 pm, Henry Townsend wrote:
    >
    >>Jack wrote:
    >>
    >>>Given what i am doing malloc takes too long (being a system call and
    >>>everything). So I rather use it as seldom as possible.

    >>
    >>Um. Malloc is not a system call. It does use a system call once in a
    >>while but that's pretty much independent of how many times you call it.
    >>The situation is analogous to fwrite(), which will use the write()
    >>system call once in a while, but the number of write() calls is a
    >>function of how much data you end up writing, not of how many times you
    >>call fwrite().
    >>
    >>Bottom line, there's no technical reason you can't do what you propose.
    >>People here are not convinced that it's the right place to apply
    >>brainpower[*] or that your reasoning makes sense, but if you're sure it
    >>needs to be done, go for it.
    >>

    >
    > Thanks for the info. Just one more question. If I am sending an int or
    > a double across the wire, should I use ntohl or ntohs since these two
    > functions are used for converting network byte order to long and
    > short. Is there another function that is to be used for variables of
    > type int or double.
    >

    ntohs converts a 16 bit number and ntohl converts a 32 bit number,
    double is undoubtably at least 64 bits, so neither is appropriate to
    covert a double.

    The name ntohl is out of touch with systems with a 64 bit long.

    --
    Ian Collins.

  10. Re: append to the recv socket buffer

    "Jack" writes:
    > On Apr 1, 6:18 pm, Ian Collins wrote:
    >>sdasto...@gmail.com wrote:
    >>> well I can not predict the size of data so I can't have a set
    >>> size. I am also trying to minimize the amount of times I call malloc
    >>> or realloc. Another thing is that even if I have a buffer for each
    >>> client, i still have to put them in to one buffer to be able to
    >>> manipulate them etc. so I thought I can skip a couple of steps and
    >>> have one buffer which has everything.

    >>
    >> Why is calling malloc a problem? There's nothing to stop you using
    >> one buffer for the reads, you just have to manage it.

    >
    > Given what i am doing malloc takes too long (being a system
    > call and everything). So I rather use it as seldom as possible.


    Well, sort of, but no... malloc() is not a system call.

    It is also not such a bad thing to call malloc() and free(). A lot of
    intelligence, implementation-specific and system-specific knowledge has
    gone into many memory allocators. Some of them are even smart enough to
    pre-allocate large chunks of memory, divide them in buckets of equally
    sized objects, and cache freed objects, so they can *save* you system
    calls, by returning a freed object the next time you ask for an
    allocation.

    For this reason, and a few others, most of the time that someone tries
    to 'optimize code' by minimizing malloc() calls, they are travelling
    under a faulty, bogus path.

    Have you actually *measured* where your program spends most of its time,
    and found that malloc() calls amount to a large percentage of your
    program's run time?

    > ok, So here is a sample code
    >
    > int sizeOfNextMsg[noMachines];
    > int temp;
    > double *mainBuffer;
    > int i;
    > for (i = 0; i < noMachines; i++){
    > recv(socket[i], &sizeOfNextMsg[i],sizeof(int), 0);
    > temp += (int)ntohl(sizeOfNextMsg[i]);
    > }


    You shouldn't just assume that an 'int' is an integral value large
    enough to hold a buffer length. The C standard allows implementations
    to choose an integer type for 'int' with limits which may be as
    constrained as the closed set of [ -32767 ... +32767 ]. If your program
    happens to run on an implementation with INT_MAX == +32767, you will
    find it very difficult to allocate buffers with a size larger than 32KB.

    Even if this problem is resolved somehow, the above loop will still have
    a problem if the *sum* of all the buffer sizes exceeds 'int' limits.
    With a sufficiently large value of 'noMachines', you are bound to
    overflow an 'int' pretty fast, even if you choose an absurdly small size
    for each buffer, like 2 bytes.

    > mainBuffer = (double*) malloc ((temp) * sizeof(double));


    You shouldn't cast the result of malloc(). See the numerous threads in
    comp.lang.c and the FAQ of comp.lang.c about the reasons why this is bad.

    > for (i = 0; i < machines; i++){
    > recv(socket[i], sampleBuffer......................,
    > sizeof(double), 0);
    > //Then I somehow go to read all the values in the sample Buffer in
    > to another buffer but converted using ntohl.
    >
    > I don't know how to append I tell recv to put the incoming data into
    > sampleBuffer.


    I don't understand the last sentence above, and I don't really
    understand what 'sampleBuffer' has to do with the malloc() call you
    showed and 'mainBuffer'. Please try to phrase it differently, or post a
    complete program, which can be compiled and tested locally.

    - Giorgos


  11. Re: append to the recv socket buffer

    On 2007-04-02, Jack wrote:
    > On Apr 1, 6:18 pm, Ian Collins wrote:

    [...]
    >> Why is calling malloc a problem? There's nothing to stop you using one
    >> buffer for the reads, you just have to manage it.
    >>

    >
    > Given what i am doing malloc takes too long (being a system call and
    > everything). So I rather use it as seldom as possible.
    >
    > ok, So here is a sample code

    [...]
    > for (i = 0; i < noMachines; i++){
    > recv(socket[i], &sizeOfNextMsg[i],sizeof(int), 0);
    > }


    So far, I've learned that calling recv too many times is much much much
    worse than using malloc. So, generally, I use large enough buffer to fit
    chunk of what the other side has sent, call recv to read as much as
    possible into that buffer, and then parse the buffer content. If the
    content is not complete, or the buffer won't fit everything, then I may
    increase the buffer and call recv again.


    --
    Minds, like parachutes, function best when open

  12. Re: append to the recv socket buffer

    On 2 Apr, 02:13, sdasto...@gmail.com wrote:
    > On Apr 1, 5:43 pm, Ian Collins wrote:
    >
    >
    >
    > > Jack wrote:
    > > > Hi all,
    > > > I have a problem.
    > > > I am writing a server code that has n number of clients (less than 10)
    > > > connected to it. I am using blocking, so i listen to one client and
    > > > wait until I receive a packet from it and then go the next and do the
    > > > same.

    >
    > > > What I do is I go through a loop and get one packet from every client
    > > > saying how much data they will be sending. Then I allocate a buffer
    > > > and read in that much data. My question is, is it possible to read how
    > > > much data I will be receiving from every client, add up the size and
    > > > then define one big buffer and append data to it as i am looping
    > > > through the clients.

    >
    > > > So first client says i am sending you 2 bytes next time. the second
    > > > client says, I am sending you 3 bytes. then I define a buffer with
    > > > size 5 and read 2 bytes from client 1 and read 3 bytes from client 2
    > > > and append it to the same buffer.

    >
    > > It's possible, but probably much harder to manage than just giving them
    > > a buffer each. What problem are you attempting to solve by doing this?

    >
    > > Would it not be easier to define a maximum message size and dispense
    > > with the first packet? Failing that, you could prepend the size to the
    > > data and read it first. Before too long, you will end up defining an on
    > > the wire protocol, which you probably should do first.

    >
    > > --
    > > Ian Collins.

    >
    > well I can not predict the size of data so I can't have a set size. I
    > am also trying to minimize the amount of times I call malloc or
    > realloc.


    If the protocol you are using has a messages size limit, it might be a
    good idea to preallocate an array of that size, and then use it as a
    ring buffer, reading into it with readv() syscall. This way reading
    data from connection would not require any user-space allocations.
    (not to mention that the ring buffers might be kept in a pool after
    use, rather than deallocated)

    > Another thing is that even if I have a buffer for each
    > client, i still have to put them in to one buffer to be able to
    > manipulate them etc. so I thought I can skip a couple of steps and
    > have one buffer which has everything.


    A custom iterator might help here. In its operator++/-- it would move
    to the next/prev array. The code would use that iterator instead of a
    plain pointer.


+ Reply to Thread