how to transfer small files? - Networking

This is a discussion on how to transfer small files? - Networking ; Hi all, I have to checkout 100000 files from an internal repository and the operation is very slow. Both server and client nics are gigabit full duplex, no errors detected by any side. I believe the problem is the tcp ...

+ Reply to Thread
Results 1 to 7 of 7

Thread: how to transfer small files?

  1. how to transfer small files?

    Hi all, I have to checkout 100000 files from an internal repository
    and the operation is very slow. Both server and client nics are
    gigabit full duplex, no errors detected by any side.

    I believe the problem is the tcp overhead created by the tcp protocol
    for each small file.

    Could you suggest me another protocol or a solution to checkout a huge
    quantity of small files?
    Maybe I could force to use udp but it's not a solution because it
    doesn't grant the integrity..


    Regards


    Marco Strullato

  2. Re: how to transfer small files?

    Marco skrev:
    > Hi all, I have to checkout 100000 files from an internal repository
    > and the operation is very slow. Both server and client nics are
    > gigabit full duplex, no errors detected by any side.
    >
    > I believe the problem is the tcp overhead created by the tcp protocol
    > for each small file.
    >
    > Could you suggest me another protocol or a solution to checkout a huge
    > quantity of small files?
    > Maybe I could force to use udp but it's not a solution because it
    > doesn't grant the integrity..
    >
    >
    > Regards
    >
    >
    > Marco Strullato


    It is most likely the tcp overhead causing you the slow transfer as you
    say. Cant you zip the files first into one large archive and then
    transfer it?

    Cheers,
    Tobias Skytte

  3. Re: how to transfer small files?

    Unfortunately the operation is a checkout so I can not compress
    files...


    > It is most likely the tcp overhead causing you the slow transfer as you
    > say. Cant you zip the files first into one large archive and then
    > transfer it?



    Marco Strullato

  4. Re: how to transfer small files?

    On Tue, 7 Oct 2008 06:48:54 -0700 (PDT), I waved a wand and this
    message magically appears in front of Marco:

    > Unfortunately the operation is a checkout so I can not compress
    > files...


    If you're using cvs, there's the -z9 option to compress files.
    --
    http://www.munted.org.uk

    Fearsome grindings.


  5. Re: how to transfer small files?

    Alex Buell writes:

    > On Tue, 7 Oct 2008 06:48:54 -0700 (PDT), I waved a wand and this
    > message magically appears in front of Marco:
    >
    >> Unfortunately the operation is a checkout so I can not compress
    >> files...

    >
    > If you're using cvs, there's the -z9 option to compress files.


    Does that compress individual files? If so, I'm not sure that will
    help that much. You still have the overhead of opening the
    connection.

    Marco - sometimes you have to start the process, and walk away from the
    computer. Come back the next morning......

  6. Re: how to transfer small files?

    On Tue, 07 Oct 2008 20:01:38 -0400, Maxwell Lol wrote:

    >Alex Buell writes:
    >
    >> On Tue, 7 Oct 2008 06:48:54 -0700 (PDT), I waved a wand and this
    >> message magically appears in front of Marco:
    >>
    >>> Unfortunately the operation is a checkout so I can not compress
    >>> files...

    >>
    >> If you're using cvs, there's the -z9 option to compress files.

    >
    >Does that compress individual files? If so, I'm not sure that will
    >help that much. You still have the overhead of opening the
    >connection.
    >
    >Marco - sometimes you have to start the process, and walk away from the
    >computer. Come back the next morning......


    That's too easy, want it now! )

    Grant.
    --
    http://bugsplatter.id.au/

  7. Re: how to transfer small files?

    Marco skrev:
    > Unfortunately the operation is a checkout so I can not compress
    > files...
    >
    >
    >> It is most likely the tcp overhead causing you the slow transfer as you
    >> say. Cant you zip the files first into one large archive and then
    >> transfer it?

    >
    >
    > Marco Strullato


    Ok, is it Subversion or something? Maybe checkout the files locally
    first then zip them (the entire directory) and transfer and unzip. In
    svn there are commandline tools to do that, so you can automate it with
    a script.

    Regards,
    Tobias

+ Reply to Thread