rsync many small files - Tools

This is a discussion on rsync many small files - Tools ; At our lab we have storage with many small files. For example a directory can contain over 15,000 files and each file averages about 75k. I would like to sync this to another filesystem on a different server but I ...

+ Reply to Thread
Results 1 to 4 of 4

Thread: rsync many small files

  1. rsync many small files

    At our lab we have storage with many small files. For example a
    directory can contain over 15,000 files and each file averages about
    75k. I would like to sync this to another filesystem on a different
    server but I am not sure if there is a rsync tuning flag I can use for
    such a intensive job. I am using rsync 3.0.3. Also, I would like to
    use little memory as possible.

    Any thoughts? TIA
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  2. Re: rsync many small files

    Tue, 23 Sep 2008 06:49:33 -0400, magawake wrote:

    > At our lab we have storage with many small files. For example a
    > directory can contain over 15,000 files and each file averages about
    > 75k. I would like to sync this to another filesystem on a different
    > server but I am not sure if there is a rsync tuning flag I can use for
    > such a intensive job. I am using rsync 3.0.3. Also, I would like to
    > use little memory as possible.


    Synchronizing many files should be no problem with rsync versions >= 3 due to
    new incremental feature; I do this regularly with around 15 million mostly small
    files. But: The initial copy might take some days. If so, you might look around
    for a somewhat faster unconditional copy method. I prefer cpio here, but cloning
    tools might be better suited.

    Sven

    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

  3. Re: rsync many small files

    On Tue 23 Sep 2008, Mag Gam wrote:

    > At our lab we have storage with many small files. For example a
    > directory can contain over 15,000 files and each file averages about
    > 75k. I would like to sync this to another filesystem on a different
    > server but I am not sure if there is a rsync tuning flag I can use for
    > such a intensive job. I am using rsync 3.0.3. Also, I would like to
    > use little memory as possible.


    You may also want to look at csync2 http://oss.linbit.com/csync2/
    for keeping the copies synchronised, it's perhaps better suited than
    rsync where you have such a large collection of files where there are
    not that many updates going on (i.e. the searching for updates takes
    significantly longer than transferring those updates).


    Paul Slootman
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  4. Re: rsync many small files

    Tue, 23 Sep 2008 13:25:10 +0200, paul+rsync wrote:
    > You may also want to look at csync2 http://oss.linbit.com/csync2/
    > for keeping the copies synchronised, it's perhaps better suited than
    > rsync where you have such a large collection of files where there are
    > not that many updates going on (i.e. the searching for updates takes
    > significantly longer than transferring those updates).


    Thanks for the link!

    Reading the documentation I got the impression that for larger
    setups (with millions of files) one should use the inotify API
    for filling the hint database (see page 6 of the csync2 paper).
    Using this inotify idea would also allow more efficient syncing
    with an rsync script.

    Has anybody explored and documented this idea, with csync2 or rsync?

    Sven
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


+ Reply to Thread