DO NOT REPLY [Bug 5727] New: rsync crashes while copying largedirectory. - Tools

This is a discussion on DO NOT REPLY [Bug 5727] New: rsync crashes while copying largedirectory. - Tools ; https://bugzilla.samba.org/show_bug.cgi?id=5727 Summary: rsync crashes while copying large directory. Product: rsync Version: 2.6.9 Platform: Other OS/Version: Linux Status: NEW Severity: normal Priority: P3 Component: core AssignedTo: wayned@samba.org ReportedBy: r.e.wolff@bitwizard.nl QAContact: rsync-qa@samba.org I apologise for running an old version: 2.6.9. Last time ...

+ Reply to Thread
Results 1 to 6 of 6

Thread: DO NOT REPLY [Bug 5727] New: rsync crashes while copying largedirectory.

  1. DO NOT REPLY [Bug 5727] New: rsync crashes while copying largedirectory.

    https://bugzilla.samba.org/show_bug.cgi?id=5727

    Summary: rsync crashes while copying large directory.
    Product: rsync
    Version: 2.6.9
    Platform: Other
    OS/Version: Linux
    Status: NEW
    Severity: normal
    Priority: P3
    Component: core
    AssignedTo: wayned@samba.org
    ReportedBy: r.e.wolff@bitwizard.nl
    QAContact: rsync-qa@samba.org


    I apologise for running an old version: 2.6.9. Last time compiling rsync was a
    real hassle, and finally got too complicated.

    I have a very large directory that needs to move from one RAID to the new RAID.

    Rsync crashes while reading the source directory. the process grows to over 1G
    (the machine has 2G RAM + 2G swap), and then crashes. Probably on a memory
    allocation failure. Probably when the process hits about 2G, but I haven't seen
    it closer to 2G than 1.2G (I get bored watching it all the time, it takes a
    long time).


    --
    Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
    ------- You are receiving this mail because: -------
    You are the QA contact for the bug, or are watching the QA contact.
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  2. DO NOT REPLY [Bug 5727] rsync crashes while copying large directory.

    https://bugzilla.samba.org/show_bug.cgi?id=5727





    ------- Comment #1 from matt@mattmccutchen.net 2008-08-31 08:44 CST -------
    In this case, you really should try a 3.0.x version because memory usage is
    significantly reduced. If rsync fails again, please provide the error message.


    --
    Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
    ------- You are receiving this mail because: -------
    You are the QA contact for the bug, or are watching the QA contact.
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  3. DO NOT REPLY [Bug 5727] rsync crashes while copying large directory.

    https://bugzilla.samba.org/show_bug.cgi?id=5727





    ------- Comment #2 from r.e.wolff@bitwizard.nl 2008-08-31 09:03 CST -------
    Thanks for trying to help me solve my problem. Here we should focus on
    improving rsync.

    Just in: I have 28 million files.

    Just upgraded to 3.0.3... After scanning 2.8 million files, it's only using
    133Mb of memory so it will likely fit due to the smaller memory footprint.
    However this doesn't mean that the bug has been fixed. It means it's harder to
    trigger.


    --
    Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
    ------- You are receiving this mail because: -------
    You are the QA contact for the bug, or are watching the QA contact.
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  4. DO NOT REPLY [Bug 5727] rsync crashes while copying large directory.

    https://bugzilla.samba.org/show_bug.cgi?id=5727





    ------- Comment #3 from r.e.wolff@bitwizard.nl 2008-08-31 10:28 CST -------
    rsync . --exclude current -e rsh driepoot:/backup/abra2_usr_src/ -avHS
    --progress --min-size 1 --max-size 93
    building file list ...
    invalid message 101:7104843 [sender]
    rsync error: error in rsync protocol data stream (code 12) at io.c(390)
    [sender=3.0.3]
    rsync . --exclude current -e rsh driepoot:/backup/abra2_usr_src/ -avHS
    --progress --min-size 93 --max-size 158
    building file list ...
    invalid message 101:7104843 [sender]
    rsync error: error in rsync protocol data stream (code 12) at io.c(390)
    [sender=3.0.3]
    rsync . --exclude current -e rsh driepoot:/backup/abra2_usr_src/ -avHS
    --progress --min-size 158 --max-size 273
    building file list ...


    --
    Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
    ------- You are receiving this mail because: -------
    You are the QA contact for the bug, or are watching the QA contact.
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  5. DO NOT REPLY [Bug 5727] rsync crashes while copying large directory.

    https://bugzilla.samba.org/show_bug.cgi?id=5727


    wayned@samba.org changed:

    What |Removed |Added
    ----------------------------------------------------------------------------
    Status|NEW |RESOLVED
    Resolution| |WORKSFORME




    ------- Comment #4 from wayned@samba.org 2008-08-31 13:16 CST -------
    This error:

    "invalid message 101:7104843 [sender]"

    Indicates that this byte sequence that was received:

    0x4b 0x69 0x6c 0x6c

    Which is the ASCII string "Kill". So, it looks like your shell likes to send
    extra text on stdout if something is killed ("Killed 1234" or some such).

    If you haven't done so already, you should investigate what is killing the
    remote process (e.g. an out-of-memory error).

    "However this doesn't mean that the bug has been fixed. It means it's harder to
    trigger."

    That's all that we can hope for with respect to to memory issues. With the way
    rsync works, it is pretty near the limit of what memory reduction is possible,
    and incremental recursion is already a huge gain for transferring larger sets
    of files than ever before. Changing rsync to not cache all the files in a
    directory would be radical surgery, and is not going to happen in this
    codebase.

    One thing that can make a difference with regard to memory issues is the alloc
    library that is used. There are some malloc libraries that will gradually lose
    memory if they don't do a good job of re-using freed memory. There was a
    recent report by a user that switching their malloc library made rsync's
    incremental recursion work for them. (If you're using glibc, this is probably
    not your problem, though.)


    --
    Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
    ------- You are receiving this mail because: -------
    You are the QA contact for the bug, or are watching the QA contact.
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  6. DO NOT REPLY [Bug 5727] rsync crashes while copying large directory.

    https://bugzilla.samba.org/show_bug.cgi?id=5727





    ------- Comment #5 from r.e.wolff@bitwizard.nl 2008-09-01 02:29 CST -------
    The remote process gets killed by the OOM killer. I've installed the newer
    version inside a chroot environment. I'm not sure yet how to do that with the
    "remote" side.


    --
    Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
    ------- You are receiving this mail because: -------
    You are the QA contact for the bug, or are watching the QA contact.
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


+ Reply to Thread