DO NOT REPLY [Bug 5567] New: REGRESSION: 3.0.3pre3 dies with - Tools

This is a discussion on DO NOT REPLY [Bug 5567] New: REGRESSION: 3.0.3pre3 dies with - Tools ; https://bugzilla.samba.org/show_bug.cgi?id=5567 Summary: REGRESSION: 3.0.3pre3 dies with "failed to open XXX, continuing: Too many open files" Product: rsync Version: 3.0.3 Platform: Other OS/Version: Linux Status: NEW Severity: normal Priority: P3 Component: core AssignedTo: wayned@samba.org ReportedBy: timo.lindfors@iki.fi QAContact: rsync-qa@samba.org I started getting ...

+ Reply to Thread
Results 1 to 6 of 6

Thread: DO NOT REPLY [Bug 5567] New: REGRESSION: 3.0.3pre3 dies with

  1. DO NOT REPLY [Bug 5567] New: REGRESSION: 3.0.3pre3 dies with

    https://bugzilla.samba.org/show_bug.cgi?id=5567

    Summary: REGRESSION: 3.0.3pre3 dies with "failed to open XXX,
    continuing: Too many open files"
    Product: rsync
    Version: 3.0.3
    Platform: Other
    OS/Version: Linux
    Status: NEW
    Severity: normal
    Priority: P3
    Component: core
    AssignedTo: wayned@samba.org
    ReportedBy: timo.lindfors@iki.fi
    QAContact: rsync-qa@samba.org


    I started getting a lot of errors like

    rsync: failed to open
    "/tftpboot/moniscreen/usr/src/linux-headers-2.6.24-19-generic/include/config/fs/posix/acl.h",
    continuing: Too many open files (24)
    rsync: failed to open
    "/tftpboot/moniscreen/usr/src/linux-headers-2.6.24-19-generic/include/config/fujitsu/laptop.h",
    continuing: Too many open files (24)
    rsync: failed to open
    "/tftpboot/moniscreen/usr/src/linux-headers-2.6.24-19-generic/include/config/fuse/fs.h",
    continuing: Too many open files (24)
    rsync: failed to open
    "/tftpboot/moniscreen/usr/src/linux-headers-2.6.24-19-generic/include/config/fusion/ctl.h",
    continuing: Too many open files (24)
    rsync: failed to open
    "/tftpboot/moniscreen/usr/src/linux-headers-2.6.24-19-generic/include/config/fusion/fc.h",
    continuing: Too many open files (24)
    rsync: failed to open
    "/tftpboot/moniscreen/usr/src/linux-headers-2.6.24-19-generic/include/config/fusion/lan.h",
    continuing: Too many open files (24)
    rsync: failed to open
    "/tftpboot/moniscreen/usr/src/linux-headers-2.6.24-19-generic/include/config/fusion/logging.h",
    continuing: Too many open files (24)
    rsync: failed to open
    "/tftpboot/moniscreen/usr/src/linux-headers-2.6.24-19-generic/include/config/fusion/sas.h",
    continuing: Too many open files (24)

    after I upgraded from rsync 3.0.3pre2 to 3.0.0pre3. I invoke rsync on a debian
    stable system on x86 with the following arguments/environment:

    execve("/usr/bin/rsync", ["rsync", "--numeric-ids", "--delete",
    "--delete-after", "-ax", "root@moniscreen:/", "/tftpboot/moniscreen"],
    ["SHELL=/bin/bash", "TERM
    =screen", "USER=root", "http_proxy=http://proxy.kurp.hut.fi:3128/",
    "SUDO_USER=lindi", "SUDO_UID=548", "ftp_proxy=http://proxy.kurp.hut.fi:3128/",
    "PATH=/usr/local/sb
    in:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/X11R6/bin"...,
    "PWD=/root", "EDITOR=/home/lindi/bin/editor", "SHLVL=1",
    "SUDO_COMMAND=/bin/bash backup2 --stdin",
    "HOME=/home/lindi", "LESSCHARSET=latin1", "LOGNAME=root", "LC_CTYPE=fi_FI",
    "SUDO_GID=500", "_=/usr/bin/strace"])

    and with the following ulimits:

    core file size (blocks, -c) unlimited
    data seg size (kbytes, -d) 1004140
    max nice (-e) 0
    file size (blocks, -f) unlimited
    pending signals (-i) unlimited
    max locked memory (kbytes, -l) 1004140
    max memory size (kbytes, -m) 1004140
    open files (-n) 1024
    pipe size (512 bytes, -p) 8
    POSIX message queues (bytes, -q) unlimited
    max rt priority (-r) 0
    stack size (kbytes, -s) 4096
    cpu time (seconds, -t) unlimited
    max user processes (-u) unlimited
    virtual memory (kbytes, -v) 1004140
    file locks (-x) unlimited

    I could raise ulimit of open files but shouldn't rsync cope just fine with the
    default limit?

    The diff between pre2 and pre3 is quite small but I was unable to figure out
    anything obvious. What extra info should I provide? The whole strace output is
    96 megabytes and contains sensitive data. I was unable to reproduce the issue
    with a simple

    mkdir a
    for i in `seq 1 4096`; do echo > a/$i; done
    rsync --numeric-ids --delete --delete-after -ax a/ b


    --
    Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
    ------- You are receiving this mail because: -------
    You are the QA contact for the bug, or are watching the QA contact.
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  2. DO NOT REPLY [Bug 5567] REGRESSION: 3.0.3pre3 dies with "failed to

    https://bugzilla.samba.org/show_bug.cgi?id=5567





    ------- Comment #1 from timo.lindfors@iki.fi 2008-06-26 06:22 CST -------
    Steps to reproduce:
    1) cd /tmp
    2) wget http://iki.fi/lindi/rsync/bug5567.tar.gz
    3) sudo tar xzf bug5567.tar.gz
    4) sudo rsync --numeric-ids --delete --delete-after -ax
    localhost:/tmp/bug5567/source/ /tmp/bug5567/target

    Expected results:
    4) rsync does not give error messages

    Actual results:
    4) rsync complains loudly with

    ....
    rsync: failed to open "/tmp/bug5567/target/scripts/mod/elfconfig.h",
    continuing: Too many open files (24)
    rsync: failed to open "/tmp/bug5567/target/scripts/mod/empty.o", continuing:
    Too many open files (24)
    rsync: failed to open "/tmp/bug5567/target/scripts/mod/file2alias.o",
    continuing: Too many open files (24)
    rsync: failed to open "/tmp/bug5567/target/scripts/mod/mk_elfconfig",
    continuing: Too many open files (24)
    rsync: failed to open "/tmp/bug5567/target/scripts/mod/modpost", continuing:
    Too many open files (24)
    rsync: failed to open "/tmp/bug5567/target/scripts/mod/modpost.o", continuing:
    Too many open files (24)
    rsync: failed to open "/tmp/bug5567/target/scripts/mod/sumversion.o",
    continuing: Too many open files (24)
    rsync: opendir "/tmp/bug5567/target/." failed: Too many open files (24)
    IO error encountered -- skipping file deletion
    rsync error: some files could not be transferred (code 23) at main.c(1506)
    [generator=3.0.3pre3]

    Please let me know if you are unable to reproduce this I am happy to provide
    more info.


    --
    Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
    ------- You are receiving this mail because: -------
    You are the QA contact for the bug, or are watching the QA contact.
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  3. DO NOT REPLY [Bug 5567] REGRESSION: 3.0.3pre3 dies with "failed to

    https://bugzilla.samba.org/show_bug.cgi?id=5567


    wayned@samba.org changed:

    What |Removed |Added
    ----------------------------------------------------------------------------
    Status|NEW |ASSIGNED




    ------- Comment #2 from wayned@samba.org 2008-06-26 09:55 CST -------
    I can run rsync on the tar you specified with "ulimit -n 16" without problem.
    Your tar file didn't contain any xattrs, though -- are there xattrs involved?
    Are any patches applied to the default source? Is selinux involved? Can you
    run "lsof -c rsync" while rsync is running to see if it has a lot of open
    files?


    --
    Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
    ------- You are receiving this mail because: -------
    You are the QA contact for the bug, or are watching the QA contact.
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  4. DO NOT REPLY [Bug 5567] REGRESSION: 3.0.3pre3 dies with "failed to

    https://bugzilla.samba.org/show_bug.cgi?id=5567





    ------- Comment #3 from timo.lindfors@iki.fi 2008-06-26 10:14 CST -------
    xattrs are probably not involved since I too can reproduce the problem by just
    untarring bug5567.tar.gz (which does not have them) to /tmp (which should be
    just regular ext3 without any special setup). No special patches have been
    applied, this was compiled from

    8b1eb7e42b36744cf580a2fba6224087 rsync-3.0.3pre3.tar.gz

    I also don't think there is any selinux setup on the debian stable system (at
    least I have never configured any). Finally, I started

    sudo sh -c 'while true; do lsof |grep rsync|wc; done'

    before running rsync and it printed

    17 153 1533
    1065 9591 126779
    1065 9587 126745
    1065 9587 126745
    1065 9587 126745
    1065 9587 126745
    1065 9587 126745
    1065 9587 126745
    1065 9587 126745
    1065 9587 126745
    1065 9587 126745
    1065 9587 126745
    1065 9587 126745
    178 1626 20262
    0 0 0

    so rsync quite rapidly opened all the files it could and then started to give
    out errors.

    If you still can't reproduce I could prepare a clean qemu image where this can
    be reproduced, would that help?


    --
    Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
    ------- You are receiving this mail because: -------
    You are the QA contact for the bug, or are watching the QA contact.
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  5. DO NOT REPLY [Bug 5567] REGRESSION: 3.0.3pre3 dies with "failed to

    https://bugzilla.samba.org/show_bug.cgi?id=5567





    ------- Comment #4 from wayned@samba.org 2008-06-26 10:14 CST -------
    Created an attachment (id=3372)
    --> (https://bugzilla.samba.org/attachmen...72&action=view)
    Close fd in the generator for empty files

    I looked at the code after my reply, and found the problem. If I add
    --no-whole-file to the command options, I too get the errors, and the attached
    patch fixes this.


    --
    Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
    ------- You are receiving this mail because: -------
    You are the QA contact for the bug, or are watching the QA contact.
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  6. DO NOT REPLY [Bug 5567] REGRESSION: 3.0.3pre3 dies with "failed to

    https://bugzilla.samba.org/show_bug.cgi?id=5567


    wayned@samba.org changed:

    What |Removed |Added
    ----------------------------------------------------------------------------
    Status|ASSIGNED |RESOLVED
    Resolution| |FIXED




    ------- Comment #5 from wayned@samba.org 2008-06-26 10:15 CST -------
    This is now fixed in the git repository.

    Thanks for your report!


    --
    Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
    ------- You are receiving this mail because: -------
    You are the QA contact for the bug, or are watching the QA contact.
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


+ Reply to Thread