out of memory in flist_expand - Tools

This is a discussion on out of memory in flist_expand - Tools ; Hello list, I get the following error using rsync 3.0.3 (both sides): ... data/xxxxxxxx/admin_docs/2/8/9/9/0/7/9/msg-75682-748.msg is uptodate data/xxxxxxxx/admin_docs/2/8/9/9/0/8/0/msg-75682-749.txt is uptodate ERROR: out of memory in flist_expand [generator] rsync error: error allocating core memory buffers (code 22) at util.c(117) [generator=3.0.3] rsync error: ...

+ Reply to Thread
Results 1 to 5 of 5

Thread: out of memory in flist_expand

  1. out of memory in flist_expand

    Hello list,

    I get the following error using rsync 3.0.3 (both sides):

    ...
    data/xxxxxxxx/admin_docs/2/8/9/9/0/7/9/msg-75682-748.msg is uptodate
    data/xxxxxxxx/admin_docs/2/8/9/9/0/8/0/msg-75682-749.txt is uptodate
    ERROR: out of memory in flist_expand [generator]
    rsync error: error allocating core memory buffers (code 22) at util.c(117)
    [generator=3.0.3]
    rsync error: received SIGUSR1 (code 19) at main.c(1286) [receiver=3.0.3]

    Command was: rsync -avvz remotehost:/var/www /backup

    These things I have tried to fix the problem:

    After starting rsync, it says 'receiving incremental file list', which I read
    indicates it won't consume too much memory.

    I have increased memory by adding an extra swap space of 4GB (I know,
    overdone..). There is 1GB of RAM in the system. There is plenty of diskspace as
    well. I have also tried modifying the maximum stack size, which is now 512 MB
    for a program.

    In the Rsync FAQ I read Rsync uses approximately 100 bytes per file for
    indexing. The number of files to copy from the remote server is 3902364, so
    rsync would need about 400MB... right?

    Both sides run FreeBSD 6.x, and rsync 3.0.3.

    What am I missing?

    --
    Rob klein Gunnewiek
    BWSS B.V. Deventer (http://www.bwss.nl)
    Tel +(31) 0570-665140
    Fax +(31) 0570-665141
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  2. Re: out of memory in flist_expand

    Could someone please help me with this problem?

    Rob klein Gunnewiek wrote:
    > Hello list,
    >
    > I get the following error using rsync 3.0.3 (both sides):
    >
    > ...
    > data/xxxxxxxx/admin_docs/2/8/9/9/0/7/9/msg-75682-748.msg is uptodate
    > data/xxxxxxxx/admin_docs/2/8/9/9/0/8/0/msg-75682-749.txt is uptodate
    > ERROR: out of memory in flist_expand [generator]
    > rsync error: error allocating core memory buffers (code 22) at util.c(117)
    > [generator=3.0.3]
    > rsync error: received SIGUSR1 (code 19) at main.c(1286) [receiver=3.0.3]
    >
    > Command was: rsync -avvz remotehost:/var/www /backup
    >
    > These things I have tried to fix the problem:
    >
    > After starting rsync, it says 'receiving incremental file list', which I read
    > indicates it won't consume too much memory.
    >
    > I have increased memory by adding an extra swap space of 4GB (I know,
    > overdone..). There is 1GB of RAM in the system. There is plenty of diskspace as
    > well. I have also tried modifying the maximum stack size, which is now 512 MB
    > for a program.
    >
    > In the Rsync FAQ I read Rsync uses approximately 100 bytes per file for
    > indexing. The number of files to copy from the remote server is 3902364, so
    > rsync would need about 400MB... right?
    >
    > Both sides run FreeBSD 6.x, and rsync 3.0.3.
    >
    > What am I missing?
    >


    --
    Rob klein Gunnewiek
    BWSS B.V. Deventer (http://www.bwss.nl)
    Tel +(31) 0570-665140
    Fax +(31) 0570-665141
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  3. Re: out of memory in flist_expand

    On Fri, Aug 15, 2008 at 05:28:43PM +0200, Rob klein Gunnewiek wrote:
    > ERROR: out of memory in flist_expand [generator]


    With -vv, you should be seeing messages like this:

    [generator] expand file_list pointer array to N bytes, did move

    That will tell you what the last realloc attempt was before the error.
    If it is a really huge value, check to see if you have a directory with
    a huge number of files in it. You should also check your process's
    memory ulimit values to see if they are too small, and try using a
    different compiler or malloc package to see if your malloc is having
    trouble with memory fragmentation.

    ...wayne..
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  4. Re: out of memory in flist_expand

    Thanks for your answer...

    Wayne Davison schreef:
    > On Fri, Aug 15, 2008 at 05:28:43PM +0200, Rob klein Gunnewiek wrote:
    >> ERROR: out of memory in flist_expand [generator]

    >
    > With -vv, you should be seeing messages like this:
    >
    > [generator] expand file_list pointer array to N bytes, did move
    >
    > That will tell you what the last realloc attempt was before the error.


    I have used -avvz, I had not noticed the '.. expand file_list pointer array to N
    bytes...' type of lines. I will try this again

    > If it is a really huge value, check to see if you have a directory with
    > a huge number of files in it. You should also check your process's
    > memory ulimit values to see if they are too small, and try using a
    > different compiler or malloc package to see if your malloc is having
    > trouble with memory fragmentation.


    I know I have a directory with a huge number of files in it. But this shouldn't
    be a big problem. Rsync from another (Linux) system works well. The linux system
    has a process stack limit of 8MB. The FreeBSD system from which this fails I
    gave 512MB max stack size.

    Also, like I said I added 4GB of swap space in the FreeBSD system.

    Here is my ulimit output:

    core file size (blocks, -c) unlimited
    data seg size (kbytes, -d) 524288
    file size (blocks, -f) unlimited
    max locked memory (kbytes, -l) unlimited
    max memory size (kbytes, -m) unlimited
    open files (-n) 11095
    pipe size (512 bytes, -p) 1
    stack size (kbytes, -s) 524288
    cpu time (seconds, -t) unlimited
    max user processes (-u) 5547
    virtual memory (kbytes, -v) unlimited

    Like you suggested, I will try using another malloc library as well.

    --
    Rob klein Gunnewiek
    BWSS B.V. Deventer (http://www.bwss.nl)
    Tel +(31) 0570-665140
    Fax +(31) 0570-665141
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  5. Re: out of memory in flist_expand

    Wayne Davison wrote:
    > [...]
    >
    > That will tell you what the last realloc attempt was before the error.
    > If it is a really huge value, check to see if you have a directory with
    > a huge number of files in it. You should also check your process's
    > memory ulimit values to see if they are too small, and try using a
    > different compiler or malloc package to see if your malloc is having
    > trouble with memory fragmentation.


    I have compiled rsync 3.0.3 with the ptmalloc library
    (http://www.malloc.de/en/), now it works!

    Thanks again for your suggestion.

    --
    Rob klein Gunnewiek
    BWSS B.V. Deventer (http://www.bwss.nl)
    Tel +(31) 0570-665140
    Fax +(31) 0570-665141
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


+ Reply to Thread