--files-from= and --delete corrent options - Tools

This is a discussion on --files-from= and --delete corrent options - Tools ; Hi, I am running the following command to backup a specific file from /tmp/BACKUPFILE which contains the name "Friday2008-09-19.bkf". rsync -ah --delete --numeric-ids --stats --delete-excluded --files-from=/tmp/BACKUPFILE /var/data/shares/vmbackups/ /mnt/usb_backup/SFSYDVS01/daily.0/VM/ The name in /tmp/BACKUPFILE changes each day for example when next run ...

+ Reply to Thread
Results 1 to 6 of 6

Thread: --files-from= and --delete corrent options

  1. --files-from= and --delete corrent options

    Hi,

    I am running the following command to backup a specific file from
    /tmp/BACKUPFILE which contains the name "Friday2008-09-19.bkf".

    rsync -ah --delete --numeric-ids --stats --delete-excluded
    --files-from=/tmp/BACKUPFILE /var/data/shares/vmbackups/
    /mnt/usb_backup/SFSYDVS01/daily.0/VM/

    The name in /tmp/BACKUPFILE changes each day for example when next run it
    will be "Monday2008-09-19.bkf" I use a script before running rsync to
    obtain the most recently modified file from /var/data/shares/vmbackups/ and
    cat it into /tmp/BACKUPFILE.

    I want the /mnt/usb_backup/SFSYDVS01/daily.0/VM/ directory to only ever
    contain the most recent backup file e.g.

    root@sfnas01:~# ls -t1 /var/data/shares/vmbackups | head -n 1 >
    /tmp/BACKUPFILE
    root@sfnas01:~# rsync [CORRECT OPTIONS] --files-from=/tmp/BACKUPFILE
    /var/data/shares/vmbackups/ /mnt/usb_backup/SFSYDVS01/daily.0/VM/
    root@sfnas01:~# ls /mnt/usb_backup/SFSYDVS01/daily.0/VM/
    Friday2008-09-19.bkf

    Skip to next day.....

    root@sfnas01:~# ls -t1 /var/data/shares/vmbackups | head -n 1 >
    /tmp/BACKUPFILE
    root@sfnas01:~# rsync [CORRECT OPTIONS] --files-from=/tmp/BACKUPFILE
    /var/data/shares/vmbackups/ /mnt/usb_backup/SFSYDVS01/daily.0/VM/
    root@sfnas01:~# ls /mnt/usb_backup/SFSYDVS01/daily.0/VM/
    Monday2008-09-19.bkf

    But currently the command grabs everything in /var/data/shares/vmbackups/
    and so I end up with:

    root@sfnas01:~# ls /mnt/usb_backup/SFSYDVS01/daily.0/VM/
    Monday2008-09-19.bkf Friday2008-09-19.bkf

    I expected the --delete option would have removed Friday2008-09-19.bkf in
    this example (with the --delete option) but it doesn't. I've read the man
    page and googled a bit but I think i'm getting a bit lost.

    Sorry if this seems long winded but I hope you understand what I'm trying
    to do.

    Thanks

    James





    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  2. Re: --files-from= and --delete corrent options

    On Mon, 2008-09-22 at 15:43 +1000, j@mesrobertson.com wrote:
    > I am running the following command to backup a specific file from
    > /tmp/BACKUPFILE which contains the name "Friday2008-09-19.bkf".
    >
    > rsync -ah --delete --numeric-ids --stats --delete-excluded
    > --files-from=/tmp/BACKUPFILE /var/data/shares/vmbackups/
    > /mnt/usb_backup/SFSYDVS01/daily.0/VM/
    >
    > The name in /tmp/BACKUPFILE changes each day for example when next run it
    > will be "Monday2008-09-19.bkf" I use a script before running rsync to
    > obtain the most recently modified file from /var/data/shares/vmbackups/ and
    > cat it into /tmp/BACKUPFILE.
    >
    > I want the /mnt/usb_backup/SFSYDVS01/daily.0/VM/ directory to only ever
    > contain the most recent backup file e.g.
    >
    > root@sfnas01:~# ls -t1 /var/data/shares/vmbackups | head -n 1 >
    > /tmp/BACKUPFILE
    > root@sfnas01:~# rsync [CORRECT OPTIONS] --files-from=/tmp/BACKUPFILE
    > /var/data/shares/vmbackups/ /mnt/usb_backup/SFSYDVS01/daily.0/VM/
    > root@sfnas01:~# ls /mnt/usb_backup/SFSYDVS01/daily.0/VM/
    > Friday2008-09-19.bkf
    >
    > Skip to next day.....
    >
    > root@sfnas01:~# ls -t1 /var/data/shares/vmbackups | head -n 1 >
    > /tmp/BACKUPFILE
    > root@sfnas01:~# rsync [CORRECT OPTIONS] --files-from=/tmp/BACKUPFILE
    > /var/data/shares/vmbackups/ /mnt/usb_backup/SFSYDVS01/daily.0/VM/
    > root@sfnas01:~# ls /mnt/usb_backup/SFSYDVS01/daily.0/VM/
    > Monday2008-09-19.bkf
    >
    > But currently the command grabs everything in /var/data/shares/vmbackups/
    > and so I end up with:
    >
    > root@sfnas01:~# ls /mnt/usb_backup/SFSYDVS01/daily.0/VM/
    > Monday2008-09-19.bkf Friday2008-09-19.bkf
    >
    > I expected the --delete option would have removed Friday2008-09-19.bkf in
    > this example (with the --delete option) but it doesn't. I've read the man
    > page and googled a bit but I think i'm getting a bit lost.


    The current meaning of --files-from is to restrict rsync to processing
    the paths in the list, and rsync deletes extraneous destination files as
    part of the processing of the parent directory (assuming -d or -r is
    on), so you can't delete unlisted files this way. See:

    https://bugzilla.samba.org/show_bug.cgi?id=3465

    Try this instead:

    rsync [OPTIONS] --include-from=/tmp/BACKUPFILE --exclude='*'
    /var/data/shares/vmbackups/ /mnt/usb_backup/SFSYDVS01/daily.0/VM/

    This way, rsync will scan the destination directory for deletions (since
    you have -a without --files-from, which implies -r) and will delete the
    old files because they are excluded with --delete-excluded.

    Matt

    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  3. Re: --files-from= and --delete corrent options


    > Try this instead:
    >
    > rsync [OPTIONS] --include-from=/tmp/BACKUPFILE --exclude='*'
    > /var/data/shares/vmbackups/ /mnt/usb_backup/SFSYDVS01/daily.0/VM/
    >
    > This way, rsync will scan the destination directory for deletions (since
    > you have -a without --files-from, which implies -r) and will delete the
    > old files because they are excluded with --delete-excluded.
    >
    > Matt
    >
    >

    Thanks Matt,

    I tried this but it doesn't seem to work for me as you describe. I just
    created a test setup. The commands I'm running are as follows:

    root@sfnas01:~# rsync --version
    rsync version 2.6.9 protocol version 29
    root@sfnas01:~# ls /tmp/source/
    Thursday2008-09-11.bkf Tuesday2008-09-09.bkf Wednesday2008-09-10.bkf
    root@sfnas01:~# ls -t1 /tmp/source | head -n 1 > /tmp/BACKUPFILE
    root@sfnas01:~# cat /tmp/BACKUPFILE
    Thursday2008-09-11.bkf
    root@sfnas01:~# rsync -ah --delete --numeric-ids --stats
    --delete-excluded -include-from=/tmp/BACKUPFILE --exclude='*'
    /tmp/source/ /tmp/destination/
    ..d..t...... ./

    Number of files: 1
    Number of files transferred: 0
    Total file size: 0 bytes
    Total transferred file size: 0 bytes
    Literal data: 0 bytes
    Matched data: 0 bytes
    File list size: 20
    File list generation time: 0.001 seconds
    File list transfer time: 0.000 seconds
    Total bytes sent: 42
    Total bytes received: 26

    sent 42 bytes received 26 bytes 136.00 bytes/sec
    total size is 0 speedup is 0.00
    root@sfnas01:~# ls /tmp/destination/
    root@sfnas01:~#

    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  4. Re: --files-from= and --delete corrent options

    On Wed, 2008-09-24 at 13:00 +1000, James Robertson wrote:
    > I tried this but it doesn't seem to work for me as you describe. I just
    > created a test setup. The commands I'm running are as follows:
    >
    > root@sfnas01:~# rsync --version
    > rsync version 2.6.9 protocol version 29
    > root@sfnas01:~# ls /tmp/source/
    > Thursday2008-09-11.bkf Tuesday2008-09-09.bkf Wednesday2008-09-10.bkf
    > root@sfnas01:~# ls -t1 /tmp/source | head -n 1 > /tmp/BACKUPFILE
    > root@sfnas01:~# cat /tmp/BACKUPFILE
    > Thursday2008-09-11.bkf
    > root@sfnas01:~# rsync -ah --delete --numeric-ids --stats
    > --delete-excluded -include-from=/tmp/BACKUPFILE --exclude='*'
    > /tmp/source/ /tmp/destination/
    > .d..t...... ./
    >
    > Number of files: 1
    > Number of files transferred: 0
    > Total file size: 0 bytes
    > Total transferred file size: 0 bytes
    > Literal data: 0 bytes
    > Matched data: 0 bytes
    > File list size: 20
    > File list generation time: 0.001 seconds
    > File list transfer time: 0.000 seconds
    > Total bytes sent: 42
    > Total bytes received: 26
    >
    > sent 42 bytes received 26 bytes 136.00 bytes/sec
    > total size is 0 speedup is 0.00
    > root@sfnas01:~# ls /tmp/destination/
    > root@sfnas01:~#


    The -include-from=/tmp/BACKUPFILE is short a dash, causing it to be
    parsed as -i -n -c -l -u -d --rsh=-from=/tmp/BACKUPFILE . Change it to
    --include-from=/tmp/BACKUPFILE .

    Matt

    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  5. Re: --files-from= and --delete corrent options

    Matt McCutchen wrote:
    > On Wed, 2008-09-24 at 13:00 +1000, James Robertson wrote:
    >
    >> I tried this but it doesn't seem to work for me as you describe. I just
    >> created a test setup. The commands I'm running are as follows:
    >>
    >> root@sfnas01:~# rsync --version
    >> rsync version 2.6.9 protocol version 29
    >> root@sfnas01:~# ls /tmp/source/
    >> Thursday2008-09-11.bkf Tuesday2008-09-09.bkf Wednesday2008-09-10.bkf
    >> root@sfnas01:~# ls -t1 /tmp/source | head -n 1 > /tmp/BACKUPFILE
    >> root@sfnas01:~# cat /tmp/BACKUPFILE
    >> Thursday2008-09-11.bkf
    >> root@sfnas01:~# rsync -ah --delete --numeric-ids --stats
    >> --delete-excluded -include-from=/tmp/BACKUPFILE --exclude='*'
    >> /tmp/source/ /tmp/destination/
    >> .d..t...... ./
    >>
    >> Number of files: 1
    >> Number of files transferred: 0
    >> Total file size: 0 bytes
    >> Total transferred file size: 0 bytes
    >> Literal data: 0 bytes
    >> Matched data: 0 bytes
    >> File list size: 20
    >> File list generation time: 0.001 seconds
    >> File list transfer time: 0.000 seconds
    >> Total bytes sent: 42
    >> Total bytes received: 26
    >>
    >> sent 42 bytes received 26 bytes 136.00 bytes/sec
    >> total size is 0 speedup is 0.00
    >> root@sfnas01:~# ls /tmp/destination/
    >> root@sfnas01:~#
    >>

    >
    > The -include-from=/tmp/BACKUPFILE is short a dash, causing it to be
    > parsed as -i -n -c -l -u -d --rsh=-from=/tmp/BACKUPFILE . Change it to
    > --include-from=/tmp/BACKUPFILE .
    >
    > Matt
    >
    >

    If /tmp/BACKUPFILE contained a directory e.g.

    root@sfnas01:~# cat /tmp/BACKUPFILE
    3/

    How would I get it to copy everything under and including
    /var/data/shares/vmbackups/3/? and delete everything else on the
    destination? The guys running the backup to /var/data/shares/vmbackups/
    are considering copying the files direct rather than sticking it into a
    *.bkf and have it create folders like 1/, 2/, 3/ with files and folders
    under that.

    I came up with running this:

    root@sfnas01:~# ls /tmp/source/
    1/ 2/ 3/
    root@sfnas01:~# ls -t1 /tmp/source/ | head -n 1 > /tmp/BACKUPFILE
    root@sfnas01:~# cat /tmp/BACKUPFILE
    3/
    root@sfnas01:~# ls -t1 /tmp/source/ | grep -f /tmp/BACKUPFILE -v >
    /tmp/BACKUPEXCLUDE
    root@sfnas01:~# cat /tmp/BACKUPEXCLUDE
    2/
    1/
    root@sfnas01:~# rsync -ah --numeric-ids --stats --delete-excluded
    --include-from=/tmp/BACKUPFILE --exclude-from=/tmp/BACKUPEXCLUDE
    /tmp/source/ /tmp/destination/
    root@sfnas01:~# ls /tmp/destination/
    3/

    Which works and copies the subdirectories and files but is there a more
    elegant solution without the need for the /tmp/BACKUPEXCLUDE file?

    I think i'll rename /tmp/BACKUPFILE to /tmp/BACKUPDIR

    Apologies as they only just decided this today after I came back with
    the solution you helped me with.





    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  6. Re: --files-from= and --delete corrent options

    On Wed, 2008-09-24 at 15:41 +1000, James Robertson wrote:
    > >> root@sfnas01:~# rsync -ah --delete --numeric-ids --stats
    > >> --delete-excluded -include-from=/tmp/BACKUPFILE --exclude='*'
    > >> /tmp/source/ /tmp/destination/


    > If /tmp/BACKUPFILE contained a directory e.g.
    >
    > root@sfnas01:~# cat /tmp/BACKUPFILE
    > 3/
    >
    > How would I get it to copy everything under and including
    > /var/data/shares/vmbackups/3/? and delete everything else on the
    > destination?


    The exclude just needs to be anchored so that it matches only the
    undesired top-level directories, not files inside the desired top-level
    directory /var/data/shares/vmbackups/3:

    rsync -ah --delete --numeric-ids --stats --delete-excluded
    --include-from=/tmp/BACKUPFILE --exclude='/*'
    /var/data/shares/vmbackups/ DEST/

    The --include-from pattern will match any descendants
    of /var/data/shares/vmbackups/3 that also happen to be directories named
    3, but that should not hurt anything. If you feel like being
    meticulous, you could anchor that one too:

    ls -t1 /var/data/shares/vmbackups/ | head -n 1 | sed 's,^,/,' >/tmp/BACKUPFILE

    Matt

    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


+ Reply to Thread