How to exclude directories from source with --relative - Tools

This is a discussion on How to exclude directories from source with --relative - Tools ; I have been unable to figure out how to exclude directories that only exist on the source side when running rsync version 3.0.2 with --relative. Here is an artificial test that shows my problem in case 3). Any help would ...

+ Reply to Thread
Results 1 to 14 of 14

Thread: How to exclude directories from source with --relative

  1. How to exclude directories from source with --relative

    I have been unable to figure out how to exclude directories that only
    exist on the source side when running rsync version 3.0.2 with --relative.

    Here is an artificial test that shows my problem in case 3).
    Any help would be appreciated.


    # mkdir -p /tmp/source/a/b /tmp/source/a/c
    # touch /tmp/source/a/b/hello /tmp/source/a/c/world


    1) Try mirroring both b/ and c/, this works:

    # rsync -avn --relative /tmp/source/./a/? /tmp/destination
    sending incremental file list
    created directory /tmp/destination
    a/
    a/b/
    a/b/hello
    a/c/
    a/c/world

    sent 119 bytes received 30 bytes 298.00 bytes/sec
    total size is 0 speedup is 0.00 (DRY RUN)


    2) Try excluding b/, this works:

    # rsync -avn --relative --exclude b/ /tmp/source/./a/? /tmp/destination
    sending incremental file list
    created directory /tmp/destination
    a/
    a/c/
    a/c/world

    sent 76 bytes received 23 bytes 198.00 bytes/sec
    total size is 0 speedup is 0.00 (DRY RUN)


    3) Try excluding a/, this does not work:

    # rsync -avn --relative --exclude a/ /tmp/source/./a/? /tmp/destination
    sending incremental file list
    created directory /tmp/destination
    a/
    a/b/
    a/b/hello
    a/c/
    a/c/world

    sent 119 bytes received 30 bytes 298.00 bytes/sec
    total size is 0 speedup is 0.00 (DRY RUN)


    --
    Stuart Anderson anderson@ligo.caltech.edu
    http://www.ligo.caltech.edu/~anderson
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  2. Re: How to exclude directories from source with --relative

    On Mon, May 26, 2008 at 09:46:04PM -0700, Stuart Anderson wrote:
    > # rsync -avn --relative --exclude b/ /tmp/source/./a/? /tmp/destination


    That excludes all "b" dirs. It's safer to exclude "/a/b/" (anchored and
    fully specified). The simple rule is to exclude the names that rsync
    outputs via -v.

    > 3) Try excluding a/, this does not work:
    >
    > # rsync -avn --relative --exclude a/ /tmp/source/./a/? /tmp/destination


    That's because 'a' is an implied directory in the longer path that you
    told rsync to transfer. Implied directories can't be excluded, but
    their attributes can be omitted via --no-implied-dirs. If you don't
    want it in the transfer, don't specify it (and if you're using wild-
    cards, specify them in such as way as to avoid matching the dir you
    don't want).

    ...wayne..
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  3. Re: How to exclude directories from source with --relative

    On Mon, May 26, 2008 at 11:40:31PM -0700, Wayne Davison wrote:
    > On Mon, May 26, 2008 at 09:46:04PM -0700, Stuart Anderson wrote:
    > > # rsync -avn --relative --exclude b/ /tmp/source/./a/? /tmp/destination

    >
    > That excludes all "b" dirs. It's safer to exclude "/a/b/" (anchored and
    > fully specified). The simple rule is to exclude the names that rsync
    > outputs via -v.
    >
    > > 3) Try excluding a/, this does not work:
    > >
    > > # rsync -avn --relative --exclude a/ /tmp/source/./a/? /tmp/destination

    >
    > That's because 'a' is an implied directory in the longer path that you
    > told rsync to transfer. Implied directories can't be excluded, but
    > their attributes can be omitted via --no-implied-dirs. If you don't
    > want it in the transfer, don't specify it (and if you're using wild-
    > cards, specify them in such as way as to avoid matching the dir you
    > don't want).


    Nuts. I was afraid that was going to be the answer. Unfortunately,
    it violates the very helpful advise given above, i.e., run with "-v"
    and add exclude rules for files you don't want to transfer based
    on the output seen.

    Is this an enhancement request you are willing to consider, i.e.,
    allow filtering of implied directories?

    The real problem I want to solve has a pattern matching requirement
    that is too complicated for a glob that specifies the implied
    directories, and while the explicit list can be generated outside
    of rsync it would be much more convenient to use rsync's --exclude
    mechanism.

    Thanks.

    --
    Stuart Anderson anderson@ligo.caltech.edu
    http://www.ligo.caltech.edu/~anderson
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  4. Re: How to exclude directories from source with --relative

    On Mon, 2008-05-26 at 23:40 -0700, Wayne Davison wrote:
    > On Mon, May 26, 2008 at 09:46:04PM -0700, Stuart Anderson wrote:
    > > 3) Try excluding a/, this does not work:
    > >
    > > # rsync -avn --relative --exclude a/ /tmp/source/./a/? /tmp/destination

    >
    > That's because 'a' is an implied directory in the longer path that you
    > told rsync to transfer. Implied directories can't be excluded, but
    > their attributes can be omitted via --no-implied-dirs. If you don't
    > want it in the transfer, don't specify it (and if you're using wild-
    > cards, specify them in such as way as to avoid matching the dir you
    > don't want).


    I can understand not being able to exclude a/ from the file list while
    still transferring stuff under it. Still, I should be able to exclude
    a/ and everything under it by passing --exclude='/a/***' . But this
    doesn't work in rsync 3.0.2: rsync excludes a/b but adds the implied dir
    a/ to the file list anyway. (Obviously, it worked in rsync 2.6.9, which
    allowed the user to exclude implied dirs.) I think rsync should check
    whether a source argument is excluded before adding any dirs it implies.

    Matt

    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.9 (GNU/Linux)

    iEYEABECAAYFAkg8XmcACgkQC+xSYN/RlfuW2QCeIAM0w+U58lyrVG1BNQRQPcqO
    7RAAn1GCGZGuE5VHiXn+4t5vRR7Q4GWZ
    =I+pR
    -----END PGP SIGNATURE-----


  5. Re: How to exclude directories from source with --relative

    I must be missing something. I've tried many different combos
    of args and still can't get --exclude to work.

    I'm mirroring CPAN and desire to exclude all subdirs/files in
    the src directory. Here's the arg list I'm using:

    /usr/bin/rsync -av --delete --exclude '/src/***' --no-implied-dirs \
    --delete-excluded --prune-empty-dirs ftp.funet.fi::CPAN /cpan-mirror/

    Let me know what I'm missing.

    Thanks.

    --
    Love feeling your best ever, all day, every day?
    Click http://RadicalHealth.com/join for the easy way.
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  6. Re: How to exclude directories from source with --relative

    On Tue, May 27, 2008 at 03:18:03PM -0400, Matt McCutchen wrote:
    > On Mon, 2008-05-26 at 23:40 -0700, Wayne Davison wrote:
    > > On Mon, May 26, 2008 at 09:46:04PM -0700, Stuart Anderson wrote:
    > > > 3) Try excluding a/, this does not work:
    > > >
    > > > # rsync -avn --relative --exclude a/ /tmp/source/./a/? /tmp/destination

    > >
    > > That's because 'a' is an implied directory in the longer path that you
    > > told rsync to transfer. Implied directories can't be excluded, but
    > > their attributes can be omitted via --no-implied-dirs. If you don't
    > > want it in the transfer, don't specify it (and if you're using wild-
    > > cards, specify them in such as way as to avoid matching the dir you
    > > don't want).

    >
    > I can understand not being able to exclude a/ from the file list while
    > still transferring stuff under it. Still, I should be able to exclude
    > a/ and everything under it by passing --exclude='/a/***' . But this
    > doesn't work in rsync 3.0.2: rsync excludes a/b but adds the implied dir
    > a/ to the file list anyway. (Obviously, it worked in rsync 2.6.9, which
    > allowed the user to exclude implied dirs.) I think rsync should check
    > whether a source argument is excluded before adding any dirs it implies.


    This would be sufficient for my needs, and I didn't realize it is a
    change in behavior from 2.6.9 to 3.0.2.

    Thanks.


    --
    Stuart Anderson anderson@ligo.caltech.edu
    http://www.ligo.caltech.edu/~anderson
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  7. Re: How to exclude directories from source with --relative

    On 28-May-2008, at 05:28, David Favor wrote:
    > /usr/bin/rsync -av --delete --exclude '/src/***'


    why all the *?

    -exclude="src/" is what I've used in the past

    /src/ would imply that src is at the root of your system, which seem
    unlikely.

    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  8. Re: (CPAN src exclusion problem)

    On Fri, 2008-05-30 at 10:00 -0600, lewis butler wrote:
    > On 28-May-2008, at 05:28, David Favor wrote:
    > > /usr/bin/rsync -av --delete --exclude '/src/***'

    >
    > why all the *?


    A trailing triple star makes the rule match /src/ and everything under
    it, which is overkill in this case but doesn't hurt anything.

    > -exclude="src/" is what I've used in the past
    >
    > /src/ would imply that src is at the root of your system, which seem
    > unlikely.


    No, the leading slash refers to the root of the file list for the
    current run of rsync, which is exactly what David wants. For the root
    of the filesystem, use --filter='-/ /src/'.

    Matt

    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.9 (GNU/Linux)

    iEYEABECAAYFAkhAL/YACgkQC+xSYN/Rlfu/KQCfXOApJxxype61PtjWQVD3k5qa
    oLQAoIYkOGhYE6vWxSyG2eA8YgdiYFTx
    =4KNq
    -----END PGP SIGNATURE-----


  9. Re: (CPAN src exclusion problem)

    Matt McCutchen wrote:
    > On Fri, 2008-05-30 at 10:00 -0600, lewis butler wrote:
    >> On 28-May-2008, at 05:28, David Favor wrote:
    >>> /usr/bin/rsync -av --delete --exclude '/src/***'

    >> why all the *?

    >
    > A trailing triple star makes the rule match /src/ and everything under
    > it, which is overkill in this case but doesn't hurt anything.
    >
    >> -exclude="src/" is what I've used in the past
    >>
    >> /src/ would imply that src is at the root of your system, which seem
    >> unlikely.

    >
    > No, the leading slash refers to the root of the file list for the
    > current run of rsync, which is exactly what David wants. For the root
    > of the filesystem, use --filter='-/ /src/'.
    >
    > Matt


    Still not working. If I remove a random file like src/fo.md5.txt all the
    following fail. In other works, each of the following exclude patterns
    recreate the file.

    # --exclude='src/'
    # --exclude='src/***'
    # --exclude='/src/'
    # --exclude='/src/***'

    I'm using rsync-3.0.1 and based on the docs this looks like a bug as it
    appears all the above should:

    1) exclude all src/* files

    2) remove all src/* files which might exist

    Neither action occurs for any of the above exclude patterns.

    --
    Love feeling your best ever, all day, every day?
    Click http://RadicalHealth.com/join for the easy way.
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  10. Re: (CPAN src exclusion problem)

    On Fri, 2008-05-30 at 12:43 -0500, David Favor wrote:
    > Still not working. If I remove a random file like src/fo.md5.txt all the
    > following fail. In other works, each of the following exclude patterns
    > recreate the file.
    >
    > # --exclude='src/'
    > # --exclude='src/***'
    > # --exclude='/src/'
    > # --exclude='/src/***'
    >
    > I'm using rsync-3.0.1 and based on the docs this looks like a bug as it
    > appears all the above should:
    >
    > 1) exclude all src/* files
    >
    > 2) remove all src/* files which might exist
    >
    > Neither action occurs for any of the above exclude patterns.


    Still can't reproduce. I ran this command to pull some files in /src.
    It's the same as yours but *without* your exclude and with some excludes
    that I added to keep the number of files reasonable:

    ~/rsync/versions/rsync-3.0.1/rsync --exclude='/[.A-Za-q]*' --exclude=/scripts --exclude='/*/*/*' -av --delete --delete-excluded --prune-empty-dirs ftp.funet.fi::CPAN cpan-mirror/

    Then:

    rm cpan-mirror/src/perl5.004_01.tar.gz.md5.txt

    And the same command but *with* your exclude:

    ~/rsync/versions/rsync-3.0.1/rsync --exclude='/[.A-Za-q]*' --exclude=/scripts --exclude='/*/*/*' -av --delete --exclude '/src/**' --delete-excluded --prune-empty-dirs ftp.funet.fi::CPAN cpan-mirror/

    Rsync deleted the rest of the files all the way up to src/ itself and
    did not recreate the one I deleted.

    Matt

    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.9 (GNU/Linux)

    iEYEABECAAYFAkhAQMAACgkQC+xSYN/Rlftt8gCbBuedlWWUA9vDz3IKKC6Rhiy6
    s5wAoIwSVwnK5mEVcIcpl/mzjbHVnC7D
    =f08F
    -----END PGP SIGNATURE-----


  11. Re: How to exclude directories from source with --relative

    On Tue, May 27, 2008 at 10:43:00AM -0700, Stuart Anderson wrote:
    > Nuts. I was afraid that was going to be the answer. Unfortunately,
    > it violates the very helpful advise given above, i.e., run with "-v"
    > and add exclude rules for files you don't want to transfer based
    > on the output seen.


    That rule is not violated (the matching is correct), it is just that you
    can't try to exclude something in the middle of a relative path that
    you've specified.

    You can work around the issue by moving higher in the directory
    structure and using includes/excludes. For example, instead of:

    rsync -avR --exclude a/ /tmp/source/./[a-z]/* /tmp/destination

    You'd change the copy to be:

    rsync -av --exclude-from=file /tmp/source/ /tmp/destination/

    and have an exclude file:

    # First, we don't want that "a" dir (anchor it with a leading slash).
    - /a/
    # Next, this is our matching rule from the path above.
    + /[a-z]/
    # Then, exclude everything else _in_this_dir_ (note leading slash).
    - /*
    # Finally, I added this rule to emulate the '*' above for a shell
    # that omits dot files from its globbing.
    - /*/.*

    If you have any questions (such as how to adapt your real copy command),
    let us know.

    ...wayne..
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  12. Re: (CPAN src exclusion problem)

    Matt McCutchen wrote:
    > On Fri, 2008-05-30 at 12:43 -0500, David Favor wrote:
    >> Still not working. If I remove a random file like src/fo.md5.txt all the
    >> following fail. In other works, each of the following exclude patterns
    >> recreate the file.
    >>
    >> # --exclude='src/'
    >> # --exclude='src/***'
    >> # --exclude='/src/'
    >> # --exclude='/src/***'
    >>
    >> I'm using rsync-3.0.1 and based on the docs this looks like a bug as it
    >> appears all the above should:
    >>
    >> 1) exclude all src/* files
    >>
    >> 2) remove all src/* files which might exist
    >>
    >> Neither action occurs for any of the above exclude patterns.

    >
    > Still can't reproduce. I ran this command to pull some files in /src.
    > It's the same as yours but *without* your exclude and with some excludes
    > that I added to keep the number of files reasonable:
    >
    > ~/rsync/versions/rsync-3.0.1/rsync --exclude='/[.A-Za-q]*' --exclude=/scripts --exclude='/*/*/*' -av --delete --delete-excluded --prune-empty-dirs ftp.funet.fi::CPAN cpan-mirror/
    >
    > Then:
    >
    > rm cpan-mirror/src/perl5.004_01.tar.gz.md5.txt
    >
    > And the same command but *with* your exclude:
    >
    > ~/rsync/versions/rsync-3.0.1/rsync --exclude='/[.A-Za-q]*' --exclude=/scripts --exclude='/*/*/*' -av --delete --exclude '/src/**' --delete-excluded --prune-empty-dirs ftp.funet.fi::CPAN cpan-mirror/
    >
    > Rsync deleted the rest of the files all the way up to src/ itself and
    > did not recreate the one I deleted.
    >
    > Matt


    I believe the problem may be ftp.funet.fi sever.

    Thanks for your assistance.

    --
    Love feeling your best ever, all day, every day?
    Click http://RadicalHealth.com/join for the easy way.
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  13. Re: How to exclude directories from source with --relative

    On Fri, May 30, 2008 at 11:14:22AM -0700, Wayne Davison wrote:
    > On Tue, May 27, 2008 at 10:43:00AM -0700, Stuart Anderson wrote:
    > > Nuts. I was afraid that was going to be the answer. Unfortunately,
    > > it violates the very helpful advise given above, i.e., run with "-v"
    > > and add exclude rules for files you don't want to transfer based
    > > on the output seen.

    >
    > That rule is not violated (the matching is correct), it is just that you
    > can't try to exclude something in the middle of a relative path that
    > you've specified.
    >
    > You can work around the issue by moving higher in the directory
    > structure and using includes/excludes. For example, instead of:


    Wayne,
    Many thanks for you help on this. I now have something working
    by giving up on the --relative option and adding a longer list of alternating
    include/exclude rules. However, I would like to request that you consider
    increasing the scope of the filtering rules to include relative paths.
    I would like to be able to use shell globing to generate a long list
    of source directories but still allow rsync to filter out some of those.

    Put another way, I would like be able to specify "--exclude abc/"
    and have the directory "abc" not be transferred under any circumstances
    unless it was first listed in an --include argument.

    P.S. The recent scalability and incremental scan improvements in 3.0 are
    working great for me, and are greatly appreciated.

    Thanks.

    --
    Stuart Anderson anderson@ligo.caltech.edu
    http://www.ligo.caltech.edu/~anderson
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  14. Re: How to exclude directories from source with --relative

    On Sat, May 31, 2008 at 10:20:35AM -0700, Stuart Anderson wrote:
    > I would like to be able to use shell globing to generate a long list
    > of source directories but still allow rsync to filter out some of those.


    That is supported, but you must exclude an item you've specified, not an
    implied component of an item you've specified. So, if you change the
    "--exclude=abc/" to "--exclude=abc/**", anything inside that dir will
    not get transferred.

    The thing that I should fix is when an "--exclude=abc/***" removes all
    the args that implied the abc dir -- then that directory should indeed
    not have been included in the transfer in the first place.

    ...wayne..
    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


+ Reply to Thread