Re: How to escape spaces in rsync source and destination paths - Tools

This is a discussion on Re: How to escape spaces in rsync source and destination paths - Tools ; On Thu, 2008-08-21 at 16:32 -0400, Carney Mimms wrote: > Thank you for taking an interest in my problem of escaping spaces. > Here is the entire script and the entire output. I am at my wits end > as ...

+ Reply to Thread
Results 1 to 3 of 3

Thread: Re: How to escape spaces in rsync source and destination paths

  1. Re: How to escape spaces in rsync source and destination paths

    On Thu, 2008-08-21 at 16:32 -0400, Carney Mimms wrote:
    > Thank you for taking an interest in my problem of escaping spaces.
    > Here is the entire script and the entire output. I am at my wits end
    > as I have tried every every trick I know or have heard of, including \
    > \\ and adding single quotes. I added —protect-args at your suggestion,
    > but it doesn’t seem to change anything. I am running the script as
    > root for now to take sudo out of the picture. I have also tried
    > putting the paths directly in the command as arguments, rather than as
    > script variables.


    > #!/bin/bash
    >
    > # Space-separated list of directories to back up; edit as needed;
    > SOURCE
    > #DIRS="/Volumes/Christine/Complete\ Rug\ Image\ Archive/Misc"
    > DIRS="admin@192.168.100.47:/Volumes/Christine/Complete\ Rug\ Image\
    > Archive/Layered\ Rooms/"
    > # Options to pass to rsync; edit as needed
    > # "--update" = update only (don't overwrite newer versions of files)
    > # "--delete"= delete files that exist on DESTINATION BUT NOT SOURCE"
    > OPTS="-avAX --progress --delete --rsync-path=/usr/local/bin/rsync
    > --protect-args --dry-run"
    >
    > # If you wish to back up to a server via ssh, change the line to
    > something like
    > #
    > BACKUPDIR="remoteusername@someserver.something:/path/to/backup/destination"
    > #BACKUPDIR="admin@192.168.110.46:/Volumes/Paris/Complete Rug Image
    > Archive/Misc/"
    > BACKUPDIR="/Volumes/Paris/Complete\ Rug\ Image\ Archive/Layered\
    > Rooms/"
    > # ignore Mac droppings
    > EXCLUDES="--exclude .DS_Store --exclude .Trash --exclude Cache
    > --exclude Caches"
    >
    > # Build the actual command
    > # NOTE the specific path to the "special" version of rsync
    > COMMAND="/usr/local/bin/rsync $OPTS $EXCLUDES $DIRS $BACKUPDIR"
    >
    > # Informative output
    > echo About to run:
    > echo $COMMAND
    > echo Please do not close this window until it is finished.
    >
    > # DO IT!
    > $COMMAND
    >
    > echo Done.
    >
    > # the end.


    Here's your problem: when you expand a variable outside of quotes (in
    this case $COMMAND), bash word-splits on whitespace in $COMMAND but does
    not recognize any form of quoting or escaping within $COMMAND. Thus,
    there is nothing you can put in $COMMAND that will generate an argument
    containing a space. This came up before:

    http://lists.samba.org/archive/rsync...il/020620.html

    In this situation, I recommend using bash arrays, which let you store,
    manipulate, and finally execute lists of arguments without the local
    shell mangling spaces in the arguments. Since you're using
    --protect-args, rsync won't let the remote shell mangle the spaces
    either, so all you have to do is quote the spaces when they are
    originally inserted into an array. Here's your script rewritten to use
    arrays (lightly tested):

    -----
    #!/bin/bash

    # Space-separated list of directories to back up; edit as needed; SOURCE
    #DIRS=("admin@192.168.100.47:/Volumes/Christine/Complete Rug Image Archive/Misc" "admin@192.168.100.47:/Volumes/Christine/Other Source")
    DIRS=("admin@192.168.100.47:/Volumes/Christine/Complete Rug Image Archive/Layered Rooms/")
    # Options to pass to rsync; edit as needed
    # "--update" = update only (don't overwrite newer versions of files)
    # "--delete"= delete files that exist on DESTINATION BUT NOT SOURCE"
    OPTS=(-avAX --progress --delete --rsync-path=/usr/local/bin/rsync --protect-args --dry-run)

    # If you wish to back up to a server via ssh, change the line to something like
    # BACKUPDIR="remoteusername@someserver.something:/path/to/backup/destination"
    #BACKUPDIR="admin@192.168.110.46:/Volumes/Paris/Complete Rug Image Archive/Misc/"
    BACKUPDIR="/Volumes/Paris/Complete Rug Image Archive/Layered Rooms/"
    # ignore Mac droppings
    EXCLUDES=(--exclude .DS_Store --exclude .Trash --exclude Cache --exclude Caches)

    # Build the actual command
    # NOTE the specific path to the "special" version of rsync
    COMMAND=(/usr/local/bin/rsync "${OPTS[@]}" "${EXCLUDES[@]}" "${DIRS[@]}" "$BACKUPDIR")

    # Informative output
    echo About to run:
    echo "${COMMAND[*]}"
    echo Please do not close this window until it is finished.

    # DO IT!
    : "${COMMAND[@]}"

    echo Done.

    # the end.
    -----

    Matt

    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.9 (GNU/Linux)

    iEYEABECAAYFAkiuFzQACgkQC+xSYN/Rlfv5cwCgk1axLk2POQLlTlG8mZPxUGAi
    7YQAnj7A4knzYslXM2TNy6fGOOfx1GQH
    =JR5h
    -----END PGP SIGNATURE-----


  2. Re: How to escape spaces in rsync source and destination paths

    Matt,

    Thank you for your suggestion, and rewrite. Bash arrays seem like a much
    better approach and once I got the syntax straight, let the script run
    without tripping over the whitespace.

    I still need some help, though, as the script now runs without actually
    doing anything or producing any feedback. When I removed the --dry-run flag
    and added the --vv flag it now runs without error, but doesn't actually copy
    anything. I added some test files to the source to be sure. I have also set
    the --progress flag. As you can see the bash command echoes back properly,
    it just doesn't actually do anything.
    Am I missing something? Is there anything I can do to at least get the
    script to produce some feedback?

    Many thanks,

    --Carneyh

    -------------------------------------------------------------
    #!/bin/bash

    # Space-separated list of directories to back up; edit as needed; SOURCE
    #DIRS variable now uses arrays instead of escaping spaces
    DIRS=("admin@192.168.100.47:/Volumes/Christine/Complete Rug Archive/Layered
    Rooms/")
    # Options to pass to rsync; edit as needed
    # "--update" = update only (don't overwrite newer versions of files)
    # "--delete"= delete files that exist on DESTINATION BUT NOT SOURCE"
    # "--protect-args"= don't parse all spaces as separators
    # "--dry-run"= run script without actually performing file operations-test
    OPTS=(-aAX --vv --progress --delete --archive
    --rsync-path=/usr/local/bin/rsync --protect-args)

    # Backup destination. In this case, it is another hard disk on the same
    machine.
    # Incidentally, it is DOS-formatted, irrelevant here.
    # If you wish to back up to a server via ssh, change the line to something
    like
    #
    BACKUPDIR="remoteusername@someserver.something:/path/to/backup/destination"
    # BACKUPDIR in this script is on local machine
    BACKUPDIR="/Volumes/Paris/Complete Rug Archive/Layered Rooms/"
    # ignore Mac droppings
    EXCLUDES=(--exclude .DS_Store --exclude .Trash --exclude Cache --exclude
    Caches)

    # Build the actual command
    # NOTE the specific path to the "special" version of rsync
    # uses arrays instead of escaping spaces
    COMMAND=(/usr/local/bin/rsync "${OPTS[@]}" "${EXCLUDES[@]}" "${DIRS[@]}"
    "$BACKUPDIR")

    # Informative output
    echo About to run:
    echo "${COMMAND[*]}"
    echo Please do not close this window until it is finished.

    # DO IT!


    ----------------
    xserve03:/Users/admin/Documents/Rsync 3.0.3/rsync_scripts root#
    ../rsync_script_1.5.sh
    About to run:
    /usr/local/bin/rsync -aAX --vv --progress --delete --archive
    --rsync-path=/usr/local/bin/rsync --protect-args --exclude .DS_Store
    --exclude .Trash --exclude Cache --exclude Caches
    admin@192.168.100.47:/Volumes/Christine/Complete Rug Archive/Layered Rooms/
    /Volumes/Paris/Complete Rug Archive/Layered Rooms/
    Please do not close this window until it is finished.
    Done.
    xserve03:/Users/admin/Documents/Rsync 3.0.3/rsync_scripts root#
    ---------------------------------


    On 8/21/08 9:32 PM, "Matt McCutchen" wrote:

    > On Thu, 2008-08-21 at 16:32 -0400, Carney Mimms wrote:
    >> Thank you for taking an interest in my problem of escaping spaces.
    >> Here is the entire script and the entire output. I am at my wits end
    >> as I have tried every every trick I know or have heard of, including \
    >> \\ and adding single quotes. I added protect-args at your suggestion,
    >> but it doesnt seem to change anything. I am running the script as
    >> root for now to take sudo out of the picture. I have also tried
    >> putting the paths directly in the command as arguments, rather than as
    >> script variables.

    >
    >> #!/bin/bash
    >>
    >> # Space-separated list of directories to back up; edit as needed;
    >> SOURCE
    >> #DIRS="/Volumes/Christine/Complete\ Rug\ Image\ Archive/Misc"
    >> DIRS="admin@192.168.100.47:/Volumes/Christine/Complete\ Rug\ Image\
    >> Archive/Layered\ Rooms/"
    >> # Options to pass to rsync; edit as needed
    >> # "--update" = update only (don't overwrite newer versions of files)
    >> # "--delete"= delete files that exist on DESTINATION BUT NOT SOURCE"
    >> OPTS="-avAX --progress --delete --rsync-path=/usr/local/bin/rsync
    >> --protect-args --dry-run"
    >>
    >> # If you wish to back up to a server via ssh, change the line to
    >> something like
    >> #
    >> BACKUPDIR="remoteusername@someserver.something:/path/to/backup/destination"
    >> #BACKUPDIR="admin@192.168.110.46:/Volumes/Paris/Complete Rug Image
    >> Archive/Misc/"
    >> BACKUPDIR="/Volumes/Paris/Complete\ Rug\ Image\ Archive/Layered\
    >> Rooms/"
    >> # ignore Mac droppings
    >> EXCLUDES="--exclude .DS_Store --exclude .Trash --exclude Cache
    >> --exclude Caches"
    >>
    >> # Build the actual command
    >> # NOTE the specific path to the "special" version of rsync
    >> COMMAND="/usr/local/bin/rsync $OPTS $EXCLUDES $DIRS $BACKUPDIR"
    >>
    >> # Informative output
    >> echo About to run:
    >> echo $COMMAND
    >> echo Please do not close this window until it is finished.
    >>
    >> # DO IT!
    >> $COMMAND
    >>
    >> echo Done.
    >>
    >> # the end.

    >
    > Here's your problem: when you expand a variable outside of quotes (in
    > this case $COMMAND), bash word-splits on whitespace in $COMMAND but does
    > not recognize any form of quoting or escaping within $COMMAND. Thus,
    > there is nothing you can put in $COMMAND that will generate an argument
    > containing a space. This came up before:
    >
    > http://lists.samba.org/archive/rsync...il/020620.html
    >
    > In this situation, I recommend using bash arrays, which let you store,
    > manipulate, and finally execute lists of arguments without the local
    > shell mangling spaces in the arguments. Since you're using
    > --protect-args, rsync won't let the remote shell mangle the spaces
    > either, so all you have to do is quote the spaces when they are
    > originally inserted into an array. Here's your script rewritten to use
    > arrays (lightly tested):
    >
    > -----
    > #!/bin/bash
    >
    > # Space-separated list of directories to back up; edit as needed; SOURCE
    > #DIRS=("admin@192.168.100.47:/Volumes/Christine/Complete Rug Image
    > Archive/Misc" "admin@192.168.100.47:/Volumes/Christine/Other Source")
    > DIRS=("admin@192.168.100.47:/Volumes/Christine/Complete Rug Image
    > Archive/Layered Rooms/")
    > # Options to pass to rsync; edit as needed
    > # "--update" = update only (don't overwrite newer versions of files)
    > # "--delete"= delete files that exist on DESTINATION BUT NOT SOURCE"
    > OPTS=(-avAX --progress --delete --rsync-path=/usr/local/bin/rsync
    > --protect-args --dry-run)
    >
    > # If you wish to back up to a server via ssh, change the line to something
    > like
    > # BACKUPDIR="remoteusername@someserver.something:/path/to/backup/destination"
    > #BACKUPDIR="admin@192.168.110.46:/Volumes/Paris/Complete Rug Image
    > Archive/Misc/"
    > BACKUPDIR="/Volumes/Paris/Complete Rug Image Archive/Layered Rooms/"
    > # ignore Mac droppings
    > EXCLUDES=(--exclude .DS_Store --exclude .Trash --exclude Cache --exclude
    > Caches)
    >
    > # Build the actual command
    > # NOTE the specific path to the "special" version of rsync
    > COMMAND=(/usr/local/bin/rsync "${OPTS[@]}" "${EXCLUDES[@]}" "${DIRS[@]}"
    > "$BACKUPDIR")
    >
    > # Informative output
    > echo About to run:
    > echo "${COMMAND[*]}"
    > echo Please do not close this window until it is finished.
    >
    > # DO IT!
    > : "${COMMAND[@]}"
    >
    > echo Done.
    >
    > # the end.
    > -----
    >
    > Matt


    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


  3. Re: How to escape spaces in rsync source and destination paths

    On Fri, 2008-08-22 at 14:56 -0400, Carney Mimms wrote:
    > I still need some help, though, as the script now runs without actually
    > doing anything or producing any feedback.


    It looks like I forgot to take out the colon in front of the
    "${COMMAND[@]}", which I had used to turn the command into a no-op for
    my own testing. Oops. The last part of the script should read:

    -----
    # DO IT!
    "${COMMAND[@]}"

    echo Done.

    # the end.
    -----

    Sorry about that.

    Matt

    > On 8/21/08 9:32 PM, "Matt McCutchen" wrote:
    > > # DO IT!
    > > : "${COMMAND[@]}"
    > >
    > > echo Done.
    > >
    > > # the end.
    > > -----


    --
    Please use reply-all for most replies to avoid omitting the mailing list.
    To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
    Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.9 (GNU/Linux)

    iEUEABECAAYFAkivdz4ACgkQC+xSYN/Rlfu8lQCgsEEaoqm0/QMQJaitTNElD7Mj
    aZ0AljV61dqHTRGDIe96wG3gNVgsDvY=
    =j5ex
    -----END PGP SIGNATURE-----


+ Reply to Thread