Using rsync on Mac OS X to mirror - Tools

This is a discussion on Using rsync on Mac OS X to mirror - Tools ; Is there a way to set up two folders, one on each machine, to mirror each other? If I drop a file in one folder in Phoenix, it will be copied to the machine in Tucson and vice versa. I've ...

+ Reply to Thread
Results 1 to 15 of 15

Thread: Using rsync on Mac OS X to mirror

  1. Using rsync on Mac OS X to mirror

    Is there a way to set up two folders, one on each machine, to mirror each
    other? If I drop a file in one folder in Phoenix, it will be copied to the
    machine in Tucson and vice versa. I've been reading these docs but am not
    seeing a simple answer.

    Any advice is greatly appreciated!


    Thanks

    Steffan


  2. Re: Using rsync on Mac OS X to mirror

    In message
    Steffan wrote:
    > Is there a way to set up two folders, one on each machine, to mirror each
    > other? If I drop a file in one folder in Phoenix, it will be copied to the
    > machine in Tucson and vice versa. I've been reading these docs but am not
    > seeing a simple answer.


    You would need a rsync task for each machine to sync the folders. The only
    issue is what to do if the same file changed in both locations. If that is
    likely (or even possible), then I would keep a Shared (Phoenix) and Shared
    (Tucson) folder. But if not, then a simple:

    rsync -a phoenix::shared/ tucson::shared/
    rsync -a tucson::shared/ phoenix::shared/

    Will keep them synced. Pop those two lines into a crontab to fire off every
    (however long you need) and setup your rsyncd.conf files and you're golden.

    I don't know where the rsyncd.conf is in 10.5 though (I use OS X, but all my
    rsync work is done on FreeBSD machines now except one backup from a laptop to
    the server, but I use rsync/ssh for that).


    --
    Advance and attack! Attack and destroy! Destroy and rejoice!

  3. Re: Using rsync on Mac OS X to mirror

    In message
    Steffan wrote:
    > Is there a way to set up two folders, one on each machine, to mirror each
    > other? If I drop a file in one folder in Phoenix, it will be copied to the
    > machine in Tucson and vice versa. I've been reading these docs but am not
    > seeing a simple answer.


    Oh, sorry, I forgot one other option.

    Write an applescript that is attached to the folder using the folder actions.

    You'll have to look up the exact syntax (maybe google: OS X Folder Actions)
    but it is something like

    on adding folder items to MyFolder after receiving NewItems
    do shell script "rsync -a " & MyFolder & "destination:/path/"
    end

    You might have to get the POSIX path to MyFolder or something, but that is
    very straight forward.

    set NewPath to POSIX file of MyFolder as alias

    Then use NewPath in the rsync command.

    If you attach this action to the folder on each machine then each time an item
    is ADDED, the script will trigger. I do not believe it will trigger when an
    item is removed or modified, however.

    If you are using the folder simply to copy files from one machine to another
    this is probably the best option as it means you don't need to mess with cron,
    you don't need to mess with rsyncd.conf, and all the prep you need is to create
    the keypair between the machines and attach the script. And since you should
    be creating the keypair *anyway*, it's minimal extra work.

    --
    "There's nothin' wrong with bein' a son of a bitch."
    -- Gaspode the Wonder Dog

  4. Re: Using rsync on Mac OS X to mirror

    in article slrnfvkdn6.b8l.g.kreme@cerebus.hsd1.co.comcast.net, Lewis at
    g.kreme@gmail.com.dontsendmecopies wrote on 4/7/08 7:59 AM:

    > In message
    > Steffan wrote:
    >> Is there a way to set up two folders, one on each machine, to mirror each
    >> other? If I drop a file in one folder in Phoenix, it will be copied to the
    >> machine in Tucson and vice versa. I've been reading these docs but am not
    >> seeing a simple answer.

    >
    > Oh, sorry, I forgot one other option.
    >
    > Write an applescript that is attached to the folder using the folder actions.
    >
    > You'll have to look up the exact syntax (maybe google: OS X Folder Actions)
    > but it is something like
    >
    > on adding folder items to MyFolder after receiving NewItems
    > do shell script "rsync -a " & MyFolder & "destination:/path/"
    > end
    >
    > You might have to get the POSIX path to MyFolder or something, but that is
    > very straight forward.
    >
    > set NewPath to POSIX file of MyFolder as alias
    >
    > Then use NewPath in the rsync command.
    >
    > If you attach this action to the folder on each machine then each time an item
    > is ADDED, the script will trigger. I do not believe it will trigger when an
    > item is removed or modified, however.
    >
    > If you are using the folder simply to copy files from one machine to another
    > this is probably the best option as it means you don't need to mess with cron,
    > you don't need to mess with rsyncd.conf, and all the prep you need is to
    > create
    > the keypair between the machines and attach the script. And since you should
    > be creating the keypair *anyway*, it's minimal extra work.

    I had already thought about using folder actions but in this case, neither
    are machines to which people will be logged into the gui. Will apple script
    folder actions work without someone being logged in? If not then may have to
    use the crontab to handle this. Is there a good tutorial on how to set up
    the rsync using ssh in the manner which you suggest?

    Thanks for the info. This is getting interesting.

    Thanks,

    Steffan


  5. Re: Using rsync on Mac OS X to mirror

    in article slrnfvkcsq.b8l.g.kreme@cerebus.hsd1.co.comcast.net, Lewis at
    g.kreme@gmail.com.dontsendmecopies wrote on 4/7/08 7:45 AM:

    > In message
    > Steffan wrote:
    >> Is there a way to set up two folders, one on each machine, to mirror each
    >> other? If I drop a file in one folder in Phoenix, it will be copied to the
    >> machine in Tucson and vice versa. I've been reading these docs but am not
    >> seeing a simple answer.

    >
    > You would need a rsync task for each machine to sync the folders. The only
    > issue is what to do if the same file changed in both locations. If that is
    > likely (or even possible), then I would keep a Shared (Phoenix) and Shared
    > (Tucson) folder. But if not, then a simple:
    >
    > rsync -a phoenix::shared/ tucson::shared/
    > rsync -a tucson::shared/ phoenix::shared/
    >
    > Will keep them synced. Pop those two lines into a crontab to fire off every
    > (however long you need) and setup your rsyncd.conf files and you're golden.
    >
    > I don't know where the rsyncd.conf is in 10.5 though (I use OS X, but all my
    > rsync work is done on FreeBSD machines now except one backup from a laptop to
    > the server, but I use rsync/ssh for that).
    >


    I thought about this. If I am syncing the same folders, meaning shared in
    phoenix with shared in tucson, would it create a situation where they would
    constantly be updating each other with the same files that no person
    touched?

    Thanks,

    Steffan


  6. Re: Using rsync on Mac OS X to mirror

    In message
    Steffan wrote:
    > in article slrnfvkcsq.b8l.g.kreme@cerebus.hsd1.co.comcast.net, Lewis at
    > g.kreme@gmail.com.dontsendmecopies wrote on 4/7/08 7:45 AM:


    >> In message
    >> Steffan wrote:
    >>> Is there a way to set up two folders, one on each machine, to mirror each
    >>> other? If I drop a file in one folder in Phoenix, it will be copied to the
    >>> machine in Tucson and vice versa. I've been reading these docs but am not
    >>> seeing a simple answer.

    >>
    >> You would need a rsync task for each machine to sync the folders. The only
    >> issue is what to do if the same file changed in both locations. If that is
    >> likely (or even possible), then I would keep a Shared (Phoenix) and Shared
    >> (Tucson) folder. But if not, then a simple:
    >>
    >> rsync -a phoenix::shared/ tucson::shared/
    >> rsync -a tucson::shared/ phoenix::shared/
    >>
    >> Will keep them synced. Pop those two lines into a crontab to fire off every
    >> (however long you need) and setup your rsyncd.conf files and you're golden.
    >>
    >> I don't know where the rsyncd.conf is in 10.5 though (I use OS X, but all my
    >> rsync work is done on FreeBSD machines now except one backup from a laptop to
    >> the server, but I use rsync/ssh for that).
    >>


    > I thought about this. If I am syncing the same folders, meaning shared in
    > phoenix with shared in tucson, would it create a situation where they would
    > constantly be updating each other with the same files that no person
    > touched?


    No, rsync is smarter than that.

    I did forget to mention, since you are on OS X, you need a -E flag.


    --
    "If this is the best God can do, I'm not impressed."

  7. Re: Using rsync on Mac OS X to mirror

    In message
    Steffan wrote:
    > I had already thought about using folder actions but in this case, neither
    > are machines to which people will be logged into the gui. Will apple script
    > folder actions work without someone being logged in?


    Ah... no, I don't think so.

    > If not then may have to
    > use the crontab to handle this. Is there a good tutorial on how to set up
    > the rsync using ssh in the manner which you suggest?


    the ssh has nothing specifically to do with rsync. You setup ssh, then you
    run rsync over it. The rysnc command is a tad different:

    rsync -aE phoenixath/ tucsonath/

    (that is if path is in ~ on both machines and the short user names are the
    same).

    I generated my public/private keypair on FreeBSD, then copied them to my OS X
    machine, but it should be the same:

    ssh-keygen -t dsa

    then simply copy the public key to the remote server, appending it to the
    authorizedkeys2 file:

    $ cat ~/.ssh/id_dsa.pub | ssh user@remote.machine.com 'sh -c "cat - >> .ssh/authorized_keys2"'

    (you will need your password for that, but should not need it again)

    You do need to enable remote login in the Sharing Control Panel to actually
    enable ssh access. And if you are going to be going both ways in logins, you
    will want the private and public keys to be on both machines.

    I did create a config file for OS X so that the ssh session does not timeout:

    $ cat .ssh/config
    ServerAliveInterval 60
    TCPKeepAlive yes



    --
    Can I borrow your underpants for 10 minutes?

  8. Re: Using rsync on Mac OS X to mirror

    Lewis wrote:
    > rsync -a phoenix::shared/ tucson::shared/ rsync -a tucson::shared/
    > phoenix::shared/
    >
    > Will keep them synced. Pop those two lines into a crontab to fire off
    > every
    > (however long you need) and setup your rsyncd.conf files and you're golden.



    I'm in a similar boat. Will this take care of deleted files/folders that
    have beem moved, deleted, etc?

    Thank

  9. Re: Using rsync on Mac OS X to mirror

    in article slrnfvkldk.b8l.g.kreme@cerebus.hsd1.co.comcast.net, Lewis at
    g.kreme@gmail.com.dontsendmecopies wrote on 4/7/08 10:11 AM:

    > In message
    > Steffan wrote:
    >> I had already thought about using folder actions but in this case, neither
    >> are machines to which people will be logged into the gui. Will apple script
    >> folder actions work without someone being logged in?

    >
    > Ah... no, I don't think so.
    >
    >> If not then may have to
    >> use the crontab to handle this. Is there a good tutorial on how to set up
    >> the rsync using ssh in the manner which you suggest?

    >
    > the ssh has nothing specifically to do with rsync. You setup ssh, then you
    > run rsync over it. The rysnc command is a tad different:
    >
    > rsync -aE phoenixath/ tucsonath/
    >
    > (that is if path is in ~ on both machines and the short user names are the
    > same).
    >
    > I generated my public/private keypair on FreeBSD, then copied them to my OS X
    > machine, but it should be the same:
    >
    > ssh-keygen -t dsa
    >
    > then simply copy the public key to the remote server, appending it to the
    > authorizedkeys2 file:
    >
    > $ cat ~/.ssh/id_dsa.pub | ssh user@remote.machine.com 'sh -c "cat - >>
    > .ssh/authorized_keys2"'
    >
    > (you will need your password for that, but should not need it again)
    >
    > You do need to enable remote login in the Sharing Control Panel to actually
    > enable ssh access. And if you are going to be going both ways in logins, you
    > will want the private and public keys to be on both machines.
    >
    > I did create a config file for OS X so that the ssh session does not timeout:
    >
    > $ cat .ssh/config
    > ServerAliveInterval 60
    > TCPKeepAlive yes
    >
    >

    Is the rsyncd.conf file even needed if I am not running it as a daemon?

    I just got the login working with out password in both directions! Cool. Now
    to get this other part working. I am confused about the proper syntax about
    specifying the IP etc

    rsync -aE hereIPath/ thereIPath/

    Does not work... Maybe I am just up too late. Any suggestions?

    Thanks,
    Steffan



  10. Re: Using rsync on Mac OS X to mirror

    In message <47fe336e$0$22152$4c368faf@roadrunner.com>
    Someone wrote:
    > Lewis wrote:
    >> rsync -a phoenix::shared/ tucson::shared/ rsync -a tucson::shared/
    >> phoenix::shared/
    >>
    >> Will keep them synced. Pop those two lines into a crontab to fire off
    >> every
    >> (however long you need) and setup your rsyncd.conf files and you're golden.



    > I'm in a similar boat. Will this take care of deleted files/folders that
    > have beem moved, deleted, etc?


    No, you need to look at --delete and --delete-after and similar flags for
    the behavior you want.



    --
    There is NO Rule six!

  11. Re: Using rsync on Mac OS X to mirror

    in article slrnfvkldk.b8l.g.kreme@cerebus.hsd1.co.comcast.net, Lewis at
    g.kreme@gmail.com.dontsendmecopies wrote on 4/7/08 10:11 AM:

    > In message
    > Steffan wrote:
    >> I had already thought about using folder actions but in this case, neither
    >> are machines to which people will be logged into the gui. Will apple script
    >> folder actions work without someone being logged in?

    >
    > Ah... no, I don't think so.
    >
    >> If not then may have to
    >> use the crontab to handle this. Is there a good tutorial on how to set up
    >> the rsync using ssh in the manner which you suggest?

    >
    > the ssh has nothing specifically to do with rsync. You setup ssh, then you
    > run rsync over it. The rysnc command is a tad different:
    >
    > rsync -aE phoenixath/ tucsonath/
    >
    > (that is if path is in ~ on both machines and the short user names are the
    > same).
    >
    > I generated my public/private keypair on FreeBSD, then copied them to my OS X
    > machine, but it should be the same:
    >
    > ssh-keygen -t dsa
    >
    > then simply copy the public key to the remote server, appending it to the
    > authorizedkeys2 file:
    >
    > $ cat ~/.ssh/id_dsa.pub | ssh user@remote.machine.com 'sh -c "cat - >>
    > .ssh/authorized_keys2"'
    >
    > (you will need your password for that, but should not need it again)
    >
    > You do need to enable remote login in the Sharing Control Panel to actually
    > enable ssh access. And if you are going to be going both ways in logins, you
    > will want the private and public keys to be on both machines.
    >
    > I did create a config file for OS X so that the ssh session does not timeout:
    >
    > $ cat .ssh/config
    > ServerAliveInterval 60
    > TCPKeepAlive yes
    >
    >


    Do you have an example of what the config file for this might look like?
    (phoenixath/) From all that I have read the config file is supposed to be
    located in /etc/rsyncd.config on both machines since I am going both ways. I
    did not see any variance in this.


  12. Re: Using rsync on Mac OS X to mirror

    Lewis wrote:
    > In message <47fe336e$0$22152$4c368faf@roadrunner.com> Someone
    > wrote:
    >> Lewis wrote:
    >>> rsync -a phoenix::shared/ tucson::shared/ rsync -a tucson::shared/
    >>> phoenix::shared/
    >>>
    >>> Will keep them synced. Pop those two lines into a crontab to fire
    >>> off every
    >>> (however long you need) and setup your rsyncd.conf files and you're
    >>> golden.

    >
    >
    >> I'm in a similar boat. Will this take care of deleted files/folders
    >> that have beem moved, deleted, etc?

    >
    > No, you need to look at --delete and --delete-after and similar flags for
    > the behavior you want.



    Will I need both --delete and --delete-after in the same line?
    rsync --delete --delete-after source destination

    on both the push and the pull (I am going to have one machine doing
    pushes and pulls) or should the push have one and pull have the other?

    Thank you

  13. Re: Using rsync on Mac OS X to mirror

    In message <4802703b$0$9537$4c368faf@roadrunner.com>
    Someone wrote:
    > Lewis wrote:
    >> In message <47fe336e$0$22152$4c368faf@roadrunner.com> Someone
    >> wrote:
    >>> Lewis wrote:
    >>>> rsync -a phoenix::shared/ tucson::shared/ rsync -a tucson::shared/
    >>>> phoenix::shared/
    >>>>
    >>>> Will keep them synced. Pop those two lines into a crontab to fire
    >>>> off every
    >>>> (however long you need) and setup your rsyncd.conf files and you're
    >>>> golden.

    >>
    >>
    >>> I'm in a similar boat. Will this take care of deleted files/folders
    >>> that have beem moved, deleted, etc?

    >>
    >> No, you need to look at --delete and --delete-after and similar flags for
    >> the behavior you want.



    > Will I need both --delete and --delete-after in the same line?
    > rsync --delete --delete-after source destination


    There's a lot of delete options. Which you choose is really kind of up
    to you:

    --delete delete extraneous files from dest dirs
    --delete-before receiver deletes before transfer (default)
    --delete-during receiver deletes during xfer, not before
    --delete-delay find deletions during, delete after
    --delete-after receiver deletes after transfer, not before
    --delete-excluded also delete excluded files from dest dirs

    I generally use --delete-excluded --delete-after but there's been some
    changes in 3.0:

    } Some options require rsync to know the full file list, so these options
    } disable the incremental recursion mode. These include:
    } --delete-before, --delete-after, --prune-empty-dirs, and
    } --delay-updates. Because of this, the default delete mode when you
    } specify --delete is now --delete-during when both ends of the
    } connection are at least 3.0.0 (use --del or --delete-during to
    } request this improved deletion mode explicitly). See also the
    } --delete-delay option that is a better choice than using
    } --delete-after.

    so I will probably siwtch to --del or --delete-delay when iI get
    everything to V3

    > on both the push and the pull (I am going to have one machine doing
    > pushes and pulls) or should the push have one and pull have the other?


    Hard to say. As I understand it:

    Machine 1
    file1
    file2
    file3

    Machine 2
    file1
    file2a
    file4

    If you sync from 1 to 2 with a delete you will lose file 4 and file2a
    will update file2.

    If you sync from 2 to 1, you will lose file3.

    If you are in the habit of creating files on both machines, you probably
    don't want --delete.

    --
    A marriage is always made up of two people who are prepared to swear that
    only the other one snores.

  14. Re: Using rsync on Mac OS X to mirror

    In message
    Steffan wrote:
    > in article slrnfvkldk.b8l.g.kreme@cerebus.hsd1.co.comcast.net, Lewis at
    > g.kreme@gmail.com.dontsendmecopies wrote on 4/7/08 10:11 AM:


    >> In message
    >> Steffan wrote:
    >>> I had already thought about using folder actions but in this case, neither
    >>> are machines to which people will be logged into the gui. Will apple script
    >>> folder actions work without someone being logged in?

    >>
    >> Ah... no, I don't think so.
    >>
    >>> If not then may have to
    >>> use the crontab to handle this. Is there a good tutorial on how to set up
    >>> the rsync using ssh in the manner which you suggest?

    >>
    >> the ssh has nothing specifically to do with rsync. You setup ssh, then you
    >> run rsync over it. The rysnc command is a tad different:
    >>
    >> rsync -aE phoenixath/ tucsonath/
    >>
    >> (that is if path is in ~ on both machines and the short user names are the
    >> same).
    >>
    >> I generated my public/private keypair on FreeBSD, then copied them to my OS X
    >> machine, but it should be the same:
    >>
    >> ssh-keygen -t dsa
    >>
    >> then simply copy the public key to the remote server, appending it to the
    >> authorizedkeys2 file:
    >>
    >> $ cat ~/.ssh/id_dsa.pub | ssh user@remote.machine.com 'sh -c "cat - >>
    >> .ssh/authorized_keys2"'
    >>
    >> (you will need your password for that, but should not need it again)
    >>
    >> You do need to enable remote login in the Sharing Control Panel to actually
    >> enable ssh access. And if you are going to be going both ways in logins, you
    >> will want the private and public keys to be on both machines.
    >>
    >> I did create a config file for OS X so that the ssh session does not timeout:
    >>
    >> $ cat .ssh/config
    >> ServerAliveInterval 60
    >> TCPKeepAlive yes
    >>
    >>


    > Do you have an example of what the config file for this might look like?
    > (phoenixath/) From all that I have read the config file is supposed to be
    > located in /etc/rsyncd.config on both machines since I am going both ways. I
    > did not see any variance in this.



    phoenix:/path will not use a rsyncd.conf file, since the single colon
    indicates a ssh connection.

    this is my backup location in rsyncd.conf (accessed via
    phoenix::backup/)

    [backup]
    path = /backup/
    comment = Backup
    readonly = no
    auth users = root
    uid = 0
    secrets file = /usr/local/etc/rsyncd.secrets

    in case you decide to go with rsync --daemon

    However, unless you are backing up an entire system, you do not waht the
    'uid = 0' line.

    [kreme]
    path = /home/kreme
    comment = kreme Home
    readonly = no
    auth users = kreme
    secrets file = /usr/local/etc/rsyncd.secrets

    also, keep in mind that that 'auth users' only refers to the username in
    the secrets file, not to any /etc/passwd users or other local users.

    --
    I get the feeling that some people's idea of heaven is an "I told you so"
    T-shirt - mmalc

  15. Re: Using rsync on Mac OS X to mirror

    Lewis wrote:
    > In message <4802703b$0$9537$4c368faf@roadrunner.com> Someone
    > wrote:
    >> Lewis wrote:
    >>> In message <47fe336e$0$22152$4c368faf@roadrunner.com> Someone
    >>> wrote:
    >>>> Lewis wrote:
    >>>
    >>>
    >>>> I'm in a similar boat. Will this take care of deleted files/folders
    >>>> that have beem moved, deleted, etc?
    >>>
    >>> No, you need to look at --delete and --delete-after and similar flags
    >>> for
    >>> the behavior you want.

    >
    >
    >> Will I need both --delete and --delete-after in the same line?
    >> rsync --delete --delete-after source destination

    >
    > There's a lot of delete options. Which you choose is really kind of up
    > to you:
    >
    > --delete delete extraneous files from dest dirs
    > --delete-before receiver deletes before transfer (default)
    > --delete-during receiver deletes during xfer, not before
    > --delete-delay find deletions during, delete after
    > --delete-after receiver deletes after transfer, not before
    > --delete-excluded also delete excluded files from dest dirs
    >
    > I generally use --delete-excluded --delete-after but there's been some
    > changes in 3.0:
    >
    > } Some options require rsync to know the full file list, so these options
    > } disable the incremental recursion mode. These include: }
    > --delete-before, --delete-after, --prune-empty-dirs, and
    > } --delay-updates. Because of this, the default delete mode when you
    > } specify --delete is now --delete-during when both ends of the
    > } connection are at least 3.0.0 (use --del or --delete-during to
    > } request this improved deletion mode explicitly). See also the }
    > --delete-delay option that is a better choice than using
    > } --delete-after.
    >
    > so I will probably siwtch to --del or --delete-delay when iI get
    > everything to V3
    >
    >> on both the push and the pull (I am going to have one machine doing
    >> pushes and pulls) or should the push have one and pull have the other?

    >
    > Hard to say. As I understand it:
    >
    > Machine 1
    > file1
    > file2
    > file3
    > Machine 2
    > file1
    > file2a
    > file4
    >
    > If you sync from 1 to 2 with a delete you will lose file 4 and file2a
    > will update file2.
    >
    > If you sync from 2 to 1, you will lose file3.
    >
    > If you are in the habit of creating files on both machines, you probably
    > don't want --delete.


    So, I'm not the only one who noticed this issue

    Maybe, one of the developers can figure a way to deal with this. If I
    could program and had enough theory I could deffeintly try to help.

+ Reply to Thread