This is a discussion on Some basic and not so basic rsync questions - Tools ; In trying to find a better way to transfer files over ssh/sftp and I've been directred towards rsync. In the past when I've looked at rsync, it just didn't seem to be a good fit for our scenario. Our situation ...
In trying to find a better way to transfer files over ssh/sftp and I've
been directred towards rsync. In the past when I've looked at rsync,
it just didn't seem to be a good fit for our scenario.
Our situation is that we are not simply mirroring/syncing two
sites/directories and that seems to complicate things in using rsync.
As I would really like to get resume functionality with ssh (something
missing from OpenSSH sftp), I'm back researching rsync again.
I'm hoping someone can help me with my possible misconceptions about
rsync and how it might be able help with our situation.
Here's my scneario and questions.
1. Files stored on a remote server are moved to different local
servers and directories.
What decides which file goes where is based on the location and name of
the file on the remote server.
I'm assuming I'm going to need to create multiple rsync type jobs for
each of these scenarios?
Is there an issue with kicking off up to 15 of these types of jobs at
the same time?
2. As files are being moved, they need to have a temporary filename at
the destination until the file transfer has been completed.
I haven't found any docs that mentions this is how rsync works. Does
rsync handle files being updated/copied/moved/syncd in the same manner
(i.e. temporary file or location on destination until it has completed
3. When files have been transferred successfully, they are moved to an
archive and removed from the original source directory.
Not sure how to handle that with rsync. I need to move the file that
has been successfully transfered to an archive and deleted from it's
I'm guessing this could be scripted in that if the rsync job completes
successfully, then do the archiving of the files it just worked with.
I suppose if the rsync job fails, then I won't necessarily know which
files were copied successfully and those that weren't?
4. Once a file has been successfully transfered, it is processed by
3rd party apps that end up removing the file from the destination.
If this happens during an rsync job, wouldn't be the equivlant of
pulling the carpet out from underneath rsync while it's trying to sync
5. Files being transfered need to happen in a sequential order. Right
now the order of the files being transfered is determined by their
filename in AlphaNumeric order.
I'm assuming rsync is handling multiple files at the same time.
If anyone can help out with this, please let me know.