Message en plusieurs parties au format MIME
--===============1470178700==
Content-Type: multipart/alternative;
boundary="=_alternative 00349CEEC125709F_="

Message en plusieurs parties au format MIME
--=_alternative 00349CEEC125709F_=
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: quoted-printable

Hi,=20

thank you for your answer.

But what do you mean by "slow rsync"? Do you mean limiting bandwith? I=20
think this will not be an option with 100Gb of data to synchronise...

However, I like your idea with the LAN: I think I will bring my mirror to=20
the remote site :-)

Just for the record, I didn't made the standard mistake, having things one =

directory level off from where they should be.

I was hoping doing a "scp -rp" would preserve all the necessary=20
information needed by rsync but apparently not... Maybe I should use tar?

Regards,

Cyrille

a =E9crit sur 19/10/2005 11:00:07 :

> Standard mistake is to somehow get things one directory level off=20
> from where they should be.
>=20
> If you have the available disk space (both ends)
> Do a fast rsync from the primary to locally available fast disk=20
> space (same computer or LAN)
> Do a slow rsync from that disk space to another set of available=20
> disk space over slow/unreliable network.
> Assuming the slow rsync completed without problems,=20
> do a fast rsync from the second disk space to the legitimate=20

target..
>=20
> Source makes a big distinction between foo/ and foo
> They both transfer the same files.
> They differ in where they transfer the file to (BY ONE LEVEL)
> Both make sense and are quite reasonable.
> If you do both, the effects are similar to untarring twice with one=20
> level difference between them.
>=20
> I know I've done it a number of times, and it is no fun over remote=20
> and often bad internet connections.
> With large volumes, it is not apparent when you've done it to yourself.
>=20
>=20
> WARNING -- these scripts work but are probably not the best nor the
> most aesthetic.
> ---just a wee bit too informative and I'm too lazy to sanitize so=20
> I'm not posting to the list.---
> ?? hosts allow/deny ignore with auth users present ??=20
> The major sucurity concerns are NOT permanently losing all my data=20
> rather than eavesdroppers.
> The internet connection(s) are ocassionally flaky to non-existent,=20
> and everything needs to assume
> that they will chose the most inopportune moment to do something I don't =


like.
> Essentially the stunt is that /home/rsync-foo is rsync'd with other=20
> such over slow and bad connections.
> With bad internet I've got (not shown) multiple off-site backups AND
> the ability to bring them all upt-to-date
> if there is a connection (any, not necessarily the preferred -- some
> connect, some don't connect)
>=20
>=20
>=20
> ---Timing depends on what is different. Typically this one is about
> 10 to 15 minutes (about 10G) total.
> If large MySql tables are rearranged, tends to take rather longer.
> real 192m41.906s
>=20
>=20
> real 12m18.512s
> sent 13750 bytes received 7577551 bytes 10279.35 bytes/sec
> total size is 14013850399 speedup is 1846.04
> Tue Oct 18 07:28:00 CDT 2005
> rsync-sjs-dwg
> [root@wf1 /root]# cat rsync-sjs-dwg
> #!/bin/sh
>=20
> rsync client
> # staged backup
> time rsync -a /home/rsync-sjs-dwg/* /home/rsync-sjs-dwg-1bk/
>=20
> time rsync -avz --progress --partial --timeout=3D1750 --password-
> file=3D/etc/rsync.secrets/rsync-sjs-dwg \
> rsync-sjs-dwg@sjs::rsync-sjs-dwg/* /home/rsync-sjs-dwg/
> date; echo rsync-sjs-dwg
>=20
> ----rsync server (actually is an intermediate, not the primary)
> This is an extra "junk" computer with a decent hard drive.
> This serves to get a fast=20
> # etc/rsyncd.conf
> pid file =3D /etc/rsyncd.pid
> [rsync-sjs-dwg]
> comment =3D backup of sjs dwg
> path =3D /home/rsync-sjs-dwg
> use chroot =3D true
> read only =3D false
> list =3D false
> uid =3D root
> gid =3D root
> auth users =3D rsync-sjs-dwg
> secrets file =3D /etc/rsyncd.secrets
> strict modes =3D true
> hosts allow =3D 10.0.0.1 pete-162 wf
> hosts deny =3D *.*.*.*
> ---rsync "client" (This is a PRIMARY)
> [root@sys001 root]# cat rsync-sjs-dwg
> #!/bin/bash
> # rsync-sjs-dwg SJS=5FDrawings/ title/
> mkdir -p /tmp/rsync ; echo `hostname` > /tmp/rsync/OPENED
> rsync -a --password-file=3D/etc/rsync.secrets/sjs-dwg --timeout=3D750 \
> /tmp/rsync/OPENED rsync-sjs-dwg@10.0.0.170::rsync-sjs-dwg/
> for name in title SJS=5FDrawings ; do
> rsync -a --password-file=3D/etc/rsync.secrets/sjs-dwg --timeout=3D750 \
> /home/dwg/$name rsync-sjs-dwg@10.0.0.170::rsync-sjs-dwg/
> done
> mkdir -p /tmp/rsync ; echo `hostname` > /tmp/rsync/CLOSED
> rsync -a --password-file=3D/etc/rsync.secrets/sjs-dwg --timeout=3D750 \
> /tmp/rsync/CLOSED rsync-sjs-dwg@10.0.0.170::rsync-sjs-dwg/
> echo 'rsync-sjs-dwg'
> date
> date -u
>=20
>=20
>=20
>=20
>=20
> -----Original Message-----
> From: rsync-bounces+tony=3Dservacorp.com@lists.samba.org [mailto:
> rsync-bounces+tony=3Dservacorp.com@lists.samba.org]On Behalf Of Cyrille=20

Bollu
> Sent: Wednesday, October 19, 2005 2:58 AM
> To: rsync@lists.samba.org
> Subject: Initialising a mirror


>=20
> Dears,=20
>=20
> I would like to mirror data from 2 servers connected together via=20
> VPN over ADSL lines (dwn 2Mbps/Upld 512Kbps).=20
>=20
> I'm sure rsync is one of the best tool to keep these data in sync=20
> but how should I use it to initialise the mirror?=20
>=20
> I'm currently testing the solution with 10Gb of data to keep in=20
> sync. But on my lines it would take more than 40 hours to initially=20
> create the mirror! And in production the data to keep in sync will=20
> reach 100Gb... :-(=20
>=20
> So, I went with a laptop on the remote site and "scp -rp" the data=20
> on it. I then got back to my site and "scp -rp" it on the mirror to=20
> initialise it.=20
>=20
> But apparently I forgot something because when rsync still sends all
> the files from the remote site to my mirror when I first run it.=20
>=20
> Does anybody has an idea about what I'm doing wrong?=20
>=20
> Does anybody has a solution to this problem?=20
>=20
> Regards,=20
>=20
> Cyrille

--=_alternative 00349CEEC125709F_=
Content-Type: text/html; charset="ISO-8859-1"
Content-Transfer-Encoding: quoted-printable



Hi,



thank you for your answer.



But what do you mean by "slow rsync&=
quot;?
Do you mean limiting bandwith? I think this will not be an option with
100Gb of data to synchronise...




However, I like your idea with the LAN: I
think I will bring my mirror to the remote site :-)




Just for the record, I didn't made the st=
andard
mistake, having things one directory level off from where they should be. font>



I was hoping doing a "scp -rp"
would preserve all the necessary information needed by rsync but apparently
not... Maybe I should use tar?




Regards,



Cyrille



<Tony@ServaCorp.com> a =E9crit sur 19/10/2005
11:00:07 :



> Standard mistake is to somehow get things one directory level off


> from where they should be.


>  

> If you have the available disk space (both ends=
)


> Do a fast rsync from the primary to locally ava=
ilable
fast disk

> space (same computer or LAN)


> Do a slow rsync from that disk space to another
set of available

> disk space over slow/unreliable network.


> Assuming the slow rsync completed without probl=
ems,


>     do a fast rsync from the second
disk space to the legitimate target..


>  

> Source makes a big distinction between foo/ and
foo


> They both transfer the same files.

> They differ in where they transfer the file to
 (BY ONE LEVEL)


> Both make sense and are quite reasonable.<=
/font>

> If you do both, the effects are similar to unta=
rring
twice with one

> level difference between them.


>  

> I know I've done it a number of times, and it
is no fun over remote

> and often bad internet connections.


> With large volumes, it is not apparent when you=
've
done it to yourself.


>  

>  

> WARNING  -- these scripts work but are pro=
bably
not the best nor the

> most aesthetic.


> ---just a wee bit too informative and I'm too
lazy to sanitize so

> I'm not posting to the list.---


>     ?? hosts allow/deny ignore with
auth users present ??


> The major sucurity concerns are NOT permanently
losing all my data

> rather than eavesdroppers.


> The internet connection(s) are ocassionally fla=
ky
to non-existent,

> and everything needs to assume


> that they will chose the most inopportune moment
to do something I don't like.


> Essentially the stunt is that /home/rsync-foo
is rsync'd with other

> such over slow and bad connections.


> With bad internet I've got (not shown) multiple
off-site backups AND

> the ability to bring them all upt-to-date


> if there is a connection (any, not necessarily
the preferred -- some

> connect, some don't connect)


>  

>  

>  

> ---Timing depends on what is different.  T=
ypically
this one is about

> 10 to 15 minutes  (about 10G) total.


> If large MySql tables are rearranged, tends to
take rather longer.


> real    192m41.906s

>  

>  

> real    12m18.512s

> sent 13750 bytes  received 7577551 bytes  10279.35 bytes/sec=


> total size is 14013850399  speedup is 1846.04

> Tue Oct 18 07:28:00 CDT 2005

> rsync-sjs-dwg


> [root@wf1 /root]# cat rsync-sjs-dwg

> #!/bin/sh


>  

> rsync client

> # staged backup

> time rsync -a /home/rsync-sjs-dwg/* /home/rsync-sjs-dwg-1bk/
t>

>  

> time rsync -avz --progress --partial --timeout=
=3D1750
--password-

> file=3D/etc/rsync.secrets/rsync-sjs-dwg \

>         rsync-sjs-dwg@sjs::rsync-sjs-dwg/* /home/r=
sync-sjs-dwg/


> date; echo rsync-sjs-dwg

>  

> ----rsync server   (actually is an interme=
diate,
not the primary)


> This is an extra "junk" computer with
a decent hard drive.


> This serves to get a fast

> # etc/rsyncd.conf

> pid file =3D /etc/rsyncd.pid

> [rsync-sjs-dwg]

>         comment =3D backup of sjs dwg

>         path =3D /home/rsync-sjs-dwg

>         use chroot =3D true

>         read only =3D false

>         list =3D false

>         uid =3D root

>         gid =3D root

>         auth users =3D rsync-sjs-dwg

>         secrets file =3D /etc/rsyncd.secrets

>         strict modes =3D true

>         hosts allow =3D 10.0.0.1 pete-162 wf

>         hosts deny =3D *.*.*.*


> ---rsync "client"  (This is a
PRIMARY)


> [root@sys001 root]# cat rsync-sjs-dwg

> #!/bin/bash

> # rsync-sjs-dwg  SJS=5FDrawings/ title/

> mkdir -p /tmp/rsync ; echo `hostname` > /tmp/rsync/OPENED

> rsync -a --password-file=3D/etc/rsync.secrets/sjs-dwg  --timeout=
=3D750
\

>         /tmp/rsync/OPENED      
rsync-sjs-dwg@10.0.0.170::rsync-sjs-dwg/


> for name in title SJS=5FDrawings ; do

> rsync -a --password-file=3D/etc/rsync.secrets/sjs-dwg --timeout=3D750
\

>         /home/dwg/$name        
rsync-sjs-dwg@10.0.0.170::rsync-sjs-dwg/

> done


> mkdir -p /tmp/rsync ; echo `hostname` > /tmp=
/rsync/CLOSED

> rsync -a --password-file=3D/etc/rsync.secrets/sjs-dwg  --timeout=
=3D750
\

>         /tmp/rsync/CLOSED      
rsync-sjs-dwg@10.0.0.170::rsync-sjs-dwg/


> echo 'rsync-sjs-dwg'

> date

> date -u


>  

>

>  


>  

>  

> -----Original Message-----

> From: rsync-bounces+tony=3Dservacorp.com@lists.samba.org [mailto:

> rsync-bounces+tony=3Dservacorp.com@lists.samba.org]On Behalf Of Cyrille
Bollu

> Sent: Wednesday, October 19, 2005 2:58 AM

> To: rsync@lists.samba.org

> Subject: Initialising a mirror



>

> Dears,

>

> I would like to mirror data from 2 servers connected together via


> VPN over ADSL lines (dwn 2Mbps/Upld 512Kbps).

>

> I'm sure rsync is one of the best tool to keep these data in sync


> but how should I use it to initialise the mirror?

>

> I'm currently testing the solution with 10Gb of data to keep in

> sync. But on my lines it would take more than 40 hours to initially


> create the mirror! And in production the data to keep in sync will


> reach 100Gb... :-(

>

> So, I went with a laptop on the remote site and "scp -rp"
the data

> on it. I then got back to my site and "scp -rp" it on the
mirror to

> initialise it.

>

> But apparently I forgot something because when rsync still sends all r>
> the files from the remote site to my mirror when I first run it.

>

> Does anybody has an idea about what I'm doing wrong?

>

> Does anybody has a solution to this problem?

>

> Regards,

>

> Cyrille

--=_alternative 00349CEEC125709F_=--

--===============1470178700==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--===============1470178700==--