[nzlug] Large scale replication

[Michael Adams]

On Wed, 29 Aug 2007 09:09:47 +1200
Cliff Pratt wrote:

> Phillip Hutchings wrote:
> > On 8/28/07, Cliff Pratt <enkidu at cliffp.com> wrote:
> >> What do people use for large scale replication? IE
> >Terabytes/millions of> files?
> > 
> > rsync
> > 
> >> I've been trying to use 'rsync' but I'm less than impressed. It
> >runs for>   a couple of hours then dies either with no message at all
> >or a> 'communications error'. It seems size related as it works if I
> >copy> directories of around 30 - 50 GB but barfs at larger ones.
> >Looks like> I'll have to code something to use 'cp' and manually
> >compare and remove> stuff.
> > 
> > Sounds like the connection is being dropped somewhere, or the
> > receiving rsync process is being terminated, it'll take some
> > debugging. Are you using an rsync server, or just rsync over SSH?
> > How many firewalls? Are they stateful? Run top on both machines and
> > see what happens to the rsync processes.
> >
> It is extremely unlikely that it is a network problem since the
> machines are connected to the same switch. Also it reports transfer of
> roughly the same number of bytes each time before it fails, but not a
> single byte arrives in the target directories.
>  >
> >> Very unimpressed with 'rsync'. It gets such good reviews, too!
> > 
> > That's because it's a great tool!
> > 
> What worries me is the failure without error message. Since I can't 
> trust it to report failures, I'll have to write something to compare
> the directories, so I might as well use simple 'cp' commands!
> 

Turn on verbose with -v and try a dummy run with -n. The dummy run
should at least check the sending end.

-- 
Michael
Linux: The OS people choose without $200,000,000 of persuasion.