[nzlug] Large scale replication

[Jim Cheetham]

On 29/08/07, Cliff Pratt <enkidu at cliffp.com> wrote:
> Simon Lyall wrote:
> > If it really is millions of files then probably rsync is running out of
> > memory since it creates a list of the files to be transfered in memory
> > ( See http://samba.anu.edu.au/rsync/FAQ.html#5 ).
> >
> The data is (literally) millions of small PDFs. As you would suspect,
> most of them don't change and changes are incremental.

Hopefully all these files are in separate directories (if not, I hope
you're not on EXT3 :-) Reiser and XFS are better at large directories
I believe). So instead of one rsync covering the root directory, slice
it up into _n_ rsyncs with a smaller range of files ...

-jim