From my understanding of the rsync changes, one could safe a lot of data transfer since rsync-3.0.0_pre* (if the server is new enough) if the option --delete-after is replaced by --del/--delete-during. Unfortunately, rsync does not support overriding earlier given options, i.e. e.g. adding --del to PORTAGE_RSYNC_EXTRA_OPTS is not possible. Of course, one can copy the whole PORTAGE_RSYNC_OPTS into /etc/make.conf and replace the option manually, but this seems rather hackish for such a useful thing. So I suggest that --delete-after should somehow be made optional (e.g. by putting this option in an extra variable). It might be perhaps even reasonable for newest portage to make --del the new default (which unfortunately is only possible if portage depends on >=rsync-3.0.0_pre1)
I don't see how this saves on data-transfer, since it is only about when the receiver deletes files which are no longer at the sender side. IOW it doesn't mean less files are transferred. man rsync here says it is "supported beginning with rsync version 2.6.4" btw. Wrt making the options more configurable; I'd have to say I think the current option (setting your own PORTAGE_RSYNC_OPTS) is enough. You only need to do it once (or whenever you want things to change) and it's not too much to ask of a Gentoo user imo. OFC if this were truly something that led to less data-transfer, I'd think it would simply be included in the default, since it would help Gentoo mirrors. (Hope I haven't misunderstood; I don't have rsync 3 installed yet.)
(In reply to comment #1) > I don't see how this saves on data-transfer, since it is only about when the > receiver deletes files which are no longer at the sender side. It prevents an improved algorithm to be used. From the rsync manpage: : Beginning with rsync 3.0.0, the recursive algorithm used is now : an incremental scan [...] Some options require rsync to know the full : file list, so these options disable the incremental recursion mode. : These include: [...] --delete-after [...] Because of this, the default : delete mode when you specify --delete is now --delete-during As I understand it, the new algorithm essentially saves the first pass of data transfer. However, this only an "educated guess", since the documentation is not very clear about it, and I did neither analyze the rsync sources/protocoll (e.g. whether filenames were actually transferred twice) nor do I have any means installed to analyze the actual traffic. Nevertheless my guess corresponds to the time needed for syncing over my not-so-fast connection (if I hit a mirror which supports the new protocoll). > OFC if this were truly something that led to less data-transfer, I'd think it > would simply be included in the default, since it would help Gentoo mirrors. Actually, I wanted to suggest this first, but it will break for users of older rsync versions. And the new versions are all ~ARCH... > Wrt making the options more configurable; I'd have to say I think the current > option (setting your own PORTAGE_RSYNC_OPTS) is enough. Yes, but setting PORTAGE_RSYNC_OPTS means that one will miss changes in the defaults (unless one checks manually).
(In reply to comment #2) > It prevents an improved algorithm to be used. Irrelevant until every single official rsync mirror in our rotations out there supports this; end of story...
(In reply to comment #3) > > It prevents an improved algorithm to be used. > > Irrelevant until every single official rsync mirror in our rotations out > there supports this Maybe there is a misunderstanding: The new option does not harm if the rsync *server* does not support this - in this case rsync automatically falls back to the old algorithm, i.e. for mirrors without new rsync there is simply no visible change in behavior (except that the deletion of files on the client takes place during the transfer instead of afterwards); however mirrors with new rsync servers will profit. The only problem which would occur with changed defaults would be that the *client* fails if it is not new enough to understand the --delete-during option.
(In reply to comment #2) > It prevents an improved algorithm to be used. From the rsync manpage: > Ah OK; now I am with you. > > Wrt making the options more configurable; I'd have to say I think the current > > option (setting your own PORTAGE_RSYNC_OPTS) is enough. > > Yes, but setting PORTAGE_RSYNC_OPTS means that one will miss changes in the > defaults (unless one checks manually). > I agree that method to get it in is not satisfactory; I had to be sure it would save on bandwidth. Seems there are two options: 1) Wait for rsync3 to be stable, so new versions of portage can depend on it and use --del, and in the meantime advanced users can use the standard config. 2) Add another variable for the delete option. I agree this is cleaner but it takes dev time. A patch would really help ;-) It might be worth you posting info on it in the forums (Docs, Tips & Tricks) so that others are aware of it in the meantime; I'll certainly be using --delete-during, so thanks from me :D
I just removed --delete-after from the default PORTAGE_RSYNC_OPTS. This allows the rsync program to choose the preferred --delete algorithm, or the user can choose one via PORTAGE_RSYNC_EXTRA_OPTS.
I'm pretty sure there was a damn good reason we had --delete-after in the rsync command, if I could only remember it ...
This has been released in 2.1.4.
