Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 365803 - Switch portage from rsync to git
Summary: Switch portage from rsync to git
Status: RESOLVED DUPLICATE of bug 333531
Alias: None
Product: Gentoo Infrastructure
Classification: Unclassified
Component: Other (show other bugs)
Hardware: All All
: Normal enhancement (vote)
Assignee: Gentoo Infrastructure
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-05-03 11:57 UTC by Rafal Kupiec
Modified: 2011-09-18 08:39 UTC (History)
7 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Rafal Kupiec 2011-05-03 11:57:33 UTC
Portage tree should be distributed via GIT instead of RSYNC, what give few more advantages over current syncing method:
* GIT is much more faster than RSYNC
* GIT provides SHA1 digests, so no meed to place them in manifest files, what makes whole portage much lightweight.
* Removing useless ebuild checksums will need less traffic to sync (only for distfiles required)
* Easier to crate local copy of portage tree for computers using gentoo in lan


For a while, there could be 2 versions of portage tree: 1 available via classic rsync, and 2nd via git, what would allow people to easily migrate. This could be implemented as in funtoo with FEATURES="mini-manifest"

Reproducible: Always
Comment 1 Chí-Thanh Christopher Nguyễn gentoo-dev 2011-05-03 12:01:12 UTC
rsync -avz -e ssh --delete --exclude="distfiles/*" /usr/portage root@otherhost:/usr

works fine here to distribute the portage tree to other hosts
Comment 2 Rafal Kupiec 2011-05-03 12:05:36 UTC
But still slower and needs more traffic, and probably more time to generate manifests as they need to include checksums of patches and ebuilds. As i have already written, git creates SHA1 checksums for everything put inside, so no need to have digests of ebuilds or patches.

This makes whole portage tree:
* less space consuming
* less traffic consuming
* less time consuming

This is enough to tell that GIT is much better than RSYNC.
Comment 3 Alex Legler (RETIRED) archtester gentoo-dev Security 2011-05-03 12:24:18 UTC
(In reply to comment #2)
> But still slower and needs more traffic, and probably more time to generate
> manifests as they need to include checksums of patches and ebuilds. As i have
> already written, git creates SHA1 checksums for everything put inside, so no
> need to have digests of ebuilds or patches.

And how do you sign your git-created SHA1 checksums?

> 
> This makes whole portage tree:
> * less space consuming
> * less traffic consuming
> * less time consuming
> 
> This is enough to tell that GIT is much better than RSYNC.

It looks all nice and speedy on funtoo, huh? github provides the server, and you're happy.
Have you compared server load on a large scale? You know, we have several dozens of mirror providers to convince to deploy git stuff for us.
Comment 4 Jeroen Roovers (RETIRED) gentoo-dev 2011-05-06 16:03:04 UTC
(In reply to comment #1)
> rsync -avz -e ssh --delete --exclude="distfiles/*" /usr/portage
> root@otherhost:/usr
> 
> works fine here to distribute the portage tree to other hosts

Actually, while we're being completely off topic anyway, I've been using NFS for that for years. One copy is enough and the client hosts of that tree only need to regenerate metadata. :)
Comment 5 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2011-05-06 16:15:34 UTC
On behalf of infra, this is a dupe.

*** This bug has been marked as a duplicate of bug 333531 ***