The Gentoo/Alt:Prefix project is currently using a portage tree which mirrors the main portage tree (gentoo-x86) with for every ebuild some small to large modifications. The tree grows and grows, as users request for new packages, and prefix development is made to explore new areas which drag in a lot of packages such as Xorg, python, perl, Java, etc. The tree modifications make the ebuilds incompatible with the gentoo-x86 ones, and the latter ones unusable for Prefixed Portage. The current tree is "big" and, looking into the future, is expected to become even bigger, making Portage slower than necessary for users, because the lack of metadata cache in the tree. Experiments have been made to put the metadata cache in the SVN tree. With support of Zac Medico, it was tried to make it work, but deemed impossible without a complete change of code, since SVN doesn't retain mtime on files which causes each checkout to differ in mtimes, which Portage looks at when determining up-to-dateness of an cache entry. Also the growing amount of users syncing their trees via anonymous SVN on overlays.gentoo.org (where the prefix tree is hosted) doesn't do the performance (of that box) good. Last but not least, users are requesting tools such as glsa-check to work, which requires metadata/glsa to be available in the portage tree. Since we cannot use the gentoo-x86 tree here, the need for a generated tree is there. Because of this, I would like to get an rsync-able tree for the Prefix project that uses the prefix SVN tree (may be moved to another box if that is necessary), runs cache-tool.py and does the other stuff that populates this tree, like rsync1 does. Of course, I don't expect a broad mirroring as the gentoo-x86 rsync tree, one place to rsync from should be fine IMO. I simply don't have the necessary upstream bandwith myself at home, hence my request here.
I'm not sure how others feel but I'm going to add my input anyway. I'm inclined to think this bug here should be marked as 'LATER' till such as the time as the project becomes a security supported arch. http://www.gentoo.org/security/en/vulnerability-policy.xml#doc_chap1 Then you have to take into account when your prefix pkg differs from the one in the portage tree and the glsa says XYZ is fixed when infact it's not. I think you would need a way to handle that. Also you would want to task somebody with being on the lookout for sec bugs that only affect your arches and be sure that glsa's get published for those to. Now to try and help you.. You could use a post_sync hook today to additionally rsync the metadata/glsa dir. Vs.. infra directly supporting this one of the dev boxes might be more fitting and could probably be used for this.
solar, I plan on allowing rsync repos on the overlays box. The way I see it we can either waste cycles on svn for uses to sync this stuff or we can save some with rsync. We need a way for users of this special "tree" to understand that they are on their own though....
We discussed this in a bit more detail. providing a rsync tree for all of overlays.gentoo.org is doable. But the metadata stuff depends on a lot of factors that we would be happy to discuss with you in detail on -infra.
ok, thanks for the hints, replies and willingness. With regard to the security aspects; for sure I don't have enough time to also maintain that for the whole of prefix. Frankly, I haven't looked at how how glsas exactly work, so no idea what I would need to change. I would just rely on the general main tree "maskings" and apply those in prefix. Note that the most important reason to have an rsync tree is that it makes it possible to do metadata/cache distribution, which allows a major speedup for normal users. Without it, rsync is as much usable to me as svn is now. I ran cache-tool.py just on the prefix tree, and it generated correct cache as far as we could see, without prefix-specific modifications. I can imagine though not all other overlays wanting/needing metadata/cache though. Also, it might need a prefix portage, I fail in that regard on my homework. I haven't researched those dependencies in-depth yet. Thanks.
Maybe it can be implemented on Gentoo hardware in the future. For now I've gone my own way (haunted by svn errors) and set up an rsync mirror myself with which I can experiment and keep people happy.
please don't close this bug. As a special case for prefix, since it's a standalone tree (vs. the rest of overlays), it would be more suited to the rsync rotation. With the new rsync1 box, I also have no objections to it being on the official (infra-run) rsync mirrors. Did you make reasonable progress with the GLSA and metadata issues?
ok, sorry for closing. Progress so far is that with some help of zmedico (and his scripts) I have an entirely populated metadata directory for the prefix tree, generated like for the gentoo-x86 tree is done. As far as I am aware now, the rsync master mirror is fully complete now, and seems to work well. I push it every 10 minutes which seems to take 1 minute and 24 seconds from start till flipping the symlink which points to the updated rsync image. Let me know if you want info or my scripts that currently do it.
So continuing the discussion from #gentoo-infra here: 1. Do we need to move the building process to a Gentoo box? 2. At the moment for boxes to serve it via rsync, I can offer boobie.g.o and potentially peacock, raven, crane (the last two might need some tuning first). 3. I'd have it on albatross (rsync1) just for distribution to the other boxes ideally (not serving users).
(In reply to comment #8) > So continuing the discussion from #gentoo-infra here: > 1. Do we need to move the building process to a Gentoo box? Ideally, yes. There is no hurry involved, though. > 2. At the moment for boxes to serve it via rsync, I can offer boobie.g.o and > potentially peacock, raven, crane (the last two might need some tuning first). At the moment we use tinderbox.dev.gentoo.org, prefix.gentooexperimental.org and another server in germany. Expected load is not extreme: http://stats.prefix.freens.org/rsync-usage.png > 3. I'd have it on albatross (rsync1) just for distribution to the other boxes > ideally (not serving users). That would make it available on the general rsync slaves, right? That is the ideal situation. If necessary for any purpose, I am in the position to offer hardware, such as e.g. an rsync and/or distfiles server next to bugzilla. (On a sidenote, our company would be interested in that due to in-network advantages.)
(In reply to comment #9) > > 2. At the moment for boxes to serve it via rsync, I can offer boobie.g.o and > > potentially peacock, raven, crane (the last two might need some tuning first). > At the moment we use tinderbox.dev.gentoo.org, prefix.gentooexperimental.org > and another server in germany. Expected load is not extreme: > http://stats.prefix.freens.org/rsync-usage.png For raven+crane it's memory pressure that I'm more worried about. boobie and peacock have a lot more free RAM. > > 3. I'd have it on albatross (rsync1) just for distribution to the other boxes > > ideally (not serving users). > That would make it available on the general rsync slaves, right? That is the > ideal situation. Available TO the rsync slaves, but they would have to update their configurations, and I'm not convinced that pushing it to every rsync server is wise - you need to be focusing on merging more aggressively into the main gentoo-portage tree. > If necessary for any purpose, I am in the position to offer hardware, such as > e.g. an rsync and/or distfiles server next to bugzilla. (On a sidenote, our > company would be interested in that due to in-network advantages.) You do know that there's already a rsync+distfile mirror at LeaseWeb right? But I certainly wouldn't object to adding an official distfiles.g.o rotation box in Europe (primarily to allow other EU mirrors to sync against it and cut the cross-Atlantic traffic down).
(In reply to comment #10) > Available TO the rsync slaves, but they would have to update their > configurations, and I'm not convinced that pushing it to every rsync server is > wise - you need to be focusing on merging more aggressively into the main > gentoo-portage tree. For that, we need a Portage that understand the EAPI that we use. To get a Portage that understands our EAPI, we need the community to agree on our "features", as well as to finish the discussion on whether prefix is an "extension" or "overlay" EAPI. Only then we can merge back ebuilds, unfortunately.
1) The prefix team has its own set of mirrors now. 2) Progress is being made on merging back to gx86 at which point this request is moot, eventually.
