| Summary: | request new rsync tree for prefix project's portage tree | ||
|---|---|---|---|
| Product: | Gentoo Infrastructure | Reporter: | Fabian Groffen <grobian> |
| Component: | Other | Assignee: | Gentoo Infrastructure <infra-bugs> |
| Status: | RESOLVED WORKSFORME | ||
| Severity: | normal | CC: | darkside, grobian, haubi |
| Priority: | High | ||
| Version: | unspecified | ||
| Hardware: | Other | ||
| OS: | Other | ||
| URL: | http://www.gentoo.org/proj/en/gentoo-alt/prefix/ | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
|
Description
Fabian Groffen
2007-08-20 15:47:59 UTC
I'm not sure how others feel but I'm going to add my input anyway. I'm inclined to think this bug here should be marked as 'LATER' till such as the time as the project becomes a security supported arch. http://www.gentoo.org/security/en/vulnerability-policy.xml#doc_chap1 Then you have to take into account when your prefix pkg differs from the one in the portage tree and the glsa says XYZ is fixed when infact it's not. I think you would need a way to handle that. Also you would want to task somebody with being on the lookout for sec bugs that only affect your arches and be sure that glsa's get published for those to. Now to try and help you.. You could use a post_sync hook today to additionally rsync the metadata/glsa dir. Vs.. infra directly supporting this one of the dev boxes might be more fitting and could probably be used for this. solar, I plan on allowing rsync repos on the overlays box. The way I see it we can either waste cycles on svn for uses to sync this stuff or we can save some with rsync. We need a way for users of this special "tree" to understand that they are on their own though.... We discussed this in a bit more detail. providing a rsync tree for all of overlays.gentoo.org is doable. But the metadata stuff depends on a lot of factors that we would be happy to discuss with you in detail on -infra. ok, thanks for the hints, replies and willingness. With regard to the security aspects; for sure I don't have enough time to also maintain that for the whole of prefix. Frankly, I haven't looked at how how glsas exactly work, so no idea what I would need to change. I would just rely on the general main tree "maskings" and apply those in prefix. Note that the most important reason to have an rsync tree is that it makes it possible to do metadata/cache distribution, which allows a major speedup for normal users. Without it, rsync is as much usable to me as svn is now. I ran cache-tool.py just on the prefix tree, and it generated correct cache as far as we could see, without prefix-specific modifications. I can imagine though not all other overlays wanting/needing metadata/cache though. Also, it might need a prefix portage, I fail in that regard on my homework. I haven't researched those dependencies in-depth yet. Thanks. Maybe it can be implemented on Gentoo hardware in the future. For now I've gone my own way (haunted by svn errors) and set up an rsync mirror myself with which I can experiment and keep people happy. please don't close this bug. As a special case for prefix, since it's a standalone tree (vs. the rest of overlays), it would be more suited to the rsync rotation. With the new rsync1 box, I also have no objections to it being on the official (infra-run) rsync mirrors. Did you make reasonable progress with the GLSA and metadata issues? ok, sorry for closing. Progress so far is that with some help of zmedico (and his scripts) I have an entirely populated metadata directory for the prefix tree, generated like for the gentoo-x86 tree is done. As far as I am aware now, the rsync master mirror is fully complete now, and seems to work well. I push it every 10 minutes which seems to take 1 minute and 24 seconds from start till flipping the symlink which points to the updated rsync image. Let me know if you want info or my scripts that currently do it. So continuing the discussion from #gentoo-infra here: 1. Do we need to move the building process to a Gentoo box? 2. At the moment for boxes to serve it via rsync, I can offer boobie.g.o and potentially peacock, raven, crane (the last two might need some tuning first). 3. I'd have it on albatross (rsync1) just for distribution to the other boxes ideally (not serving users). (In reply to comment #8) > So continuing the discussion from #gentoo-infra here: > 1. Do we need to move the building process to a Gentoo box? Ideally, yes. There is no hurry involved, though. > 2. At the moment for boxes to serve it via rsync, I can offer boobie.g.o and > potentially peacock, raven, crane (the last two might need some tuning first). At the moment we use tinderbox.dev.gentoo.org, prefix.gentooexperimental.org and another server in germany. Expected load is not extreme: http://stats.prefix.freens.org/rsync-usage.png > 3. I'd have it on albatross (rsync1) just for distribution to the other boxes > ideally (not serving users). That would make it available on the general rsync slaves, right? That is the ideal situation. If necessary for any purpose, I am in the position to offer hardware, such as e.g. an rsync and/or distfiles server next to bugzilla. (On a sidenote, our company would be interested in that due to in-network advantages.) (In reply to comment #9) > > 2. At the moment for boxes to serve it via rsync, I can offer boobie.g.o and > > potentially peacock, raven, crane (the last two might need some tuning first). > At the moment we use tinderbox.dev.gentoo.org, prefix.gentooexperimental.org > and another server in germany. Expected load is not extreme: > http://stats.prefix.freens.org/rsync-usage.png For raven+crane it's memory pressure that I'm more worried about. boobie and peacock have a lot more free RAM. > > 3. I'd have it on albatross (rsync1) just for distribution to the other boxes > > ideally (not serving users). > That would make it available on the general rsync slaves, right? That is the > ideal situation. Available TO the rsync slaves, but they would have to update their configurations, and I'm not convinced that pushing it to every rsync server is wise - you need to be focusing on merging more aggressively into the main gentoo-portage tree. > If necessary for any purpose, I am in the position to offer hardware, such as > e.g. an rsync and/or distfiles server next to bugzilla. (On a sidenote, our > company would be interested in that due to in-network advantages.) You do know that there's already a rsync+distfile mirror at LeaseWeb right? But I certainly wouldn't object to adding an official distfiles.g.o rotation box in Europe (primarily to allow other EU mirrors to sync against it and cut the cross-Atlantic traffic down). (In reply to comment #10) > Available TO the rsync slaves, but they would have to update their > configurations, and I'm not convinced that pushing it to every rsync server is > wise - you need to be focusing on merging more aggressively into the main > gentoo-portage tree. For that, we need a Portage that understand the EAPI that we use. To get a Portage that understands our EAPI, we need the community to agree on our "features", as well as to finish the discussion on whether prefix is an "extension" or "overlay" EAPI. Only then we can merge back ebuilds, unfortunately. 1) The prefix team has its own set of mirrors now. 2) Progress is being made on merging back to gx86 at which point this request is moot, eventually. |
