Hola. Specify, section #4 of the doc needs modification- no manual intervention should occur for distfiles cleansing, sucker's carefully setup to handle it automatically, folks reading a doc and following the instructions screwing things up would be a bad thing ;)
actually... whole doc needs updating. aside from that, I'm probably adding mirror-dist to the tree since it's useful beyond just infra uses (now would be the time to sqawk if you dislike) ;)
Mind offering a diff? (or new file) :) If not, it'll probably take a bit to find time to update the doc.
Don't have a diff to offer- just stick in something as straight forward "for report of current master mirror status, go here". Main thing is getting the report linked up rather then me and a few others being the only ones to know about it...
I have a very rough draft started. If you would look at it and make sure that it is technically correct so far. Also, if you would look at the flow of the doc and make sure that it makes sense, is it presenting the info correctly or would you like a different format, etc... It's basically broken down into two sections. The first section is relevant to devs wanting to put files on the mirrors. The second section is for Infra. There is still info that I haven't put in yet but comments on anything you feel is important that I don't have yet would be welcome. Especially on the Step by Step part, I hacked that together and it needs to be more ummmm better :-) I do have my irc logs with ferringb and zmedico so the Common Fetch Errors part will be filled in. Any suggestions for that will be appreciated though. Information needed: * Where do the scripts reside on osprey (/usr/local/)? * Where is the failure.xml report being written to? * Where is the fail log (/var/log/$) ToDo: * Set up rsync module on osprey for failure.xml(not sure where this is on osprey) This needs to be done ASAP. * Set up rsync on loon to copy failure.xml to loon.gentoo.org:/var/www/www.gentoo.org/htdocs/dyn/ instead of dev.gentoo.org/~zmedico/infra/distfiles/ (I can do this as soon as the rsync on osprey is set up) * "--disable-central-pid-manager" is being used on mirror-dist.sh due to grsec issues on osprey, specifically waitpid failing in an unknown way. This needs to be researched and a permanent solution applied. (should I open a new bug for this or is there one already, other?) * gen-report-xml.py currently renders failure.xml directly instead of using an XSL transform. zmedico, open a new bug and give me details on how you want that accomplished. No account on osprey means I have no idea of whats inside the scripts so no clue how/what I should do for this. * Ramereth, is it possible for me to get an account on osprey with the proper perms? http://wwwredesign.gentoo.org/proj/en/infrastructure/mirrors/overview-distfile.xml In CVS - (Should show up in about an hour)
(In reply to comment #4) > * Where do the scripts reside on osprey (/usr/local/)? /home/distfiles/scripts > * Where is the failure.xml report being written to? /home/distfiles/reports/failure.xml > * Where is the fail log (/var/log/$) /home/distfiles/logs/{failure,success}.log > * gen-report-xml.py currently renders failure.xml directly instead of using > an XSL transform. zmedico, open a new bug and give me details on how you want > that accomplished. No account on osprey means I have no idea of whats inside > the scripts so no clue how/what I should do for this. Here is a tarball of the current scripts: http://dev.gentoo.org/~zmedico/infra/distfiles/distfiles-scripts.tar.bz2 I've opened bug 139213 to address the problem with gen-report-xml.py.
I have it fleshed out now I think. I'm still not quite sure about the entire process so please look carefully when you proofread: http://wwwredesign.gentoo.org/proj/en/infrastructure/mirrors/overview-distfile.xml Thanks for the tarball zac. I was hinting at ramereth to set me up an account but I appreciate it. ;-) Is there still an issue with dev.g.o:/space/distfiles-whitelist/ not being deleted correctly? Is someone doing it manually? Do we know how to fix it or does it need to be researched?
Bleh, pardon the delay. suggestions- 1) The description of the tier updating in section #1 is a bit off; implies it's a push setup, when in reality it's all mirrors servers doing pulls at offset cronjobs- it's not bandwidth issues that limit files getting on the mirrors, it's A) timing, B) unable to pull a file for the master image, C) broken distfiles mirror (no longer doing cronjob'd pulls). I'd rephrase that to reflect that, since the "bandwidth constraints" is a bit of smoke ;) 2) whitelist file wasn't designed to be maintained indefinitely- that was borked in the switch to the new d.g.o. Original agreed upon period for whitelist files was 6 months- after 6 months of whitelisting, have to move the files into the appropriate months whitelisting to continue exempting them. If y'all are changing it, hey cool, just commenting on what it was originally setup as. 3) common fetch errors addition- host isn't valid (happens a lot), related, host is too laggy for usage (timeouts on fetching). Aside from that, looks good, ty curtis.
Thanks ferringb, I've completed the changes and commited (in CVS). I've left the bug open pending resolution of bug#135098. I will update this doc when that bug is fixed.
"Instructions for new developers"[1] also needs to be updated. [1] http://www.gentoo.org/proj/en/devrel/new-dev-training.xml#doc_chap2
(In reply to comment #9) > "Instructions for new developers"[1] also needs to be updated. > > [1] http://www.gentoo.org/proj/en/devrel/new-dev-training.xml#doc_chap2 What specifically needs to be updated?
(In reply to comment #10) > (In reply to comment #9) > > "Instructions for new developers"[1] also needs to be updated. > > > > [1] http://www.gentoo.org/proj/en/devrel/new-dev-training.xml#doc_chap2 > > What specifically needs to be updated? The "Putting files on mirrors" section tells the developer that copying a file to dev.gentoo.org:/space/distfiles-local is the proper way of getting files to the mirrors. That, according to overview-distfile.xml, is wrong. Files are automatically fetched and there is no need to copy them to dev.g.o yourself unless they are for distfiles with mirror://. It should be updated to reflect what's on overview-distfile.xml.
(In reply to comment #11) > (In reply to comment #10) > > (In reply to comment #9) > > > "Instructions for new developers"[1] also needs to be updated. > > > > > > [1] http://www.gentoo.org/proj/en/devrel/new-dev-training.xml#doc_chap2 > > > > What specifically needs to be updated? > > The "Putting files on mirrors" section tells the developer that copying a file > to dev.gentoo.org:/space/distfiles-local is the proper way of getting files to > the mirrors. That, according to overview-distfile.xml, is wrong. Files are > automatically fetched and there is no need to copy them to dev.g.o yourself > unless they are for distfiles with mirror://. It should be updated to reflect > what's on overview-distfile.xml. > I updated that doc. See bug#140138
The new-dev-training.xml seems to be duplicated in the handbook: http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?part=1&chap=4 It may be better to remove the duplication, because it needs more than just fixing the distfile instructions.
(In reply to comment #13) > The new-dev-training.xml seems to be duplicated in the handbook: > http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?part=1&chap=4 > > It may be better to remove the duplication, because it needs more than just > fixing the distfile instructions. > I updated that page in the dev-handbook so it's fixed in the short term. Long term someone in devrel should make a decision and probably remove /proj/en/devrel/new-dev-training.xml (since it's basically the same exact information as /proj/en/devrel/handbook/hb-introduction-new-devs.xml). If you do make sure to let GDP know **FIRST** since it will break the metadoc which causes havoc on the website.
(In reply to comment #14) > Long term someone in devrel should make a decision and probably remove > /proj/en/devrel/new-dev-training.xml (since it's basically the same exact > information as /proj/en/devrel/handbook/hb-introduction-new-devs.xml). > > If you do make sure to let GDP know **FIRST** since it will break the metadoc > which causes havoc on the website. FYI, it b0rks the doc index, not the whole web site. It wouldn't be a problem with the current version of gorg. We might upgrade before Easter 2009. Until then, please do let GDP know *before* you remove the file.
zmedico: what's still relevant in this bug, or can we close it?
(In reply to comment #16) > zmedico: what's still relevant in this bug, or can we close it? > ping ^^
I guess we can close it. @ferringb: Are we done here? I guess you can update overview-distfile.xml yourself now if necessary.
closing bug per previous comments.
