Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 135098 - distfiles-mirroring report is horked.
Summary: distfiles-mirroring report is horked.
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Infrastructure
Classification: Unclassified
Component: Other (show other bugs)
Hardware: All Linux
: High normal (vote)
Assignee: Gentoo Infrastructure
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: 118952
  Show dependency tree
 
Reported: 2006-05-31 19:56 UTC by Brian Harring (RETIRED)
Modified: 2007-12-18 01:56 UTC (History)
3 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Brian Harring (RETIRED) gentoo-dev 2006-05-31 19:56:17 UTC
http://dev.gentoo.org/distfile-mirroring/failure.xml is 404; horkage point seems to have been 02/06/06 going by google cache roughly (last copy it knows of).

Sucker works by generating reports and shoving them in a dir, then said dir is rsync pulled from d.g.o- since dev.gentoo.org/distfile-mirroring/ is 404 (rather then 403, which is what it used to chuck since there was no index), looks like either url rewriting got screwed up, or directories were shifted around.

Report generation is fine (zmedico verified it)- most likely the transfer to wood pecker left out this specific bit (first bet would be the cronjob'd rsync'ing got left out).

Meanwhile, cc'ing QA- y'all probably would have use for the report for spotting folks commiting ebuilds without uploading for when upstream SRC_URI is extremely crappy (berlios.de, in this case).
Comment 1 Lance Albertson (RETIRED) gentoo-dev 2006-05-31 20:00:04 UTC
I suggest we move the file from syncing on dev.g.o to www.g.o much like how the devaway stuff is. It should be a simple thing to add to the nodes for the most part. That way its much more centralized and zmedico/qa can create a page linking to it.

Thoughts?
Comment 2 Brian Harring (RETIRED) gentoo-dev 2006-05-31 21:36:55 UTC
works for me (the d.g.o location was a bit obscure/hackish anyways ;)
Comment 3 Lance Albertson (RETIRED) gentoo-dev 2006-05-31 22:15:54 UTC
(In reply to comment #2)
> works for me (the d.g.o location was a bit obscure/hackish anyways ;)

Yup, totally agree. I'll put that on the todo list.
Comment 4 Brian Harring (RETIRED) gentoo-dev 2006-06-21 13:56:08 UTC
*nudge*
status?  see bug #134938, easy to state what's going on if the report is available (no report, means have to track down zmedico which wastes his time for trivial stuff)....
Comment 5 Zac Medico gentoo-dev 2006-06-21 20:50:28 UTC
As a temporary solution, until we have an official place for this, I've setup a local cron job so that osprey.gentoo.org:/home/distfiles/reports/ is synced to my dev space:

http://dev.gentoo.org/~zmedico/infra/distfiles/failure.xml
Comment 6 Brian Harring (RETIRED) gentoo-dev 2006-07-02 03:44:32 UTC
(In reply to comment #3)
> (In reply to comment #2)
> > works for me (the d.g.o location was a bit obscure/hackish anyways ;)
> 
> Yup, totally agree. I'll put that on the todo list.

any progress?
Comment 7 Lance Albertson (RETIRED) gentoo-dev 2006-07-02 08:46:19 UTC
(In reply to comment #6)
> (In reply to comment #3)
> > (In reply to comment #2)
> > > works for me (the d.g.o location was a bit obscure/hackish anyways ;)
> > 
> > Yup, totally agree. I'll put that on the todo list.
> 
> any progress?

Nope, still on the todo list. Don't expect this to get done quickly. We have several other things on our list to get done before this. So please be patient
Comment 8 Brian Harring (RETIRED) gentoo-dev 2006-08-13 08:39:49 UTC
(In reply to comment #7)
> Nope, still on the todo list. Don't expect this to get done quickly. We have
> several other things on our list to get done before this. So please be patient

H'okay... been 3 months for a one line cronjob entry.

Seriously, what's the hold up here?
Comment 9 Kurt Lieber (RETIRED) gentoo-dev 2006-08-13 08:56:04 UTC
where's the script that needs to be run and what's the cronjob entry for running it?  I know nothing about this whole thing (first time I've seen the bug) but if this is just a matter of adding another script to the www.g.o rotation and someone can provide some pretty clear instructions on what needs to be done, I'll add it.
Comment 10 Zac Medico gentoo-dev 2006-08-13 11:20:16 UTC
The report, located at osprey.gentoo.org:/home/distfiles/reports/failure.xml, is generated every 4 hours by the distfiles user's only cron job:

20 */4 * * *    /usr/local/bin/pidlock /home/distfiles/scripts/update_distfiles.sh

The failure.xml file should be synced to some webspace somewhere.  A cronjob on my desktop system currently syncs it to my dev space:

http://dev.gentoo.org/~zmedico/infra/distfiles/failure.xml
Comment 11 Kurt Lieber (RETIRED) gentoo-dev 2006-08-13 11:50:14 UTC
ok, some notes:

a) this doesn't belong on www.g.o.  It's for dev use, not general user consumption.  (no big deal if they see it, but www.g.o should contain things primarily aimed at our user base)

b) it's already working on woodpecker.  It's just apache that's screwed up and not displaying it.  See:

woodpecker distfile-mirroring # pwd
/var/www/dev.gentoo.org/distfile-mirroring
woodpecker distfile-mirroring # ls -alh failure.xml
-rw-r--r-- 1 root root 28K Aug 13 17:00 failure.xml

So, I'll beat on apache a bit and see if I can get it to show up correctly.

--kurt
Comment 12 Kurt Lieber (RETIRED) gentoo-dev 2006-08-13 11:56:39 UTC
amazing what happens when you move the directory to the correct location (i.e. under htdocs/)

works now, kind of.  images are still screwed up, but that's XSL-fu.  Anyone know how to fix that?

http://dev.gentoo.org/distfile-mirroring/failure.xml
Comment 13 Kurt Lieber (RETIRED) gentoo-dev 2006-08-13 11:58:02 UTC
changed /etc/crontab entry on woodpecker to sync to the correct path.  that's why this wasn't working before.
Comment 14 Zac Medico gentoo-dev 2006-08-13 12:03:13 UTC
The latest copy of failure.xml might be better.  I've updated the xml to use "/xsl/guide.xsl" instead of just "guide.xsl".
Comment 15 Xavier Neys (RETIRED) gentoo-dev 2006-08-13 12:06:58 UTC
(In reply to comment #12)
> works now, kind of.  images are still screwed up, but that's XSL-fu.  Anyone
> know how to fix that?

It would work if it was allowed to work its fu on it :)

1. Remove the processing instruction from the file (2nd line) or use "/xsl/guide.xsl"
2. Let the server process it, ?passthru=1 does not work which tells me it's not being passed to gorg

Same file with the p.i. http://dev.gentoo.org/~neysx/failure.xml
Comment 16 Kurt Lieber (RETIRED) gentoo-dev 2006-08-13 12:34:21 UTC
(In reply to comment #15)
> 1. Remove the processing instruction from the file (2nd line) or use
> "/xsl/guide.xsl"

OK, so can the folks responsible for the upstream script on osprey take care of this?  Once it's changed there, it should propogate down to woodpecker the next time it syncs.
Comment 17 Xavier Neys (RETIRED) gentoo-dev 2006-08-13 15:24:45 UTC
Just before I leave,

> 2. Let the server process it

Same files, different locations:
http://dev.gentoo.org/~neysx/failure.xml
http://dev.gentoo.org/distfile-mirroring/failure.xml

Check the apache config, look for "handler .xml"
Comment 18 Xavier Neys (RETIRED) gentoo-dev 2006-08-28 08:33:23 UTC
Ping.
How many Gentoo devs does it need to change one config line in an Apache config?
Comment 19 Lance Albertson (RETIRED) gentoo-dev 2006-08-28 12:06:36 UTC
(In reply to comment #18)
> Ping.
> How many Gentoo devs does it need to change one config line in an Apache
> config?
> 

Can you find me on IRC in the next few hours? I'll see if I can resolve this. I tried mucking with it, and its still working. I'm kind of busy, so I can't spend a lot of time debugging it, but I'm sure its a one-off thing. Thanks.
Comment 20 Xavier Neys (RETIRED) gentoo-dev 2006-08-30 15:15:03 UTC
(In reply to comment #19)

> Can you find me on IRC in the next few hours? I'll see if I can resolve this. I
> tried mucking with it, and its still working. I'm kind of busy, so I can't
> spend a lot of time debugging it, but I'm sure its a one-off thing. Thanks.

I'll try later. Could I have a look at the vhost config for a start?
distfile-mirroring is not under htdocs and I guess gorg is only set as a handler for xml files under htdocs.
Comment 21 Brian Harring (RETIRED) gentoo-dev 2006-10-17 17:46:21 UTC
*bump*
status?
Comment 22 Brian Harring (RETIRED) gentoo-dev 2007-01-19 00:47:20 UTC
file is back, but now is giving the finger for xslt transformation...
Comment 23 Lars Weiler (RETIRED) gentoo-dev 2007-03-09 01:14:02 UTC
(In reply to comment #22)
> file is back, but now is giving the finger for xslt transformation...

Infra: woodpecker-admin required
Comment 24 Mark Loeser (RETIRED) gentoo-dev 2007-11-04 21:58:02 UTC
So...its been quite awhile since the last comment on here, it this still looks like its broken.  Could someone please fix this? :)
Comment 25 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2007-12-17 02:43:02 UTC
after a LOT of head scratching, I finally figured out what the problem was - and Apache was NOT any help at all.

The gorg handler was undefined in that part of the Apache config.
Comment 26 Xavier Neys (RETIRED) gentoo-dev 2007-12-17 19:25:31 UTC
(In reply to comment #25)
> after a LOT of head scratching, I finally figured out what the problem was -
> and Apache was NOT any help at all.
> 
> The gorg handler was undefined in that part of the Apache config.

Reading the comments above would have saved you a lot of head scratching.
Thanks for fixing this.

Comment 27 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2007-12-18 01:56:52 UTC
neysx:
In answer to your comment, I did read the entire bug, including your comment #17.
"AddHandler gorg .xml" existed already in the relevant section of the Apache config. However it did NOT do anything: No XML processing, nor error messages.
?passthru=1 worked fine, but served the identical, non-processed XML.

I had to add "Action gorg /fcgi-bin/gorg.fcgi" in the relevant scope for Apache to do anything. That is what I meant that the handler was undefined. Somebody please skewer upsteam Apache for not giving errors when you use AddHandler with an undefined value.