Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 187971 - Gentoo Website Command Injection Issue
Summary: Gentoo Website Command Injection Issue
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Infrastructure
Classification: Unclassified
Component: Other web server issues (show other bugs)
Hardware: All Linux
: High critical with 8 votes (vote)
Assignee: Tavis Ormandy (RETIRED)
URL: http://www.gentoo.org/proj/en/infrast...
Whiteboard:
Keywords:
: 187973 188052 (view as bug list)
Depends on:
Blocks: 194189
  Show dependency tree
 
Reported: 2007-08-07 02:59 UTC by bannedit
Modified: 2008-08-26 10:41 UTC (History)
72 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
fix for sql injection and the crash problem in query_package.py (query_package.py.diff,606 bytes, patch)
2007-08-29 14:53 UTC, Christian Hoffmann (RETIRED)
Details | Diff
fix for sql injection in query_ebuild.py (query_ebuild.py.diff,711 bytes, patch)
2007-08-29 14:54 UTC, Christian Hoffmann (RETIRED)
Details | Diff
Screenshot of new packages.gentoo.org (Gentoo Packages.png,261.32 KB, image/png)
2007-11-18 06:49 UTC, Alexander Skwar
Details

Note You need to log in before you can comment on or make changes to this bug.
Description bannedit 2007-08-07 02:59:33 UTC
The gentoo packages web app contains a command injection vulnerability within the "similar" links.

Reproducible: Always

Steps to Reproduce:
1.Visit the http://packages.gentoo.org page 
2.Click on any package's Similar link
3.Add a semi-colan to the URL followed by the command you'd like to execute. If spaces are required use ${IFS} as a replacement for spaces.

Actual Results:  
At the bottom of the page the output of the command will be shown.

Expected Results:  
Commands should not be executed
Comment 1 bannedit 2007-08-07 03:06:12 UTC
*** Bug 187973 has been marked as a duplicate of this bug. ***
Comment 2 solar (RETIRED) gentoo-dev 2007-08-07 06:45:45 UTC
bannedit,

Thank you very much for reporting this problem. we have taken the servers offline
for now as we simply can't risk a full compromise to any our servers for all the 
obvious reasons. Our admins are looking into this right now and chances are we 
will re-image the servers for safety's sake. This ticket is marked as private for 
now till full details can be researched and the problematic code is re-coded and 
or a decision is made to take said code off-line forever.

Again thank you for reporting this problem.
Comment 3 Mike Doty (RETIRED) gentoo-dev 2007-08-07 22:30:04 UTC
path to packages cominb back online:
1. fix the damn code.
2. full audit by our security team.
3. profit?
Comment 4 Shyam Mani (RETIRED) gentoo-dev 2007-08-08 03:14:59 UTC
Adding marduk to the bug for his comment(s).
Comment 5 Albert Hopkins (RETIRED) gentoo-dev 2007-08-08 12:42:55 UTC
Hi.  Sorry I am out of town and don't have access to Gentoo servers.  So naturally this is the time for something like this to occur.

Many thanks for the reporter of this unfortunate bug.

I'm sending this comment and then I'll be AFK for about 12 hours so hopefully I get enough info to someone who can resolve the issue and have packages.g.o back up before then :-|

Static pages are generated by the mksite.py script on site creation.  It creates, e.g. the index.[s]html page in /similar.

The offending code is basically this:

    <!--#exec cmd="./similar.py $QUERY_STRING" -->

Probably immediately obvious by now why this is a problem.  As for why we (I) am using static pages/SSI, etc.  Well, there are historical reasons (this is *old*) code.

Now for the resolution, the solution should be something more to the effect:

   <!--#exec cmd="export QUERY_STRING;./similar.py" -->

That way cgi.FieldStorage() gets the values from QUERY_STRING a little more directly and there are no command-line parameter passing.  The reason for using exec cmd instead of exec cgi is because it's returning part of the whole page not the whole page so the HTTP headers are not wanted.

Anyway hope this is enough for someone to run with.  I'm sorry I can't do more right now and even more sorry to have allowed this to slip through.

Thanks again to the reporter.
Comment 6 Albert Hopkins (RETIRED) gentoo-dev 2007-08-08 12:44:17 UTC
Oh, forgot to mention... they /similar page is not the only one where "exec cmd" is used.  So this will have to be fixed in all occurrences.  They're all found in mksite.py though.

Comment 7 Lars Weiler (RETIRED) gentoo-dev 2007-08-08 19:17:57 UTC
(In reply to comment #3)
> path to packages cominb back online:
> 1. fix the damn code.
> 2. full audit by our security team.
> 3. profit?

You might want to add
0. link packages.gentoo.org + other domains on that server to a static page which describe the downtime.

Otherwise we will get more dups of bug #188052.

Comment 8 Albert Hopkins (RETIRED) gentoo-dev 2007-08-09 09:04:09 UTC
Is anyone actively working on this?  I still have no idea when I'll have access again.
Comment 9 Mike Doty (RETIRED) gentoo-dev 2007-08-09 16:31:35 UTC
(In reply to comment #8)
> Is anyone actively working on this?  I still have no idea when I'll have access
> again.
> 
all the infra people are at LWE this week.  we'll not be working on it until next week
Comment 10 Albert Hopkins (RETIRED) gentoo-dev 2007-08-14 03:51:13 UTC
(In reply to comment #9)
> all the infra people are at LWE this week.  we'll not be working on it until
> next week
> 

Can anyone give any info as to what the status of this is?  I'm getting all kinds of emails asking what happened with the site and thus far I haven't any reasonable explanation to explain why it is taking so long to come online, or why there is no  "be back soon" page when users come to the site.

Could someone own this and be a little more professional about it?
Comment 11 Mike Doty (RETIRED) gentoo-dev 2007-08-14 17:51:33 UTC
*** Bug 188052 has been marked as a duplicate of this bug. ***
Comment 12 Albert Hopkins (RETIRED) gentoo-dev 2007-08-14 19:25:31 UTC
For infra- ... if they are in fact listening (it appears not to be the case)

I am in the middle of relocating.  My development machine (the one that gives me access to Gentoo's servers) has been packed away in a box since 4 August and I have put myself on devaway.  Because I'm in a state of transition right now I won't realistically be able to do anything until the evening of 20 August (hopefully).

Therefore someone else needs to take this on.  I've described the technical issue as well as the resolution.  Someone has to be willing to take it on from here.
Comment 13 Mike Doty (RETIRED) gentoo-dev 2007-08-14 19:57:57 UTC
(In reply to comment #12)
> For infra- ... if they are in fact listening (it appears not to be the case)
> 
> I am in the middle of relocating.  My development machine (the one that gives
> me access to Gentoo's servers) has been packed away in a box since 4 August and
> I have put myself on devaway.  Because I'm in a state of transition right now I
> won't realistically be able to do anything until the evening of 20 August
> (hopefully).
> 
> Therefore someone else needs to take this on.  I've described the technical
> issue as well as the resolution.  Someone has to be willing to take it on from
> here.
> 
Marduk-

infra is listening, please stop insinuating that we aren't.  As I already explained, the infra folks were at a conference last week.  I will be attempting to fix the code.  After that I'll hand it over to security to do a complete audit.  After all that is done we'll put it back online.
Comment 14 SpanKY gentoo-dev 2007-08-15 00:20:07 UTC
dont suppose we could get it prioritized so the non-packages.g.o machines are rebuilt and brought up while packages.g.o stays dead ?  i'm thinking of archives.g.o in particular ...
Comment 15 Mike Doty (RETIRED) gentoo-dev 2007-08-15 00:24:13 UTC
(In reply to comment #14)
> dont suppose we could get it prioritized so the non-packages.g.o machines are
> rebuilt and brought up while packages.g.o stays dead ?  i'm thinking of
> archives.g.o in particular ...
> 

already in the works
Comment 16 Abe E 2007-08-18 21:35:05 UTC
In regards to Infra not listening. Marduk, I undertsand you have a move in the works and the fact the rest of infra are at a conference. 2 questions arise from this: 1) Who's watching the store, so to speak? 2) Why hasn't an alternative mirror been posted?
Comment 17 Alex Howells (RETIRED) gentoo-dev 2007-08-19 16:29:28 UTC
(In reply to comment #16)
> In regards to Infra not listening. Marduk, I undertsand you have a move in the
> works and the fact the rest of infra are at a conference. 2 questions arise
> from this: 1) Who's watching the store, so to speak? 2) Why hasn't an
> alternative mirror been posted?
> 

#1: Emergency issues can still be responded to, as evidenced by the fact that this server was immediately taken down once Infra became aware of the problem.

#2: Alternative mirror to compromised code? Could you be a bit more clear? :) packages.gentoo.org simply displays the current tree in a 'user friendly' format online allowing users to see what's stable / ~arch easily; this information can all be gleaned from the CLI, and alternative sites (ie: gentoo-portage.com) also provide vastly similar functionality if you're desperate.

Article about this mess:
http://www.theregister.co.uk/2007/08/17/gentoo_disconnects_vulnerable_server/

To quote from that article;  "We will update this story if we receive responses to emails sent to Gentoo members."

That story has received no update despite the fact a response has been sent to The Register, and the response was as follows:

  "There was no possibility of any leak of personal information
   or meddling with the Gentoo Portage tree. The attack was limited
   to one service on one server."  -- Mike Doty
Comment 18 Tom Knight (RETIRED) gentoo-dev 2007-08-19 19:08:48 UTC
Fixed all exec calls in in mksite.py as per comment 5. The code for this is gentoo/src/packages branch v1_3.
Comment 19 Thomas Anderson (tanderson) (RETIRED) gentoo-dev 2007-08-20 11:35:43 UTC
> #2: Alternative mirror to compromised code? Could you be a bit more clear? :)
> packages.gentoo.org simply displays the current tree in a 'user friendly'
> format online allowing users to see what's stable / ~arch easily; this
> information can all be gleaned from the CLI, and alternative sites (ie:
> gentoo-portage.com) also provide vastly similar functionality if you're
> desperate.

Not to be picky, but you do realize that gentoo-portage was down at the same time?
Comment 20 Albert Hopkins (RETIRED) gentoo-dev 2007-08-20 12:24:39 UTC
Thanks.  Just an update for me: This morning I'll get the keys to my new place and the movers should come to deliver my stuff, including my PC.  But I still have to call to get my Internet turned on. I'm thinking another 1 or 2 days for that. 
Comment 21 Albert Hopkins (RETIRED) gentoo-dev 2007-08-21 22:54:24 UTC
Just an update re my situation.  I have my machines (still in boxes) but I wont have Internet connectivity until Saturday 25 August.
Comment 22 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2007-08-24 10:58:38 UTC
The analysis writeup is now published:
http://www.gentoo.org/proj/en/infrastructure/nuthatch-writeup/

tomk has the code fixed, I don't know if security has reviewed it yet.
Comment 23 Christian Hoffmann (RETIRED) gentoo-dev 2007-08-29 14:40:30 UTC
I was bored and so I took a look at the code. As the fix for the initial issue was committed to the v1_3 branch I took that as a base. I'm not sure whether this code should really get online without major rewrites again...
The issues I found are not as critical as the initial one, but they should be fixed nonetheless:

At a lot of places supplied data is not properly escaped before being used (a lot of examples in gentoo.py). While this is not directly exploitable (as the data inserted to the query often comes from the database) it is incorrect to rely on the data being ready for usage in sql queries (checked and/or escaped properly).
I found two places where this is actually exploitable:
  1. /query_package.py?category=sys-apps&offset=1,1000/*
     It can be fixed by forcing the offset parameter to be an integer.
     Patch attached.
  2. /query_ebuild.py?%22+OR+%22foo-1
     (you cannot use any non-int/rc/pre stuff after the first dash as
     portage's pkgsplit() is used...)
     The name should be escaped properly. Patch attached.
Don't know how critical they are.
It is also possible to crash the CGI (= produce a traceback) wherever cgi.FieldStorage.getvalue is used. Simply specifying one parameter mulitply times makes getvalue() return a list instead of a string which breaks the following code. getfirst() should be used in those places (as my patch for sql injection #1 already does).

Again, I don't think all vulnerabilities are fixed with these two patches; there are probably more, but I'm not really keen on tracking that down further...
Comment 24 Christian Hoffmann (RETIRED) gentoo-dev 2007-08-29 14:53:43 UTC
Created attachment 129536 [details, diff]
fix for sql injection and the crash problem in query_package.py
Comment 25 Christian Hoffmann (RETIRED) gentoo-dev 2007-08-29 14:54:57 UTC
Created attachment 129537 [details, diff]
fix for sql injection in query_ebuild.py
Comment 26 solar (RETIRED) gentoo-dev 2007-08-30 17:31:38 UTC
taviso of the sec audit team has been asked to audit the code. 
The packages.gentoo.org service will not be coming back online 
till he gives it the thumbs up.
Comment 27 solar (RETIRED) gentoo-dev 2007-09-08 08:04:07 UTC
(In reply to comment #21)
> Just an update re my situation.  I have my machines (still in boxes) but I wont
> have Internet connectivity until Saturday 25 August.
> 

http://starship.python.net/crew/marduk/blog/entry/1189186730,3524

Today he retired. He was the author and maintainer of this service.

Infra..
Guess it's safe for us to say the p.g.o code-base wont be coming back at all now?
Comment 28 Gian Luca Dalla Torre 2007-09-10 19:14:22 UTC
Any news on this issue?
Is it possible to have a timeline for the fix?

Comment 29 Alex Howells (RETIRED) gentoo-dev 2007-09-10 19:16:38 UTC
(In reply to comment #28)
> Any news on this issue?
> Is it possible to have a timeline for the fix?

Er, it is fixed; the vulnerable service was taken down as soon as Infra became aware of a problem and other affected services on the same box have now been restored to operational status.

Since the developer of packages.gentoo.org has now retired from Gentoo, it seems unlikely that this service will return anytime soon.

Perhaps you should try somewhere like http://www.gentoo-portage.com?
Comment 30 Gian Luca Dalla Torre 2007-09-10 19:22:57 UTC
Now I am using gentoo-portage but I prefer packages.gentoo.org since it is (was) an internal service and it display much more info (and in a better manner) than gentoo-portage.

The questions are:

- will, in the near future, this service come back with a new mantainer?
- if no, will it be rewritten from scratch to ensure that a new programmer could mantain this service?
- Is this service lost forever?

(In reply to comment #29)
> (In reply to comment #28)
> > Any news on this issue?
> > Is it possible to have a timeline for the fix?
> 
> Er, it is fixed; the vulnerable service was taken down as soon as Infra became
> aware of a problem and other affected services on the same box have now been
> restored to operational status.
> 
> Since the developer of packages.gentoo.org has now retired from Gentoo, it
> seems unlikely that this service will return anytime soon.
> 
> Perhaps you should try somewhere like http://www.gentoo-portage.com?
> 

Comment 31 Alexander Skwar 2007-09-10 19:53:15 UTC
(In reply to comment #29)

> Since the developer of packages.gentoo.org has now retired from Gentoo, it
> seems unlikely that this service will return anytime soon.
> 
> Perhaps you should try somewhere like http://www.gentoo-portage.com?

Would it be possible to add a statement pretty much like the one you made to http://packages.gentoo.org/? That would like a lot better than the current 404. Or maybe even remove packages.gentoo.org completely from DNS?

Comment 32 Gordon Malm (RETIRED) gentoo-dev 2007-09-10 21:06:04 UTC
I have to agree with comment #30.  packages.g.o is far superior to gentoo-portage and others and I would hate to see it go.  Perhaps many are not aware of this newly developed situation?  Maybe a call could go out on the next GWN for a maintainer?
Comment 33 Mike Doty (RETIRED) gentoo-dev 2007-09-10 21:29:38 UTC
(In reply to comment #27)
> Infra..
> Guess it's safe for us to say the p.g.o code-base wont be coming back at all
> now?
> 
If security gives us the OK we'll put it back online.


(In reply to comment #28)
> Any news on this issue?
> Is it possible to have a timeline for the fix?
> 

No, there is no current time line.  When security finishes their audit we will have more information.

(In reply to comment #29)
> Since the developer of packages.gentoo.org has now retired from Gentoo, it
> seems unlikely that this service will return anytime soon.

If the security team gives the package the OK it will go back online.

(In reply to comment #30)
> - will, in the near future, this service come back with a new mantainer?
unknown.
> - if no, will it be rewritten from scratch to ensure that a new programmer
> could mantain this service?
unknown.
> - Is this service lost forever?
unknown.

(In reply to comment #31)
> Would it be possible to add a statement pretty much like the one you made to
> http://packages.gentoo.org/? That would like a lot better than the current 404.
> Or maybe even remove packages.gentoo.org completely from DNS?
> 

There is no 404 and we will not be removing it from DNS.
Comment 34 Codo 2007-09-10 21:30:49 UTC
(In reply to comment #29)
> Since the developer of packages.gentoo.org has now retired from Gentoo, it
> seems unlikely that this service will return anytime soon.
Alex, can someone point me to the source code to have a look at it?  Maybe I
can maintain it for a while.  I don't think it is rocket science, is it?

Thanks!
Comment 35 Gordon Malm (RETIRED) gentoo-dev 2007-09-10 22:09:13 UTC
(In reply to comment #34)
> (In reply to comment #29)
> > Since the developer of packages.gentoo.org has now retired from Gentoo, it
> > seems unlikely that this service will return anytime soon.
> Alex, can someone point me to the source code to have a look at it?  Maybe I
> can maintain it for a while.  I don't think it is rocket science, is it?
> 
> Thanks!
>

http://sources.gentoo.org/viewcvs.py/gentoo/src/packages/
Comment 36 Codo 2007-09-11 17:39:19 UTC
(In reply to comment #23)
> I was bored and so I took a look at the code. As the fix for the initial
...snip...  In addition to hoffies comments...

  Some of my comments may be redundant due to my ignorance.  Please ignore them if this is the case.  And, I am no python programmer (but I wasn't ruby, C#, C, Smalltalk, etc... either)

  Most of the SQL functions are not escaped.  This should be easy to fix.  Just patience and a good beer on a Saturday afternoon should do.  Example: p_objects.py line 214

  The maintenance scripts should be kept well away from the reaches of the webserver, only invoked by cron.  We can identify and make sure infra knows of this.

  query_ebuild.py needs a bit more of escaping when querying the DB IMHO.

  query_package.py has been sorted, so I guess it's alright...  Checking will do no harm.

  What happens in python when I do a sys.stdout.write(`cat /etc/passwd`) ???  This needs a bit more attention...

  search.py seems corrected.  Will do a double check.

  similar.py needs to be escaped I think...

  What you think, shall we give a push and bring back p.g.o?  I will go ahead and try to fix some issues, but it would be good if someone can test them, and if someone from Sec/Infra promises to check the code so it can go Live.  If you want to get ahead of me, please do so and let know in the bug (I have to finish a few things today and tomorrow so I will probably start on Friday)

  In my opinion p.g.o can be brought back with not much effort.  Rewrite == no-go.  Code seems allright to me, just needs TLC.

  Arturo.
Comment 37 Daniel Carosone 2007-09-11 21:51:23 UTC
I don't want to pester or hurry folks into reopening the site before they're sure it's safe.

However, given the time is dragging on, I (and I think many others) would appreciate some more helpful text on the static page that's there now.  Some more specific pointers on where/how else to get the information the site used to provide.  In my particular case, I'm interested in seeing the changelogs for particular packages I depend on.  If the best option for that right now is to fetch the svn repository, so be it.  If there are other options as well, all the better - but please provide a link to guide users there
Comment 38 Albert Hopkins (RETIRED) gentoo-dev 2007-09-12 00:37:12 UTC
I think that this is a perfect example right here of lack of responsibility and leadership among those who have been been given the honor of carrying Gentoo.  This site has been down for over a month.  If this were a business Gentoo would probably be closing its doors now.  There seems to be no responsibility to the customer (yes, Gentoo has customers) and no drive to respond to the needs of the customer.

The lack of disclosure/communication is also a major contributing factor for the frustration.  I'm saying this as a user/customer as well as a developer.  I realize that many people in the infra- group were away at LWE when the vulnerability was found but 1) that was over a month ago and 2) that's not an excuse.  When the "decision" was made to take down the site I wasn't even notified.  I had to scurry around and ask wtf happened like everyone else.  When this bug was created not even *I* was given access to it.  Meanwhile I have tons of people emailing and messaging me me asking me what's going on and it makes me look like an idiot.  Now it's not that difficult to make me look like and idiot, so I'm used to it.  But in this case it makes *Gentoo* look like complete idiots.

Then it was difficult to get the ball rolling, in fact we as developers completely dropped the ball.  No communication to the users.  Total irresponsibility.  Then when a user asks when the problem will be resolved given , the time lapse, is perfectly valid, we get responses from Gentoo developers like: we took the site down therefore the issue is resolved.  No, the issue is not resolve, you idiot.  The issue is that Gentoo users have been without a service that they used to depend on.  We've already *shown* the users that we don't give a fuck about them; we don't actually have to come out and *say* it.

Anyway, it's really disappointing the way this has carried out, and I don't go without taking some of the responsibility.  I think if Gentoo were a company I'd say that someone needs to just come in, fire every last one of us, and just start hiring/re-hiring people from scratch.  I've done my part by resigning, but I'm definitely not the only rotten apple here.
Comment 39 Lance Albertson (RETIRED) gentoo-dev 2007-09-12 02:34:53 UTC
(In reply to comment #38)
> I think that this is a perfect example right here of lack of responsibility and
> leadership among those who have been been given the honor of carrying Gentoo. 
> This site has been down for over a month.  If this were a business Gentoo would
> probably be closing its doors now.  There seems to be no responsibility to the
> customer (yes, Gentoo has customers) and no drive to respond to the needs of
> the customer.

Frankly, I don't understand why you had to reply to a bug like this in such a negative manner. I understand that its been a big deal for it being down, but I would rather the proper security audit be completed since this service has already be targeted (more eyes are looking at it now). If you (or anyone else) wants to do a quick rewrite that just generates the static site to at least some of the functionality back sooner, please feel free to do that. 

> The lack of disclosure/communication is also a major contributing factor for
> the frustration.  I'm saying this as a user/customer as well as a developer.  I
> realize that many people in the infra- group were away at LWE when the
> vulnerability was found but 1) that was over a month ago and 2) that's not an
> excuse.  When the "decision" was made to take down the site I wasn't even
> notified.  I had to scurry around and ask wtf happened like everyone else. 
> When this bug was created not even *I* was given access to it.  Meanwhile I
> have tons of people emailing and messaging me me asking me what's going on and
> it makes me look like an idiot.  Now it's not that difficult to make me look
> like and idiot, so I'm used to it.  But in this case it makes *Gentoo* look
> like complete idiots.

Forgive me if I'm incorrect on this, but you were in the middle of moving while this was happening so we had issues contacting you as well. So I'd appreciate it if you wouldn't put a single blame on infra. It also took you a few weeks to get your systems set back up at home as well. Yes, we could have done a better job of communicating but our biggest concern was protecting our assets and ensuring that the user base was *safe*. 

> Then it was difficult to get the ball rolling, in fact we as developers
> completely dropped the ball.  No communication to the users.  Total
> irresponsibility.  Then when a user asks when the problem will be resolved
> given , the time lapse, is perfectly valid, we get responses from Gentoo
> developers like: we took the site down therefore the issue is resolved.  No,
> the issue is not resolve, you idiot.  The issue is that Gentoo users have been
> without a service that they used to depend on.  We've already *shown* the users
> that we don't give a fuck about them; we don't actually have to come out and
> *say* it.

Please don't use this type of language in a technical bug like this. If you have frustrations you can talk to me online directly if you want. I don't understand why you suddenly gave this type of a response to a bug that seemed to be (slowly) progressing. 

> Anyway, it's really disappointing the way this has carried out, and I don't go
> without taking some of the responsibility.  I think if Gentoo were a company
> I'd say that someone needs to just come in, fire every last one of us, and just
> start hiring/re-hiring people from scratch.  I've done my part by resigning,
> but I'm definitely not the only rotten apple here.
> 

Well, that's why we're a volunteer organization. We're not here to make money or keep customers always happy 100%. Yes this is an important service, but Gentoo isn't directly dependent on it to run. I hope we can get all the details worked out soon, but as most volunteer organizations are, we're time strapped.

I'm sorry to see you retire but I do wish you luck on your next venture.
Comment 40 Onkobu 2007-09-12 05:03:01 UTC
Are you able to restore the service after resigning/searching for responsibles/committing possible fixes? How much does it take to contribute in any way? I could spend 4 hours on this - for freeeeeeEEe (Maybe it's an effort if somebody with zero Gentoo experience has a look at the possible vulnerabilities/fixes and don't start something like an interview, just hand over some CVS/SVN read access and in return you'll get an impact analysis.)

BTW: I totally agree with the responsibility-discussion-arguments - nothing professional, just a bunch of freelancers, heavy rotation and no core competences/core developers, resp. nobody to blame. Resigned developers aren't bad people, but being the only person watching after an important service is an inacceptable state.
Comment 41 Onkobu 2007-09-12 05:19:32 UTC
I could mirror the portage tree browsing on my own server, a) if it's possible and b) if there are more testers than me - if it passes: return to normal business and put packages.gentoo.org back online, if it fails: I'd offer it as testing facility for 2 or 3 days (a gentoo system *g*).
Comment 42 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2007-09-12 06:03:38 UTC
Onkobu: the location of the source has been mentioned before.
http://sources.gentoo.org/viewcvs.py/gentoo/src/packages/
You can pull it from there using the anoncvs instructions that should be at the top of all the sources pages. 

Codo and hoffie posted an initial set of patches and highlighted the remaining problems (beyond tomk's original fix).
Comment 43 Codo 2007-09-12 06:04:29 UTC
(In reply to comment #41)
> I could mirror the portage tree browsing on my own server, a) if it's possible
> and b) if there are more testers than me - if it passes: return to normal
> business and put packages.gentoo.org back online, if it fails: I'd offer it as
> testing facility for 2 or 3 days (a gentoo system *g*).
> 
Onkobu, can you try to mirror on your system?  Browse the code, there is a
mksite.py around and two scripts to create the MySQL db.  I WILL correct the
issues but if you can get experience in getting it up and running that would be
great.  Please feel free to conctact me straight to my E-mail.

Marduk, I totally agree with you, but could you give a bit of a hand here as a
sort of handover?  I totally agree with you but that is a topic for gentoo-proj
I think.  It would be great if for a while you can resolve a few questions for
me.  Thanks!

And, can someone on infra/security PLEASE COMMIT HIM/HERSELF TO CHECK AND GIVE
GREEN LIGHT WHEN THE CODE IS READY?  PLEASE?
Comment 44 Onkobu 2007-09-13 21:29:50 UTC
First schedule:
Friday (Sept, 14th), MySQL Setup, Get CVS code, Apache Setup
Saturday (Sept, 15th) 06:00-10:00 review/apply patches
                      19:00-23:00 activate access to mirrored page(s)
Anyone interested in IRC channels/sessions/chats (all times in GMT+2:00, CEST)

Second security topics:
- is it necessary to use SSI Exec instead of SSI CGI?
- does SSI EXEC/CGI run as unprivileged user/ WebServer suExec-capable [1]?
- is packages.gentoo.org separated from other (gentoo) services (virtual hosts, separate server)[2]?

What I'm trying to say: running 'rm -rf /' does not cause any harm if you're in a chroot jail/resp. if you're an unprivileged user (suexec). Esp. Apache has good protection mechanisms to keep scripts from doing anything outside their "DocumentRoot"...well "cat /dev/zero > spamfile" would be a nice DoS thing, though...

[1] http://httpd.apache.org/docs/2.0/suexec.html
[2] http://httpd.apache.org/docs/2.2/en/vhosts/
Comment 45 Codo 2007-09-13 22:05:24 UTC
(In reply to comment #44)
> First schedule:
...snip...
Hi Onkobu... I'm puzzled by your post.  Are you part of gentoo/sec?  Cool! Do you have a test environment set-up?  I'm setting up one tomorrow as well.  It would be nice to discuss so we can move things forward.

Though a chjail is good I don't think p.g.o will be a good deal.  We just have to escape properly user-input.  What do you mean by those times there?  A.
Comment 46 Tom Knight (RETIRED) gentoo-dev 2007-09-14 00:05:13 UTC
(In reply to comment #44)
> - is packages.gentoo.org separated from other (gentoo) services (virtual hosts,
> separate server)[2]?

It's on a separate vhost but on the same box as other services.
Comment 47 Albert W. Hopkins 2007-09-14 00:21:27 UTC
(In reply to comment #43)


> Marduk, I totally agree with you, but could you give a bit of a hand here as a
> sort of handover?  I totally agree with you but that is a topic for gentoo-proj
> I think.  It would be great if for a while you can resolve a few questions for
> me.  Thanks!

It shouldn't be that difficult to set up.  I still have an instance running on one of my local development machines.  I'd been considering parting ways with Gentoo for a while now, so one of the things I tried to do is get the code base halfway organized and clear so that someone else could take it over.  Anyway if you have any questions I'll try to help.  I of course would still like to see the site come back online (with the necessary fixes, of course) if at all possible. 
Comment 48 solar (RETIRED) gentoo-dev 2007-09-14 01:10:35 UTC
Please take further code design topics to another bug. This one can more 
or less be CLOSED as FIXED. The orig service was taken offline. The 
nuthatch box was painstakingly formatted rebuilt by me and is online waiting 
to be rotated back into production.

I think you will want a tracker for p.g.o stuff. Chris G (wolf31o2) reassigned 
a bunch bugs related to that service to infra yesterday of which most all 
looked like they were feature requests. Infra itself is not going to do 
anything with them. The other bug that should be filed is for the 
official security audit.

If a tracker is filed please let infra know the bug number.

PS: we need a dev to be willing to be the maintainer of this code.
Comment 49 Onkobu 2007-09-14 04:57:46 UTC
(In reply to comment #45)
> (In reply to comment #44)
> > First schedule:
> ...snip...
> Hi Onkobu... I'm puzzled by your post.  Are you part of gentoo/sec?
Nope, I'm just fed up with packages.gentoo.org being down for a month!
> Cool! Do
> you have a test environment set-up?  I'm setting up one tomorrow as well.  It
> would be nice to discuss so we can move things forward.
I'll give my best to tweak my web server a bit, maybe others are scared of "just try if it works" or there are no resources to do so (people to set it up/machines to set up on).

> Though a chjail is good I don't think p.g.o will be a good deal.  We just have
> to escape properly user-input.  What do you mean by those times there?  A.
P.G.O.? I prefer double- or even tripple-checking, that's why my concepts are based on a broader view (not only write a few lines but also look around it, maybe there are fundamental design flaws.)
The times are a "schedule" when I'll do what. I'm not sure when I'll start on Friday, could be 17:00 or one or two hours later. On Saturday I've got another "date", so I'll be busy then, except in the morning and in the evening. It'd be a good idea to be present in an IRC channel at this time, to discuss and coordinate testing/improvements...another (asynchronous) way could be a discussion thread. (All previous posts did not contain deadlines.)
Comment 50 Onkobu 2007-09-14 05:02:04 UTC
(In reply to comment #46)
> (In reply to comment #44)
> > - is packages.gentoo.org separated from other (gentoo) services (virtual hosts,
> > separate server)[2]?
> 
> It's on a separate vhost but on the same box as other services.
> 

At least a separate DocumentRoot (I guess), no suexec? Not really necessary, but would be another barrier. (I'll setup my Apache with suexec and have look how it's done and what it's about, even with vulnerable code.)
Comment 51 Codo 2007-09-14 06:46:57 UTC
(In reply to comment #48)
> Please take further code design topics to another bug. This one can more 
> or less be CLOSED as FIXED.
Guys, if the current code (with hoffie's patches) is ready for production, then what are we waiting for.  Has SEC checked the stuff?  What is going on?  Don't close this bug if p.g.o is not coming online.  This is the smelly one that is bringing p.g.o back to life (and the bug number is on p.g.o page at the moment).

> I think you will want a tracker for p.g.o stuff. [wolf31o2] reassigned 
> a bunch bugs related to that service to infra yesterday of which most all 
> looked like they were feature requests. Infra itself is not going to do 
> anything with them. 
For all future things, I was thinking of writing an ebuild and make packages.gentoo.org behave just like any other package, so anybody can get their hands onto it, install it on their local machines, etc...  And, it would be easier to upgrade in production if we want to apply further patches.

> The other bug that should be filed is for the 
> official security audit.
Lets keep things together.  Don't divert it to security.  Can security come to this bug please?

> PS: we need a dev to be willing to be the maintainer of this code.
Yes, there will be.  Priority number 1 now is back online.  I think the community will keep p.g.o codebase in the worst case.
Comment 52 Pacho Ramos gentoo-dev 2007-09-15 09:11:21 UTC
(In reply to comment #51)
> For all future things, I was thinking of writing an ebuild and make
> packages.gentoo.org behave just like any other package, so anybody can get
> their hands onto it, install it on their local machines, etc...  And, it would
> be easier to upgrade in production if we want to apply further patches.
> 

If it was possible, it would be nice ;-)
Comment 53 Onkobu 2007-09-15 09:30:36 UTC
(In reply to comment #50)
> (In reply to comment #46)
> > (In reply to comment #44)
> > > - is packages.gentoo.org separated from other (gentoo) services (virtual hosts,
> > > separate server)[2]?
> > 
> > It's on a separate vhost but on the same box as other services.
> > 
> 
> At least a separate DocumentRoot (I guess), no suexec? Not really necessary,
> but would be another barrier. (I'll setup my Apache with suexec and have look
> how it's done and what it's about, even with vulnerable code.)
> 

Right now (11:30am, CEST) I'm migrating portage tree to database, set up apache with suexec and maybe we'll have a mirror soon.
Comment 54 Onkobu 2007-09-15 10:22:23 UTC
My first impression: absolutely necessary to rework the whole service. There are INSERT statements which do not refer to column names but to the sequence columns were created (INSERT INTO table Values(...)). The CREATE TABLE scripts miss columns (is_masked and prevarch) and primary keys as well as joins are (based on) VARCHARs. I'll write a sort of report and host it somewhere on the mirror (including patch impact analysis) so maybe the code maintainer has a point to start from. (Sometimes you need a bit fast hacking, but this thing running for years - obviously luck or never extended).
Comment 55 Codo 2007-09-15 14:41:11 UTC
(In reply to comment #54)
> My first impression: absolutely necessary to rework the whole service. There
...snip...

Onkobu: Great, if you have the service running, can you drop me an E-mail?  I am heavily patching the whole thing but I haven't been able to put up the site working on my lappy, so it needs testing.  Will talk by E-mail.  Thanks again.

A.
Comment 56 Alec Warner (RETIRED) archtester gentoo-dev Security 2007-09-16 01:49:36 UTC
I'd like to kindly ask once again to take the development for this to a different place.  Bugzilla is not a mailing list.

Recruiting people and explaining what you are up to sounds like it would fit well into the gentoo-project mailing list.

-Alec
Comment 57 Gian Luca Dalla Torre 2007-09-19 07:35:41 UTC
(In reply to comment #56)
> I'd like to kindly ask once again to take the development for this to a
> different place.  Bugzilla is not a mailing list.
> 
> Recruiting people and explaining what you are up to sounds like it would fit
> well into the gentoo-project mailing list.
> 
> -Alec
> 

I understand your request Alec, but, since end - users normally do not subscribe the gentoo ml, is it possible to use this bug as a diary of the work-in-progress? This bug is very critical and, in my opinion, it should be considered in some vay "special"...

I agree with you that the development matters should be discusses through private emails, but I think that Gentoo users have to be informed on how the work is proceeeding. If this bug is not the right place, please update the special page that Gentoo Infra has written (http://www.gentoo.org/proj/en/infrastructure/nuthatch-writeup/) or give access to it to Codo or Onkubu, they surely will do it.

By the way, is the work proceeding? Will PGO come back soon?

Thanks in advance.
Gian Luca

Comment 58 Codo 2007-09-19 08:08:11 UTC
Alec:

  If you provide some platform so we can use instead of this bug we would love to use it.  This bug is great because a lot of people are CCd to it.

> By the way, is the work proceeding? Will PGO come back soon?
It is proceeding, but we are finding new things.  Will come buck to the bug soon.

Arturo.
Comment 59 Alex Howells (RETIRED) gentoo-dev 2007-09-19 08:51:05 UTC
>   If you provide some platform so we can use instead of this bug we would love
> to use it.  This bug is great because a lot of people are CCd to it.

... and there lies the problem, with all of your internal "updates" you are sending out >100 emails for no good reason.  If you wish to use Bugzilla rather than subscribing to a mailing-list, create a tracker bug, don't use this one.

> > By the way, is the work proceeding? Will PGO come back soon?

I believe the statement from Infrastructure said it'd be back once the Security guys have approved it for use again, and preferably also once a developer has stepped forward to agree to maintain it in future.

Once again, could folks please move these conversations elsewhere? As has been previously noted this bug may as well be CLOSED FIXED, at the very least until a later date when taviso / Security have approved things...
Comment 60 Steve B 2007-09-19 23:31:02 UTC
I'm tasked with running numerous distributed servers that run Gentoo and I frequent the packages site monitoring updates and such. I understand what Alec is saying but I must point out the obvious that everyone else has pointed out... This is the page that infra has linked to from the pgo page. 

Alec also mentions that one of the problems is that it generates too much mail. I notice that there is a remove CC option...

I would gladly offer a hand getting pgo back online but my area of expertise is of no help in this matter. I commend the people that are working on it though and will continue to monitor the bug in case there comes a point where I may be able to add something.

May I also politely insert that if I adamently insisted that turning off a service meant "it is fixed" it to any of the companies I contract to, I wouldn't be working for them long.

Once again, hats off to the people working on fixing this issue.

Steve B.
Comment 61 solar (RETIRED) gentoo-dev 2007-09-20 00:24:06 UTC
This bug is CLOSED. 

Infra has done it's part. We now await others to do theirs.
Comment 62 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2007-09-20 09:11:01 UTC
*** Bug 188052 has been marked as a duplicate of this bug. ***
Comment 63 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2007-09-20 09:11:51 UTC
reopen to reassign (properly this time)
Comment 64 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2007-09-20 09:13:10 UTC
Reassigned to taviso, because he needs to audit the code first.
He already has where the test site is online.

infra-bugs has been removed from this, because we are tired of the bugspam.
Comment 65 Onkobu 2007-09-20 17:02:46 UTC
Go on moving this bug from team to team, keep others out with comments like «this chit chat does not belong in here» and don't go nowhere. After 8 weeks NOTHING happened. My service's down 'til tomorrow and I'm off this bug - not pissed but disappointed 'bout lack of professional problem sollution. Fully remove exec-includes and replace with virtual (cgi-bin)-calls, wrap CGI with suexec and you won't have any problems. If the CC-list is absolutely busy for whatever reason, why are they on CC at all?
Comment 66 Kevin Lyles 2007-09-20 17:37:44 UTC
(In reply to comment #65)
> Go on moving this bug from team to team, keep others out with comments like
> «this chit chat does not belong in here» and don't go nowhere. After 8 weeks
> NOTHING happened. My service's down 'til tomorrow and I'm off this bug - not
> pissed but disappointed 'bout lack of professional problem sollution. Fully
> remove exec-includes and replace with virtual (cgi-bin)-calls, wrap CGI with
> suexec and you won't have any problems. If the CC-list is absolutely busy for
> whatever reason, why are they on CC at all?
 
They explained why they are done with the bug, and who needs to do the rest.  I don't see the problem with them transferring the bug to taviso after that.  They are correct that the complaints, however justified (and most of them are), do not belong on bugzilla (try the forums or e-mail, instead).  Finally, they just stated that they removed themselves from the CC list.
Comment 67 christian 2007-09-26 21:00:56 UTC
this bug should NOT be CLOSED.  this bug caused a WANTED (by many) service to go offline.  until full functionality is restored, gentoo is LESS than it was.  i am now embarrassed to recommend gentoo to anyone as an operating system.  gentoo is built upon ebuilds.  computer usability depends upon software functionality.  p.g.o was a great place to find software.  it's 2007.  if something doesn't grow, it stagnates and dies.  for gentoo to grow, a friendly web interface is a necessity.  equery using the command line equals no regular users equals gentoo being a niche equals microsoft ruling forever.  and, gentoo IS a BUSINESS, a non-profit organization, with a board of trustees.  gentoo SELLS discs, clothing, mugs, stickers, etcetera.  money is made.  donations are received.  volunteers are used.  advertising is done.  customers are made.  customers want a great system.  if i were a programmer, i'd help.  unfortunately i'm not, so i can't.  over the years, i have ever more proudly pitched gentoo to friends, co-worker, and the like.  hopefully i made a few converts.  but now i keep my mouth shut.  no longer can i pull up packages.g.o and show someone all the great software -- easily searchable all in one place with a ton of needed info.  so, gentoo trustees,  
*  Michael Cummings, mcummings
* Chris Gianelloni, wolf31o2
* Grant Goodyear, g2boojum
* Renat Lumpau, rl03
* Paul de Vrieze, pauldv
  leave this bug open until gentoo is as good as or better than it was, until we have a great p.g.o or equivalent.  and until then, i'll keep on keeping my mouth shut.
Comment 68 Alex Howells (RETIRED) gentoo-dev 2007-09-26 21:16:49 UTC
I swear we need some kind of 'Mute all but developers' button which leaves a bug read-only for the world to see; the amount of signal:noise on this bug is insane. Please can we stop with the "OMFG! Fix this now! It's essential" waterfall of comments, they are not helpful and frankly the bugspam is annoying.

As has been stated countless times, the service will go back online if and when Gentoo Security (taviso) approves and not before.

It'd also be very helpful if a Gentoo Developer stepped forward who was willing to be the maintainer, so future issues are fixed in a timely fashion.

Please be patient, and please be quiet. Thank you :)
Comment 69 christian 2007-09-26 23:40:11 UTC
(In reply to comment #68)

"if and when..." is a lazy attitude.  you imply that only the opinion of "developers" count and the voice of all others should be censored out.  i did NOT say fix it now.  i said it NEEDS to be fixed.  read through the above and you'll see that the general opinion of those responsible for fixing this is basically "we've done what we will for now, we're tired of messin' with it, we know how to use equery to suite our needs, the rest go eat a jelly baby."  and my MAIN point is THIS... if gentoo can't fix this and have it's system up and ready after almost 60 days down, gentoo has LOST my confidence in it's viability as distro/meta-distro/however-you-wanna-call-it.  MANY people USE gentoo and need it to WORK... which it still DOES.  but gentoo won't work for LONG with the kinda attitude that's been shown by this problem.  and personally, when i invest SO much time in my system, i want my system to have a lifespan as long as mine.  and guy, i may keep my mouth shut as far as promoting gentoo.  but, i'll never be MUTED.  and now, i'm gonna do a lot to bring attention to gentoo's deficit(s).  and i'm already eyeing other distros that i CAN promote.  
Comment 70 Dawid Węgliński (RETIRED) gentoo-dev 2007-09-26 23:52:18 UTC
Oh please. Read this bug activity. You will note it is assigned and NEW, so what on earth do you write this whole flame for? There is a bunch of people that try to make p.g.o up, so please... Stop with bugspam. This bug will be resolved when the time comes.
Comment 71 Daniel Carosone 2007-09-27 02:42:02 UTC
By all means take the time you need to fix this properly.  By all    
means track that activity as you see fit.  By all means spend that
time arguing with eachother instead if you prefer. That's all fine.
                                                                                                                                                                                     
What I find most disappointing is that you spend time arguing with                                                                                                                   
eachother instead of acting on simple user requests, eg my #37.
                                                         
Just maybe, a better place-holder page with useful information might
reduce the number people feeling the need to express their frustration
in the bug.                                
Comment 72 JTRiley 2007-09-27 21:07:37 UTC
(In reply to comment #71)
> By all means take the time you need to fix this properly.  By all    
> means track that activity as you see fit.  By all means spend that
> time arguing with eachother instead if you prefer. That's all fine.
> 
> What I find most disappointing is that you spend time arguing with              
> eachother instead of acting on simple user requests, eg my #37.
> 
> Just maybe, a better place-holder page with useful information might
> reduce the number people feeling the need to express their frustration
> in the bug.                                
> 

2-cents: 

Something I think we should add to the place-holder page is a link to www.gentoo-portage.com so that users needing to search packages via the web can still find this functionality until p.g.o comes back online.  

~jtriley

Comment 73 Marco Qualizza 2007-09-28 19:25:16 UTC
Out of curiousity, what's the current state of the p.g.o. code right now?  Do we have a maintainer?  Is the current codebase "repaired" and simply waiting to pass security?
Comment 74 Daniele Bortoluzzi 2007-10-08 23:28:10 UTC
(In reply to comment #73)
> Out of curiousity, what's the current state of the p.g.o. code right now?  Do
> we have a maintainer?  Is the current codebase "repaired" and simply waiting to
> pass security?
> 

My 2 cents, don't call it bugspam please:

just read all the posts, and seems like Gentoo (infra, security or anybody else) lacks organization, coordination, open source team spirit.
Oh yeah, it's not an enterprise business but it's REAL, tons of sysadmins, developers, poor users work with Gentoo. 
Synergy is the power of Open Source, so why aren't you taking users' contribute seriously? Just let them do their work, for free, for the community, it's a pleasure to help Gentoo, isn't it? Isn't it Open Source? Let the volunteers do this dirty work.

Thank you, and excuse me for "spamming" this bugzilla item. I fell into temptation.
Comment 75 Gordon Malm (RETIRED) gentoo-dev 2007-10-08 23:47:22 UTC
He somehow just removed almost everyone from CC.
Comment 76 Joshua Kinard gentoo-dev 2007-10-09 04:59:39 UTC
> Thank you, and excuse me for "spamming" this bugzilla item. I fell into
> temptation.

You also removed a ton of people from the CC List.  Please watch what you click on next time you fall into "temptation".
Comment 77 Dawid Węgliński (RETIRED) gentoo-dev 2007-10-09 05:05:05 UTC
(In reply to comment #74)
> 
> My 2 cents, don't call it bugspam please:
> 
> just read all the posts, and seems like Gentoo (infra, security or anybody
> else) lacks organization, coordination, open source team spirit.
> Oh yeah, it's not an enterprise business but it's REAL, tons of sysadmins,
> developers, poor users work with Gentoo. 
> Synergy is the power of Open Source, so why aren't you taking users' contribute
> seriously? Just let them do their work, for free, for the community, it's a
> pleasure to help Gentoo, isn't it? Isn't it Open Source? Let the volunteers do
> this dirty work.
> 
> Thank you, and excuse me for "spamming" this bugzilla item. I fell into
> temptation.
> 
First. Can you tell me why on earth did you remove all people from CC list? Second. Can you tell me, what's wrong with this bug you all are so frustrated? Please keep in mind, there *are* developers, who are writing p.g.o almost from scrach plus Taviso is currently away. Third. I have NFC why you feel like not permited to contribute? Code of packages is available in our cvs repository. Write a patch, submit it. You are always more than welcome to do that.
Comment 78 Alexander Skwar 2007-10-09 05:33:39 UTC
(In reply to comment #77)

> Second. Can you tell me, what's wrong with this bug you all are so frustrated?

Simple - the whole handling of this bug is "suboptimal". Users rely on pgo or rather, on the service provided by pgo. When users now go to pgo, they don't find the service they are looking for.

Why don't you, the Gentoo folks, just add a line pointing the users to http://gentoo-portage.com/? I understand that gpc isn't a Gentoo site, but for the time being, it's the best available alternative solution, as far as I know.

Comment 79 Dawid Węgliński (RETIRED) gentoo-dev 2007-10-09 05:46:14 UTC
(In reply to comment #78)
> Why don't you, the Gentoo folks, just add a line pointing the users to
> http://gentoo-portage.com/? I understand that gpc isn't a Gentoo site, but for
> the time being, it's the best available alternative solution, as far as I know.
> 
That's the same reason, why we don't link to gentoo-wiki.com 'for time being'. We do not support it and we don't take responsibility from external services.
Comment 80 Alexander Skwar 2007-10-09 06:45:26 UTC
(In reply to comment #79)
> (In reply to comment #78)
> > Why don't you, the Gentoo folks, just add a line pointing the users to
> > http://gentoo-portage.com/? I understand that gpc isn't a Gentoo site, but for
> > the time being, it's the best available alternative solution, as far as I know.
> > 
> That's the same reason, why we don't link to gentoo-wiki.com 'for time being'.

Exactly. That should be done as well. It would serve your users a good service, "for the time being". 

> We do not support it and we don't take responsibility from external services.

So, what? Of course you don't. Make that clearly visible by a disclaimer. No problem.

Comment 81 Dawid Węgliński (RETIRED) gentoo-dev 2007-10-09 07:37:43 UTC
(In reply to comment #80)
> Exactly. That should be done as well. It would serve your users a good service,
> "for the time being". 
"For time being" methods are not good anyways.
> > We do not support it and we don't take responsibility from external services.
> 
> So, what? Of course you don't. Make that clearly visible by a disclaimer. No
> problem.
Following your mind, we could link to porno sites, and make such a disclaimer. As far as we have no controle under any service, which can be confusing to users, we won't link to it. And please, stop putting pressure on us. From my side, p.g.o will be up as soon as we test it and infra approve it.
Comment 82 Alexander Skwar 2007-10-09 07:40:58 UTC
(In reply to comment #81)
> (In reply to comment #80)
> > Exactly. That should be done as well. It would serve your users a good service,
> > "for the time being". 
> "For time being" methods are not good anyways.
> > > We do not support it and we don't take responsibility from external services.
> > 
> > So, what? Of course you don't. Make that clearly visible by a disclaimer. No
> > problem.
> Following your mind, we could link to porno sites, and make such a disclaimer.

Correct - if Gentoo offered a porno site and that site is down. If Gentoo does not offer a Porno service, then I don't get what you want to try to say.

> As far as we have no controle under any service, which can be confusing to
> users, we won't link to it. And please, stop putting pressure on us. 

No, I won't. It's not as if Gentoo were the first "company" that would link to outside resources, because of a (temporal) lack of own resources. What's so bad about offering users a good service?

> From my
> side, p.g.o will be up as soon as we test it and infra approve it.

Fine. And for the time being, a link from http://packages.gentoo.org/ to http://gentoo-portage.com/ would be a good service.
Comment 83 Codo 2007-10-09 09:17:13 UTC
Stop it guys...

Tavis, have you tested?
Comment 84 Jakub Moc (RETIRED) gentoo-dev 2007-10-09 10:32:19 UTC
(In reply to comment #83)
> Tavis, have you tested?

If you actually *did* read the previous comments, you'd know that Tavis is away ATM. (Plus, this whole p.g.o. thing needs a rewrite from scratch anyway).
Comment 85 Markus Ullmann (RETIRED) gentoo-dev 2007-10-09 11:09:30 UTC
(In reply to comment #84)
> (Plus, this whole p.g.o. thing needs a rewrite from scratch anyway).

Indeed and I'm actively working on it.
Someone interested helping instead of moaning?

http://orion7.digital-server.de/cgi-bin/gitweb.cgi?p=packages.git
preview: http://packages.gentooext.net
Comment 86 Albert W. Hopkins 2007-10-09 12:05:26 UTC
Actually packages.g.o *was* re-written about 3 years ago and was put in the pre_2.0 branch.  However when presented to infra I was told that it wouldn't go in because infra did not want to support another language/framework (in its case, Quixote).

Early this year I offered again to re-write packages.g.o using whatever language/platform infra was able to support (even offering it write it in PHP). Though I never receieved a response.  So pretty much then my perception was that packages.g.o was pretty much not going to be supported by infra (and the events of the past couple of months also this) so I tried to at least get the existing code "packaged" a little nicer in anticipation of my eventual retirement.

I would be happy if the site could eventually get a re-write and hopefully infra will give some backing to it.

OTOH to give the community something here-and-now only requires a few changes to the existing code to make it secure.  IMO they're relatively easy fixes and that should be the priority now.  But again, I don't think the Gentoo developer community agrees with that priority, and that is unfortunate for the Gentoo community as a whole.
Comment 87 Marco Qualizza 2007-10-09 21:09:57 UTC
The reason that I asked is because I do want to help.  I'm a developer with a few years experience (well, 11 professionally, plus a bunch more before that :-) ).  Albert, I'm not trying to doubt your judgement (comment 86) with these next questions, but I don't know the codebase:  Is what Albert said accurate?  *Can* the current codebase be put back live, and be safe, with only a few minor changes?  If so, is there a significant reason for not going down that path as a short-term fix, with the p.g.o. rewrite as the long-term (and "proper") sol'n?
Comment 88 Christian Hoffmann (RETIRED) gentoo-dev 2007-10-10 08:36:20 UTC
(In reply to comment #87)
> The reason that I asked is because I do want to help.  I'm a developer with a
> few years experience (well, 11 professionally, plus a bunch more before that
> :-) ).
So jokey@gentoo.org / #gentoo-guis is your friend :)

>  Albert, I'm not trying to doubt your judgement (comment 86) with these
> next questions, but I don't know the codebase:  Is what Albert said accurate? 
> *Can* the current codebase be put back live, and be safe, with only a few minor
> changes?
I don't think so. The initial bug seems pretty easy to fix, but as already noted above (comment #23), I think there are more problems regarding the handling of user input (SQL injection possibilities, "crashs") and it's hard to detect and fix all of them without ending up with problems like double-escaped input (as escaping seems to be pretty inconsistent). Please note that this is just my personal impression -- maybe I overlooked some checks or it is easier to fix than I think.

The second reason is probably that infra wants a Gentoo developer who maintains the code -- and I did not hear of anyone who would be interested in maintaing the current code base.

If you want to help it would be best to concentrate on one, good solution and that would be jokey's rewrite in this case, IMO.

(Also, this isn't meant as an offence though I know that some comments on the old code sound quite harsh)
Comment 89 Albert W. Hopkins 2007-10-10 12:37:19 UTC
This is a small app, really just an extension of a script I wrote 4 years ago. The code is not that complex or difficult to understand.  Most functions are less than a page long and contain docstrings. The security issues have been identified and most have either have already had fixes published or are trivial.  They were easy to identify and just as easy to fix, something I regret not having done earlier. The few places that require user input or query strings just need a to call the mysql escape function.

Obviously the Gentoo devs call what they want to do.  My comments were only to point out that it is possible to get the site safely back into production quicker than a total rewrite, even though I do agree that a rewrite is due.  My primary concerns about downtime, service to the community, and of course security.

I don't feel offended by any code criticism.  Most of it I already knew anyway.  The code has some history you probably don't know about but I won't talk about it here.
Comment 90 Codo 2007-10-10 14:25:30 UTC
(In reply to comment #88)
> (In reply to comment #87)
> >  Albert, I'm not trying to doubt your judgement (comment 86) with these
> > next questions, but I don't know the codebase:  Is what Albert said accurate? 
> > *Can* the current codebase be put back live, and be safe, with only a few minor
> > changes?
> I don't think so. The initial bug seems pretty easy to fix, but as already
> noted above (comment #23), I think there are more problems regarding the
> handling of user input (SQL injection possibilities, "crashs") and it's hard to
> detect and fix all of them without ending up with problems like double-escaped
> input (as escaping seems to be pretty inconsistent). Please note that this is
> just my personal impression -- maybe I overlooked some checks or it is easier
> to fix than I think.
Hi hoffie.  I disagree with you and agree with marduk.  Though it would nice to rewrite p.g.o, and there were/are a few vulnerabilities in the code, as it stands, the public-facing part of p.g.o is just a few scripts that if fixed (with your comments and perhaps some other few things that may have been overlooked) is good enough to go live and I'm sure it will work as well as it used to.  I think some of the 'don't bring it back because of security concerns' is unfounded.

 
> The second reason is probably that infra wants a Gentoo developer who maintains
> the code -- and I did not hear of anyone who would be interested in maintaing
> the current code base.
I offered to maintain it many times, and Onkobu was very keen as well, and I had an E-mail from someone else (Roeland) that wanted to maintain it as well.  There's plenty of people that want to give a hand here, so lack of maintainers is no excuse.  In fact, when talking to Infra, they have never mentioned that.  The only thing they want is gentoo-security to do what they were meant to (test the site at packagestest.gentoo.org).  I have been bugging them numerous times, but all seem to be away?!?

gentoo-security has to give green light, and with this, the site will go live.  That is all we are asking to close this bug.  That's it.  In fact, all this conversation belongs somewhere else.  Gentoo-security mailing list has been dead for a long time, by the way (I think my post there was the last one).  Maybe we should pursue other ways for it to be tested.  Any dev want to step forward?


> If you want to help it would be best to concentrate on one, good solution and
> that would be jokey's rewrite in this case, IMO.
Here totally agree with you, but in my opinion p.g.o codebase has to come out with an ebuild that anybody can just emerge, and Infra shouldn't have to be involved in anything other than configuring the thing.  If it is not released into the community as something that anybody can hack, modify and submit patches, then with time we will end up at the same place as we are now.

Jokey is releasing p.g.o in [python I believe?].  I am doing it with RubyOnRails.  So you will have two versions in the future to emerge (in your own box).  Infra should just choose and install.  I don't think we should ask Infra to do stuff that they shouldn't be doing in the first place (like looking after it).


> (Also, this isn't meant as an offence though I know that some comments on the
> old code sound quite harsh)
Bring them on!

** And Jakub, keep those kind of comments to yourself please.  Thanks.
Comment 91 David Lütolf 2007-10-31 08:56:50 UTC
Remember what Albert said in his <a href="http://bugs.gentoo.org/show_bug.cgi?id=187971#c38">comment #38</a>... this is pretty much how I felt after reading the first posts.

This space should be left for technical talk, right. The fact is, p.g.o has been down for now almost 3 (!) months, the notice hasn't changed a bit since then and (at least) the first (half of the) posts in this page show no progress. Where do you communicate about how the works are going?

Some time ago, I emerged acx100. The ebuild version was 20070101, the current version being 20071006. That's WAY OVER TEN months old. I posted a note on bugs.g.o but nothing changed until now. It really feels like I'm back to my first Debian Stable and its (almost) 10-year package lifespan.

Albert may have been harsh, but I think he was right. Gentoo may not be a commercial firm, but it does have its "clients" and "adepts". Even some very passionate users and detractors. These deserve to be well-served, or at least informed, and maybe some are even ready to help.

Maybe I should do like Albert - "retire" from gentoo, and switch to slackware?

Now please, I really think you (the gentoo developpers in general) have done some great work. Gentoo definitely has great advantages over about every single distro I can think of. Don't let it become some would-have-been-cool-but-now-history distro, move your butts and do something about it. Would it have been so hard to just remove the "related" feature and tell people what's going on??
Comment 92 younker 2007-11-05 09:14:14 UTC
What's the problem of p.g.o?

I don't want this become the annoucement of gentoo death, 
I love gentoo and I use it everyday, please do help to fix p.g.o, 
it is important to me to check the packages that I need to use. 
Comment 93 James Lamanna 2007-11-07 23:59:24 UTC
You may want to try http://gentoo-portage.com in the meantime.
It's an unofficial p.g.o. stand-in.
Comment 94 Kelly Price 2007-11-08 00:03:52 UTC
gentoo-portage.com is currently down -- inaccessible.  It maybe dead behind a squid reverse proxy.
Comment 95 Timothy J. Warren 2007-11-12 22:13:31 UTC
(In reply to comment #5)

I can't believe no one else has seen this in the three months that this has been going on (and it really torques me that I have to register to give you free advice), but changing
>     <!--#exec cmd="./similar.py $QUERY_STRING" -->
to
>    <!--#exec cmd="export QUERY_STRING;./similar.py" -->

WILL NOT FIX ANY PROBLEMS.  The second piece of code is logically equivalent to the first.  Both can be compromised by passing a semi-colon into QUERY_STRING.  The only difference is that the injected code will run BEFORE similar.py, whereas it used to run AFTER similar.py.

I see that you are re-writing this software, and that may be necessary, but PLEASE do not rely on this suggested change as a fix -- it does not fix anything.

Comment 96 Ilya Eremin 2007-11-18 02:26:00 UTC
packages.gentoo.org is now up, so may be this bug can finally be closed. Though how do you search for a package on the new site??
Comment 97 Lucius Chiaraviglio 2007-11-18 02:54:39 UTC
(In reply to comment #96)

Partial workaround for this (for instance, to search for package firefox):  on
Google, use:

        site:packages.gentoo.org firefox

Note that (continuing with the above example) this doesn't pull up firefox-bin,
so it is not a complete solution to the above problem.  (Not sure if this is
just because the resurrection of packages.gentoo.org is so recent that Google
hasn't finished indexing the site.)
Comment 98 Alexander Skwar 2007-11-18 06:49:20 UTC
Created attachment 136220 [details]
Screenshot of new packages.gentoo.org

Yuck! The looks of pgo is *TERRIBLE*.

Where to report bugs about this?
Comment 99 Dawid Węgliński (RETIRED) gentoo-dev 2007-11-18 08:44:45 UTC
First of all, *please* read the fine FAQ (http://packages.gentoo.org/faq/). 

Alexander: There is already known rendering problem, so once again, refer to the FAQ and you will see short todo list and more details which you may be interested in (like where to file bugs). 

Taviso is currently away, so let me close this bug. Reopen if you feel like it shouldn't be closed yet. Thanks everyone for your patience.
Comment 100 Christian Strahl 2008-08-26 10:32:18 UTC
Hi everyone,

will packages2.gentoo.org get a search function soon? (maybe with regular expressions :D )

Its on the TODO list (http://packages2.gentoo.org/faq/) for about one year, but nothing happend yet.
Comment 101 Christian Hoffmann (RETIRED) gentoo-dev 2008-08-26 10:41:02 UTC
Hallo,

(In reply to comment #100)
> will packages2.gentoo.org get a search function soon? (maybe with regular
> expressions :D )
> 
> Its on the TODO list (http://packages2.gentoo.org/faq/) for about one year, but
> nothing happend yet.
Please ask (or even better, give meaningful comments) on the relevant bug (bug 208376). This bug just handled security issue, which is fixed now and any comments here create unnecessary noise.
BTW: I'd say it's rather unlikely that the search is ever going to support regexps, as using user-supplied regexps is a Bad Thing (tm).