Currently planned features: - recompile all new ebuilds upon commit - compile with all combos of the ebuild's supported use settings - generate a tbz2 and an rpm for each of - the ebuild settings - optimization profiles (AMD, PIII, P5, 386) - generate bug reports directly into bugzilla when builds fail
This is an excellent idea for a lot of reasons. First it will be an excellent way to verify the compilability of ebuilds, and second it will, given a set of standard tests, verify the functionality of portage itself. A tinderbox designed to bootstrap and emerge system would also be highly advisable. On several occasions portage has been blessed with bugs that allowed either bootstrap or emerge system to finish, but left the filesystem in a highly inconsistent state (all symlinks dead, broken man-pages, missing library files, etc). I have an Athlon system I can put toward this if it's needed. Please let me know if it's useful.
Since being educated on the concept of a tinderbox, I think it's a very excellent idea for Gentoo. With the growing pains Gentoo is sure to endure, it would be a great step to begin thorough testing. I'll throw my workstations' spare CPU cycles at it if need be.
It's just a matter of getting around to it, as usual....
Okay, so an update on the situation is in order. I actually have a design for this, and a very rudimentary prototype for it running on my setup. It is based on the idea of distributed tasks (using a DHT), where client tinderboxes can subscribe to one central point of failure, err, server, to obtain tasks. I've also planned, but not implemented a facility for PKI, so that we may know the trust-level of the participating tinderbox clients. All of this is publicly available in my source directory on my laptop for now:/
So this would pull in updated packages (from world) and test builds? Have you looked at John Keiser's Tinderbox 3? http://www.johnkeiser.com/mozilla/tbox3.html
It's a really nice idea. I can help to test it on a few pcs, just let me know when it will be ready. "- generate a tbz2 and an rpm for each of - the ebuild settings - optimization profiles (AMD, PIII, P5, 386)" Why is this needed?
Not sure about the rpm part. But I've got the basis of several other aspects touched on in this bug for several profiles and arches going at. http://tinderbox.x86.dev.gentoo.org/ Testing every single combo of every single USE= flag on every single commit simply wont work as you will run into blockers and the likes. Plus you would need a pretty damn big cluster of build hosts for that.
One could generate a few sets of "average" USE/CFLAG combinations from the (currently non-existent?) gentoo-statistics project so not every single combination is checked. This way, one would avoid silly cflag- and/or use-combinations
karltk retired.
*** Bug 240199 has been marked as a duplicate of this bug. ***
QA should provide a tinderbox.
(In reply to Julian Ospald (hasufell) from comment #11) > QA should provide a tinderbox. Probably, but if you want it official - we need support from Infra as well
(In reply to Sergey Popov from comment #12) > (In reply to Julian Ospald (hasufell) from comment #11) > > QA should provide a tinderbox. > > Probably, but if you want it official - we need support from Infra as well The last tinderbox (run by solar) wasn't 'official.' I don't want Infra support to be some sort of blocker here (we have other commitments to handle at the moment.) Get some hardware, play around, get it working. Infra can move to own if (if we even want to own it) once it is running and proven somewhat stable. -A
Depends on whether you consider build failures and stuff like that to be QA's problem. Further depends on getting hardware--I for one don't exactly have boxes lying around which have nothing better to do than build packages all day. We can discuss this at the next meeting, but I wouldn't suggest getting your hopes up.
(In reply to Chris Reffett from comment #14) > Depends on whether you consider build failures and stuff like that to be > QA's problem. Further depends on getting hardware--I for one don't exactly > have boxes lying around which have nothing better to do than build packages > all day. We can discuss this at the next meeting, but I wouldn't suggest > getting your hopes up. I am not really sure if I understand that reasoning. Testing is the very center of Quality Assurance (that's what I learned... correct me if I am wrong). Gentoo as a distribution ships the portage tree. If no one tests the tree on a global basis (arch testers don't), then there is not much QA overall. The quality of our tree inherently depends on the compileability of it's packages. Further, there are a lot of use cases where a developer might want/need to request a tinderbox run with a certain package unmasked, a certain eclass changed etc. In the end, it directly affects the user.
I have a tinderbox class system. I build 4400 packages a day, give or take. Problem is, it's the same 4400 packages, and bug reports are entirely non-automated. If you would like to help setup something better, here I am.
(In reply to Rick Farina (Zero_Chaos) from comment #16) > I have a tinderbox class system. I build 4400 packages a day, give or take. > Problem is, it's the same 4400 packages, and bug reports are entirely > non-automated. If you would like to help setup something better, here I am. We should fix bugs first before adding more of them; reviving Tinderbox would be nice for when we run out of bugs, but that's definitely not the case yet today. Consider to mark this RESOLVED LATER again...
(In reply to Tom Wijsman (TomWij) from comment #17) > (In reply to Rick Farina (Zero_Chaos) from comment #16) > > I have a tinderbox class system. I build 4400 packages a day, give or take. > > Problem is, it's the same 4400 packages, and bug reports are entirely > > non-automated. If you would like to help setup something better, here I am. > > We should fix bugs first before adding more of them; reviving Tinderbox > would be nice for when we run out of bugs, but that's definitely not the > case yet today. > > Consider to mark this RESOLVED LATER again... lolwat?
(In reply to Tom Wijsman (TomWij) from comment #17) > We should fix bugs first before adding more of them; reviving Tinderbox > would be nice for when we run out of bugs, but that's definitely not the > case yet today. Should we replace enter_bug.cgi with a website saying "sorry, we have enough bugs for now" as well?
QA team, please decide what we do with this bug; given that the recent answers by others are unacceptable behavior per the Gentoo CoC, this bug is private for now.
How about you guys fight this out and talk to infra when there's actually something we can do except getting our inboxes filled with this nonsense. (In reply to Tom Wijsman (TomWij) from comment #20) > QA team, please decide what we do with this bug; given that the recent > answers by others are unacceptable behavior per the Gentoo CoC, this bug is > private for now. <hat type="infra"> I don't really see anything that needs to be hidden from public view, especially not to QA. Social contract says we don't hide things, CoC says it's up to comrel to take measures, so until they think this needs to not be public, it won't be restricted. </hat>
(In reply to Julian Ospald (hasufell) from comment #18) > lolwat? (In reply to Alexander Berntsen from comment #19) > Should we replace enter_bug.cgi with a website saying "sorry, we have enough > bugs for now" as well? Can you two please appropriately use this bug and take your jokes elsewhere? https://wiki.gentoo.org/wiki/Project:Council/Code_of_conduct (In reply to Alex Legler from comment #21) > it won't be restricted. So, why do we have that checkbox then? You've just publicized a private QA vote.
> (In reply to Alex Legler from comment #21) > > it won't be restricted. > > So, why do we have that checkbox then? You've just publicized a private QA > vote. There is no voting going on as far as I can see. If you need to hold a vote, do so on a new private bug or even better via email. Locking out the interested audience from the complete discussion is not a valid choice.
(In reply to Alex Legler from comment #23) > There is no voting going on as far as I can see. If you need to hold a vote, > do so on a new private bug or even better via email. Okay, thank you. > Locking out the > interested audience from the complete discussion is not a valid choice. True, we could only lock those out who are no longer interested; similar to what you've told in Comment #21, I'm tired as well of inboxes filling with "lolwat".
I've talked a few times on conferences about getting testing done. Part of the testing could be based on a tinderbox. Part of the tests I'd like to have are best done through a virtualization environment or involve chroots. AFAIK, besides Ned (solar) and Diego (flameeyes), the developers that have looked at and keep doing this type of testing are Patrick (bonsaikitten) and Magnus (zorry). I suggest anyone really interested in this bug to talk to them. Rick already said he's doing the same scope of testing, so he's someone else to contact.
Tom, your comment about running out of bugs is just plain silly. Julian, please refrain from unnecessary lolwuts, that's also not helping. Otherwise I see absolutely no business for comrel here. I suggest from now on only people who have actually set up a large-scale tinderbox and are willing to do that again should comment with real content. Set up a build system, build a few thousand packages, file a few hundred real bugs. Then, come back here and we get talking. No, no "ping" reminders either.
(In reply to Andreas K. Hüttel from comment #26) > I suggest from now on only people who have actually set up a large-scale > tinderbox and are willing to do that again should comment with real content. > Set up a build system, build a few thousand packages, file a few hundred > real bugs. Then, come back here and we get talking. I understand and agree that the behaviour of Tom and Julian above was childish. But that comment raises a few questions. Is this the kind of Gentoo we want? What I mean is, just because we have a few badly behaving developers, others with ideas or constructive criticism can't contribute to a particular area because they're not part of a happy few? Is this a notion we want to extend to all of Gentoo? Thanks, Denis.
(In reply to Andreas K. Hüttel from comment #26) > Julian, please refrain from unnecessary lolwuts, that's also not helping. > I'm sorry about that. But in fact... that was exactly what I was thinking in the moment I hit "save changes". There was no intention of derailing/trolling, it was just what I was thinking.
I've been doing some work on the collection/reporting side of this. In the qa-scripts repo I committed a portage bashrc[1] that collects and stores some information about packages. This can be processed to produce reports like this: https://dev.gentoo.org/~kensington/tinderbox/ Testing and feedback on the script / output is of course welcomed. It's easy to combine output from multiple sources, so I'd be keen to receive output tarballs from anyone building a decent number of packages to test the parser. 1: http://git.overlays.gentoo.org/gitweb/?p=proj/qa-scripts.git;a=blob;f=tinderbox/bashrc
