Bug 480408 – Add /releases/${arch}/verified tree for tested autobuilds

• Product: Gentoo Linux
• Status: RESOLVED WONTFIX

[Description Matt Turner 2013-08-09 18:18:31 UTC]

We've had various cases of autobuild ISOs being unusable.

Although the release media is built from the stable portage tree, this alone does not ensure that it is working. Often unworking release media is due to newly required kernel options (recently DEVTMPFS) or fleeting breakage in the stable portage tree.

The autobuild ISOs are mirrored in the /releases/ folder on the mirrors, which implies a certain expectation of quality. That expectation has been too frequently incorrect.

A potential new user's first experience with Gentoo will likely be booting one of these autobuild ISOs. If that proves to be a bad experience, we're likely to lose a new user.

I simply propose that autobuild release media be stored in /experimental/ rather than /releases/ initially. Architecture teams would be tasked periodically with verifying that some recent release media is working and capable of installing Gentoo. Once verified, the media would be moved to the /releases/ folder.

We have various quality-assuring policies and guidelines (30 days in tree before stabilization, et al) for individual packages, but have none for the installation media required to install those packages. My proposal attempts to put in place a policy that will improve release media quality.

I received a unanimously positive response to this suggestion on gentoo-dev@.

Jorge (Cc'd) disagreed with the proposal for reasons I do not understand. I will let him restate them here.

[Comment 1 Tomáš Chvátal (RETIRED) 2013-08-09 18:29:22 UTC]

As our build is pretty straight forward we could include openQA testing we do at openSUSE.

Basically it works like you sent some keystrokes to virtual and it screenshots the progress here and there and you compare the whole screen or parts to match the reality.

Results should be published here http://openqa.opensuse.org/, but they are not right now.

If it works for testing whole yast-install thingy there should be no prob this could not test the gentoo iso.

Btw we have os-autoinst package which is backend in portage (the deps need bit tweaking) and openqa web interface is not ebuilded but available at github.

[Comment 2 Jorge Manuel B. S. Vicetto (RETIRED) 2013-08-09 19:30:03 UTC]

I've already told you I won't support this.
As I'm the one that has been taking care of this for the 3 years or so, I find this bug very "insulting" to me. If your goal is to drive me away from building stages, you're getting very close to be successful.

[Comment 3 Matt Turner 2013-08-09 19:31:56 UTC]

(In reply to Jorge Manuel B. S. Vicetto from comment #2)
> I've already told you I won't support this.
> As I'm the one that has been taking care of this for the 3 years or so, I
> find this bug very "insulting" to me. If your goal is to drive me away from
> building stages, you're getting very close to be successful.

Well... I'm not sure what this is about.

[Comment 4 Jorge Manuel B. S. Vicetto (RETIRED) 2013-08-13 22:19:49 UTC]

I'm reassigning this bug back to release as after talking with Matt we agreed on a proposal and this bug is really a releng issue.

@council:
I've left you on CC so you can choose whether to stay around or not. I don't believe there's anything here for you, but that's your call.

So, here are my comments to the initial proposal:

 * I don't want us to change the locations of autobuilds
     They current go under /releases/${arch}/autobuilds[1] and that's where they should stay (imho)

 * I have no objections to more testing. I've talked about it myself a few times before, including on my talks on FOSDEM and Prague last year.

 * One reason we need to provide "stable" releases is so that we make easier for server providers to offer Gentoo images - such as vr.org and hetzner.de

 [1] - http://distfiles.gentoo.org/releases/amd64/autobuilds/


So here's our proposal for going forward:

 * We keep the current automated release model, having weekly builds and pushing the stages / CDs to the /releases/${arch}/autobuilds

 * periodically we get people testing an automated built stage / CD, and after verification, we push it to /releases/${arch}/stable . I believe getting that done every 6 months would be "good enough" and a reasonable improvement over the current status.

 * we should improve the documentation over the current release dir structure to reflect the following:
    ^ /experimental - custom made stages / CDs or automated builds with experimental features / using experimental tools
    ^ /releases/${arch}/autobuilds - automated weekly builds running on release team hardware
    ^ /releases/${arch}/verified - automated builds that were verified as working (hopefully every 6 months). This could be further split into {testing,stable} so we could have builds for ~arch and arch.
    ^ /testing - we might in the future add weekly ~arch builds here


As I've mentioned with Fabian and Graff on Prague[2][3], we need much more testing, both for releng and for gentoo as a whole. It's important to note the build failures help identify issues in the tree,

 [2] - http://www.gentoo.org/proj/en/miniconf/schedule-2012.xml?style=printable
 [3] - http://blogs.gentoo.org/miniconf-2012/files/presentations/miniconf-2012-automated_testing-jmbsvicetto-grobian-graaff.pdf

[Comment 5 Jorge Manuel B. S. Vicetto (RETIRED) 2013-08-13 22:42:07 UTC]

Sorry for the bugspam, but I forgot to update the title. I changed the proposal while writing the comment.

[Comment 6 Ulrich Müller 2013-09-24 20:49:17 UTC]

http://www.gentoo.org/proj/en/council/meeting-logs/20130813-summary.txt>
"No action to be taken by the council. This should be sorted out within the releng team."

Removing council from CC.

[Comment 7 Jorge Manuel B. S. Vicetto (RETIRED) 2017-02-06 21:10:54 UTC]

Is anyone still interested in this?
The release team kept doing the automated releases, but I don't recall anyone asking us to work on the testing and helping with the "stable" releases.
I'm still open / interested in providing those "stable" releases, as long as someone else is willing to help / do the testing.

[Comment 8 Matt Turner 2017-02-06 21:13:30 UTC]

(In reply to Jorge Manuel B. S. Vicetto from comment #7)
> Is anyone still interested in this?
> The release team kept doing the automated releases, but I don't recall
> anyone asking us to work on the testing and helping with the "stable"
> releases.
> I'm still open / interested in providing those "stable" releases, as long as
> someone else is willing to help / do the testing.

After:

(In reply to Jorge Manuel B. S. Vicetto from comment #2)
> I've already told you I won't support this.
> As I'm the one that has been taking care of this for the 3 years or so, I
> find this bug very "insulting" to me. If your goal is to drive me away from
> building stages, you're getting very close to be successful.

I cannot imagine how you could possibly move forward on this suggestion in a positive way.

[Comment 9 Jorge Manuel B. S. Vicetto (RETIRED) 2017-02-06 23:49:30 UTC]

(In reply to Matt Turner from comment #8)
> (In reply to Jorge Manuel B. S. Vicetto from comment #7)
> > Is anyone still interested in this?
> > The release team kept doing the automated releases, but I don't recall
> > anyone asking us to work on the testing and helping with the "stable"
> > releases.
> > I'm still open / interested in providing those "stable" releases, as long as
> > someone else is willing to help / do the testing.
> 
> After:
> 
> (In reply to Jorge Manuel B. S. Vicetto from comment #2)
> > I've already told you I won't support this.
> > As I'm the one that has been taking care of this for the 3 years or so, I
> > find this bug very "insulting" to me. If your goal is to drive me away from
> > building stages, you're getting very close to be successful.
> 
> I cannot imagine how you could possibly move forward on this suggestion in a
> positive way.

Matt,

you ignored comment 4. I thought we had cleared any misunderstandings back in 2013, but now see that might not have been the case.