Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 725906 - request: amd64 CI builder machine at AWS
Summary: request: amd64 CI builder machine at AWS
Status: RESOLVED WONTFIX
Alias: None
Product: Gentoo Infrastructure
Classification: Unclassified
Component: Dev box issues (show other bugs)
Hardware: All Linux
: Normal normal
Assignee: Gentoo Infrastructure
URL:
Whiteboard: aws
Keywords:
Depends on:
Blocks:
 
Reported: 2020-05-28 17:14 UTC by Robin Johnson
Modified: 2022-06-16 17:24 UTC (History)
5 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2020-05-28 17:14:05 UTC
Copying from ago's email:

Hello Robin,

the aws machine that I'm using produced well results:
https://bugs.gentoo.org/buglist.cgi?bug_status=UNCONFIRMED&bug_status=CONFIRMED&bug_status=IN_PROGRESS&bug_status=RESOLVED&f1=longdesc&f2=reporter&limit=0&list_id=4630028&o1=substring&o2=equals&order=changeddate%2Cpriority%2Cbug_severity&query_format=advanced&resolution=---&resolution=FIXED&resolution=INVALID&resolution=WONTFIX&resolution=LATER&resolution=REMIND&resolution=DUPLICATE&resolution=WORKSFORME&resolution=CANTFIX&resolution=NEEDINFO&resolution=TEST-REQUEST&resolution=UPSTREAM&resolution=OBSOLETE&v1=discovered%20on%20amd64&v2=ago%40gentoo.org

Since we have credits I'm wondering if it is possible to have another machine.

My idea is that while the first machine compiles the tree against some certain
problems ( e.g. fno-common, static-libs, CC respect ) and takes a while until
all packages are compiled, the second machine will follow the git log. So if a
package A is touched, then it will be compiled, this includes also when new
packages joins the tree.
It will act as CI

On power side, for that task, I don't need the same machine (a1.2xlarge I
guess) but something small like a1.large is fine.
100GB disk is fine

Obviously is aws won't renew the credit we can shutdown it without problems as
it is a 'plus'
Comment 1 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2020-05-28 17:14:45 UTC
ago: request granted.

slashbeast: have you got enough tooling to help fill this request in an automated way now?
Comment 2 Agostino Sarubbo gentoo-dev 2020-05-28 17:29:41 UTC
I confused the aws instance names. I need amd64.
2 cores / 4gb ram is fine
Comment 3 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2020-05-28 17:54:59 UTC
(In reply to Agostino Sarubbo from comment #2)
> I confused the aws instance names. I need amd64.
> 2 cores / 4gb ram is fine

Yeah, I was wondering why you'd do that on arm64 ;-).
Comment 4 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2020-05-28 17:56:01 UTC
That said, it'd be nice to sync this with Toralf.  Having two instances do the same thing is a waste of energy.  Plus, his reports are much nicer.
Comment 5 Piotr Karbowski (RETIRED) gentoo-dev 2020-05-28 20:07:49 UTC
(In reply to Robin Johnson from comment #1)
> slashbeast: have you got enough tooling to help fill this request in an
> automated way now?

There's a concept that is not yet implemented as we're a bit stuck with antarus on the integration with SAML, once that's sorted out I will.

To unblock ago I will just re-use the terraform code that was used for him already to spawn another machine, with 2nd block device just like the first one.
Comment 6 Agostino Sarubbo gentoo-dev 2020-05-29 07:19:19 UTC
(In reply to Michał Górny from comment #4)
> That said, it'd be nice to sync this with Toralf.  Having two instances do
> the same thing is a waste of energy.  Plus, his reports are much nicer.

The difference is his reports are filed manually while mine not. So I put a generic summary e.g. (fails to compile) and if maintainer want to specify the exact error is on his own.

Apart that a bug report should contain the log and emerge --info. If you have suggestions or want additions no problem but keep in sync with the others because as you may see, we always have people that want things in different way
Comment 7 Piotr Karbowski (RETIRED) gentoo-dev 2020-05-30 10:51:41 UTC
VM created, terraform present in infra-as-code.git commit c5d4098
Comment 8 Agostino Sarubbo gentoo-dev 2020-08-05 14:02:35 UTC
If possible I would like to request a volume of 200GB. Atm it is 100GB.
TIA.
Comment 9 Piotr Karbowski (RETIRED) gentoo-dev 2020-08-07 19:36:05 UTC
Done in 49e693b
Comment 10 Agostino Sarubbo gentoo-dev 2020-09-10 06:41:01 UTC
Hello,

as discussed on irc, if possible I would like to move from 16 cores to 32 or 64.

machine ip is: 18.195.254.18


Thanks
Comment 11 Alec Warner (RETIRED) archtester gentoo-dev Security 2020-09-10 16:04:43 UTC
(In reply to Agostino Sarubbo from comment #10)
> Hello,
> 
> as discussed on irc, if possible I would like to move from 16 cores to 32 or
> 64.
> 
> machine ip is: 18.195.254.18
> 
> 
> Thanks

18.159.62.53 is the new IP; now with more cores.

-A
Comment 12 Agostino Sarubbo gentoo-dev 2020-10-30 09:08:15 UTC
Hello,

since the aws machines will be suspended tonight, I'm wondering if it is possible to request a machine for Continuous Integration (for example one from here: https://www.hetzner.com/dedicated-rootserver/matrix-ax)

Thanks
Comment 13 Alec Warner (RETIRED) archtester gentoo-dev Security 2020-11-04 23:28:20 UTC
(In reply to Agostino Sarubbo from comment #12)
> Hello,
> 
> since the aws machines will be suspended tonight, I'm wondering if it is
> possible to request a machine for Continuous Integration (for example one
> from here: https://www.hetzner.com/dedicated-rootserver/matrix-ax)
> 
> Thanks

I think we are interested in the source code for the CI system; are you willing to made it available if we fund this effort?

-A
Comment 14 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2020-11-05 08:04:22 UTC
For the record, QA has received multiple complaints about the quality of ago's work. I would think twice before allocating money to such a controversial effort.
Comment 15 Agostino Sarubbo gentoo-dev 2020-11-05 16:13:47 UTC
Since people complained about quality of report, I will not spend my time into doing this.
Comment 16 Piotr Karbowski (RETIRED) gentoo-dev 2020-11-05 22:19:10 UTC
Would that be possible that you will share the sources and scripts that you used, so maybe someone would be willing to contribute to the quality of records and resume project?
Comment 17 Piotr Karbowski (RETIRED) gentoo-dev 2020-11-29 18:11:35 UTC
AWS is back, ago, reach out to me on IRC so I can bring the hosts back for you. Their public DNS and IP will change so you need to configure it, if they're interconnected.
Comment 18 Piotr Karbowski (RETIRED) gentoo-dev 2020-12-15 22:02:46 UTC
i-060acc4e268dcee0c and i-01f507d0cd20034fa are back online, both in eu-central-1.
Comment 19 Aaron Bauman (RETIRED) gentoo-dev 2021-06-28 18:58:24 UTC
This request has morphed into providing a Hetzner box for ago's efforts. 

As a trustee, I am currently not satisfied with the request that ago publish ALL of his source code for the tinderbox. As such, I am against funding any resources to support ago's box until all source code is published. This includes the use of any "free" AWS credits (as they are given for open source work) or foundation monies to purchase a server from Hetzner.

This is a very simple request based on the cornerstone of our community... open source code.

This can be published in a personal repo that remains accessible, but preferably on a Gentoo owned dev repo to preserve the code.
Comment 20 Agostino Sarubbo gentoo-dev 2021-06-28 20:23:13 UTC
(In reply to Aaron Bauman from comment #19)
> This request has morphed into providing a Hetzner box for ago's efforts. 
> 
> As a trustee, I am currently not satisfied with the request that ago publish
> ALL of his source code for the tinderbox. As such, I am against funding any
> resources to support ago's box until all source code is published. This
> includes the use of any "free" AWS credits (as they are given for open
> source work) or foundation monies to purchase a server from Hetzner.
> 
> This is a very simple request based on the cornerstone of our community...
> open source code.
> 
> This can be published in a personal repo that remains accessible, but
> preferably on a Gentoo owned dev repo to preserve the code.

Aaron, thanks for the feedback.

As already stated on irc let's write a summary.

The main code for the tinderbox/ci has been published. Atm there are auxiliary parts that are not in a good state and I would like to refactor before publish.

Trustees please vote if it is worth rent an Hetzner machine for this purpose considering what Aaron said.

Result of tinderboxing/CI are always visible here:
https://bit.ly/3dn9j9Y
Comment 21 Aaron Bauman (RETIRED) gentoo-dev 2021-06-29 00:18:52 UTC
(In reply to Agostino Sarubbo from comment #20)
> (In reply to Aaron Bauman from comment #19)
> > This request has morphed into providing a Hetzner box for ago's efforts. 
> > 
> > As a trustee, I am currently not satisfied with the request that ago publish
> > ALL of his source code for the tinderbox. As such, I am against funding any
> > resources to support ago's box until all source code is published. This
> > includes the use of any "free" AWS credits (as they are given for open
> > source work) or foundation monies to purchase a server from Hetzner.
> > 
> > This is a very simple request based on the cornerstone of our community...
> > open source code.
> > 
> > This can be published in a personal repo that remains accessible, but
> > preferably on a Gentoo owned dev repo to preserve the code.
> 
> Aaron, thanks for the feedback.
> 
> As already stated on irc let's write a summary.
> 
> The main code for the tinderbox/ci has been published. Atm there are
> auxiliary parts that are not in a good state and I would like to refactor
> before publish.
> 
> Trustees please vote if it is worth rent an Hetzner machine for this purpose
> considering what Aaron said.
> 
> Result of tinderboxing/CI are always visible here:
> https://bit.ly/3dn9j9Y

As stated on IRC, this same reasoning has been used for several months (at least). Publishing the code assures others that you are abiding by community interests and principles (free and open software) and others can join in helping improve it.
Comment 22 John Helmert III archtester Gentoo Infrastructure gentoo-dev Security 2022-06-16 17:24:28 UTC
No longer using AWS devboxes.