Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 814281 - dev-lang/rust improve makeopts for distcc hosts
Summary: dev-lang/rust improve makeopts for distcc hosts
Status: RESOLVED INVALID
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Georgy Yakovlev
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-09-21 21:55 UTC by nic
Modified: 2021-09-24 20:06 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
rust-distcc-support-ebuild patch (rust-ebuild-distcc.patch,1.41 KB, patch)
2021-09-21 21:56 UTC, nic
Details | Diff
rust-1.53.0.ebuild with distcc patch applied (rust-1.53.0.ebuild,20.24 KB, text/plain)
2021-09-21 21:57 UTC, nic
Details

Note You need to log in before you can comment on or make changes to this bug.
Description nic 2021-09-21 21:55:32 UTC
Compiling dev-lang/rust on a distcc configured host consumes all system resources.

 MAKEOPTS="-j22 -l2"


What I am proposing is if makeopts_loadavg float is defined, we round to int and pass as the -j value
Otherwise, if --load-average is undefined we pass makeopts_jobs -j as we are doing now

This maintains a reasonable parallel build in respect to the local load, without excessive io consumption.
Ignoring -j which would normally available for additional distcc jobs

This maintains the current rust workaround in respect to makeopts_jobs vs MAKEOPTS

Reproducible: Always
Comment 1 nic 2021-09-21 21:56:42 UTC
Created attachment 740616 [details, diff]
rust-distcc-support-ebuild patch

applies cleanly to 1.53.0 and 1.55.0
Comment 2 nic 2021-09-21 21:57:50 UTC
Created attachment 740619 [details]
rust-1.53.0.ebuild with distcc patch applied
Comment 3 Georgy Yakovlev archtester gentoo-dev 2021-09-24 17:49:02 UTC
this does not seem like a correct solution to suddenly interpret load-avg as number of jobs, it breaks principle of least surprise.

I suggest to maintain an package.env override for rust and other ebuilds that inhibit similar behavior under distcc.

https://wiki.gentoo.org/wiki/Knowledge_Base:Overriding_environment_variables_per_package

you can just re-define MAKEOPTS there for local-only compilation.
Comment 4 nic 2021-09-24 18:35:31 UTC
Thank you for an alternative approach.

For clarity.
The guidance set for configuring portage with distcc aligns wit the spirit of this patch; "set the value of --load-average to the number of local CPU cores"

https://wiki.gentoo.org/wiki/Distcc#With_Portage

Overall, Gentoo has always advised setting --jobs to the number of local CPU cores. So there's no sudden re-interpretation of either these parameters

https://wiki.gentoo.org/wiki/Handbook:AMD64/Working/Features

The current rust package astonishing disrupts distcc users without clear documentation that a manual local working around is required.
Comment 5 Georgy Yakovlev archtester gentoo-dev 2021-09-24 19:10:19 UTC
yeah I understand it's not very pleasant experience.
but distcc for non-C programs is always tricky, so AFAIK most users are aware of local overrides, it's not new and distcc is not default.

we can easily mention workaround for rust on distcc wiki page for awareness.


btw, we tried integrating https://github.com/mozilla/sccache (it's kinda ccache + distcc hybrid for rust)
with portage, but it's a bit tricky. and looks like project is stalled and there's no recent developments, so I would not recommend relying on it.
I still use it from time to time if I need to build rust or mozilla 10+ times in a row, but never use it for system, just for development compiles.
Comment 6 nic 2021-09-24 19:50:04 UTC
Thanks, I really do appreciate your time and insights.

I dropped an edit into the wiki
https://wiki.gentoo.org/index.php?title=Rust&oldid=1009410
Comment 7 Georgy Yakovlev archtester gentoo-dev 2021-09-24 20:02:05 UTC
btw, I just remembered.
you can override it even simpler


CARGO_BUILD_JOBS=2

in make.conf

will always take priority over MAKEOPTS

I have it partially documented here, but have not moved to main rust article

https://wiki.gentoo.org/wiki/User:GYakovlev/Rust
Comment 8 Georgy Yakovlev archtester gentoo-dev 2021-09-24 20:06:55 UTC
and just realized it may not work for rust itself, but will work for apps compiled with cargo, mixed it up a little bit.

maybe I can move number of jobs to toml file instead of passing on command line.


the option priority is toml -> env -> cmdline