| Summary: | sys-devel/gcc should check available disk space via check-reqs.eclass | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Paweł Stankowski <aambitny> |
| Component: | Current packages | Assignee: | Gentoo Toolchain Maintainers <toolchain> |
| Status: | UNCONFIRMED --- | ||
| Severity: | minor | CC: | arsen, enribnt, gelihay319, gentoo-bugs-augustin, info, j.spam, limanski, martin.dummer, shuber, waebbl-gentoo, whissi |
| Priority: | Normal | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Attachments: |
add disc space requirement check
Results for the size check |
||
|
Description
Paweł Stankowski
2014-10-26 23:13:51 UTC
most likely i'm not going to look into this. but if someone wanted to post a patch ... *** Bug 568280 has been marked as a duplicate of this bug. *** Created attachment 425184 [details, diff]
add disc space requirement check
This patch adds a disc space requirement check.
(In reply to Martin Dummer from comment #3) what USE flags did you check against ? what target ? if you build with just C/C++ on x86, the requirements will be much lighter than including fortran, openmp, sanitizer, etc... on x86_64. (In reply to SpanKY from comment #4) > (In reply to Martin Dummer from comment #3) > > what USE flags did you check against ? what target ? if you build with > just C/C++ on x86, the requirements will be much lighter than including > fortran, openmp, sanitizer, etc... on x86_64. On a amd64/13.0/desktop/plasma system with cxx fortran nls nptl openmp sanitize vtv flags compilation of sys-devel/gcc-5.4.0 required more than 3.8G disk space thus a requirement of having free 4G sounds sane. *** Bug 636858 has been marked as a duplicate of this bug. *** A check would be very nice. It is annoying when emerge will fail after 30+ minutes because you just run out of disk space... I've started a script on my machine to build gcc 7.2.0-r1 with only one use flag enabled at a time to compare results of `du -hs` on each case, and do one final build with all use flags enabled and see if it adds up more or less. I'll post the results when done (expect a day or more if my script sucks and fails). It is important that you continually record the size while the PM is building the package, not just at the end, to get the peak. Noted, I added a watcher on the directory size every 15 seconds. Some data from my amd64 machine: /gentoo/build-logs/cross-x86_64-HEAD-linux-gnu:gcc-7.2.0-r1:20180109-052250.log: * Final size of build directory: 1635164 KiB ( 1.5 GiB) /gentoo/build-logs/cross-x86_64-HEAD-linux-gnu:gcc-7.2.0-r1:20180109-052250.log: * Final size of installed tree: 239204 KiB (233.5 MiB) /gentoo/build-logs/cross-x86_64-UNREG-linux-gnu:gcc-7.2.0-r1:20180109-054312.log: * Final size of build directory: 1635456 KiB ( 1.5 GiB) /gentoo/build-logs/cross-x86_64-UNREG-linux-gnu:gcc-7.2.0-r1:20180109-054312.log: * Final size of installed tree: 239220 KiB (233.6 MiB) /gentoo/build-logs/cross-x86_64-w64-mingw32:gcc-7.2.0-r1:20180109-061748.log: * Final size of build directory: 1440976 KiB ( 1.3 GiB) /gentoo/build-logs/cross-x86_64-w64-mingw32:gcc-7.2.0-r1:20180109-061748.log: * Final size of installed tree: 173700 KiB (169.6 MiB) /gentoo/build-logs/sys-devel:gcc-6.4.0-r1:20180109-013559.log: * Final size of build directory: 4777812 KiB (4.5 GiB) /gentoo/build-logs/sys-devel:gcc-6.4.0-r1:20180109-013559.log: * Final size of installed tree: 1221312 KiB (1.1 GiB) /gentoo/build-logs/sys-devel:gcc-7.2.0-r1:20180106-205620.log: * Final size of build directory: 4339428 KiB (4.1 GiB) /gentoo/build-logs/sys-devel:gcc-7.2.0-r1:20180106-205620.log: * Final size of installed tree: 1261860 KiB (1.2 GiB) Here cross-* builds are built on amd64 with '-O2 -pipe' and native builds are '-O2 -ggdb3' (opposite sides). The baseline seems to be about 1.5G of build tree including install it's about 2G. -ggdb3 presence does it about 6G (~3x larger). I've ran the tests on a fresh stage3 chroot install. The size of /var/tmp/portage/sys-devel/gcc has been checked with `du -hs` every 15 seconds, there have been some loss when the command was ran and the make job removed some temporary files but I doubt the difference would be more than ~300MB as this is the greatest variation I've noticed between two checks. Ok, here comes the peak sizes when a *single* use flag is enabled. $USE Max size cilk 2.1G cxx 2.1G debug 2.4G doc 2.1G fortran 2.3G go 2.1G graphite 2.1G mpx 2.1G nptl 2.1G objc 2.3G objc++ 2.1G objc-gc 2.1G openmp 2.2G pch 2.9G pgo 3.0G pie 2.1G regression-test 2.1G sanitize 2.1G ssp 2.1G vanilla 2.1G vtv 2.2G Same thing ordered go 2.1G cxx 2.1G doc 2.1G mpx 2.1G pie 2.1G ssp 2.1G cilk 2.1G nptl 2.1G objc++ 2.1G objc-gc 2.1G vanilla 2.1G graphite 2.1G sanitize 2.1G regression-test 2.1G vtv 2.2G openmp 2.2G objc 2.3G fortran 2.3G debug 2.4G pch 2.9G pgo 3.0G I am now running the config with all useflags disabled, then enabled. If the differences are too high I’ll see if we don't have to do some other tests with mixed useflag configurations. I'm correcting myself, the greatest difference of size between two checks was 0.5GB. Sorry for the double post noise. (In reply to Nado from comment #13) > I'm correcting myself, the greatest difference of size between two checks > was 0.5GB. > Sorry for the double post noise. Now the same with -g in C*FLAGS please... :) (In reply to Andreas K. Hüttel from comment #14) > (In reply to Nado from comment #13) > > I'm correcting myself, the greatest difference of size between two checks > > was 0.5GB. > > Sorry for the double post noise. > > Now the same with -g in C*FLAGS please... :) As soon as the current build (all flags) is done ;) gcc without any use flag enabled has a peak of 2.1GB Also what makes a big difference is multilib vs no-multilib. Alright, I’ll try to test the 4 configs then : {no-,}multilib * with{,out} -g
I’ll report all of them together after it's done, in a week or two at most.
If you have any other suggestion let me know.
Created attachment 517184 [details] Results for the size check Here are the results, they’re not complete, I realized I got some build failures, there must be some in the previous results since I forgot to check for errors before. I also detected that USE="-* go" is useless without cxx (see https://bugs.gentoo.org/645090) so I'll set other builds to cover it better. *** Bug 643172 has been marked as a duplicate of this bug. *** This now hit me too for gcc-8.2.0-r6 on my machine after ~4 hours of compilation. Kinda frustrating. The output at the end of src_install tells me it needs 8.4 GiB for the build directory plus 472.8 MiB for the install tree. I'm using the USE flags [ebuild R ] sys-devel/gcc-8.2.0-r6 USE="cxx doc fortran graphite (multilib) nls nptl openmp pch pgo (pie) sanitize ssp vtv (-altivec) -debug (-fixed-point) -go (-hardened) (-jit) (-libssp) -mpx -objc -objc++ -objc-gc -regression-test -systemtap -vanilla" I'm aware that the check-reqs concept is a fragile one, the size depending on many factors. But I wonder how many ebuild maintainers manage to give at least a rough estimate for this value, while maintainers of a critical system package don't get this done. IMO it'd be better to have a false positive, slightly increasing the compile time, than having double the compile time due to ebuild failure because of a OOM event when the build runs out of the tmp disk size. Please consider adding a check-reqs to packages with prolonged build times. (In reply to Bernd from comment #20) > This now hit me too for gcc-8.2.0-r6 on my machine after ~4 hours of > compilation. Kinda frustrating. > > The output at the end of src_install tells me it needs 8.4 GiB for the build > directory plus 472.8 MiB for the install tree. > > I'm using the USE flags > [ebuild R ] sys-devel/gcc-8.2.0-r6 USE="cxx doc fortran graphite > (multilib) nls nptl openmp pch pgo (pie) sanitize ssp vtv (-altivec) -debug > (-fixed-point) -go (-hardened) (-jit) (-libssp) -mpx -objc -objc++ -objc-gc > -regression-test -systemtap -vanilla" > > I'm aware that the check-reqs concept is a fragile one, the size depending > on many factors. But I wonder how many ebuild maintainers manage to give at > least a rough estimate for this value, while maintainers of a critical > system package don't get this done. > > IMO it'd be better to have a false positive, slightly increasing the compile > time, than having double the compile time due to ebuild failure because of a > OOM event when the build runs out of the tmp disk size. > > Please consider adding a check-reqs to packages with prolonged build times. 2-to-9 GB is not a very useful error margin. It's hard to put a reasonable number here. (In reply to Sergei Trofimovich from comment #21) > 2-to-9 GB is not a very useful error margin. It's hard to put a reasonable > number here. I stopped trying to compile gcc in a tmpfs due to this wide range and rather compile it on disk, taking reasonably longer, but still at a reasonable time as well and without risk of running out of space. *** Bug 816792 has been marked as a duplicate of this bug. *** *** Bug 830568 has been marked as a duplicate of this bug. *** There seems to be no progress on this bug because the interval [MIN, MAX] of all possible space footprints is too large. I would like to add two thoughts: 1. Even MIN is considerably high, so why not at least add this, which is an improvement over no check at all? 2. Why not take the footprint required for building the amd64/23.0 binpkg as a sane guideline? is there some reason not to use bootstrap-lean on all arches? that seems like it'd reduce the MIN. We already do, right? https://github.com/gentoo/gentoo/blob/master/eclass/toolchain.eclass#L1781 yes - misread the code. disregard |
