Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 526986 - sys-devel/gcc should check available disk space via check-reqs.eclass
Summary: sys-devel/gcc should check available disk space via check-reqs.eclass
Status: UNCONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All All
: Normal minor with 1 vote (vote)
Assignee: Gentoo Toolchain Maintainers
URL:
Whiteboard:
Keywords:
: 568280 636858 643172 (view as bug list)
Depends on:
Blocks:
 
Reported: 2014-10-26 23:13 UTC by Paweł Stankowski
Modified: 2019-12-27 22:36 UTC (History)
8 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
add disc space requirement check (gcc-5.3.0-check-diskspace.diff,766 bytes, patch)
2016-02-11 06:59 UTC, Martin Dummer
Details | Diff
Results for the size check (results,3.87 KB, text/plain)
2018-01-30 20:56 UTC, Nado
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Paweł Stankowski 2014-10-26 23:13:51 UTC
When emerging gcc, available disk space for compilation is not checked, and gcc compilation takes excessive amount of it.

I don't have precise maeasurements done, but I noticed that at the end of stage2 build folder was 2.6 GB big for following use flags on amd64 architecture: cxx nls nptl openmp sanitize. Just after end of stage size occupied by package was decreasing rapidly.
Comment 1 SpanKY gentoo-dev 2014-10-27 00:06:21 UTC
most likely i'm not going to look into this.  but if someone wanted to post a patch ...
Comment 2 SpanKY gentoo-dev 2015-12-14 22:43:00 UTC
*** Bug 568280 has been marked as a duplicate of this bug. ***
Comment 3 Martin Dummer 2016-02-11 06:59:26 UTC
Created attachment 425184 [details, diff]
add disc space requirement check

This patch adds a disc space requirement check.
Comment 4 SpanKY gentoo-dev 2016-05-11 21:49:35 UTC
(In reply to Martin Dummer from comment #3)

what USE flags did you check against ?  what target ?  if you build with just C/C++ on x86, the requirements will be much lighter than including fortran, openmp, sanitizer, etc... on x86_64.
Comment 5 MarisN 2016-06-28 17:24:43 UTC
(In reply to SpanKY from comment #4)
> (In reply to Martin Dummer from comment #3)
> 
> what USE flags did you check against ?  what target ?  if you build with
> just C/C++ on x86, the requirements will be much lighter than including
> fortran, openmp, sanitizer, etc... on x86_64.

On a amd64/13.0/desktop/plasma system with cxx fortran nls nptl openmp sanitize vtv flags compilation of sys-devel/gcc-5.4.0 required more than 3.8G disk space thus a requirement of having free 4G sounds sane.
Comment 6 Andreas K. Hüttel gentoo-dev 2017-12-16 23:21:12 UTC
*** Bug 636858 has been marked as a duplicate of this bug. ***
Comment 7 Thomas Deutschmann gentoo-dev Security 2018-01-11 02:18:54 UTC
A check would be very nice. It is annoying when emerge will fail after 30+ minutes because you just run out of disk space...
Comment 8 Nado 2018-01-11 16:14:02 UTC
I've started a script on my machine to build gcc 7.2.0-r1 with only one use flag enabled at a time to compare results of `du -hs` on each case, and do one final build with all use flags enabled and see if it adds up more or less.

I'll post the results when done (expect a day or more if my script sucks and fails).
Comment 9 Thomas Deutschmann gentoo-dev Security 2018-01-11 16:16:20 UTC
It is important that you continually record the size while the PM is building the package, not just at the end, to get the peak.
Comment 10 Nado 2018-01-11 16:40:42 UTC
Noted, I added a watcher on the directory size every 15 seconds.
Comment 11 Sergei Trofimovich gentoo-dev 2018-01-11 22:43:47 UTC
Some data from my amd64 machine:

/gentoo/build-logs/cross-x86_64-HEAD-linux-gnu:gcc-7.2.0-r1:20180109-052250.log: * Final size of build directory: 1635164 KiB (  1.5 GiB)
/gentoo/build-logs/cross-x86_64-HEAD-linux-gnu:gcc-7.2.0-r1:20180109-052250.log: * Final size of installed tree:   239204 KiB (233.5 MiB)
/gentoo/build-logs/cross-x86_64-UNREG-linux-gnu:gcc-7.2.0-r1:20180109-054312.log: * Final size of build directory: 1635456 KiB (  1.5 GiB)
/gentoo/build-logs/cross-x86_64-UNREG-linux-gnu:gcc-7.2.0-r1:20180109-054312.log: * Final size of installed tree:   239220 KiB (233.6 MiB)
/gentoo/build-logs/cross-x86_64-w64-mingw32:gcc-7.2.0-r1:20180109-061748.log: * Final size of build directory: 1440976 KiB (  1.3 GiB)
/gentoo/build-logs/cross-x86_64-w64-mingw32:gcc-7.2.0-r1:20180109-061748.log: * Final size of installed tree:   173700 KiB (169.6 MiB)
/gentoo/build-logs/sys-devel:gcc-6.4.0-r1:20180109-013559.log: * Final size of build directory: 4777812 KiB (4.5 GiB)
/gentoo/build-logs/sys-devel:gcc-6.4.0-r1:20180109-013559.log: * Final size of installed tree:  1221312 KiB (1.1 GiB)
/gentoo/build-logs/sys-devel:gcc-7.2.0-r1:20180106-205620.log: * Final size of build directory: 4339428 KiB (4.1 GiB)
/gentoo/build-logs/sys-devel:gcc-7.2.0-r1:20180106-205620.log: * Final size of installed tree:  1261860 KiB (1.2 GiB)

Here cross-* builds are built on amd64 with '-O2 -pipe' and native builds are '-O2 -ggdb3' (opposite sides).

The baseline seems to be about 1.5G of build tree including install it's about 2G. -ggdb3 presence does it about 6G (~3x larger).
Comment 12 Nado 2018-01-13 17:17:19 UTC
I've ran the tests on a fresh stage3 chroot install. The size of /var/tmp/portage/sys-devel/gcc has been checked with `du -hs` every 15 seconds, there have been some loss when the command was ran and the make job removed some temporary files but I doubt the difference would be more than ~300MB as this is the greatest variation I've noticed between two checks.

Ok, here comes the peak sizes when a *single* use flag is enabled.

$USE             Max size
cilk             2.1G
cxx              2.1G
debug            2.4G
doc              2.1G
fortran          2.3G
go               2.1G
graphite         2.1G
mpx              2.1G
nptl             2.1G
objc             2.3G
objc++           2.1G
objc-gc          2.1G
openmp           2.2G
pch              2.9G
pgo              3.0G
pie              2.1G
regression-test  2.1G
sanitize         2.1G
ssp              2.1G
vanilla          2.1G
vtv              2.2G

Same thing ordered

go               2.1G
cxx              2.1G
doc              2.1G
mpx              2.1G
pie              2.1G
ssp              2.1G
cilk             2.1G
nptl             2.1G
objc++           2.1G
objc-gc          2.1G
vanilla          2.1G
graphite         2.1G
sanitize         2.1G
regression-test  2.1G
vtv              2.2G
openmp           2.2G
objc             2.3G
fortran          2.3G
debug            2.4G
pch              2.9G
pgo              3.0G

I am now running the config with all useflags disabled, then enabled. If the differences are too high I’ll see if we don't have to do some other tests with mixed useflag configurations.
Comment 13 Nado 2018-01-13 17:20:50 UTC
I'm correcting myself, the greatest difference of size between two checks was 0.5GB.
Sorry for the double post noise.
Comment 14 Andreas K. Hüttel gentoo-dev 2018-01-13 18:30:48 UTC
(In reply to Nado from comment #13)
> I'm correcting myself, the greatest difference of size between two checks
> was 0.5GB.
> Sorry for the double post noise.

Now the same with -g in C*FLAGS please... :)
Comment 15 Nado 2018-01-13 18:38:43 UTC
(In reply to Andreas K. Hüttel from comment #14)
> (In reply to Nado from comment #13)
> > I'm correcting myself, the greatest difference of size between two checks
> > was 0.5GB.
> > Sorry for the double post noise.
> 
> Now the same with -g in C*FLAGS please... :)

As soon as the current build (all flags) is done ;)

gcc without any use flag enabled has a peak of 2.1GB
Comment 16 Andreas K. Hüttel gentoo-dev 2018-01-13 19:04:59 UTC
Also what makes a big difference is multilib vs no-multilib.
Comment 17 Nado 2018-01-13 19:18:33 UTC
Alright, I’ll try to test the 4 configs then : {no-,}multilib * with{,out} -g

I’ll report all of them together after it's done, in a week or two at most.

If you have any other suggestion let me know.
Comment 18 Nado 2018-01-30 20:56:43 UTC
Created attachment 517184 [details]
Results for the size check

Here are the results, they’re not complete, I realized I got some build failures, there must be some in the previous results since I forgot to check for errors before.

I also detected that USE="-* go" is useless without cxx (see https://bugs.gentoo.org/645090) so I'll set other builds to cover it better.
Comment 19 Sergei Trofimovich gentoo-dev 2019-02-14 23:39:51 UTC
*** Bug 643172 has been marked as a duplicate of this bug. ***
Comment 20 Bernd 2019-02-27 14:45:50 UTC
This now hit me too for gcc-8.2.0-r6 on my machine after ~4 hours of compilation. Kinda frustrating. 

The output at the end of src_install tells me it needs 8.4 GiB for the build directory plus 472.8 MiB for the install tree.

I'm using the USE flags
[ebuild   R   ] sys-devel/gcc-8.2.0-r6  USE="cxx doc fortran graphite (multilib) nls nptl openmp pch pgo (pie) sanitize ssp vtv (-altivec) -debug (-fixed-point) -go (-hardened) (-jit) (-libssp) -mpx -objc -objc++ -objc-gc -regression-test -systemtap -vanilla" 

I'm aware that the check-reqs concept is a fragile one, the size depending on many factors. But I wonder how many ebuild maintainers manage to give at least a rough estimate for this value, while maintainers of a critical system package don't get this done.

IMO it'd be better to have a false positive, slightly increasing the compile time, than having double the compile time due to ebuild failure because of a OOM event when the build runs out of the tmp disk size.

Please consider adding a check-reqs to packages with prolonged build times.
Comment 21 Sergei Trofimovich gentoo-dev 2019-12-26 11:38:22 UTC
(In reply to Bernd from comment #20)
> This now hit me too for gcc-8.2.0-r6 on my machine after ~4 hours of
> compilation. Kinda frustrating. 
> 
> The output at the end of src_install tells me it needs 8.4 GiB for the build
> directory plus 472.8 MiB for the install tree.
> 
> I'm using the USE flags
> [ebuild   R   ] sys-devel/gcc-8.2.0-r6  USE="cxx doc fortran graphite
> (multilib) nls nptl openmp pch pgo (pie) sanitize ssp vtv (-altivec) -debug
> (-fixed-point) -go (-hardened) (-jit) (-libssp) -mpx -objc -objc++ -objc-gc
> -regression-test -systemtap -vanilla" 
> 
> I'm aware that the check-reqs concept is a fragile one, the size depending
> on many factors. But I wonder how many ebuild maintainers manage to give at
> least a rough estimate for this value, while maintainers of a critical
> system package don't get this done.
> 
> IMO it'd be better to have a false positive, slightly increasing the compile
> time, than having double the compile time due to ebuild failure because of a
> OOM event when the build runs out of the tmp disk size.
> 
> Please consider adding a check-reqs to packages with prolonged build times.

2-to-9 GB is not a very useful error margin. It's hard to put a reasonable number here.
Comment 22 Bernd 2019-12-27 22:36:06 UTC
(In reply to Sergei Trofimovich from comment #21)
> 2-to-9 GB is not a very useful error margin. It's hard to put a reasonable
> number here.

I stopped trying to compile gcc in a tmpfs due to this wide range and rather compile it on disk, taking reasonably longer, but still at a reasonable time as well and without risk of running out of space.