Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 570534 - dev-qt/qtwebkit & dev-qt/qtwebengine: use check-reqs.eclass
Summary: dev-qt/qtwebkit & dev-qt/qtwebengine: use check-reqs.eclass
Status: CONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All All
: Low enhancement (vote)
Assignee: Qt Bug Alias
URL:
Whiteboard:
Keywords:
: 669934 (view as bug list)
Depends on:
Blocks:
 
Reported: 2016-01-01 22:36 UTC by Davide Pesavento
Modified: 2021-03-18 17:38 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Davide Pesavento gentoo-dev 2016-01-01 22:36:14 UTC
Building qtwebkit requires lots of memory. We should use check-reqs.eclass to warn users when their hardware does not satisfy some minimum requirements.
Comment 1 Guy 2016-02-20 00:53:11 UTC
I'd like to point out that while building webkit does require lots of memory, I've recently gone through the revdep-rebuild process resulting from the gcc-5.3 upgrade on several different x86 based PCs and I'm finishing up with the last one now. All three x86 based PCs have successfully rebuilt 
dev-qt/qtwebkit-5.5.1-r1.

Two have finished and the third {after some issues} is finishing it's gcc-5.3 induced revdep-rebuilds now.

These snippets are from the 3rd PC's still currently running revdep-rebuilds now. {I expect at least another day of running ..}

cat gcc.cleanup | xargs emerge -1 
Calculating dependencies... done!
>>> Verifying ebuild manifests
>>> Running pre-merge checks for media-libs/libraw-0.17.1
>>> Running pre-merge checks for media-libs/ffmpegsource-2.22
>>> Running pre-merge checks for media-sound/pulseaudio-8.0
 * Determining the location of the kernel source code
 * Found kernel source directory:
 *     /usr/src/linux
 * Found kernel object directory:
 *     /lib/modules/4.3.5-gentoo/build
 * Found sources for kernel version:
 *     4.3.5-gentoo
 * Checking for suitable kernel configuration options...                                                                                             [ ok ]
 * A preallocated buffer-size of 2048 (kB) or higher is recommended for the HD-audio driver!
 * CONFIG_SND_HDA_PREALLOC_SIZE=64
>>> Running pre-merge checks for dev-db/mariadb-10.1.11
>>> Running pre-merge checks for sys-devel/llvm-3.7.1
 * Checking for at least 550 MiB disk space at "/var/tmp/portage/sys-devel/llvm-3.7.1/temp" ...                                                      [ ok ]
 * Trying to build a C++11 test program ...                                                                                                          [ ok ]
>>> Running pre-merge checks for kde-frameworks/kwindowsystem-5.19.0
>>> Running pre-merge checks for kde-frameworks/solid-5.17.0
>>> Running pre-merge checks for kde-frameworks/kguiaddons-5.19.0
>>> Running pre-merge checks for kde-plasma/libkscreen-5.5.4
>>> Running pre-merge checks for lxqt-base/liblxqt-0.10.0
>>> Running pre-merge checks for app-office/akonadi-server-1.13.0-r2
>>> Running pre-merge checks for net-libs/webkit-gtk-2.4.9
>>> Running pre-merge checks for net-libs/webkit-gtk-2.4.9-r200
>>> Running pre-merge checks for kde-base/kdelibs-4.14.16
>>> Running pre-merge checks for media-video/aegisub-9999
>>> Emerging (1 of 354) dev-qt/qtchooser-0_p20151008::gentoo
>>> Installing (1 of 354) dev-qt/qtchooser-0_p20151008::gentoo

..

>>> Installing (75 of 354) dev-qt/qt3support-4.8.7::gentoo
>>> Emerging (76 of 354) dev-qt/qtwebkit-4.8.7::gentoo
>>> Installing (76 of 354) dev-qt/qtwebkit-4.8.7::gentoo
>>> Emerging (77 of 354) dev-qt/qtsvg-4.8.7::gentoo
>>> Installing (77 of 354) dev-qt/qtsvg-4.8.7::gentoo

..

>>> Emerging (130 of 354) dev-qt/qtwebkit-5.5.1-r1::gentoo
>>> Installing (130 of 354) dev-qt/qtwebkit-5.5.1-r1::gentoo
>>> Emerging (131 of 354) net-libs/webkit-gtk-2.4.9::gentoo

{this is the current progress point for this set of revdep-rebuilds}

PetaMem R&D's comment here: https://bugs.gentoo.org/show_bug.cgi?id=570430#c12 is relevant. My 3rd x86 PC will only build large ebuilds like {qt}webkit, libreoffice etc with MAKEOPTS="-j1". PCs 1 & 2 are set with MAKEOPTS="-j2" and usually don't have a problem with these builds. They just take for-e-ver.

While you're thinking about what pre-emerge checks you may want to perform, you may also want to read this: https://forums.gentoo.org/viewtopic-t-1039248-highlight-.html

Not every MMU works all that well. Personal, multiple PCs, real world observation: For problem PCs where all memory tests succeed but memory still seems an issue, reducing MAKEOPTS more often then not 'solves' the problem.

gcc with large packages like qtwebkit, chromium, libreoffice always seem to overly stress memory subsystems.

I'd suggest a pre-emerge check of physical memory versus MAKEOPTS setting would make sense with a reduction of MAKEOPTS="-j1" for systems with less than 2G RAM.

PetaMem R&D's suggestion of a formula for verifying max MAKEOPTS versus physical memory setting might also be a good idea.

On the other hand, I've completed these builds with only 1G RAM so I can't suggest putting a lower cutoff limit on RAM which is more than 1G.



Informational notes:

For x86 PC 1:firebook src # emerge --info
Portage 2.2.27 (python 2.7.11-final-0, default/linux/x86/13.0/desktop, gcc-5.3.0, glibc-2.22-r2, 4.3.5-gentoo i686)
=================================================================
System uname: Linux-4.3.5-gentoo-i686-Intel-R-_Atom-TM-_CPU_N270_@_1.60GHz-with-gentoo-2.2
KiB Mem:     1018132 total,    269132 free
KiB Swap:    2047996 total,   2036880 free
Timestamp of repository gentoo: Wed, 17 Feb 2016 10:00:01 +0000
sh bash 4.3_p42-r2
ld GNU ld (Gentoo 2.25.1 p1.1) 2.25.1
app-shells/bash:          4.3_p42-r2::gentoo
dev-java/java-config:     2.2.0::gentoo
dev-lang/perl:            5.22.1::gentoo
dev-lang/python:          2.7.11-r2::gentoo, 3.4.3-r7::gentoo, 3.5.1-r2::gentoo
dev-util/cmake:           3.4.3::gentoo
dev-util/pkgconfig:       0.29::gentoo
sys-apps/baselayout:      2.2::gentoo
sys-apps/openrc:          0.20.5::gentoo
sys-apps/sandbox:         2.10-r1::gentoo
sys-devel/autoconf:       2.13::gentoo, 2.69-r1::gentoo
sys-devel/automake:       1.11.6-r2::gentoo, 1.13.4-r1::gentoo, 1.14.1-r1::gentoo, 1.15-r2::gentoo
sys-devel/binutils:       2.25.1-r1::gentoo
sys-devel/gcc:            5.3.0::gentoo
sys-devel/gcc-config:     1.8-r1::gentoo
sys-devel/libtool:        2.4.6-r2::gentoo
sys-devel/make:           4.1-r1::gentoo
sys-kernel/linux-headers: 4.4::gentoo (virtual/os-headers)
sys-libs/glibc:           2.22-r2::gentoo

firebook src # eix qtwebkit
[I] dev-qt/qtwebkit
     Available versions:  
     (4)    4.8.6-r1 (~)4.8.7 [M]~*4.10.4
     (5)    (~)5.4.2^mt{tbz2} (~)5.5.1-r1^mt{tbz2}
       {aqua debug +exceptions geolocation (+)gstreamer gstreamer010 icu +jit multimedia opengl orientation pch printsupport qml test udev webchannel webp ABI_MIPS="n32 n64 o32" ABI_PPC="32 64" ABI_S390="32 64" ABI_X86="32 64 x32"}
     Installed versions:  5.5.1-r1(5)^mt{tbz2}(09:07:39 AM 01/28/2016)(gstreamer jit opengl printsupport -debug -geolocation -gstreamer010 -multimedia -orientation -qml -test -webchannel -webp)
     Homepage:            https://www.qt.io/
     Description:         WebKit rendering library for the Qt5 framework (deprecated)

For x86 PC 2: qemu_gnn_01 ~ # emerge --info
Portage 2.2.27 (python 2.7.11-final-0, default/linux/x86/13.0/desktop, gcc-5.3.0, glibc-2.22-r2, 4.3.5-gentoo i686)
=================================================================
System uname: Linux-4.3.5-gentoo-i686-AMD_A10-7850K_Radeon_R7,_12_Compute_Cores_4C+8G-with-gentoo-2.2
KiB Mem:     1028108 total,    895004 free
KiB Swap:     983036 total,    983036 free
Timestamp of repository gentoo: Wed, 17 Feb 2016 18:30:01 +0000
sh bash 4.3_p42-r2
ld GNU ld (Gentoo 2.25 p1.2) 2.25
app-shells/bash:          4.3_p42-r2::gentoo
dev-java/java-config:     2.2.0::gentoo
dev-lang/perl:            5.22.1::gentoo
dev-lang/python:          2.7.11-r2::gentoo, 3.4.3-r7::gentoo, 3.5.1-r2::gentoo
dev-util/cmake:           3.4.3::gentoo
dev-util/pkgconfig:       0.29::gentoo
sys-apps/baselayout:      2.2::gentoo
sys-apps/openrc:          0.20.5::gentoo
sys-apps/sandbox:         2.10-r1::gentoo
sys-devel/autoconf:       2.13::gentoo, 2.69-r1::gentoo
sys-devel/automake:       1.11.6-r2::gentoo, 1.13.4-r1::gentoo, 1.14.1-r1::gentoo, 1.15-r2::gentoo
sys-devel/binutils:       2.25-r1::gentoo
sys-devel/gcc:            5.3.0::gentoo
sys-devel/gcc-config:     1.8-r1::gentoo
sys-devel/libtool:        2.4.6-r2::gentoo
sys-devel/make:           4.1-r1::gentoo
sys-kernel/linux-headers: 4.4::gentoo (virtual/os-headers)
sys-libs/glibc:           2.22-r2::gentoo

qemu_gnn_01 ~ # eix qtwebkit
[I] dev-qt/qtwebkit
     Available versions:  
     (4)    4.8.6-r1 (~)4.8.7{tbz2} [M]~*4.10.4
     (5)    (~)5.4.2^mt{tbz2} (~)5.5.1-r1^mt{tbz2}
       {aqua debug +exceptions geolocation (+)gstreamer gstreamer010 icu +jit multimedia opengl orientation pch printsupport qml test udev webchannel webp ABI_MIPS="n32 n64 o32" ABI_PPC="32 64" ABI_S390="32 64" ABI_X86="32 64 x32"}
     Installed versions:  4.8.7(4){tbz2}(06:54:07 PM 01/27/2016)(exceptions gstreamer icu jit -aqua -debug -pch ABI_MIPS="-n32 -n64 -o32" ABI_PPC="-32 -64" ABI_S390="-32 -64" ABI_X86="32 -64 -x32") 5.5.1-r1(5)^mt{tbz2}(06:03:16 PM 01/27/2016)(gstreamer jit opengl printsupport -debug -geolocation -gstreamer010 -multimedia -orientation -qml -test -webchannel -webp)
     Homepage:            https://www.qt.io/
     Description:         WebKit rendering library for the Qt5 framework (deprecated)

For x86 PC 3: slizard ~ # emerge --info
Portage 2.2.27 (python 2.7.11-final-0, default/linux/x86/13.0/desktop/kde, gcc-5.3.0, glibc-2.22-r2, 4.3.5-gentoo i686)
=================================================================
System uname: Linux-4.3.5-gentoo-i686-Intel-R-_Pentium-R-_4_CPU_2.66GHz-with-gentoo-2.2
KiB Mem:     2069972 total,    598700 free
KiB Swap:    2047996 total,   2029380 free
Timestamp of repository gentoo: Wed, 17 Feb 2016 10:00:01 +0000
sh bash 4.3_p42-r2
ld GNU ld (Gentoo 2.25 p1.2) 2.25
app-shells/bash:          4.3_p42-r2::gentoo
dev-java/java-config:     2.2.0::gentoo
dev-lang/perl:            5.22.1::gentoo
dev-lang/python:          2.7.11-r2::gentoo, 3.4.3-r7::gentoo, 3.5.1-r2::gentoo
dev-util/cmake:           3.4.3::gentoo
dev-util/pkgconfig:       0.29::gentoo
sys-apps/baselayout:      2.2::gentoo
sys-apps/openrc:          0.19.1::gentoo
sys-apps/sandbox:         2.10-r1::gentoo
sys-devel/autoconf:       2.13::gentoo, 2.69-r1::gentoo
sys-devel/automake:       1.11.6-r2::gentoo, 1.13.4-r1::gentoo, 1.14.1-r1::gentoo, 1.15-r2::gentoo
sys-devel/binutils:       2.25-r1::gentoo
sys-devel/gcc:            5.3.0::gentoo
sys-devel/gcc-config:     1.8-r1::gentoo
sys-devel/libtool:        2.4.6-r2::gentoo
sys-devel/make:           4.1-r1::gentoo
sys-kernel/linux-headers: 4.4::gentoo (virtual/os-headers)
sys-libs/glibc:           2.22-r2::gentoo

slizard ~ # eix qtwebkit
[I] dev-qt/qtwebkit
     Available versions:  
     (4)    4.8.6-r1{tbz2} (~)4.8.7{tbz2} [M]~*4.10.4
     (5)    (~)5.4.2^mt{tbz2} (~)5.5.1-r1^mt{tbz2}
       {aqua debug +exceptions geolocation (+)gstreamer gstreamer010 icu +jit multimedia opengl orientation pch printsupport qml test udev webchannel webp ABI_MIPS="n32 n64 o32" ABI_PPC="32 64" ABI_S390="32 64" ABI_X86="32 64 x32"}
     Installed versions:  4.8.7(4){tbz2}(09:32:24 PM 02/18/2016)(exceptions gstreamer icu jit -aqua -debug -pch ABI_MIPS="-n32 -n64 -o32" ABI_PPC="-32 -64" ABI_S390="-32 -64" ABI_X86="32 -64 -x32") 5.5.1-r1(5)^mt{tbz2}(11:36:31 AM 02/19/2016)(gstreamer jit opengl printsupport -debug -geolocation -gstreamer010 -multimedia -orientation -qml -test -webchannel -webp)
     Homepage:            https://www.qt.io/
     Description:         WebKit rendering library for the Qt5 framework (deprecated)
Comment 2 Guy 2016-02-20 01:54:49 UTC
For the "TL:DR" readers:

For packages where large numbers of simultaneous sub builds are possible, gcc will increase stress on memory sub-systems in a logarithmic fashion with each single MAKEOPTS increment.

High MAKEOPTS values, insufficient memory, overclocked systems or poorly designed/built memory subsystems will alway be at higher risk for memory related failures. Failures will be either to run out of memory or to experience segfaults. Segfaults on a system with memory known to pass all memory tests indicates a problematic memory susbsystem due to either poor design or too high overclocking.

qtwebit is an example of such a package which allows gcc to perform large numbers of simultaneous sub builds depending on the value of MAKEOPTS.

pre-emerge checks for packages like qtwebkit should consider some relationship between MAKEOPTS value and installed memory with a warning to the user that builds that fail due to OOM or segfaults should be retried with lower MAKEOPTS values than whatever is accepted by the pre-emerge check.

I tend to be chatty and verbose in my writing. Sorry about that.
Comment 3 Michael Palimaka (kensington) gentoo-dev 2019-03-11 07:09:00 UTC
*** Bug 669934 has been marked as a duplicate of this bug. ***
Comment 4 Ben 2021-03-18 17:38:16 UTC
While RAM usage is one thing (seems some threads can use 2+GB RAM on x86_64, not sure about x86 and other architectures, plus this is somewhat controllable with makeopts and swap): 

  PID USER      NI    VIRT    RES   SWAP P S  %CPU nTH   TIME COMMAND           
25796 distcc    15 2626312   2.4g      0 2 R 100.0   1   0:56 cc1plus           
25804 distcc    15 2213876   2.0g      0 0 R  99.7   1   0:46 cc1plus           

(and later...)

26265 distcc    15 3163132   2.9g      0 0 R 100.0   1   1:06 cc1plus           

(Incidentally, I'm DoS'ing my distcc helpers with qtwebengine as they too are heavily getting into swap to get these built, and though they do eventually get their piece done, sometimes they seem to time out distcc ... )

But disk usage is something else.  Would be nice to get a warning on how much disk it uses.  I'm seeing a 6-7GB used, perhaps this should be enough for 5.15?