Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 773622 - games-strategy/0ad-0.0.24b_alpha does not utilize all processor cores while building
Summary: games-strategy/0ad-0.0.24b_alpha does not utilize all processor cores while b...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal
Assignee: Gentoo Games
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-03-01 18:52 UTC by Vasilis Lourdas
Modified: 2021-03-02 18:35 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Vasilis Lourdas 2021-03-01 18:52:06 UTC
The latest ebuild does not utilize all processor cores while building, so the building process takes a while. I tried copying the ebuild to my local repository and manually changed the $(makeopts_jobs) variable to a fixed value (my number of processor cores) and still the build uses one processor core only.

emerge --info
Portage 3.0.13 (python 3.8.7-final-0, default/linux/amd64/17.1/desktop/plasma/systemd, gcc-9.3.0, glibc-2.32-r7, 5.10.18-gentoo x86_64)
=================================================================
System uname: Linux-5.10.18-gentoo-x86_64-Intel-R-_Core-TM-_i5-8600_CPU_@_3.10GHz-with-glibc2.2.5
KiB Mem:    16263052 total,   4494076 free
KiB Swap:    8388600 total,   8381740 free
Timestamp of repository gentoo: Mon, 01 Mar 2021 17:00:01 +0000
Head commit of repository gentoo: 7af267ce25a26015c4d37cb24d81807d3c827419
sh bash 5.0_p18
ld GNU ld (Gentoo 2.35.1 p2) 2.35.1
app-shells/bash:          5.0_p18::gentoo
dev-java/java-config:     2.3.1::gentoo
dev-lang/perl:            5.30.3::gentoo
dev-lang/python:          2.7.18-r6::gentoo, 3.6.12-r2::gentoo, 3.7.9-r2::gentoo, 3.8.7-r1::gentoo, 3.9.1-r1::gentoo
dev-util/cmake:           3.18.5::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.7::gentoo
sys-apps/sandbox:         2.20::gentoo
sys-devel/autoconf:       2.13-r1::gentoo, 2.69-r5::gentoo
sys-devel/automake:       1.11.6-r3::gentoo, 1.15.1-r2::gentoo, 1.16.2-r1::gentoo
sys-devel/binutils:       2.34-r2::gentoo, 2.35.1-r1::gentoo
sys-devel/gcc:            9.3.0-r2::gentoo
sys-devel/gcc-config:     2.3.3::gentoo
sys-devel/libtool:        2.4.6-r6::gentoo
sys-devel/make:           4.3::gentoo
sys-kernel/linux-headers: 5.10::gentoo (virtual/os-headers)
sys-libs/glibc:           2.32-r7::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    sync-rsync-extra-opts:
    sync-rsync-verify-max-age: 24
    sync-rsync-verify-jobs: 1
    sync-rsync-verify-metamanifest: no

local_repo
    location: /usr/local/portage
    masters: gentoo
    priority: 0

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="@FREE @FREE bh-luxi Skype-TOS intel-ucode free-noncomm MSttfEULA vmware unRAR MagentaMgOpen CCPL-ShareAlike-1.0 Oracle-BCLA-JavaSE TeamViewer PUEL google-chrome RAR Vivaldi as-is all-rights-reserved MPEG-4 IUPAC-InChi VIGRA CC-Sampling-Plus-1.0 BUILDLIC HRP DUKE3D viber NPSL"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe -march=skylake -mtune=skylake -mmmx -msse -msse2 -msse3 -mssse3 -mcx16 -msahf -mmovbe -maes -mpclmul -mpopcnt -mabm -mfma -mbmi -msgx -mbmi2 -mavx -mavx2 -msse4.2 -msse4.1 -mlzcnt -mrtm -mhle -mrdrnd -mf16c -mfsgsbase -mrdseed -mprfchw -madx -mfxsr -mxsave -mxsaveopt -mclflushopt -mxsavec -mxsaves --param=l1-cache-size=32 --param=l1-cache-line-size=64 --param=l2-cache-size=9216"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php7.3/ext-active/ /etc/php/apache2-php7.4/ext-active/ /etc/php/cgi-php7.3/ext-active/ /etc/php/cgi-php7.4/ext-active/ /etc/php/cli-php7.3/ext-active/ /etc/php/cli-php7.4/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-O2 -pipe -march=skylake -mtune=skylake -mmmx -msse -msse2 -msse3 -mssse3 -mcx16 -msahf -mmovbe -maes -mpclmul -mpopcnt -mabm -mfma -mbmi -msgx -mbmi2 -mavx -mavx2 -msse4.2 -msse4.1 -mlzcnt -mrtm -mhle -mrdrnd -mf16c -mfsgsbase -mrdseed -mprfchw -madx -mfxsr -mxsave -mxsaveopt -mclflushopt -mxsavec -mxsaves --param=l1-cache-size=32 --param=l1-cache-line-size=64 --param=l2-cache-size=9216"
DISTDIR="/usr/portage/distfiles"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs candy cgroup config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://mirror.init7.net/gentoo/ http://mirror.dkm.cz/gentoo/ http://gentoo.mirror.web4u.cz/ http://mirror.leaseweb.com/gentoo/ http://mirror.netcologne.de/gentoo/ http://ftp.fi.muni.cz/pub/linux/gentoo/"
LANG="C"
LDFLAGS="-Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed"
LINGUAS="en en_GB en_US el"
MAKEOPTS="-j6"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="256-color X a52 aac acl acpi activities additions admin aio alsa amd64 apache2 apng archive argon2 asf bash-completion bcmath berkdb branding bzip2 bzlib cairo calendar caps cdinstall cdr cgmanager cgroups charconv chm chromium cleartype_hinting cli contrast corefonts crypt cryptsetup css ctype cups cupsddk curl dba dbus dcc_video declarative device-mapper dia dmi dri dri3 dts dvd dvdr dvdread ebook effects efi emboss encode epub exif expat extended extensions extras faac faad fam fastcgi fat fbcon ffmpeg fftw filecaps filter firefox3 flac fontconfig foomaticdb fpm freerdp ftp g3dvl gallium gd gdbm gif gimp gimpprint git glib glx gmp gnome-keyring gnomedb gocr gold gphoto2 gpm graphviz gs gstreamer gtk gui hddtemp hotpixels howl http2 hwaccel iconv icu id3tag imagemagick imap imlib innodb inotify intl ipv6 iso14755 jabber java java6 javascript jemalloc jpeg jpeg2k json jumbo-build kde kipi kwallet lame lcdfilter lcms leaps-timezone lensfun libass libglvnd libkms libnotify libtirpc libwww lm-sensors lzma mad madwifi maildir mailwrapper maps mariadb matroska mdadm melt mime minizip mmap mmx mmxext motif mozbranding mozdevelop mozilla mozsvg mp3 mp4 mpeg mplayer mpm-worker multilib multiuser mysql mysqli mysqlnd ncurses newspr nlpsolver nls no-old-linux nodrm nomalloccheck nptl ntfs ntfsprogs nvidia objc offensive ofx ogg oldpim opcache opengl openmp openssl optimization opus oqgraph osmesa pam pango pcmcia pcntl pcre pdf pdfimport pdo pear pgo phonon php pie plasma plotutils pmu png policykit posix postgres postproc ppds pulseaudio python qml qt4 qt5 quicktime quota rar raw rdesktop rdoc readline redeyes reports romio rpc samba sasl scanner script sddm sdl seccomp server sha512 sharedext sharedmem simplexml smp sna sndfile soap sockets sodium sound speex spell sphinx split-usr spreadsheet sqlite sqlite3 srt sse sse2 sse3 sse4_1 ssl ssse3 startup-notification subtitles subversion svg symlink sysfs syslog systemd sysv-utils sysvipc tcltk tcpd templates terminal tetex theora thin-splines threads tiff tokenizer tordns truetype tty-helpers udev udisks unicode upower urandom usb userlocales uuid v4l vaapi vda vdpau vhost-net video vkd3d vmware-tools vmware-tools-linux vmware-tools-windows vncviewer vorbis vpx vulkan wddx webm webp widgets windeco wv wxgtk1 wxwidgets x264 x265 xcb xcomposite xetex xft xml xmlreader xmlrpc xmlwriter xmp xorg xpdf-headers xpm xprint xscreensaver xsl xslt xv xvfb xvid xvnc zip zlib" ABI_X86="64" ADA_TARGET="gnat_2018" ALSA_CARDS="hda-intel" APACHE2_MODULES="alias auth_basic authn_core authn_dbm authn_file authz_core authz_dbm authz_groupfile authz_host authz_owner authz_user autoindex cgid deflate dir env expires filter headers http2 log_config logio mime mime_magic negotiation proxy proxy_fcgi rewrite setenvif socache_shmcb unixd userdir" APACHE2_MPMS="event" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64" INPUT_DEVICES="evdev" KERNEL="linux" L10N="en el" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-minimizer nlpsolver" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-3 php7-4" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_8" PYTHON_TARGETS="python2_7 python3_7 python3_8" RUBY_TARGETS="ruby25" SANE_BACKENDS="snapscan" USERLAND="GNU" VIDEO_CARDS="intel i965" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 1 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-03-01 18:58:17 UTC
I'd have thought https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=be214f59a2387da65ae4665cbe1639a9c3e6bb42 would solve this, but your investigation re changing the actual contents tells us that we need to be passing JOBS= differently.

Thank you!
Comment 2 Vasilis Lourdas 2021-03-01 19:00:52 UTC
This is from my one machine:

genlop -t 0ad
 * games-strategy/0ad

     Sun Apr 19 18:28:21 2020 >>> games-strategy/0ad-0.0.23b_alpha
       merge time: 13 minutes and 27 seconds.

     Fri May 22 16:28:15 2020 >>> games-strategy/0ad-0.0.23b_alpha
       merge time: 8 minutes and 48 seconds.

     Mon Oct 26 14:06:41 2020 >>> games-strategy/0ad-0.0.23b_alpha
       merge time: 14 minutes and 36 seconds.

     Mon Mar  1 20:57:49 2021 >>> games-strategy/0ad-0.0.24b_alpha
       merge time: 44 minutes and 10 seconds.
Comment 3 Emily Rowlands 2021-03-01 19:02:15 UTC
(In reply to Sam James from comment #1)
> I'd have thought
> https://gitweb.gentoo.org/repo/gentoo.git/commit/
> ?id=be214f59a2387da65ae4665cbe1639a9c3e6bb42 would solve this, but your
> investigation re changing the actual contents tells us that we need to be
> passing JOBS= differently.
> 
> Thank you!

>	XARGS="${EPREFIX}/usr/bin/xargs" ./build.sh JOBS="-j(makeopts_jobs)" || die "Failed to build bundled SpiderMonkey"

It looks like in this patch should be JOBS="-j$(makeopts_jobs)" to me?
Comment 4 Larry the Git Cow gentoo-dev 2021-03-01 19:06:51 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=0ad51cffc272f53e4a4fef804c218eaae395cfa6

commit 0ad51cffc272f53e4a4fef804c218eaae395cfa6
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2021-03-01 19:06:07 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2021-03-01 19:06:44 +0000

    games-strategy/0ad: fix makeopts_jobs typo
    
    This is at least one issue...
    
    Bug: https://bugs.gentoo.org/773622
    Thanks-to: Emily Rowlands <emily.rowlands+gentoo@mailbox.org>
    Signed-off-by: Sam James <sam@gentoo.org>

 games-strategy/0ad/0ad-0.0.24b_alpha.ebuild | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Comment 5 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-03-01 19:07:03 UTC
(In reply to Emily Rowlands from comment #3)
> 
> It looks like in this patch should be JOBS="-j$(makeopts_jobs)" to me?

Good spot!
Comment 6 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-03-01 19:07:28 UTC
(In reply to Vasilis Lourdas from comment #2)
> This is from my one machine:
> 
> genlop -t 0ad
>  * games-strategy/0ad
> 
>      Sun Apr 19 18:28:21 2020 >>> games-strategy/0ad-0.0.23b_alpha
>        merge time: 13 minutes and 27 seconds.
> 
>      Fri May 22 16:28:15 2020 >>> games-strategy/0ad-0.0.23b_alpha
>        merge time: 8 minutes and 48 seconds.
> 
>      Mon Oct 26 14:06:41 2020 >>> games-strategy/0ad-0.0.23b_alpha
>        merge time: 14 minutes and 36 seconds.
> 
>      Mon Mar  1 20:57:49 2021 >>> games-strategy/0ad-0.0.24b_alpha
>        merge time: 44 minutes and 10 seconds.

To be fair, given we're now building SpiderMonkey (probably the biggest chunk), this was always going to be a bit slower. Can you try with the latest typo fix?
Comment 7 Vasilis Lourdas 2021-03-01 19:27:50 UTC
Trying the ebuild with the latest change. I notice that one core is used while building the bundled spidermonkey. But spidermonkey 78.6 is already installed in that system and it takes about 12-13 minutes to build. I tried building spidermonkey and it seems to use all available processor cores, so I think there must be something wrong with the build procedure of the bundled spidermonkey.
Comment 8 Larry the Git Cow gentoo-dev 2021-03-01 19:44:33 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=8f24ae86e614567ae2ffcd316a2115da31d72214

commit 8f24ae86e614567ae2ffcd316a2115da31d72214
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2021-03-01 19:44:07 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2021-03-01 19:44:26 +0000

    games-strategy/0ad: pass JOBS as env var
    
    Closes: https://bugs.gentoo.org/773622
    Signed-off-by: Sam James <sam@gentoo.org>

 games-strategy/0ad/0ad-0.0.24b_alpha.ebuild | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
Comment 9 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-03-01 19:46:58 UTC
(In reply to Vasilis Lourdas from comment #7)
> Trying the ebuild with the latest change. I notice that one core is used
> while building the bundled spidermonkey. But spidermonkey 78.6 is already
> installed in that system and it takes about 12-13 minutes to build. I tried
> building spidermonkey and it seems to use all available processor cores, so
> I think there must be something wrong with the build procedure of the
> bundled spidermonkey.

Ah, good point. Try now?
Comment 10 Vasilis Lourdas 2021-03-01 20:10:33 UTC
(In reply to Sam James from comment #9)
> Ah, good point. Try now?

Huge difference:

genlop -t 0ad
 * games-strategy/0ad

     Sun Feb 23 13:13:35 2020 >>> games-strategy/0ad-0.0.23b_alpha
       merge time: 3 minutes and 53 seconds.

     Fri May 22 18:06:51 2020 >>> games-strategy/0ad-0.0.23b_alpha
       merge time: 3 minutes and 56 seconds.

     Sat Feb 27 16:41:29 2021 >>> games-strategy/0ad-0.0.24_alpha_pre20210116040036-r2
       merge time: 36 minutes and 29 seconds.

     Mon Mar  1 21:12:59 2021 >>> games-strategy/0ad-0.0.24b_alpha
       merge time: 32 minutes and 22 seconds.

     Mon Mar  1 22:09:36 2021 >>> games-strategy/0ad-0.0.24b_alpha
       merge time: 11 minutes and 5 seconds.
Comment 11 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-03-01 20:11:51 UTC
Excellent. Thanks for your patience, both, and fast feedback. It really helps a lot.

Give me a shout if I can do anything more!
Comment 12 Vasilis Lourdas 2021-03-02 18:35:24 UTC
(In reply to Sam James from comment #11)
> Excellent. Thanks for your patience, both, and fast feedback. It really
> helps a lot.
> 
> Give me a shout if I can do anything more!

All seems fine, except the fact that (in both PCs), when I click on exit to exit from the game, it just freezes. From top, I can see the game is stuck (1 cpu core is at 100%) and the only way to exit the game is by killing the process (not gracefully). Other than that, all seems ok. I tried launching the game from a terminal just in case some error or message appears upon clicking on exit, but no luck.