Bug 823798 – sys-libs/glibc-2.34-r2: fails to install with debug info (ln: failed to create hard link '/var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/lib/debug/usr/lib64/misc/glibc/getconf/getconf.debug': File exists)

• Status: RESOLVED FIXED
• Product: Portage Development

[Description Sam James 2021-11-15 06:58:07 UTC]

[...]
ln: failed to create hard link '/var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/lib/debug/usr/lib64/misc/glibc/getconf/getconf.debug': File exists
 * ERROR: sys-libs/glibc-2.34-r2::gentoo failed:
 *   (no error message)
 *

----

Portage 3.0.28 (python 3.10.0-final-0, default/linux/amd64/17.1/desktop/plasma/systemd, gcc-11.2.0, glibc-2.34-r1, 5.10.78-gentoo-dist-hardened x86_64)
=================================================================
System uname: Linux-5.10.78-gentoo-dist-hardened-x86_64-AMD_Ryzen_9_3950X_16-Core_Processor-with-glibc2.34
KiB Mem:    16363384 total,   2117944 free
KiB Swap:   16777212 total,  14525376 free
Timestamp of repository gentoo: Mon, 15 Nov 2021 05:21:39 +0000
Head commit of repository gentoo: 0c04740882cd514f0675b2804103a91244e75069

Timestamp of repository kde: Sat, 13 Nov 2021 20:21:36 +0000
Head commit of repository kde: 681a0a9522de222a1e97a6bbe799153fdf83f41f

Timestamp of repository qt: Tue, 09 Nov 2021 10:53:05 +0000
Head commit of repository qt: 25a2f91a063cef0e68a5c413f7c6ef6710cdaa82

Timestamp of repository sam_c: Sun, 31 Oct 2021 19:53:36 +0000
Head commit of repository sam_c: b61733eb2b48c41ad12d55f528603ad446484c17

Timestamp of repository steam-overlay: Tue, 09 Nov 2021 10:53:07 +0000
Head commit of repository steam-overlay: b21d724e69a176a398cae2db70172cb9dfede48a

sh dash 0.5.11.5
ld GNU ld (Gentoo 2.37_p1 p1) 2.37
ccache version 4.5 [disabled]
app-shells/bash:          5.1_p8::gentoo
dev-lang/perl:            5.34.0-r5::gentoo
dev-lang/python:          3.8.12_p1::gentoo, 3.9.8::gentoo, 3.10.0_p1::gentoo
dev-lang/rust-bin:        1.56.1::gentoo
dev-util/ccache:          4.5::gentoo
dev-util/cmake:           3.21.4::gentoo
sys-apps/baselayout:      2.8::gentoo
sys-apps/sandbox:         2.29::gentoo
sys-devel/autoconf:       2.13-r1::gentoo, 2.71-r1::gentoo
sys-devel/automake:       1.16.5::gentoo
sys-devel/binutils:       2.37_p1-r1::gentoo
sys-devel/gcc:            9.4.0::gentoo, 10.3.0-r2::gentoo, 11.2.0::gentoo
sys-devel/gcc-config:     2.4::gentoo
sys-devel/libtool:        2.4.6-r6::gentoo
sys-devel/make:           4.3::gentoo
sys-kernel/linux-headers: 5.15::gentoo (virtual/os-headers)
sys-libs/glibc:           2.34-r1::gentoo
Repositories:

gentoo
    location: /var/db/repos/gentoo
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/gentoo.git
    priority: -1000
    sync-git-clone-extra-opts: -b stable -c gc.reflogExpire=0 -c gc.reflogExpireUnreachable=0 -c gc.rerereresolved=0 -c gc.rerereunresolved=0 -c gc.pruneExpire=now
    sync-git-verify-commit-signature: yes

crossdev
    location: /var/db/repos/crossdev
    masters: gentoo

kde
    location: /var/db/repos/kde
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/kde.git
    masters: gentoo

qt
    location: /var/db/repos/qt
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/qt.git
    masters: gentoo

sam_c
    location: /var/db/repos/sam_c
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/sam_c.git
    masters: gentoo

steam-overlay
    location: /var/db/repos/steam-overlay
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/steam-overlay.git
    masters: gentoo

ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="@FREE"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe -march=native -fdiagnostics-color=always -frecord-gcc-switches"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-O2 -pipe -march=native -fdiagnostics-color=always -frecord-gcc-switches"
DISTDIR="/var/cache/distfiles"
EMERGE_DEFAULT_OPTS="--keep-going --with-bdeps=y --complete-graph --deep --changed-deps-report=y --usepkg-exclude sys-fs/zfs --usepkg-exclude sys-fs/zfs-kmod --usepkg=y"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe -march=native -fdiagnostics-color=always -frecord-gcc-switches"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg cgroup config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox mount-sandbox multilib-strict network-sandbox news parallel-fetch parallel-install pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe -march=native -fdiagnostics-color=always -frecord-gcc-switches"
GENTOO_MIRRORS="http://mirror.bytemark.co.uk/gentoo/ http://www.mirrorservice.org/sites/distfiles.gentoo.org/ http://mirrors.soeasyto.com/distfiles.gentoo.org/ http://mirrors.gethosted.online/gentoo"
LANG="en_GB.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,--defsym=__gentoo_check_ldflags__=0"
LINGUAS="en en_GB"
MAKEOPTS="-j24"
PKGDIR="/var/cache/binpkgs"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="PIC X a52 aac acl acpi activities aes alsa amd64 avx avx2 bash-completion bluetooth branding bzip2 cairo caps cdda cdr cli crypt dbus declarative dist-kernel dri dts dvd dvdr emacs emboss encode exif f16c filecaps firewalld flac fma3 fortran gdbm gif gmp gpm graphite gtk gui hardened hunspell iconv icu ipv6 jit jpeg kde kdesu kipi kwallet lcms libglvnd libnotify libtirpc llvm-libunwind mad mmx mmxext mng mp3 mp4 mpeg multilib ncurses nls nptl ogg opengl openmp pam pango pclmul pcre pdf pgo pie plasma png policykit popcnt ppds pulseaudio qml qt5 rdrand readline sdl seccomp semantic-desktop sha spell split-usr sse sse2 sse3 sse4_1 sse4_2 sse4a ssl ssse3 startup-notification svg system-av1 system-binutils system-boost system-bootstrap system-cairo system-clang system-digest system-ffmpeg system-harfbuzz system-heimdal system-icu system-jpeg system-leveldb system-libevent system-libs system-libvpx system-libyaml system-lz4 system-mitkrb5 system-sqlite system-ssl system-tbb system-uulib system-webp system-zlib systemd threads tiff truetype udev udisks unicode upower usb verify-sig vorbis vulkan wayland widgets wxwidgets x264 xattr xcb xml xv xvid zfs zlib zsh-completion" ABI_X86="32 64" ADA_TARGET="gnat_2019" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt rdrand sha sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput" KERNEL="linux" L10N="en en-GB" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-3 php7-4" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_9" PYTHON_TARGETS="python3_9 python3_10 python3_8" RUBY_TARGETS="ruby26 ruby27" USERLAND="GNU" VIDEO_CARDS="amdgpu radeonsi radeon" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RUSTFLAGS

[Comment 1 Sam James 2021-11-15 06:58:58 UTC]

Created attachment 751218 [details]
build.log.bz2

[Comment 2 Sam James 2021-11-15 06:59:12 UTC]

Whissi reported the same error with current tip of portage.git.

[Comment 3 Mike Gilbert 2021-11-15 15:36:34 UTC]

I am unable to reproduce this.

Please confirm the exact portage commit you are using.

Also, please create a tarball with the contents of /var/tmp/portage/sys-libs/glibc-2.34-r2 and I will take a look.

[Comment 4 Thomas Deutschmann (RETIRED) 2021-11-16 17:04:35 UTC]

Portage at 704bcd1581e49432f363f0eda648d58411775d7f. However, after tree update today, glibc merged.

[Comment 5 Zac Medico 2021-11-16 23:14:14 UTC]

This appears to be a regression in commit 84206c6200eb003314cf4f2d640bf73f04654012.  With commit 84206c6200eb003314cf4f2d640bf73f04654012 getconf.debug is missing when I build glibc-2.34-r2 with splitdebug, as shown by this find command:

> $ find /var/tmp/portage/sys-libs/glibc-2.34-r2/image -name 'getconf*'
> /var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/lib64/misc/glibc/getconf
> /var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/bin/getconf
> /var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/lib/debug/usr/lib64/misc/glibc/getconf
> /var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/lib/debug/usr/lib/misc/glibc/getconf
> /var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/lib/misc/glibc/getconf

If I revert commit 84206c6200eb003314cf4f2d640bf73f04654012, then getconf.debug appears as expected:

>  $ find /var/tmp/portage/sys-libs/glibc-2.34-r2/image -name 'getconf*'
> /var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/lib64/misc/glibc/getconf
> /var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/bin/getconf
> /var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/lib/debug/usr/lib64/misc/glibc/getconf
> /var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/lib/debug/usr/bin/getconf.debug
> /var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/lib/debug/usr/lib/misc/glibc/getconf
> /var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/lib/misc/glibc/getconf

[Comment 6 Zac Medico 2021-11-16 23:20:28 UTC]

Apparently 84206c6200eb003314cf4f2d640bf73f04654012 triggers this getconf problem because of getconf hardlinks created in /usr/lib64/misc/glibc/getconf/, as shown by the identical inode numbers here:

> $ stat /var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/bin/getconf
>   File: /var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/bin/getconf
>   Size: 34952     	Blocks: 72         IO Block: 4096   regular file
> Device: 40h/64d	Inode: 376622      Links: 4
> Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
> Access: 2021-11-16 15:08:46.581141261 -0800
> Modify: 2021-11-16 15:08:45.465156116 -0800
> Change: 2021-11-16 15:08:46.498142366 -0800
>  Birth: -
> $ stat /var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/lib64/misc/glibc/getconf/POSIX_V6_LP64_OFF64
>   File: /var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/lib64/misc/glibc/getconf/POSIX_V6_LP64_OFF64
>   Size: 34952     	Blocks: 72         IO Block: 4096   regular file
> Device: 40h/64d	Inode: 376622      Links: 4
> Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
> Access: 2021-11-16 15:08:46.581141261 -0800
> Modify: 2021-11-16 15:08:45.465156116 -0800
> Change: 2021-11-16 15:08:46.498142366 -0800
>  Birth: -
> $ stat /var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/lib64/misc/glibc/getconf/POSIX_V7_LP64_OFF64
>   File: /var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/lib64/misc/glibc/getconf/POSIX_V7_LP64_OFF64
>   Size: 34952     	Blocks: 72         IO Block: 4096   regular file
> Device: 40h/64d	Inode: 376622      Links: 4
> Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
> Access: 2021-11-16 15:08:46.581141261 -0800
> Modify: 2021-11-16 15:08:45.465156116 -0800
> Change: 2021-11-16 15:08:46.498142366 -0800
>  Birth: -
> $ stat /var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/lib64/misc/glibc/getconf/XBS5_LP64_OFF64
>   File: /var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/lib64/misc/glibc/getconf/XBS5_LP64_OFF64
>   Size: 34952     	Blocks: 72         IO Block: 4096   regular file
> Device: 40h/64d	Inode: 376622      Links: 4
> Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
> Access: 2021-11-16 15:08:46.581141261 -0800
> Modify: 2021-11-16 15:08:45.465156116 -0800
> Change: 2021-11-16 15:08:46.498142366 -0800
>  Birth: -

[Comment 7 Zac Medico 2021-11-16 23:28:04 UTC]

It looks like the dst_basename variable has the wrong value.

[Comment 8 Mike Gilbert 2021-11-17 02:38:24 UTC]

(In reply to Zac Medico from comment #5)
> This appears to be a regression in commit
> 84206c6200eb003314cf4f2d640bf73f04654012.  With commit
> 84206c6200eb003314cf4f2d640bf73f04654012 getconf.debug is missing when I
> build glibc-2.34-r2 with splitdebug, as shown by this find command:

That commit changes the way the debug file names are generated. This is not  "regression", but an intentional change.

[Comment 9 Mike Gilbert 2021-11-17 02:43:28 UTC]

(In reply to Zac Medico from comment #7)
> It looks like the dst_basename variable has the wrong value.

Please elaborate.

[Comment 10 Mike Gilbert 2021-11-17 02:57:55 UTC]

I would like to add some additional messaging to help further diagnose this.

https://github.com/gentoo/portage/pull/770

[Comment 11 Zac Medico 2021-11-17 03:22:35 UTC]

Created attachment 751742 [details]
ln: failed to create symbolic link

(In reply to Mike Gilbert from comment #9)
> (In reply to Zac Medico from comment #7)
> > It looks like the dst_basename variable has the wrong value.
> 
> Please elaborate.

I assumed that the missing getconf.debug file was due to some bug in the dst_basename logic, but maybe I was wrong.

I've just observed a different failure:

strip: x86_64-pc-linux-gnu-strip --strip-unneeded -N __gentoo_check_ldflags__ -R .comment -R .GCC.command.line -R .note.gnu.gold-version
   /usr/lib/crti.o
   /usr/lib/crt1.o
   /usr/lib/crtn.o
   /usr/lib/libBrokenLocale.a
   /usr/lib/Mcrt1.o
   /usr/lib/Scrt1.o
   /lib/libBrokenLocale.so.1
   /usr/lib/gcrt1.o
   /usr/lib/gconv/ISO8859-1.so
ln: failed to create symbolic link '/var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/lib/debug/.build-id/a4/8463e4bad2e0b9367bb9913a9fb3c63f4ae897': File exists
   /usr/lib/gconv/ISO8859-9.so
 * ERROR: sys-libs/glibc-2.34-r2::gentoo failed:
 *   (no error message)

[Comment 12 Mike Gilbert 2021-11-17 03:51:50 UTC]

(In reply to Zac Medico from comment #11)

This feels like some race condition when creating links with the same name.

[Comment 13 Mike Gilbert 2021-11-17 04:15:51 UTC]

I was able to reproduce the problem by moving PORTAGE_TMPDIR from a tmpfs to a btrfs filesystem.

[Comment 14 Mike Gilbert 2021-11-17 04:32:43 UTC]

Created attachment 751746 [details]
debug log

From my build log:

ln: failed to create hard link '/var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/lib/debug/usr/lib64/misc/glibc/getconf/getconf.debug': File exists
ln: failed to create hard link '/var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/lib/debug/usr/lib64/misc/glibc/getconf/getconf.debug': File exists
 * ERROR: sys-libs/glibc-2.34-r2::gentoo failed:
 *   failed to create hard link '/var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/lib/debug/usr/lib64/misc/glibc/getconf/getconf.debug'
 * ERROR: sys-libs/glibc-2.34-r2::gentoo failed:
 * 
 *   failed to create hard link '/var/tmp/portage/sys-libs/glibc-2.34-r2/image/usr/lib/debug/usr/lib64/misc/glibc/getconf/getconf.debug'


It looks like estrip is trying to process these links simultaneously:

image/usr/lib64/misc/glibc/getconf/POSIX_V7_LP64_OFF64 22471252_debug
image/usr/lib64/misc/glibc/getconf/POSIX_V6_LP64_OFF64 22471252_debug
image/usr/lib64/misc/glibc/getconf/XBS5_LP64_OFF64 22471252_debug

These all share the same inode (22471252), and therefore they must share the same debug file (getconf.debug).

estrip attempts to create image/usr/lib64/misc/glibc/getconf/getconf.debug in 3 concurrent processes, and 2 of processes fail.

A simple solution would be to just make the ln calls non-fatal. Alternatively, we could serialize processing of these links somehow.

[Comment 15 Michał Górny 2021-11-17 07:22:52 UTC]

We could also make the whole thing serial and remove some ugly code in the process.

[Comment 16 Mike Gilbert 2021-11-17 18:29:57 UTC]

Ok, so the locking in estrip was faulty. I pushed a fix to the linked PR. Please review/test.

https://github.com/gentoo/portage/pull/770

[Comment 17 Mike Gilbert 2021-11-17 18:38:36 UTC]

(In reply to Michał Górny from comment #15)
> We could also make the whole thing serial and remove some ugly code in the
> process.

I'm not sure what kind of performance impact that would have.

[Comment 18 Michał Górny 2021-11-17 19:50:02 UTC]

(In reply to Mike Gilbert from comment #17)
> (In reply to Michał Górny from comment #15)
> > We could also make the whole thing serial and remove some ugly code in the
> > process.
> 
> I'm not sure what kind of performance impact that would have.

Compared to compiling glibc or other large packages?  I think we're trying to microoptimize here, at the cost of correctness.

[Comment 19 Mike Gilbert 2021-11-17 22:33:21 UTC]

I timed estrip when installing sys-libs/glibc-2.34-r2 with 3 enabled abis.

1 thread: 1m51.715s
6 threads: 0m21.342s

Without ccache, the entire glibc build takes around 19-20 minutes on the same system. 
With ccache, it takes around 8-9 minutes.

My thoughts: multi-threaded estrip is nice if you happen to be staring at the terminal while it is running. Otherwise, the speedup is small compared to the overall build time.

[Comment 20 Mike Gilbert 2021-11-17 22:39:46 UTC]

I created a second PR to drop the multijob code. We should decide which of these we want to merge.

https://github.com/gentoo/portage/pull/771

[Comment 21 Michał Górny 2021-11-18 07:17:37 UTC]

I should also point out that parallel estrip used to have very high memory consumption -- before my hardware upgrade, it used to cause intensive swapping when stripping e.g. LLVM (back when it installed split libraries).

[Comment 22 SpanKY 2021-11-18 08:08:24 UTC]

(In reply to Mike Gilbert from comment #19)

it isn't a small speedup when you're on slower systems with many cores (like often happens with mips).  or you have a package that installs hundreds of objects (like the kernel).  the speedup in parallel builds is significant, especially when comparing with hot caches.

(In reply to Michał Górny from comment #21)

so does building C++ projects (especially with -pipe).  the answer isn't to force -j1.

[Comment 23 Mike Gilbert 2021-11-18 16:03:55 UTC]

(In reply to Michał Górny from comment #21)

We could introduce a new environment variable to override the job count for estrip to address that issue if necessary.

[Comment 24 Mike Gilbert 2021-11-18 16:06:45 UTC]

(In reply to SpanKY from comment #22)

Can you share some benchmarks for the examples you have given?

[Comment 25 SpanKY 2021-11-19 07:35:08 UTC]

i don't have access to the mips box anymore, but it was a system where clock speeds were <1 GHz, but had it had like 64 cores.  so any single threaded process was painful, but any parallel process was fine.  coreutils that installs ~100 ELFs would see dramatic speedups.  even current mips/arm/arm64 systems are being built in such configurations.  if you look back in bug history, you can see a time where kumba@ was fixing a lot of `emake -j1` bugs in the tree because they were painful for him.

this can be important even for developers working on fast machines with incremental builds.  on a very expensive machine i have from work (Intel Xeon Gold 6154 @ 3GHz which is like 36 total cores w/SMT for 72 hyperthreads), these are the times i have building the kernel (actually building it, not just installing source).  repeated runs shows deviation of <5 seconds on the wallclock, so it's not that noisy.

# parallel build+strip w/72 jobs
real    1m39.737s
user    23m0.377s
sys     9m52.825s

# parallel build w/72 jobs + single threaded strip
real    2m35.281s
user    23m12.628s
sys     8m55.144s

even these times look high ... it seems we've regressed in some of our optimizations as we used to be <1min.  these cycle times are important for devs who are doing compile+edit+test.

the strip helper is really begging to be rewritten in Python.  it should have been long ago before its shell code got out of hand.

[Comment 26 Sam James 2021-11-19 07:37:43 UTC]

(In reply to SpanKY from comment #25) if you look back in bug
> you can see a time where kumba@ was fixing a lot of `emake -j1`
> bugs in the tree because they were painful for him.

yeah, this was a while back IIRC, but I do remember it. This kind of thing is true for arm64 boxes we still have now. Until very recently, the Packet hardware we had had 96 cores but had pretty slow single-core perf. Anything parallel zipped along. I have an older arm64 box (one of the first server-grade CPUs) which fits this profile. Some sparc hardware is like this as well.

[Comment 27 Larry the Git Cow 2021-11-19 16:17:31 UTC]

The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=fe3e50025fee15d59c4956a39b675faf50555c46

commit fe3e50025fee15d59c4956a39b675faf50555c46
Author:     Mike Gilbert <[e-mail]>
AuthorDate: 2021-11-17 18:23:07 +0000
Commit:     Mike Gilbert <[e-mail]>
CommitDate: 2021-11-19 16:16:07 +0000

    estrip: fix lockfile handling
    
    The previous code would wait for the first link to an inode to be
    processed, and would then allow multiple processes to process duplicate
    links to the same inode. This behavior leads to a race condition in
    save_elf_debug.
    
    The new code ensures that each inode is accessed by a single process at
    a time.
    
    Closes: https://bugs.gentoo.org/823798
    Signed-off-by: Mike Gilbert <[e-mail]>

 bin/estrip | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

Additionally, it has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=2cd58ac5a96541d0b1ba0440666114d48dc2c889

commit 2cd58ac5a96541d0b1ba0440666114d48dc2c889
Author:     Mike Gilbert <[e-mail]>
AuthorDate: 2021-11-16 19:58:42 +0000
Commit:     Mike Gilbert <[e-mail]>
CommitDate: 2021-11-19 16:16:07 +0000

    estrip: add death messages and debug prints in save_elf_debug
    
    Bug: https://bugs.gentoo.org/823798
    Signed-off-by: Mike Gilbert <[e-mail]>

 bin/estrip | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)