Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 912843 - <dev-util/pkgconf-2: poor performance with pkg-config --cflags grpc++, pkg-config --cflags protobuf, etc
Summary: <dev-util/pkgconf-2: poor performance with pkg-config --cflags grpc++, pkg-co...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Gentoo's Team for Core System packages
URL:
Whiteboard:
Keywords:
Depends on: 911721 923778
Blocks: 912819
  Show dependency tree
 
Reported: 2023-08-22 17:35 UTC by Sam James
Modified: 2024-06-08 15:36 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
build.log (file_912843.txt,9.13 KB, text/plain)
2023-08-22 17:35 UTC, Sam James
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-08-22 17:35:09 UTC
* https://github.com/grpc/grpc/issues/29137 (they seem to have been using fd.o pkgconfig here though)
* https://github.com/pkgconf/pkgconf/issues/229
* https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=265542

pkgconf-2 should fix this, but it's masked because of other bugs
Comment 1 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-08-22 17:35:51 UTC
Created attachment 868502 [details]
build.log

Report on IRC was:

[18:30:06]  <lts> Compiling dev-libs/protobuf-c-1.4.1-r1:0/1.0.0 seems to get stuck running "/usr/bin/x86_64-pc-linux-gnu-pkg-config --cflags protobuf >= 3.0.0". One core is at 100% and nothing happens. Tips? Build log https://bpa.st/WIZA , emerge --info https://bpa.st/GJMA
Comment 2 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-08-22 17:35:59 UTC
# emerge --info
Portage 3.0.51 (python 3.11.4-final-0, default/linux/amd64/17.1/desktop, gcc-13, glibc-2.37-r4, 5.15.124-gentoo-dist x86_64)
=================================================================
System uname: Linux-5.15.124-gentoo-dist-x86_64-AMD_Ryzen_9_3900X_12-Core_Processor-with-glibc2.37
KiB Mem:    32789856 total,  15757384 free
KiB Swap:          0 total,         0 free
Timestamp of repository gentoo: Tue, 22 Aug 2023 17:17:19 +0000
Head commit of repository gentoo: 883e9eb5f4f4b919acee762de49664c654df04a9

Timestamp of repository steam-overlay: Tue, 22 Aug 2023 17:16:56 +0000
Head commit of repository steam-overlay: d31b5771c4004728bf93708d30b58de4de734990

sh bash 5.2_p15-r6
ld GNU ld (Gentoo 2.41 p2) 2.41.0
app-misc/pax-utils:        1.3.7::gentoo
app-shells/bash:           5.2_p15-r6::gentoo
dev-lang/perl:             5.38.0-r1::gentoo
dev-lang/python:           3.11.4::gentoo
dev-lang/rust:             1.71.1::gentoo
dev-util/cmake:            3.27.3-r1::gentoo
dev-util/meson:            1.2.1-r1::gentoo
sys-apps/baselayout:       2.14::gentoo
sys-apps/openrc:           0.48::gentoo
sys-apps/sandbox:          2.38::gentoo
sys-devel/autoconf:        2.13-r8::gentoo, 2.71-r7::gentoo
sys-devel/automake:        1.16.5-r1::gentoo
sys-devel/binutils:        2.41-r1::gentoo
sys-devel/binutils-config: 5.5::gentoo
sys-devel/clang:           15.0.7-r3::gentoo, 16.0.6::gentoo
sys-devel/gcc:             13.2.0::gentoo
sys-devel/gcc-config:      2.11::gentoo
sys-devel/libtool:         2.4.7-r1::gentoo
sys-devel/lld:             15.0.7::gentoo, 16.0.6::gentoo
sys-devel/llvm:            15.0.7-r3::gentoo, 16.0.6::gentoo
sys-devel/make:            4.4.1-r1::gentoo
sys-kernel/linux-headers:  5.15-r3::gentoo (virtual/os-headers)
sys-libs/glibc:            2.37-r4::gentoo
Repositories:

gentoo
    location: /var/db/repos/gentoo
    sync-type: git
    sync-uri: https://anongit.gentoo.org/git/repo/sync/gentoo.git
    priority: -1000
    volatile: False

steam-overlay
    location: /var/db/repos/steam-overlay
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/steam-overlay.git
    masters: gentoo
    volatile: False

ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="*"
AR="llvm-ar"
CBUILD="x86_64-pc-linux-gnu"
CC="clang"
CFLAGS="-march=native -O3 -pipe -flto=thin"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXX="clang++"
CXXFLAGS="-march=native -O3 -pipe -flto=thin"
DISTDIR="/var/cache/distfiles"
EMERGE_DEFAULT_OPTS="--jobs 4 --load-average 6"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance buildpkg-live config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch parallel-install pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="https://mirrors.dotsrc.org/gentoo/ https://ftp.fau.de/gentoo http://distfiles.gentoo.org"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O2 -Wl,--as-needed"
LEX="flex"
MAKEOPTS="-j12 -l11"
NM="llvm-nm"
PKGDIR="/var/cache/binpkgs"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
RANLIB="llvm-ranlib"
SHELL="/bin/zsh"
USE="X a52 aac acl acpi alsa amd64 branding bzip2 cairo cdda cdr clang cli crypt cups dav1d dbus dri dts dvd dvdr elogind encode exif flac fortran gdbm gif gpm gtk gui iconv icu jpeg jumbo-build lcms libnotify libtirpc lm-sensors lto mad mng mp3 mp4 mpeg multilib ncurses nls nptl ogg opengl openh264 openmp opus pam pango pcre pdf pipewire png ppds pulseaudio qt5 readline sdl seccomp sound spell split-usr ssl startup-notification svg test-rust tiff truetype udev unicode upower usb vaapi vdpau vorbis vulkan webp wxwidgets x264 x265 xattr xcb xft xml xv xvid zlib" ABI_X86="64" ADA_TARGET="gnat_2021" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2 aes avx avx2 f16c fma3 pclmul popcnt rdrand sha sse3 sse4_1 sse4_2 sse4a ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64" INPUT_DEVICES="libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php8-1" POSTGRES_TARGETS="postgres15" PYTHON_SINGLE_TARGET="python3_11" PYTHON_TARGETS="python3_11" RUBY_TARGETS="ruby31" SANE_BACKENDS="pixma" VIDEO_CARDS="amdgpu radeonsi" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  ADDR2LINE, ARFLAGS, AS, ASFLAGS, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXXFILT, ELFEDIT, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LFLAGS, LIBTOOL, LINGUAS, MAKE, MAKEFLAGS, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS
Comment 3 Xuefer 2023-08-26 14:53:03 UTC
please use "pkgconf --libs protobuf --simulate" and/or "pkgconf --cflags protobuf --simulate" to reproduce. the output is huge compared to others *.pc as protobuf contains nested "Requires". please back port any patch from 2.* to fix this problem since 2.* is masked
Comment 5 Xuefer 2023-08-26 16:10:38 UTC
problem gone by using 1.9.5 (copy pkgconf-1.8.1.ebuild to pkgconf-1.9.5.ebuild local repo)
Comment 6 Larry the Git Cow gentoo-dev 2023-09-03 04:33:07 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=5dea9db406aea15a140294d30af8317762f69461

commit 5dea9db406aea15a140294d30af8317762f69461
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2023-09-03 04:32:00 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2023-09-03 04:32:51 +0000

    profiles: unmask pkgconf-2 again
    
    We really do need this unmasked if we can given the protobuf performance issue.
    
    Bug: https://bugs.gentoo.org/911721
    Bug: https://bugs.gentoo.org/912843
    Bug: https://bugs.gentoo.org/912461
    Bug: https://bugs.gentoo.org/912581
    Signed-off-by: Sam James <sam@gentoo.org>

 profiles/package.mask | 4 ----
 1 file changed, 4 deletions(-)

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=8f31c5a32dc20a57764a9b073cf989e9cf2ea615

commit 8f31c5a32dc20a57764a9b073cf989e9cf2ea615
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2023-09-03 04:30:44 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2023-09-03 04:32:49 +0000

    dev-util/pkgconf: add 2.0.3
    
    Bug: https://bugs.gentoo.org/911721
    Bug: https://bugs.gentoo.org/912843
    Closes: https://bugs.gentoo.org/912461
    Closes: https://bugs.gentoo.org/912581
    Signed-off-by: Sam James <sam@gentoo.org>

 dev-util/pkgconf/Manifest             |  1 +
 dev-util/pkgconf/pkgconf-2.0.3.ebuild | 69 +++++++++++++++++++++++++++++++++++
 2 files changed, 70 insertions(+)
Comment 7 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-09-03 06:48:34 UTC
If issues persist with pkgconf-2, we can try backport stuff to 1.8.x, although I haven't checked if any of the commits break ABI (something did between 1.x and 2.x, I just haven't checked what and if it's related to this).
Comment 8 Allen Webb 2023-11-08 20:30:48 UTC
I wonder if this might be related to abseil-cpp being overly redundant with its cflags and libs. On ChromeOS we added this to src_install(){} for abseil-cpp

```
	# absl adds all the Cflags to all the pc files even though almost all the
	# .pc files require abs_config.pc. This causes an explosion of flags when
	# including multiple abseil sub-libraries through pkg-config. Until the
	# issue is fixed upstream, strip them out here since this is easier to
	# maintain than a version specific patch.
	find "${D}/usr/$(get_libdir)/pkgconfig" -type f -print0 -not -name 'absl_config.pc' | xargs -0 sed -i '/^Cflags: /d' || die
```
Comment 9 Larry the Git Cow gentoo-dev 2023-11-22 19:06:34 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6294bda3ed20ec6ccc9186ef9c2612ce32a79323

commit 6294bda3ed20ec6ccc9186ef9c2612ce32a79323
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2023-11-22 19:05:44 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2023-11-22 19:05:52 +0000

    dev-util/pkgconf: add 2.1.0
    
    Bug: https://bugs.gentoo.org/912843
    Bug: https://bugs.gentoo.org/911721
    Signed-off-by: Sam James <sam@gentoo.org>

 dev-util/pkgconf/Manifest             |  1 +
 dev-util/pkgconf/pkgconf-2.1.0.ebuild | 69 +++++++++++++++++++++++++++++++++++
 dev-util/pkgconf/pkgconf-9999.ebuild  |  2 +-
 3 files changed, 71 insertions(+), 1 deletion(-)
Comment 10 Matt Turner gentoo-dev 2023-12-21 16:06:28 UTC
A similar thing was reported internally at Google -- https://issuetracker.google.com/317357322
Comment 11 Volkmar W. Pogatzki 2024-01-22 09:28:26 UTC
What is the status here?  Why is it still blocking bug #912819?
Didn't those 3 commits solve the problem?
Comment 12 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-01-22 09:35:00 UTC
(In reply to Volkmar W. Pogatzki from comment #11)
> What is the status here?  Why is it still blocking bug #912819?
> Didn't those 3 commits solve the problem?

Again, we don't tend to remove things from blocking trackers when they're fixed, we just close the bug.

Anyway, no, there is another problem:
* https://issuetracker.google.com/317357322
* https://issuetracker.google.com/issues/319549559
* https://github.com/pkgconf/pkgconf/pull/339
Comment 13 Larry the Git Cow gentoo-dev 2024-01-22 15:59:12 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=5bc2303601b0dc6bc4113f75292371518f8f2453

commit 5bc2303601b0dc6bc4113f75292371518f8f2453
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2024-01-22 15:57:36 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2024-01-22 15:57:36 +0000

    dev-util/pkgconf: backport upstream fixes, add Google performance fix
    
    * Backport upstream fixes for --modversion
    * Pull in pending upstream PR for poor performance hit by Google
    * Pull in another pending upstream PR which adds another test for a previously
      problematic area
    
    Bug: https://bugs.gentoo.org/912843
    Signed-off-by: Sam James <sam@gentoo.org>

 .../pkgconf/files/pkgconf-2.1.0-digraph-test.patch | 41 ++++++++++
 .../files/pkgconf-2.1.0-fix-modversion.patch       | 95 ++++++++++++++++++++++
 .../files/pkgconf-2.1.0-traverse-only-once.patch   | 71 ++++++++++++++++
 dev-util/pkgconf/pkgconf-2.1.0-r1.ebuild           | 75 +++++++++++++++++
 4 files changed, 282 insertions(+)
Comment 14 Larry the Git Cow gentoo-dev 2024-02-05 02:21:32 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=97fa4bd219d50b5aa3a9a296dc5ce25c731f9f0e

commit 97fa4bd219d50b5aa3a9a296dc5ce25c731f9f0e
Author:     Sam James <sam@gentoo.org>
AuthorDate: 2024-02-05 02:19:10 +0000
Commit:     Sam James <sam@gentoo.org>
CommitDate: 2024-02-05 02:21:23 +0000

    dev-util/pkgconf: add 2.1.1
    
    This has tweaked versions of the fixes we pulled in from PRs in 2.1.0-r1.
    
    Bug: https://bugs.gentoo.org/912843
    Bug: https://bugs.gentoo.org/923778
    Signed-off-by: Sam James <sam@gentoo.org>

 dev-util/pkgconf/Manifest             |  1 +
 dev-util/pkgconf/pkgconf-2.1.1.ebuild | 69 +++++++++++++++++++++++++++++++++++
 dev-util/pkgconf/pkgconf-9999.ebuild  |  2 +-
 3 files changed, 71 insertions(+), 1 deletion(-)
Comment 15 Matt Turner gentoo-dev 2024-06-08 15:27:36 UTC
Is this fixed?
Comment 16 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2024-06-08 15:36:02 UTC
I think so.