Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 702146

Summary: sys-apps/portage: consider removing FEATURES=distcc-pump feature from portage
Product: Portage Development Reporter: Sergei Trofimovich (RETIRED) <slyfox>
Component: Core - ConfigurationAssignee: Portage team <dev-portage>
Status: RESOLVED FIXED    
Severity: normal CC: cluster, mattst88, mgorny, whissi
Priority: Normal Keywords: InVCS
Version: unspecified   
Hardware: All   
OS: Linux   
See Also: https://bugs.gentoo.org/show_bug.cgi?id=636806
https://bugs.gentoo.org/show_bug.cgi?id=672754
https://bugs.gentoo.org/show_bug.cgi?id=671950
https://bugs.gentoo.org/show_bug.cgi?id=702162
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 701268    
Attachments: 0001-emerge-drop-FEATURES-distcc-pump-support-bug-702146.patch

Description Sergei Trofimovich (RETIRED) gentoo-dev 2019-12-06 08:33:21 UTC
Me and mattst88 spent some time debugging miscompilation of glibc on #gentoo-mips. We found out distcc-pump to be the failure trigger.

Looking at pump's assumptions they seem to be very strong: https://www.systutorials.com/docs/linux/man/1-pump/

"""
Note that distcc's pump-mode assumes that sources files will not be
 modified during the lifetime of the include server, so modifying source files during a build may cause inconsistent results.
"""

I don't think it holds even for ./configure scripts.

A few supporting cases for removal:
- bug #636806: llvm fails at cmake configuration time
- bug #672754#c3: statement by upstream maintainer that pump should not be ran on arbitrary code

I suggest removing the FEATURES=distcc-pump entirely from portage and leave any plumbing to end-users if they need to.

WDYT?
Comment 1 Sergei Trofimovich (RETIRED) gentoo-dev 2019-12-06 08:58:52 UTC
I think #507482 mostly broke distcc-pump as it added it to src_configure(). That difectly violates assumption of immutable sources.

But other phases (src_compile()) can do it just as well. For example gcc builds stage2 and stage3 in the same directory and generates intermediate source and header files there.
Comment 2 Thomas Deutschmann (RETIRED) gentoo-dev 2019-12-06 13:09:25 UTC
FYI: I found one package, dev-util/cmake, which doesn't compile _without_ distcc-pump.
Comment 3 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2019-12-06 13:28:28 UTC
(In reply to Thomas Deutschmann from comment #2)
> FYI: I found one package, dev-util/cmake, which doesn't compile _without_
> distcc-pump.

Don't want to kill your enthusiasm but it probably just means your system is so broken you need to --emptytree.
Comment 4 Sergei Trofimovich (RETIRED) gentoo-dev 2019-12-06 18:15:39 UTC
(In reply to Thomas Deutschmann from comment #2)
> FYI: I found one package, dev-util/cmake, which doesn't compile _without_
> distcc-pump.

Compared to FEATURES=distcc? I think it's a bug and should not happen. Or there is an extra negation in the sentence?
Comment 5 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2019-12-06 18:46:30 UTC
It turned out to be a PEBKAC.
Comment 6 Sergei Trofimovich (RETIRED) gentoo-dev 2019-12-06 21:01:59 UTC
Created attachment 598656 [details, diff]
0001-emerge-drop-FEATURES-distcc-pump-support-bug-702146.patch
Comment 7 Sergei Trofimovich (RETIRED) gentoo-dev 2019-12-07 10:13:01 UTC
Sent https://archives.gentoo.org/gentoo-portage-dev/message/87fade273f739fb913942867cf27bbc9 for review.
Comment 8 Larry the Git Cow gentoo-dev 2019-12-08 21:31:22 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=bc0c28bbcdb03506e6c0bafac9e63aff5ebb6894

commit bc0c28bbcdb03506e6c0bafac9e63aff5ebb6894
Author:     Sergei Trofimovich <slyfox@gentoo.org>
AuthorDate: 2019-12-06 21:00:46 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2019-12-08 21:29:45 +0000

    emerge: drop FEATURES=distcc-pump support, bug #702146
    
    'distcc' distributes code generation for preprocessed files.
    
    'pump' distributes preprocessing and code generation of files
    and imposes very strict requirement:
    
    """
    Note that distcc's pump-mode assumes that sources files will
    not be modified during the lifetime of the include server, so
    modifying source files during a build may cause inconsistent
    results.
    """
    
    `src_configure()` (where we used to start include server before
    this change) almost always violates that requirement.
    
    It is not uncommon to generate more intermediate source files
    as a package builds (`bison`, `flex`, child `./configure` calls
    from `make`) and thus quite unsafe to use `pump`.
    
    This change drops `FEATURES=distcc-pump` and leaves only
    FEATURES=distcc. This way all the proprocessing happens as expected
    and only code generation is offloaded.
    
    Bug: https://bugs.gentoo.org/702146
    Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 bin/phase-functions.sh     | 17 -----------------
 lib/_emerge/EbuildPhase.py |  2 +-
 lib/portage/const.py       |  1 -
 man/make.conf.5            |  3 ---
 4 files changed, 1 insertion(+), 22 deletions(-)
Comment 9 Larry the Git Cow gentoo-dev 2019-12-15 23:00:37 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=9557f104aa85b65c7d394c52c5c8d8727a111651

commit 9557f104aa85b65c7d394c52c5c8d8727a111651
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2019-12-15 22:53:50 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2019-12-15 22:56:34 +0000

    sys-apps/portage: Bump to version 2.3.82
    
     #310009 emerge: Show package USE in conflict messages
     #680456 display relevant FEATURES when unshare fails
     #693454 emerge-webrsync: support gentoo-YYYYMMDD snapshots
     #702146 emerge: drop FEATURES=distcc-pump support
     #702970 emerge-webrsync: enable xz snapshots for tarsync
    
    Bug: https://bugs.gentoo.org/701268
    Bug: https://bugs.gentoo.org/310009
    Bug: https://bugs.gentoo.org/680456
    Bug: https://bugs.gentoo.org/693454
    Bug: https://bugs.gentoo.org/702146
    Bug: https://bugs.gentoo.org/702970
    Package-Manager: Portage-2.3.82, Repoman-2.3.20
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-2.3.82.ebuild | 272 +++++++++++++++++++++++++++++++++
 2 files changed, 273 insertions(+)
Comment 10 Reva Denis 2020-01-21 18:59:43 UTC
Should remove distcc-pump usage from Handbook
Comment 11 Sergei Trofimovich (RETIRED) gentoo-dev 2020-01-21 19:49:12 UTC
(In reply to Reva Denis from comment #10)
> Should remove distcc-pump usage from Handbook

Can you link to a section where it's mentioned?
Comment 12 carbohydrates 2021-10-20 13:52:03 UTC
This is a not a good solution to the problem.  Rather, it is better to modify /etc/portage/package.env to accommodate the unfriendly packages.  For example, in package.env:

dev-util/cmake nodistcc.conf

And then in /etc/portage/env/nodistcc.conf to match:

FEATURES="-distcc"

This way, distcc and pump mode can remain enabled for all other packages.  

Please do change the fix here to this more appropriate solution.  No need to remove entire features when such a simple accommodation is available.
Comment 13 Matt Turner gentoo-dev 2021-10-21 02:39:35 UTC
(In reply to carbohydrates from comment #12)
> This is a not a good solution to the problem.  Rather, it is better to
> modify /etc/portage/package.env to accommodate the unfriendly packages.  For
> example, in package.env:
> 
> dev-util/cmake nodistcc.conf
> 
> And then in /etc/portage/env/nodistcc.conf to match:
> 
> FEATURES="-distcc"
> 
> This way, distcc and pump mode can remain enabled for all other packages.  
> 
> Please do change the fix here to this more appropriate solution.  No need to
> remove entire features when such a simple accommodation is available.

It's dead, Jim.

FWIW, I've got a semi-defunct project like you mention here: https://gitweb.gentoo.org/proj/no-distcc-env.git/