Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 702146 - sys-apps/portage: consider removing FEATURES=distcc-pump feature from portage
Summary: sys-apps/portage: consider removing FEATURES=distcc-pump feature from portage
Status: IN_PROGRESS
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core - Configuration (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on:
Blocks: 701268
  Show dependency tree
 
Reported: 2019-12-06 08:33 UTC by Sergei Trofimovich
Modified: 2020-02-02 21:07 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
0001-emerge-drop-FEATURES-distcc-pump-support-bug-702146.patch (0001-emerge-drop-FEATURES-distcc-pump-support-bug-702146.patch,3.53 KB, patch)
2019-12-06 21:01 UTC, Sergei Trofimovich
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Sergei Trofimovich gentoo-dev 2019-12-06 08:33:21 UTC
Me and mattst88 spent some time debugging miscompilation of glibc on #gentoo-mips. We found out distcc-pump to be the failure trigger.

Looking at pump's assumptions they seem to be very strong: https://www.systutorials.com/docs/linux/man/1-pump/

"""
Note that distcc's pump-mode assumes that sources files will not be
 modified during the lifetime of the include server, so modifying source files during a build may cause inconsistent results.
"""

I don't think it holds even for ./configure scripts.

A few supporting cases for removal:
- bug #636806: llvm fails at cmake configuration time
- bug #672754#c3: statement by upstream maintainer that pump should not be ran on arbitrary code

I suggest removing the FEATURES=distcc-pump entirely from portage and leave any plumbing to end-users if they need to.

WDYT?
Comment 1 Sergei Trofimovich gentoo-dev 2019-12-06 08:58:52 UTC
I think #507482 mostly broke distcc-pump as it added it to src_configure(). That difectly violates assumption of immutable sources.

But other phases (src_compile()) can do it just as well. For example gcc builds stage2 and stage3 in the same directory and generates intermediate source and header files there.
Comment 2 Thomas Deutschmann gentoo-dev Security 2019-12-06 13:09:25 UTC
FYI: I found one package, dev-util/cmake, which doesn't compile _without_ distcc-pump.
Comment 3 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2019-12-06 13:28:28 UTC
(In reply to Thomas Deutschmann from comment #2)
> FYI: I found one package, dev-util/cmake, which doesn't compile _without_
> distcc-pump.

Don't want to kill your enthusiasm but it probably just means your system is so broken you need to --emptytree.
Comment 4 Sergei Trofimovich gentoo-dev 2019-12-06 18:15:39 UTC
(In reply to Thomas Deutschmann from comment #2)
> FYI: I found one package, dev-util/cmake, which doesn't compile _without_
> distcc-pump.

Compared to FEATURES=distcc? I think it's a bug and should not happen. Or there is an extra negation in the sentence?
Comment 5 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2019-12-06 18:46:30 UTC
It turned out to be a PEBKAC.
Comment 6 Sergei Trofimovich gentoo-dev 2019-12-06 21:01:59 UTC
Created attachment 598656 [details, diff]
0001-emerge-drop-FEATURES-distcc-pump-support-bug-702146.patch
Comment 8 Larry the Git Cow gentoo-dev 2019-12-08 21:31:22 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=bc0c28bbcdb03506e6c0bafac9e63aff5ebb6894

commit bc0c28bbcdb03506e6c0bafac9e63aff5ebb6894
Author:     Sergei Trofimovich <slyfox@gentoo.org>
AuthorDate: 2019-12-06 21:00:46 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2019-12-08 21:29:45 +0000

    emerge: drop FEATURES=distcc-pump support, bug #702146
    
    'distcc' distributes code generation for preprocessed files.
    
    'pump' distributes preprocessing and code generation of files
    and imposes very strict requirement:
    
    """
    Note that distcc's pump-mode assumes that sources files will
    not be modified during the lifetime of the include server, so
    modifying source files during a build may cause inconsistent
    results.
    """
    
    `src_configure()` (where we used to start include server before
    this change) almost always violates that requirement.
    
    It is not uncommon to generate more intermediate source files
    as a package builds (`bison`, `flex`, child `./configure` calls
    from `make`) and thus quite unsafe to use `pump`.
    
    This change drops `FEATURES=distcc-pump` and leaves only
    FEATURES=distcc. This way all the proprocessing happens as expected
    and only code generation is offloaded.
    
    Bug: https://bugs.gentoo.org/702146
    Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 bin/phase-functions.sh     | 17 -----------------
 lib/_emerge/EbuildPhase.py |  2 +-
 lib/portage/const.py       |  1 -
 man/make.conf.5            |  3 ---
 4 files changed, 1 insertion(+), 22 deletions(-)
Comment 9 Larry the Git Cow gentoo-dev 2019-12-15 23:00:37 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=9557f104aa85b65c7d394c52c5c8d8727a111651

commit 9557f104aa85b65c7d394c52c5c8d8727a111651
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2019-12-15 22:53:50 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2019-12-15 22:56:34 +0000

    sys-apps/portage: Bump to version 2.3.82
    
     #310009 emerge: Show package USE in conflict messages
     #680456 display relevant FEATURES when unshare fails
     #693454 emerge-webrsync: support gentoo-YYYYMMDD snapshots
     #702146 emerge: drop FEATURES=distcc-pump support
     #702970 emerge-webrsync: enable xz snapshots for tarsync
    
    Bug: https://bugs.gentoo.org/701268
    Bug: https://bugs.gentoo.org/310009
    Bug: https://bugs.gentoo.org/680456
    Bug: https://bugs.gentoo.org/693454
    Bug: https://bugs.gentoo.org/702146
    Bug: https://bugs.gentoo.org/702970
    Package-Manager: Portage-2.3.82, Repoman-2.3.20
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-2.3.82.ebuild | 272 +++++++++++++++++++++++++++++++++
 2 files changed, 273 insertions(+)
Comment 10 Reva Denis 2020-01-21 18:59:43 UTC
Should remove distcc-pump usage from Handbook
Comment 11 Sergei Trofimovich gentoo-dev 2020-01-21 19:49:12 UTC
(In reply to Reva Denis from comment #10)
> Should remove distcc-pump usage from Handbook

Can you link to a section where it's mentioned?