Bug 288597 – Full support for distcc's pump mode

• Product: Portage Development
• Status: RESOLVED FIXED

[Description Tobias Klausmann (RETIRED) 2009-10-11 17:38:02 UTC]

Classic distcc preprocesses source files locally, then sends of the files to the compile hosts. For some projects (mostly C++ stuff), this means that a large amount of work still remains on the "slow" host.

Thus, the distcc devs invented pump mode, where a locally running server provides the the compile hosts with all include files and thus enbales preprocessing to be done on the compile hosts, resulting in a much higher speed gain than with classic distcc.

Usually, pump mode is invoked by prefixing make or emerge with the pump command. The pump command sets up a shared directory with pipes etc. for the actual compile process. This works very well with classic compilation where only make is involved.

With portage, this fails due to protage switching UIDs post-invocation. Pump is started as UID root (as "pump emerge foo/bar"), but portage soon switches to a non-0 UID if the userpriv and/or usersandbox features are active. As a result, the compile process can not access the pump dir and the whole process fails (is degraded to local compilation).

Thus, it would be nice if pump support would be intergrated with portage so that the pump "prefix command" is used after portage has switched UIDs. This, naturally would be activated with another feature flag (e.g. distcc-pump) or by looking at /etc/distcc/hosts (the host-flag "cpp" tells distcc to try distributing preprocessing).

[Comment 1 Szpadel 2010-01-04 15:21:57 UTC]

pump --help

if you run 
   eval `pump --startup`
berore emerge it's work fine
after emerge you shuld type 
   pump --shutdown

its forwarding pumps comunication into distcc executed by emerge


---
sorry for my english

[Comment 2 Tobias Klausmann (RETIRED) 2011-01-05 10:59:12 UTC]

(In reply to comment #1)
>    eval `pump --startup`

How does this help with the UID problem? I could start pump with sudo -u portage pump, but I still wouldn't be able to get the portage-internal environment changed easily - yes, there's env.d, but I think it'd be better to just have the call to pump be integrated with portage/emerge.

[Comment 3 niogic 2011-04-02 00:22:23 UTC]

Thumb up!
Pump+lzo may make distcc worth over a 1 Mbps link (I gotta try).

By the way, do distcc work with c++ files?
Or you still need ,cpp option appended to hosts that can compile c++ ?

[Comment 4 Tobias Klausmann (RETIRED) 2011-04-02 12:59:18 UTC]

(In reply to comment #3)
> By the way, do distcc work with c++ files?
> Or you still need ,cpp option appended to hosts that can compile c++ ?

Yes, distcc works with C++ automagically, the option/attribute "cpp" stands for C/C++ pre-processor (i.e. include files, defines etc) and has nothing to do with the distinction between C and C++ (distcc mostly doesn't care about that, it basically tells the helping host compile this file with these options using a binary named like that).

[Comment 5 MATSUU Takuto (RETIRED) 2011-05-20 14:46:35 UTC]

Created attachment 274111 [details, diff]
[RFC]distcc-pump support

Added '$(/usr/bin/pump --startup)' and 'trap "/usr/bin/pump --shutdown" EXIT' before src_compile.

[Comment 6 Zac Medico 2011-05-20 19:22:23 UTC]

(In reply to comment #5)
> Created attachment 274111 [details, diff]
> [RFC]distcc-pump support
> 
> Added '$(/usr/bin/pump --startup)' and 'trap "/usr/bin/pump --shutdown" EXIT'
> before src_compile.

Thanks, this is in git:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=729531f12e097f8bcbbe12d86bad169f27aa8aca

[Comment 7 Zac Medico 2011-05-23 07:19:16 UTC]

This is included in portage-2.2.0_alpha35, but I'll leave this bug open until it's released in an unmasked version.

[Comment 8 niogic 2011-05-23 07:22:34 UTC]

How to use this new feature?
FEATURES="distcc distcc-pump"

or just FEATURES="distcc-pump" ?

Will it work out of the box?

[Comment 9 Zac Medico 2011-05-23 07:26:04 UTC]

(In reply to comment #8)
> How to use this new feature?
> FEATURES="distcc distcc-pump"
> 
> or just FEATURES="distcc-pump" ?

You need both enabled.

> Will it work out of the box?

I haven't tested it myself.

[Comment 10 Zac Medico 2011-05-27 04:59:40 UTC]

This is in 2.1.9.50.

[Comment 11 Jeremy Murphy 2011-06-08 00:57:55 UTC]

Could you change this to call 'pump' rather than '/usr/bin/pump'?  Thanks.

[Comment 12 Zac Medico 2011-06-09 14:36:28 UTC]

(In reply to comment #11)
> Could you change this to call 'pump' rather than '/usr/bin/pump'?  Thanks.

Ok, this is in git:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=12dd82b64c23764fab5a185bcfba446af3fcbf46

[Comment 13 Zac Medico 2012-07-12 18:09:13 UTC]

*** Bug 426290 has been marked as a duplicate of this bug. ***

[Comment 14 Zac Medico 2018-07-16 20:08:42 UTC]

Please comment if the distcc-pump feature is useful to you. There's a proposal to remove the distcc-pump feature:

https://archives.gentoo.org/gentoo-portage-dev/message/3f3d3edc76819e131ac0a821a5725f69