Classic distcc preprocesses source files locally, then sends of the files to the compile hosts. For some projects (mostly C++ stuff), this means that a large amount of work still remains on the "slow" host. Thus, the distcc devs invented pump mode, where a locally running server provides the the compile hosts with all include files and thus enbales preprocessing to be done on the compile hosts, resulting in a much higher speed gain than with classic distcc. Usually, pump mode is invoked by prefixing make or emerge with the pump command. The pump command sets up a shared directory with pipes etc. for the actual compile process. This works very well with classic compilation where only make is involved. With portage, this fails due to protage switching UIDs post-invocation. Pump is started as UID root (as "pump emerge foo/bar"), but portage soon switches to a non-0 UID if the userpriv and/or usersandbox features are active. As a result, the compile process can not access the pump dir and the whole process fails (is degraded to local compilation). Thus, it would be nice if pump support would be intergrated with portage so that the pump "prefix command" is used after portage has switched UIDs. This, naturally would be activated with another feature flag (e.g. distcc-pump) or by looking at /etc/distcc/hosts (the host-flag "cpp" tells distcc to try distributing preprocessing).
pump --help if you run eval `pump --startup` berore emerge it's work fine after emerge you shuld type pump --shutdown its forwarding pumps comunication into distcc executed by emerge --- sorry for my english
(In reply to comment #1) > eval `pump --startup` How does this help with the UID problem? I could start pump with sudo -u portage pump, but I still wouldn't be able to get the portage-internal environment changed easily - yes, there's env.d, but I think it'd be better to just have the call to pump be integrated with portage/emerge.
Thumb up! Pump+lzo may make distcc worth over a 1 Mbps link (I gotta try). By the way, do distcc work with c++ files? Or you still need ,cpp option appended to hosts that can compile c++ ?
(In reply to comment #3) > By the way, do distcc work with c++ files? > Or you still need ,cpp option appended to hosts that can compile c++ ? Yes, distcc works with C++ automagically, the option/attribute "cpp" stands for C/C++ pre-processor (i.e. include files, defines etc) and has nothing to do with the distinction between C and C++ (distcc mostly doesn't care about that, it basically tells the helping host compile this file with these options using a binary named like that).
Created attachment 274111 [details, diff] [RFC]distcc-pump support Added '$(/usr/bin/pump --startup)' and 'trap "/usr/bin/pump --shutdown" EXIT' before src_compile.
(In reply to comment #5) > Created attachment 274111 [details, diff] > [RFC]distcc-pump support > > Added '$(/usr/bin/pump --startup)' and 'trap "/usr/bin/pump --shutdown" EXIT' > before src_compile. Thanks, this is in git: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=729531f12e097f8bcbbe12d86bad169f27aa8aca
This is included in portage-2.2.0_alpha35, but I'll leave this bug open until it's released in an unmasked version.
How to use this new feature? FEATURES="distcc distcc-pump" or just FEATURES="distcc-pump" ? Will it work out of the box?
(In reply to comment #8) > How to use this new feature? > FEATURES="distcc distcc-pump" > > or just FEATURES="distcc-pump" ? You need both enabled. > Will it work out of the box? I haven't tested it myself.
This is in 2.1.9.50.
Could you change this to call 'pump' rather than '/usr/bin/pump'? Thanks.
(In reply to comment #11) > Could you change this to call 'pump' rather than '/usr/bin/pump'? Thanks. Ok, this is in git: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=12dd82b64c23764fab5a185bcfba446af3fcbf46
*** Bug 426290 has been marked as a duplicate of this bug. ***
Please comment if the distcc-pump feature is useful to you. There's a proposal to remove the distcc-pump feature: https://archives.gentoo.org/gentoo-portage-dev/message/3f3d3edc76819e131ac0a821a5725f69
