Bug 267290 – app-office/openoffice ebuild should export MAXPROCESS

• Product: Gentoo Linux
• Status: RESOLVED FIXED

[Description Matt Whitlock 2009-04-24 02:28:14 UTC]

The app-office/openoffice ebuild should export MAXPROCESS to take advantage of multiple CPU cores when building OpenOffice, as per:

http://wiki.services.openoffice.org/wiki/Building_OpenOffice.org#set_MAXPROCESS.3D.3Cnumer_or_processes.3E

This results in a dramatic speed-up of the build.

I'd recommend adding the following to the ebuild:

let MAXPROCESS=$(grep -c processor /proc/cpuinfo)+1
export MAXPROCESS

Alternatively, the user's preferred make parallelism might be extracted from ${MAKEOPTS}.  One possible way to do that:

export MAXPROCESS=$(echo "${MAKEOPTS}" |
    sed -re 's/^.*-([^-]*j *|-jobs=)([0-9]+).*$/\2/;t;d')

[Comment 1 Lars Wendler (Polynomial-C) (RETIRED) 2009-04-25 16:04:28 UTC]

This snippet already takes care of parallel builds in openoffice ebuilds:

  # Use multiprocessing by default now, it gets tested by upstream
  export JOBS=$(echo "${MAKEOPTS}" | sed -e "s/.*-j\([0-9]\+\).*/\1/")

[Comment 2 Matt Whitlock 2009-04-25 18:36:18 UTC]

(In reply to comment #1)
> This snippet already takes care of parallel builds in openoffice ebuilds:
> 
>   # Use multiprocessing by default now, it gets tested by upstream
>   export JOBS=$(echo "${MAKEOPTS}" | sed -e "s/.*-j\([0-9]\+\).*/\1/")
> 

You are incorrect.  It does not.  In a few hours, I will post another comment showing a comparison of the ebuild times with MAXPROCESS set and unset.  (It'll take that long to run the builds.)

[Comment 3 Matt Whitlock 2009-04-25 22:57:30 UTC]

# time ebuild /usr/portage/app-office/openoffice/openoffice-3.0.0.ebuild clean compile && uptime

real    172m6.369s
user    152m34.935s
sys     19m6.173s
 17:31:16 up 1 day, 19:40,  0 users,  load average: 1.40, 1.36, 1.29

# time MAXPROCESS=5 ebuild /usr/portage/app-office/openoffice/openoffice-3.0.0.ebuild clean compile && uptime

real    72m30.378s
user    155m50.827s
sys     19m46.905s
 18:50:06 up 1 day, 20:59,  0 users,  load average: 4.53, 4.84, 4.50

Both /tmp and /var/tmp/portage are tmpfs on my system, and I have no swap, so disk I/O is not a factor at all.  I do not use ccache or distcc.  I did not use the system for anything else during the builds.

As you can see, MAXPROCESS makes a huge difference.

Also, on a separate but related point, the sed script that creates the value for ${JOBS} in the existing ebuild will break if ${MAKEOPTS} does not contain a '-j' option, or if there's space between the '-j' and its argument, of if the '--jobs=' form of the option is used.  I'd recommend changing it to the script I gave in comment #1, which handles all of these cases.

[Comment 4 Matt Whitlock 2009-04-25 22:59:47 UTC]

Sorry, I meant the script I gave in the bug description, not in comment #1.

[Comment 5 Andreas Proschofsky (RETIRED) 2009-11-25 12:16:52 UTC]

(In reply to comment #2)

> You are incorrect.  It does not.  In a few hours, I will post another comment
> showing a comparison of the ebuild times with MAXPROCESS set and unset.  (It'll
> take that long to run the builds.)
> 

Hmm, that's "interesting". The current way it works definitely uses multiple processes on my system, you can even see that in the build output in lots of places. (and in the configure-part of the ebuild: --with-num-cpus="${JOBS}"). So with which MAKEOPTS-setting did you test the first case? 

[Comment 6 Matt Whitlock 2009-11-25 17:31:32 UTC]

(In reply to comment #5)
> Hmm, that's "interesting". The current way it works definitely uses multiple
> processes on my system, you can even see that in the build output in lots of
> places. (and in the configure-part of the ebuild: --with-num-cpus="${JOBS}").
> So with which MAKEOPTS-setting did you test the first case?

Set MAKEOPTS to "" (empty) and just compare the difference of MAXPROCESS="" and MAXPROCESS="4" (or however many CPU cores you have).

MAKEOPTS='-j#' controls how many libraries are built simultaneously.  MAXPROCESS controls how many compilation units are built simultaneously within a single library.  The build seems much stabler when you only do one library at a time.

For time comparison, you can try all four combinations of MAKEOPTS and MAXPROCESS.

[Comment 7 Tomáš Chvátal (RETIRED) 2011-07-26 21:20:32 UTC]

libreoffice-3.4.2.2 use jobserver correctly and should work as expected.
For older release the behaviour can't be backported as this is dependant on the buildsystem updates.