Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 519700 - sys-cluster/{openmpi,mpich2}: migrate to multilib eclasses
Summary: sys-cluster/{openmpi,mpich2}: migrate to multilib eclasses
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Gentoo Cluster Team
URL:
Whiteboard:
Keywords:
Depends on: 540036 540294 540392 541738 541746 592468
Blocks: gx86-multilib 510180
  Show dependency tree
 
Reported: 2014-08-12 10:13 UTC by Sergey Popov
Modified: 2016-09-05 21:21 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
patch to add multilib support to openmpi (openmpi-1.8.4-r4.ebuild.patch,5.38 KB, patch)
2016-08-09 22:53 UTC, Christoph Junghans
Details | Diff
patch to add multilib support to mpich (mpich-3.1.3.ebuild.patch,2.17 KB, patch)
2016-08-09 22:57 UTC, Christoph Junghans
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Sergey Popov gentoo-dev 2014-08-12 10:13:47 UTC
Hi. Recently, in bug #512884 dev-libs/boost was ported to use multilib features. But neigher sys-cluster/openmpi nor sys-cluster/mpich2 ebuilds have them - those two packages are dependendices for Boost, builded with USE="mpi".

I have added dirty hack to workaround this issue for now - MPI is used only for native ABI.

Please provide multilib-aware ebuilds for those two packages or explain if it can not be done, thanks in advance
Comment 1 Christoph Junghans gentoo-dev 2014-08-12 14:57:52 UTC
Please, we had similar problem for fftw in bug #519094 and bug #483758.
Comment 2 Christoph Junghans gentoo-dev 2014-08-13 21:25:16 UTC
sys-cluster/mpich deps:
- dev-libs/libaio (already multilib)
- sys-apps/hwloc (not multilib, but bug #456984)
- net-fs/nfs-utils (not multilib, but only USE=romio)
- sys-devel/libtool (already multilib)

sys-cluster/openmpi:
- dev-libs/libevent (already multilib)
- >=sys-apps/hwloc-1.7.2 (not multilib)
- sys-libs/zlib (already multilib)
- dev-util/nvidia-cuda-toolkit (not multilib, but only USE=cuda)
- ... many more.
Comment 3 Christoph Junghans gentoo-dev 2015-02-17 00:15:51 UTC
Ok, I implemented a version for testing in the science overlay.
Comment 4 Christoph Junghans gentoo-dev 2015-02-17 15:46:15 UTC
@pinkbyte: I have added =dev-libs/boost-1.56.0-r2::science with multilib mpi for testing. 

However, currently there are two major showstoppers
1.) need to export multilib FC/F77 in toolchain.eclass (bug #540294)
2.) wrong path in gcc's 32bit libtool files (bug #540392)
Comment 5 Justin Bronder (RETIRED) gentoo-dev 2015-02-17 16:23:41 UTC
(In reply to Christoph Junghans from comment #3)
> Ok, I implemented a version for testing in the science overlay.

Please continue to use mpi.eclass in the overlay.
Comment 6 Justin Bronder (RETIRED) gentoo-dev 2015-02-18 06:47:09 UTC
https://github.com/gentoo-science/sci/issues/372
Comment 7 Christoph Junghans gentoo-dev 2015-02-18 18:42:32 UTC
(In reply to Justin Bronder from comment #5)
> Please continue to use mpi.eclass in the overlay.
Sorry, I overlooked that, is mpi.eclass already multilib capable?
Comment 8 Justin Bronder (RETIRED) gentoo-dev 2015-02-18 21:42:02 UTC
(In reply to Christoph Junghans from comment #7)
> (In reply to Justin Bronder from comment #5)
> > Please continue to use mpi.eclass in the overlay.
> Sorry, I overlooked that, is mpi.eclass already multilib capable?

I have no idea :)  I haven't really looked at multilib too closely as the deps were not multilib.  Then you guys all got motivated to work on this so who was I to complain... until I noticed mpi.eclass got left out.

But people are depending on being able to use empi to install multiple MPI implementations from the overlay.  It's not a huge issue right now as empi will ignore stuff that doesn't use mpi.eclass.  So no one is 'broken', but eventually they will want upgrades.

As I try to keep things somewhat in sync between the main tree and the overlay, not being able to use mpi.eclass with multilib will be a blocker for merging this stuff to gx86 until the two can play nicely together.

I can look closer at multilib later, but if it's also playing games with the configure arguments like mpi.eclass then there's probably going to be some work involved.  Hopefully we can pass stuff like mpi_root() and mpi_class() to multilib to come up with paths both eclasses are happy with.
Comment 9 Christoph Junghans gentoo-dev 2015-02-18 22:36:34 UTC
(In reply to Justin Bronder from comment #8)
> (In reply to Christoph Junghans from comment #7)
> > (In reply to Justin Bronder from comment #5)
> > > Please continue to use mpi.eclass in the overlay.
> > Sorry, I overlooked that, is mpi.eclass already multilib capable?
> 
> I have no idea :)  I haven't really looked at multilib too closely as the
> deps were not multilib.  Then you guys all got motivated to work on this so
> who was I to complain... until I noticed mpi.eclass got left out.
Sorry to have broken your stuff! Is there any reason to not port mpi.eclass to gx86? Maybe this would be the right opportunity to at least add support for multiple simultaneously installed mpi versions.
Comment 10 Justin Bronder (RETIRED) gentoo-dev 2015-02-18 23:49:49 UTC
(In reply to Christoph Junghans from comment #9)
> (In reply to Justin Bronder from comment #8)
> > (In reply to Christoph Junghans from comment #7)
> > > (In reply to Justin Bronder from comment #5)
> > > > Please continue to use mpi.eclass in the overlay.
> > > Sorry, I overlooked that, is mpi.eclass already multilib capable?
> > 
> > I have no idea :)  I haven't really looked at multilib too closely as the
> > deps were not multilib.  Then you guys all got motivated to work on this so
> > who was I to complain... until I noticed mpi.eclass got left out.
> Sorry to have broken your stuff! Is there any reason to not port mpi.eclass
> to gx86? Maybe this would be the right opportunity to at least add support
> for multiple simultaneously installed mpi versions.

Not really.  Way back when mpi.eclass was originally written posting an eclass to -dev was just asking for flames and bikeshedding so I didn't bother.  People using it typically install the overlay anyway and seemed happy enough with the current setup.

It might be worth making sure that people still want this functionality.  When it was under development I got a lot of feedback but I haven't heard much recently especially from the larger sites.
Comment 11 Christoph Junghans gentoo-dev 2016-08-09 22:53:42 UTC
Created attachment 442882 [details, diff]
patch to add multilib support to openmpi
Comment 12 Christoph Junghans gentoo-dev 2016-08-09 22:57:57 UTC
Created attachment 442884 [details, diff]
patch to add multilib support to mpich
Comment 13 Christoph Junghans gentoo-dev 2016-08-11 19:36:18 UTC
Stuff now in the main tree, let's wait a couple of weeks and then re-add keywords.
Comment 14 Justin Bronder (RETIRED) gentoo-dev 2016-08-11 20:59:36 UTC
(In reply to Christoph Junghans from comment #13)
> Stuff now in the main tree, let's wait a couple of weeks and then re-add
> keywords.

Assuming no calls to keep empi/mpi.eclass around from the science mailing list that is ;)
Comment 15 Christoph Junghans gentoo-dev 2016-08-25 18:20:50 UTC
(In reply to Justin Bronder from comment #14)
> (In reply to Christoph Junghans from comment #13)
> > Stuff now in the main tree, let's wait a couple of weeks and then re-add
> > keywords.
> 
> Assuming no calls to keep empi/mpi.eclass around from the science mailing
> list that is ;)
Two weeks later and only one response (by a gentoo dev) later, what should we do?
Comment 16 David Seifert gentoo-dev 2016-08-25 18:28:28 UTC
(In reply to Christoph Junghans from comment #15)
> (In reply to Justin Bronder from comment #14)
> > (In reply to Christoph Junghans from comment #13)
> > > Stuff now in the main tree, let's wait a couple of weeks and then re-add
> > > keywords.
> > 
> > Assuming no calls to keep empi/mpi.eclass around from the science mailing
> > list that is ;)
> Two weeks later and only one response (by a gentoo dev) later, what should
> we do?

The problem with unkeyworded ebuilds is that don't really get any exposure. I would opt for keywording them all and unleashing them on unstable users.
Comment 17 Justin Bronder (RETIRED) gentoo-dev 2016-08-25 19:18:14 UTC
Well that's two production HPC clusters with some number of users depending on mpi.eclass which I believe are exactly the sort of people we should be supporting with openmpi and mpich.  I'm still not sure what the advantage of multilib.eclass is for that target audience.  I still have no problem with it being added but I'm very hesitant to break current functionality that anyone is using.  I'd ask that you guys look at getting it working in the overlay with mpi.eclass first [1] or let me know why that's not a workable solution.  Really, all that eclass does is install to a different prefix.

1.  If I don't get to it first, but let's be honest, I'm not all that motivated to spend the limited amount of time I have on this.
Comment 18 Christoph Junghans gentoo-dev 2016-08-26 00:05:08 UTC
I don't want to hinder development in the main tree due to some eclass in the science overlay.

Why aren't we just masking "sys-cluster/{openmpi,mpich2}::gentoo abi_x86_32" and "sci-libs/fftw::gentoo abi_x86_32" in science overlay and put "!multilib_is_native_abi" guard around the extra stuff we added to multilib sys-cluster/{openmpi,mpich2} and merge these changes back in the science overlay.
Comment 19 Justin Bronder (RETIRED) gentoo-dev 2016-08-26 01:41:00 UTC
The actual users of these packages are using the science overlay.

But, if I understand your proposal correctly, I think it satisfies my concerns.

1.  Preserve functionality people are currently using
2.  Make it easy to move changes back and forth from the overlay (the diff between ebuilds should be small and very obvious as to what can be left out).

So, if the patch ends up looking like something similar to the difference between pkg-9999.ebuild and pkg-1.2.3.ebuild where the former pulls from a vcs and the latter is using tarballs, I like it.  Please post something for review.
Comment 20 Christoph Junghans gentoo-dev 2016-08-26 22:49:50 UTC
Ok, package.use{,.mask} is a per-overlay thing, so we cannot mask the abi_x86_32 in gx86 from the science overlay, but we could explicitly block against sys-cluster/openmpi[abi_x86_32] in mpi.eclass on amd64 and the same in virtual/mpi.

Concerning the changes, I would add a variable to empi.eclass e.g. IS_EMPI=yes, which we can use differentiate the overlays. The advantage of a variable over a functions that the default is empty and hence gx86 doesn't need to know about mpi.eclass.
Comment 21 Christoph Junghans gentoo-dev 2016-08-27 00:47:37 UTC
Work in progress on the multilib_mpi branch of the science overlay:
https://github.com/gentoo-science/sci/compare/multilib_mpi

I think I will port sys-cluster/nullmpi to gx86 and hence remove virtual/mpi from the science overlay.

@soap: Can you back-port your clean-up to mpich and openmpi to the science overlay and put the multilib addition in  [[ ${EMPI} ]] guards?

On a related though mpi.eclass should maybe export MULTILIB_COMPAT="abi_x86_64", too?
Comment 22 Christoph Junghans gentoo-dev 2016-08-29 16:55:35 UTC
I found a much simpler solution using REQUIRED_USE and ported the changes from gx86 back into the science overlay, see:
https://github.com/gentoo-science/sci/pull/664

I didn't tested mpich yet!
Comment 23 Christoph Junghans gentoo-dev 2016-08-29 21:05:13 UTC
Multilib support merge in science overlay.

Re-added keywords to ebuild in gx86, let's see what breaks!
Comment 24 Christoph Junghans gentoo-dev 2016-08-29 21:51:09 UTC
@toralf: Could please you do a tinderbox run using multilib mpi?
Comment 25 Toralf Förster gentoo-dev 2016-08-30 08:31:32 UTC
(In reply to Christoph Junghans from comment #24)
I think it is already in place :

tinderbox@mr-fox ~ $ grep -i ' mpi' amd64-*/etc/portage/make.conf
amd64-hardened-libressl-unstable_20160821-203105/etc/portage/make.conf:  libinput mad midi mikmod mod modplug mpeg2 mpg123 mpi multitarget
amd64-hardened-unstable_20160821-190619/etc/portage/make.conf:  libvirtd midi mod mpeg mpi multimedia multitarget nss opencv -pam pdo