Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 473838 - app-emulation/emul-linux-* could install into /opt to ease the transition to multilib eclasses
Summary: app-emulation/emul-linux-* could install into /opt to ease the transition to ...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: AMD64 Project
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-06-19 15:31 UTC by Alexis Ballier
Modified: 2013-07-14 23:56 UTC (History)
6 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Alexis Ballier gentoo-dev 2013-06-19 15:31:04 UTC
This list is growing and growing:

# Samuli Suominen <ssuominen@gentoo.org> (16 Mar 2013)
# Waiting for new emul-linux-x86-{media,sound}libs without
# the duplicate files.
>=media-libs/audiofile-0.3.6-r1
>=media-libs/alsa-lib-1.0.27-r3
>=media-libs/flac-1.2.1-r5
>=media-libs/libmikmod-3.2.0-r1
>=media-libs/libmodplug-0.8.8.4-r1
>=media-libs/libogg-1.3.0-r1
>=media-libs/libsamplerate-0.1.8-r1
>=media-libs/libsndfile-1.0.25-r1
>=media-libs/libv4l-0.9.5-r1
>=media-libs/libvorbis-1.3.3-r1
>=media-libs/webrtc-audio-processing-0.1-r1
>=media-sound/portaudio-19_pre20111121-r1
>=media-sound/twolame-0.3.13-r1
>=media-sound/wavpack-4.60.1-r1
>=media-sound/gsm-1.0.13-r1
>=media-libs/xvid-1.3.2-r1



emul-linux could install its libraries to /opt (they're prebuilt binaries after all) such as /opt/${PN} and install an env.d file containing:
LDPATH=/opt/${PN}/usr/lib32

then, for instance for {media,sound}libs, only wine needs to build against them so could use a temporary append-ldflags -L/opt/... if needed.

That way we could unmask the multilib-ready packages, hurry on getting packages needed by wine to be multilib-ready to drop the -L hack for example.

Any thoughts on this?
Comment 1 Alexandre Rostovtsev (RETIRED) gentoo-dev 2013-06-19 16:20:58 UTC
(In reply to Alexis Ballier from comment #0)
> Any thoughts on this?

If I understood you correctly, you are proposing giving wine a dozen automagic dependencies. Things like RDEPEND="|| ( media-libs/flac[abi_x86_32] app-emulation/emul-linux-x86-soundlibs )" etc, and then using an append-ldflags hack so that /opt/emul-linux-x86-soundlibs/usr/lib32/libFLAC.so gets picked up when /usr/lib32/libFLAC.so is not available.

That sounds like a bad idea unless you can guarantee mutual ABI compatibility for the two libraries.

Counterproposal: when a bunch of libraries are ready to go multilib in ~arch, simply release a new revision of emul-linux-x86-*libs that doesn't install them, adjust ~arch wine's dependencies, and if needed, ensure that wine gets rebuilt using subslot operators.
Comment 2 Alexis Ballier gentoo-dev 2013-06-19 17:02:23 UTC
(In reply to Alexandre Rostovtsev from comment #1)
> (In reply to Alexis Ballier from comment #0)
> > Any thoughts on this?
> 
> If I understood you correctly, you are proposing giving wine a dozen
> automagic dependencies. Things like RDEPEND="|| (
> media-libs/flac[abi_x86_32] app-emulation/emul-linux-x86-soundlibs )" etc,
> and then using an append-ldflags hack so that
> /opt/emul-linux-x86-soundlibs/usr/lib32/libFLAC.so gets picked up when
> /usr/lib32/libFLAC.so is not available.

The append-ldflags is just for linking, the env.d file takes care of this at runtime but that's the idea yes. If the deps are in a || then it's not automagic btw :)

> That sounds like a bad idea unless you can guarantee mutual ABI
> compatibility for the two libraries.

We're already ***ked with this if that's not the case: It'll use system headers (ie the ABI/API from the ebuild, not emul-linux) but link to the lib from emul-linux. It'll even get the wrong soname recorded in DT_NEEDED with emul-linux.

> Counterproposal: when a bunch of libraries are ready to go multilib in
> ~arch, simply release a new revision of emul-linux-x86-*libs that doesn't
> install them, adjust ~arch wine's dependencies, and if needed, ensure that
> wine gets rebuilt using subslot operators.

Be my guest then :=) I'm not maintaining emul-linux package; the pasted mask from comment #0 is 3 months old and growing. comment #0 is mainly for getting this done more smoothly.
Comment 3 Alexandre Rostovtsev (RETIRED) gentoo-dev 2013-06-19 18:01:30 UTC
(In reply to Alexis Ballier from comment #2)
> We're already ***ked with this if that's not the case: It'll use system
> headers (ie the ABI/API from the ebuild, not emul-linux) but link to the lib
> from emul-linux. It'll even get the wrong soname recorded in DT_NEEDED with
> emul-linux.

At least for dbus and glib, the header issue is addressed by using pkg-config and ABI-specific include paths (/usr/lib64/glib-2.0/include vs. /usr/lib32/glib-2.0/include)
Comment 4 Alexis Ballier gentoo-dev 2013-06-19 18:06:20 UTC
(In reply to Alexandre Rostovtsev from comment #3)

this is another issue: I'm talking about a header which is ABI-agnostic and in /usr/include but is made for libfoo.so.1 while emul-linux provides libfoo.so.0
Comment 5 Alexis Ballier gentoo-dev 2013-06-19 18:07:14 UTC
(In reply to Alexis Ballier from comment #4)
> (In reply to Alexandre Rostovtsev from comment #3)
> 
> this is another issue: I'm talking about a header which is ABI-agnostic and
> in /usr/include but is made for libfoo.so.1 while emul-linux provides
> libfoo.so.0

which was your question about ABI compatibility btw
Comment 6 Alexandre Rostovtsev (RETIRED) gentoo-dev 2013-06-19 18:17:13 UTC
Alternative possibility: how realistic is it to replace *all* of e.g. emul-linux-x86-soundlibs in one move in the observable future? Or are there specific libraries that are difficult to multilibize and would need to wait longer?

(In reply to Alexis Ballier from comment #4)
> this is another issue

Right. I realized that immediately after hitting "save changes" :/
Comment 7 Alexis Ballier gentoo-dev 2013-06-19 18:35:29 UTC
(In reply to Alexandre Rostovtsev from comment #6)
> Alternative possibility: how realistic is it to replace *all* of e.g.
> emul-linux-x86-soundlibs in one move in the observable future? Or are there
> specific libraries that are difficult to multilibize and would need to wait
> longer?

a couple of them need stuff from baselibs (zlib,bzip2,etc.) see bug #457134 for only the first of them; it might take long since those are usually not easy to multilibize
Comment 8 Pacho Ramos gentoo-dev 2013-06-20 06:38:15 UTC
(In reply to Alexis Ballier from comment #0)
> This list is growing and growing:
> 
> # Samuli Suominen <ssuominen@gentoo.org> (16 Mar 2013)
> # Waiting for new emul-linux-x86-{media,sound}libs without
> # the duplicate files.

I think the easiest think to do is to revbump affected emul packages removing colliding files and adding a dep on new multilib ebuilds. But would be nice to wait a bit (if a full emul set can be moved, it would be great) to not need to have thousands of revbumps to drop each lib.

What is blocking full soundlibs set migration? Couldn't it be done without waiting for moving baselibs (like I think has been done other times)?
Comment 9 Pacho Ramos gentoo-dev 2013-06-20 06:47:13 UTC
think -> thing :P
Comment 10 Alexis Ballier gentoo-dev 2013-06-20 11:45:50 UTC
(In reply to Pacho Ramos from comment #8)
> Couldn't it be done without
> waiting for moving baselibs (like I think has been done other times)?

If you want to do it properly, not have to go back later and fix these ebuilds if you happen to remember them, and not have a broken tree when adding support for a new multilib arch: No.
Comment 11 Alexis Ballier gentoo-dev 2013-06-20 11:53:06 UTC
(In reply to Pacho Ramos from comment #8)
> (In reply to Alexis Ballier from comment #0)
> > This list is growing and growing:
> > 
> > # Samuli Suominen <ssuominen@gentoo.org> (16 Mar 2013)
> > # Waiting for new emul-linux-x86-{media,sound}libs without
> > # the duplicate files.
> 
> I think the easiest think to do is to revbump affected emul packages
> removing colliding files and adding a dep on new multilib ebuilds. But would
> be nice to wait a bit (if a full emul set can be moved, it would be great)
> to not need to have thousands of revbumps to drop each lib.

Oh, and the current bug was precisely to avoid to do this, since, 3 months later it became clear it's gonna be long: That way multilib ebuilds can be properly tested in ~arch instead of unmasking them all at once and see what happens.

Also, since this is a matter of months and not days, this is starting to be annoying for maintainers: What to do when a new version is out? Don't bump it? Remove multilib? Make two revisions? That'd be cool to avoid burdening maintainers like that too.
Comment 12 Pacho Ramos gentoo-dev 2013-06-21 06:40:30 UTC
What I think you can do is to group your new multilib additions a bit, for example, when you add two or three new multilib packages, revbump emul set removing colliding files from it
Comment 13 Pacho Ramos gentoo-dev 2013-06-21 06:56:34 UTC
(In reply to Pacho Ramos from comment #12)
> What I think you can do is to group your new multilib additions a bit, for
> example, when you add two or three new multilib packages, revbump emul set
> removing colliding files from it

And you can, of course, commit that kind of changes for testing version of the emul set without needing to ping me every time
Comment 14 Alexis Ballier gentoo-dev 2013-06-21 14:09:55 UTC
(In reply to Pacho Ramos from comment #13)

this seems good to me but the way I see it is to remove colliding files from a new emul package (whatever is the proper way to do it, it doesnt matter for now) and then either:
- make the emul package depend on the relevant libs[abi_x86_32]
- fix the revdeps of said emul package to || dep on  ( libfoo[abi_x86_32] <emul-package-without-libs )

either way, this means that, in order to work properly, abi_x86_32 should be enabled globally by default, which is not the case atm AFAIK.

I would expect protests against enabling it by default for now, esp. if done that way: we would be moving hard-masked stuff straight to default enabled in ~arch. IMHO it'd be better to have a sort of transition/testing period in ~arch before, that'd justify "it has been widely tested and can be enabled by default".
Installing emul libs to /opt just seemed a nice way to achieve that.
Comment 15 Pacho Ramos gentoo-dev 2013-06-21 15:42:55 UTC
(In reply to Alexis Ballier from comment #14)
> (In reply to Pacho Ramos from comment #13)
> 
> this seems good to me but the way I see it is to remove colliding files from
> a new emul package (whatever is the proper way to do it, it doesnt matter
> for now) and then either:
> - make the emul package depend on the relevant libs[abi_x86_32]

That would make sense indeed :)

> - fix the revdeps of said emul package to || dep on  ( libfoo[abi_x86_32]
> <emul-package-without-libs )
>

I think this is the way it's being handled now, for example, I think this is the case for xlibs rdeps
 
> either way, this means that, in order to work properly, abi_x86_32 should be
> enabled globally by default, which is not the case atm AFAIK.
> 

Since people will usually don't require all libs with abi_x86_32 but only the minimum required to run some specific app (like acroread for example), I would simply let portage autounmask feature to get USE enabled only in what really needed
Comment 16 Alexis Ballier gentoo-dev 2013-06-21 15:47:11 UTC
(In reply to Pacho Ramos from comment #15)
> Since people will usually don't require all libs with abi_x86_32 but only
> the minimum required to run some specific app (like acroread for example), I
> would simply let portage autounmask feature to get USE enabled only in what
> really needed

last I tried this still required manual intervention and at a first shoot emerge will bail out telling you package has unsatisfied deps, no? this is not a very good user experience
Comment 17 Alexis Ballier gentoo-dev 2013-06-21 15:54:52 UTC
(In reply to Alexis Ballier from comment #16)
> (In reply to Pacho Ramos from comment #15)
> > Since people will usually don't require all libs with abi_x86_32 but only
> > the minimum required to run some specific app (like acroread for example), I
> > would simply let portage autounmask feature to get USE enabled only in what
> > really needed
> 
> last I tried this still required manual intervention and at a first shoot
> emerge will bail out telling you package has unsatisfied deps, no? this is
> not a very good user experience

note: this is what happens with xlibs atm if you unmask 20130224-r1:
[...]
[ebuild     U #] app-emulation/emul-linux-x86-xlibs-20130224-r1 [20130224] USE="(development) opengl" 0 kB
[blocks b      ] <=app-emulation/emul-linux-x86-xlibs-20130224 ("<=app-emulation/emul-linux-x86-xlibs-20130224" is blocking x11-libs/libXdmcp-1.1.1-r1, x11-proto/renderproto-0.11.1-r1, x11-libs/libXcomposite-0.4.4-r1, x11-libs/libvdpau-0.6, x11-libs/libXvMC-1.0.8, x11-proto/xineramaproto-1.2.1-r1, x11-libs/libXpm-3.5.10-r1, media-libs/fontconfig-2.10.2-r1, x11-proto/xproto-7.0.24, x11-proto/inputproto-2.3, x11-libs/libXdamage-1.1.4-r1, x11-libs/libX11-1.6.0, x11-proto/xf86bigfontproto-1.2.0-r1, x11-libs/libXaw-1.0.11-r2, x11-libs/libXinerama-1.1.3, x11-libs/libICE-1.0.8-r1, x11-libs/libXext-1.3.2, x11-libs/libXrandr-1.4.1, x11-libs/libXtst-1.2.2, x11-libs/libXp-1.0.2, x11-libs/libXt-1.1.4, x11-libs/libXmu-1.1.1-r1, x11-proto/damageproto-1.2.1-r1, x11-proto/recordproto-1.14.2-r1, x11-proto/scrnsaverproto-1.2.2-r1, x11-libs/libXScrnSaver-1.2.2-r1, dev-libs/libpthread-stubs-0.3-r1, x11-libs/libXfixes-5.0.1, x11-proto/xf86vidmodeproto-2.3.1-r1, x11-libs/libxcb-1.9.1, x11-libs/libSM-1.2.1-r1, x11-libs/libXrender-0.9.8, x11-proto/kbproto-1.0.6-r1, x11-proto/compositeproto-0.4.2-r1, x11-proto/xcb-proto-1.8-r3, x11-libs/libXv-1.0.8, x11-proto/randrproto-1.4.0-r1, x11-proto/xf86dgaproto-2.1-r2, x11-proto/fixesproto-5.0-r1, x11-libs/libXi-1.7.1, x11-libs/libXxf86vm-1.1.3, x11-proto/printproto-1.0.5-r1, x11-libs/libXau-1.0.8, x11-libs/libpciaccess-0.13.1-r1, x11-libs/libXcursor-1.1.13-r1, media-libs/freetype-2.4.12, x11-proto/xextproto-7.2.1-r1, x11-libs/libXxf86dga-1.1.4, x11-proto/videoproto-2.3.1-r1, x11-libs/libXft-2.3.1-r1)

Total: 52 packages (1 upgrade, 3 new, 48 reinstalls), Size of downloads: 572 kB
Conflict: 1 block

The following USE changes are necessary to proceed:
 (see "package.use" in the portage(5) man page for more details)
# required by x11-libs/libXinerama-1.1.3
# required by app-emulation/emul-linux-x86-xlibs-20130224-r1
[... tons of usechanges needed...]



and this is precisely what I'm trying to avoid.
Comment 18 Pacho Ramos gentoo-dev 2013-06-21 15:56:32 UTC
What does portage when you simply mask all versions of already migrated package? In that situation, portage should simply try to respect rdep relying on new multilib migrated packages
Comment 19 Alexis Ballier gentoo-dev 2013-06-21 16:04:22 UTC
(In reply to Pacho Ramos from comment #18)
> What does portage when you simply mask all versions of already migrated
> package? In that situation, portage should simply try to respect rdep
> relying on new multilib migrated packages

again, if abi_x86_32 were enabled by default this wouldnt be a problem and portage would use the new multilib migrated packages: 'ABI_X86="32 64" emerge -uDNa world' works fine. If you want to propose to enable this by default, then this has to go through -dev ml. I suppose people will oppose to this and they will be right because so far most of it has stayed in pmask. Again, having emul-libs in /opt allows giving it a wide ~arch testing.
Comment 20 Pacho Ramos gentoo-dev 2013-06-23 08:15:47 UTC
Will CC portage team as they will know much more than me about dependencies resolution. My question is, once rdeps are migrated to something like:
               || (
                        (
                                media-libs/fontconfig[abi_x86_32]
                                x11-libs/libICE[abi_x86_32]
                                x11-libs/libSM[abi_x86_32]
                                x11-libs/libXcomposite[abi_x86_32]
                                x11-libs/libXcursor[abi_x86_32]
                                x11-libs/libXdamage[abi_x86_32]
                                x11-libs/libXext[abi_x86_32]
                                x11-libs/libXfixes[abi_x86_32]
                                x11-libs/libXft[abi_x86_32]
                                x11-libs/libXi[abi_x86_32]
                                x11-libs/libXrandr[abi_x86_32]
                                x11-libs/libXrender[abi_x86_32]
                                x11-libs/libXxf86vm[abi_x86_32]
                        )
                        app-emulation/emul-linux-x86-xlibs
                )

What will occur when the package goes to stable, the libs providing abi_x86_32 too, *AND* app-emulation/emul-linux-x86-xlibs is hardmasked (even keeping abi_x86_32 default off globally)?
Comment 21 Zac Medico gentoo-dev 2013-06-24 00:00:00 UTC
(In reply to Pacho Ramos from comment #20)
> What will occur when the package goes to stable, the libs providing
> abi_x86_32 too, *AND* app-emulation/emul-linux-x86-xlibs is hardmasked (even
> keeping abi_x86_32 default off globally)?

If emul-linux-x86-xlibs is hardmasked, then portage should try to pull in the foo[abi_x86_32] packages as replacements. If you have --autounmask enabled (default), then it will generate package.use settings for you. Otherwise, it will just bail out due to the unsatisfied foo[abi_x86_32] deps.
Comment 22 Alexis Ballier gentoo-dev 2013-06-24 14:51:07 UTC
ok, so it seems I was wrong on some parts: I thought emul-linux-xlibs deps were all migrated to these || and I couldn't enable abi_x86_32 without unmasking emul-linux-xlibs. It was due to emul-linux-{opengl,gtklibs} not having been migrated. (This should be done IMHO otherwise abi_x86_32 is kind of useless with a pure ~arch system. Tweaking these deps allow me to enable abi_x86_32 and portages solves it correctly.)

So, the migration that had been proposed some time ago will work:
- emul-linux reverse deps get their deps changed to || ( lib[abi_x86_32] emul-linux )
- multilib enabled packages can be unmasked safely


Then comes the problem of a partial conversion of an emul-linux bundle: either way there will be a problem.

Let's say I convert libfoo:
- If I remove libfoo from the emul-linux-fooers bundle then either the packages needing libfoo[abi_x86_32] or/and emul-linux-fooers have to depend on libfoo[abi_x86_32]. Since abi_x86_32 is not enabled by default, those packages will be uninstallable by default (autounmask is not that great)
- If I leave libfoo in emul-linux-fooers (with a proper blocker) and convert packages needing libfoo[abi_x86_32] to a proper || dep, then libbar, which is part of emul-linux-fooers, is not yet converted and thus packages needing libbar[abi_x86_32] will have to depend on emul-linux-fooers. This will make libfoo[abi_x86_32] uninstallable on systems that need libbar[abi_x86_32]. Not that great either.
Comment 23 Pacho Ramos gentoo-dev 2013-06-24 18:41:02 UTC
(In reply to Alexis Ballier from comment #22)
[...]
> Let's say I convert libfoo:
> - If I remove libfoo from the emul-linux-fooers bundle then either the
> packages needing libfoo[abi_x86_32] or/and emul-linux-fooers have to depend
> on libfoo[abi_x86_32]. Since abi_x86_32 is not enabled by default, those
> packages will be uninstallable by default (autounmask is not that great)

I would still rely on autounmask, doesn't it work in that case? This would be like any other USE requirement. On the other hand, we can default enable abi_x86_32 on libs that are usually required to have multilib enabled in most setups (I think it's possible... but I am not fully sure :S), while keeping other libs without multilib enabled by default as they are more ready for corner cases.

Other option could be to have a profile with it enabled by default for people willing to have all available libs installed for 32 bits also... or ask them to enable it in their make.conf if not willing to let autounmask do its job 

> - If I leave libfoo in emul-linux-fooers (with a proper blocker) and convert
> packages needing libfoo[abi_x86_32] to a proper || dep, then libbar, which
> is part of emul-linux-fooers, is not yet converted and thus packages needing
> libbar[abi_x86_32] will have to depend on emul-linux-fooers. This will make
> libfoo[abi_x86_32] uninstallable on systems that need libbar[abi_x86_32].
> Not that great either.

We need to drop libfoo as soon as it's migrated (at least in the same keywording level) -> emul packages will die as soon as possible
Comment 24 Zac Medico gentoo-dev 2013-06-24 18:52:11 UTC
(In reply to Pacho Ramos from comment #23)
> I would still rely on autounmask, doesn't it work in that case? This would
> be like any other USE requirement.

For deps like your example in comment #20, portage will choose emul-linux-x86-xlibs if abi_x86_32 is disabled for the other packages, unless emul-linux-x86-xlibs is hardmasked. If emul-linux-x86-xlibs is hardmasked, then portage will choose foo[abi_x86_32] and that will trigger autounmask.
Comment 25 Alexis Ballier gentoo-dev 2013-06-24 18:56:15 UTC
(In reply to Pacho Ramos from comment #23)
> (In reply to Alexis Ballier from comment #22)
> [...]
> > Let's say I convert libfoo:
> > - If I remove libfoo from the emul-linux-fooers bundle then either the
> > packages needing libfoo[abi_x86_32] or/and emul-linux-fooers have to depend
> > on libfoo[abi_x86_32]. Since abi_x86_32 is not enabled by default, those
> > packages will be uninstallable by default (autounmask is not that great)
> 
> I would still rely on autounmask, doesn't it work in that case? This would
> be like any other USE requirement.

Please try it and tell me what you think about it.

# echo "app-emulation/emul-linux-x86-xlibs" > /etc/portage/package.unmask/emulxlibs
# emerge  app-emulation/emul-linux-x86-xlibs
[...tens of required use / blocker lines like...]
# required by media-libs/fontconfig-2.10.2-r1
# required by app-emulation/emul-linux-x86-xlibs-20130224-r1
# required by app-emulation/emul-linux-x86-xlibs (argument)
>=media-libs/freetype-2.4.12 abi_x86_32

Use --autounmask-write to write changes to config files (honoring
CONFIG_PROTECT). Carefully examine the list of proposed changes,
paying special attention to mask or keyword changes that may expose
experimental or unstable packages.

# emerge  app-emulation/emul-linux-x86-xlibs --autounmask-write
...
# dispatch-conf
...
# emerge  app-emulation/emul-linux-x86-xlibs
it works at least!

this is certainly not what I call a nice user experience, esp. when there _IS_ a way to avoid this for now.

> On the other hand, we can default enable
> abi_x86_32 on libs that are usually required to have multilib enabled in
> most setups (I think it's possible... but I am not fully sure :S), while
> keeping other libs without multilib enabled by default as they are more
> ready for corner cases.

As far as I know, most, if not all, multilib based ebuilds have some multilib consumers.

> Other option could be to have a profile with it enabled by default for
> people willing to have all available libs installed for 32 bits also... or
> ask them to enable it in their make.conf if not willing to let autounmask do
> its job 

This doesn't solve the problem of uninstallable packages for people using the other profile.
Comment 26 Pacho Ramos gentoo-dev 2013-06-24 19:58:22 UTC
(In reply to Alexis Ballier from comment #25)
[...]
> # emerge  app-emulation/emul-linux-x86-xlibs --autounmask-write
> ...
> # dispatch-conf
> ...
> # emerge  app-emulation/emul-linux-x86-xlibs
> it works at least!

Isn't emul-linux-x86-xlibs completely migrated yet? In that case, don't use emul-linux-x86-xlibs at all, simply mask it completely and force rdeps to pull only only the libs their need with 32 bits support


> > On the other hand, we can default enable
> > abi_x86_32 on libs that are usually required to have multilib enabled in
> > most setups (I think it's possible... but I am not fully sure :S), while
> > keeping other libs without multilib enabled by default as they are more
> > ready for corner cases.
> 
> As far as I know, most, if not all, multilib based ebuilds have some
> multilib consumers.
> 

But not all libs needs to have 32 bits support enabled (I think also in portage-multilib approach, it wasn't installing 32 bits libs for *all*), for example, I think I wont ever need jni 32 bits in my lib, while xlibs stuff probably yes.

> > Other option could be to have a profile with it enabled by default for
> > people willing to have all available libs installed for 32 bits also... or
> > ask them to enable it in their make.conf if not willing to let autounmask do
> > its job 
> 
> This doesn't solve the problem of uninstallable packages for people using
> the other profile.

They are installable, people simply need to toggle USE flag on either letting autounmask to do that job or enabling it globally (like occurs when you need to enable a USE flag required by other package)
Comment 27 Pacho Ramos gentoo-dev 2013-06-24 20:02:14 UTC
Anyway, I don't fully understand how enabling or not abi_x86_32 globally has anything to do here -> that problem is more general and would affect us even when all emul sets have been removed :/
Comment 28 Alexis Ballier gentoo-dev 2013-06-24 20:14:18 UTC
I think you don't understand my concerns here: I _do_ want to avoid relying on autounmask because autounmask requires manual intervention and portage's (sane) bahavior is to bail out when usedeps are not satisfied: package is uninstallable by default.

Since you recently migrated zlib/bzip2, why did you keep them masked?
Proper testing could be done in ~arch. The reality, as I see it, is that zlib[abi_x86_32] is uninstallable on most multilib installs because it collides with emul-linux-baselibs which is required by most 32bits packages.
Comment 29 Pacho Ramos gentoo-dev 2013-06-24 20:27:38 UTC
(In reply to Alexis Ballier from comment #28)
> I think you don't understand my concerns here

Probably

>: I _do_ want to avoid relying
> on autounmask because autounmask requires manual intervention and portage's
> (sane) bahavior is to bail out when usedeps are not satisfied: package is
> uninstallable by default.
> 
> Since you recently migrated zlib/bzip2, why did you keep them masked?

Because mgorny explicetly demanded it to me via IRC

> Proper testing could be done in ~arch. The reality, as I see it, is that
> zlib[abi_x86_32] is uninstallable on most multilib installs because it
> collides with emul-linux-baselibs which is required by most 32bits packages.

Because we need a revbump of emul-linux-baselibs removing colliding files from it (like needs to be done in medialibs/soundlibs...)
Comment 30 Alexis Ballier gentoo-dev 2013-06-24 20:45:29 UTC
(In reply to Pacho Ramos from comment #29)
> > Proper testing could be done in ~arch. The reality, as I see it, is that
> > zlib[abi_x86_32] is uninstallable on most multilib installs because it
> > collides with emul-linux-baselibs which is required by most 32bits packages.
> 
> Because we need a revbump of emul-linux-baselibs removing colliding files
> from it (like needs to be done in medialibs/soundlibs...)

...and then you hit bug #461608
Comment 31 Pacho Ramos gentoo-dev 2013-06-24 21:07:20 UTC
But, how do you get blockers between:
app-emulation/emul-linux-x86-xlibs-20130224 (you don't need -r1 since it's fully migrated)
- Xorg libs migrated to native multilib
Comment 32 Alexis Ballier gentoo-dev 2013-06-24 22:53:24 UTC
(In reply to Pacho Ramos from comment #31)
> But, how do you get blockers between:
> app-emulation/emul-linux-x86-xlibs-20130224 (you don't need -r1 since it's
> fully migrated)
> - Xorg libs migrated to native multilib

see comment #22

we're really going nowhere with this bug. it is a chicken and egg problem and we're turning round. the proposal from comment #0 was only meant to break that chicken-egg problem.

I still strongly believe relying on autounmask (which is the solution you've been advocating and what it implies to remove stuff from emul-libs) is a very poor way of enabling abi_x86_32 by default on multilib profiles (that's the reason why migrated emul-linux-xlibs is still masked i think)

i'm bored of arguing about this, i'll start unmasking packages i maintain as i see fit; 3 months is more than enough for "testing" and some of them have had version bumps (see bug #472950 )
Comment 33 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2013-06-26 08:59:44 UTC
I have no real opinion about this. It could work, it may help somehow but it feels like a QA violation. On the other hand, I don't think it could break anything that wasn't broken already.

I feel like we're on a good way to do this properly. We started migrating baselibs. Now it's all a matter of time and work. But I don't think we have enough people to do all the work, so the interim solution with emul-linux co-installable with multilib libs sounds acceptable.

Just please think about it three times and think about all possible problems. If we're going to go that way, we need to be prepared for least expected issues. For example, 32-bit libraries which load plugins will need to be hacked to use alternate plugin location.

@Alexis, I can understand your irritation but please don't do something that would result in our users hitting unsolvable blockers.
Comment 34 Alexis Ballier gentoo-dev 2013-06-26 12:28:33 UTC
(In reply to Michał Górny from comment #33)
> @Alexis, I can understand your irritation but please don't do something that
> would result in our users hitting unsolvable blockers.

unmasking all these media-libs do not make the blockers more unsolvable than for xlibs, that are unmasked, on a pure ~arch system.
Comment 35 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2013-06-26 13:03:05 UTC
(In reply to Alexis Ballier from comment #34)
> (In reply to Michał Górny from comment #33)
> > @Alexis, I can understand your irritation but please don't do something that
> > would result in our users hitting unsolvable blockers.
> 
> unmasking all these media-libs do not make the blockers more unsolvable than
> for xlibs, that are unmasked, on a pure ~arch system.

Unless I'm missing something, there's a big difference of xlibs being converted completely and medialibs only partially. So unless we have a new release of emul-linux (which would end up requiring multilib packages anyway), user's going to hit an unsolvable blocker using ABI_X86=32 and a package requiring emul-linux.

Autounmask in this case will disable abi_x86_32 per-package which isn't anywhere near good ideas IMO.
Comment 36 Alexandre Rostovtsev (RETIRED) gentoo-dev 2013-06-26 13:12:21 UTC
Maybe we should create some new amd64 profiles that have ABI_X86=32 set globally?
Comment 37 Alexis Ballier gentoo-dev 2013-06-26 13:16:56 UTC
(In reply to Michał Górny from comment #35)
> > unmasking all these media-libs do not make the blockers more unsolvable than
> > for xlibs, that are unmasked, on a pure ~arch system.
> 
> Unless I'm missing something, there's a big difference of xlibs being
> converted completely and medialibs only partially. 

yes but e.g. emul-linux-opengl requires -xlibs. anyone needing it will have unsolvable deps.

> So unless we have a new
> release of emul-linux (which would end up requiring multilib packages
> anyway), user's going to hit an unsolvable blocker using ABI_X86=32 and a
> package requiring emul-linux.

you can also revbump emul-linux-medialibs, remove stuff that have been migrated in src_unpack and make it depend on the correct [abi_x86_32] package. It'd take you 5 mins. I have not done this because I do not believe this is the right solution.
Comment 38 Alexis Ballier gentoo-dev 2013-06-26 15:23:45 UTC
another way much satisfactory solution from a maintainer pov is to p.u.mask abi_x86_32 for partial updates.
Comment 39 Alexis Ballier gentoo-dev 2013-06-26 15:52:27 UTC
other proposal that seems to have the advantages of pacho's proposal without its problems: add abi_x86_32 useflag to emul-libs; depend on migrated packages and remove relevant files when enabled, otherwise keep as is.

Example:

--- emul-linux-x86-medialibs-20130224.ebuild	2013-03-16 13:56:48.000000000 -0300
+++ emul-linux-x86-medialibs-20130224-r1.ebuild	2013-06-26 11:50:11.626658434 -0400
@@ -6,7 +6,8 @@
 inherit emul-linux-x86
 
 LICENSE="APL-1.0 GPL-2 BSD BSD-2 public-domain LGPL-2 MPL-1.1 LGPL-2.1 MPEG-4"
-KEYWORDS="-* amd64"
+KEYWORDS="-* ~amd64"
+IUSE="abi_x86_32"
 
 DEPEND=""
 RDEPEND="~app-emulation/emul-linux-x86-baselibs-${PV}
@@ -14,11 +15,22 @@
 	~app-emulation/emul-linux-x86-db-${PV}
 	!<=app-emulation/emul-linux-x86-sdl-20081109
 	!<=app-emulation/emul-linux-x86-soundlibs-20110101
-	!>=media-libs/libv4l-0.8.9-r1[abi_x86_32]"
+	!>=media-libs/libv4l-0.8.9-r1[abi_x86_32]
+	abi_x86_32? (
+		>=media-sound/lame-3.99.5-r1[abi_x86_32(-)]
+		>=media-libs/libvpx-1.2.0_pre[abi_x86_32(-)]
+	)
+	"
 PDEPEND="~app-emulation/emul-linux-x86-soundlibs-${PV}"
 
 src_prepare() {
 	# Include all libv4l libs, bug #348277
 	ALLOWED="${S}/usr/lib32/libv4l/"
 	emul-linux-x86_src_prepare
+
+	# Remove migrated stuff.
+	if use abi_x86_32 ; then
+		rm -f usr/lib32/libmp3lame.* #media-sound/lame
+		rm -f usr/lib32/libvpx.* usr/lib32/pkgconfig/vpx.pc #media-libs/libvpx
+	fi
 }
Comment 40 Pacho Ramos gentoo-dev 2013-06-26 16:29:06 UTC
(In reply to Alexis Ballier from comment #39)
> other proposal that seems to have the advantages of pacho's proposal without
> its problems: add abi_x86_32 useflag to emul-libs; depend on migrated
> packages and remove relevant files when enabled, otherwise keep as is.
[...]

Looks interesting... only one suggestion:

>  src_prepare() {
>  	# Include all libv4l libs, bug #348277
>  	ALLOWED="${S}/usr/lib32/libv4l/"
>  	emul-linux-x86_src_prepare
> +
> +	# Remove migrated stuff.
> +	if use abi_x86_32 ; then
> +		rm -f usr/lib32/libmp3lame.* #media-sound/lame
> +		rm -f usr/lib32/libvpx.* usr/lib32/pkgconfig/vpx.pc #media-libs/libvpx
> +	fi
>  }

As the list of libs will grow a lot when more and more packages are migrated, I would instead create a file in FILESDIR containing files to remove (instead of making ebuild grow)
Comment 41 Alexis Ballier gentoo-dev 2013-06-26 16:46:47 UTC
(In reply to Pacho Ramos from comment #40)
> >  src_prepare() {
> >  	# Include all libv4l libs, bug #348277
> >  	ALLOWED="${S}/usr/lib32/libv4l/"
> >  	emul-linux-x86_src_prepare
> > +
> > +	# Remove migrated stuff.
> > +	if use abi_x86_32 ; then
> > +		rm -f usr/lib32/libmp3lame.* #media-sound/lame
> > +		rm -f usr/lib32/libvpx.* usr/lib32/pkgconfig/vpx.pc #media-libs/libvpx
> > +	fi
> >  }
> 
> As the list of libs will grow a lot when more and more packages are
> migrated, I would instead create a file in FILESDIR containing files to
> remove (instead of making ebuild grow)

why not, but how would you do this?
Comment 42 Pacho Ramos gentoo-dev 2013-06-26 16:56:35 UTC
the idea would be to list colliding files in "${FILESDIR}/remove-native" (or the name you prefer) and run something like "rm -f $(cat ${FILESDIR}/remove-native)"
Comment 43 Alexis Ballier gentoo-dev 2013-06-26 17:00:41 UTC
(In reply to Pacho Ramos from comment #42)
> the idea would be to list colliding files in "${FILESDIR}/remove-native" (or
> the name you prefer) and run something like "rm -f $(cat
> ${FILESDIR}/remove-native)"

okey, will do that way and commit (+ unmask xvid + update blockers to [-abi_x86_32])
Comment 44 Pacho Ramos gentoo-dev 2013-06-26 17:11:29 UTC
(In reply to Alexis Ballier from comment #43)
> (In reply to Pacho Ramos from comment #42)
> > the idea would be to list colliding files in "${FILESDIR}/remove-native" (or
> > the name you prefer) and run something like "rm -f $(cat
> > ${FILESDIR}/remove-native)"
> 
> okey, will do that way and commit (+ unmask xvid + update blockers to
> [-abi_x86_32])

That is fine from my part, also, any one is free to modify that files to remove their desired libs when needed
Comment 45 Alexis Ballier gentoo-dev 2013-06-26 17:24:26 UTC
should be all set up now :)
Comment 46 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2013-07-13 21:33:52 UTC
This all looks neat so far but I have one important remark: we need to make sure that all blockers are in place and working. Especially that they allow both collision-free upgrades and downgrades.

We also need to work on avoiding tiny revision mismatches. That is, if libfoo is removed from emul-linux in -r3, its multilib ebuild needs to block <=-r2[abi_x86_32] as well.
Comment 47 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2013-07-13 22:15:26 UTC
And to clarify my previous thought: I've just added the following blocker to libv4l:

+	abi_x86_32? ( !<=app-emulation/emul-linux-x86-medialibs-20130224-r2
+		!app-emulation/emul-linux-x86-medialibs[-abi_x86_32(-)] )"

so it needs both to block all emul-linux without ABI_X86=32 and emul-linux old enough not to strip that library.
Comment 48 Alexis Ballier gentoo-dev 2013-07-14 23:56:59 UTC
(In reply to Michał Górny from comment #47)

this is what i've done in all unmasked multilib ebuilds from sound/medialibs i think