we've seen it breaks a lot of packages; pkg-config is not the solution to everything: there is a standard libdir and search path and there is a standard includedir and search path. Lots of packages rely on this and not the pkg-config pseudo-invention :) For multilib, this is indeed a problem to have abi specific headers in /usr/include. One solution I had seen in the past (but cannot find it anymore? maybe linux-headers?) would be to do, for a foo.h header that goes to dir $d but is abi-specific: mkdir $d/$abi install foo.h in $d/$abi get abi #ifdef directive for $abi (this should likely be done manually in a table): you can distinguish from x86, amd64 and x32 with: #ifdef __i386__ x86 stuff #endif #ifdef __x86_64__ #ifdef __ILP32__ x32 stuff #else amd64 stuff #endif #endif and here, $d/foo.h will be exactly the above code where '* stuff' is replaced by #include <$d/$abi/foo.h> that way, including the usual header will work and will include the correct abi specific header depending on the abi we are building for. I think multilib-portage does this, thus grabbing the code from them should be easier. multilib_check_headers() should be extended to honour a given variable like 'ABI_SPECIFIC_HEADERS' and do this wrapping for this list of files that ebuilds can define (or maybe detect it automatically) and do not check their checksum. For simplicity, you can likely put such a header template that you sed when installing it in the eclass dir ala ELT-patches.
some references for writing a nice header wrapper: http://sourceforge.net/p/predef/wiki/Architectures/
There was some discussion about it on the tracker bug. Long story short, Diego didn't like the idea at first and I'm waiting for his reply/explanation.
We used to have this for glibc, and I remember hitting at least a couple of packages at the time, that used headers to find information but did _not_ use the full standard pre-processor, which means that it didn't define _any_ of the arch-specific macros. Are they more or less than those we have to fix now? I have no idea, tbh, but at least this way we have a quick way to find the issue of the problem and fix it. I'm not keen to debug the other issue if it happens in front of me again.
(In reply to comment #3) > We used to have this for glibc, and I remember hitting at least a couple of > packages at the time, that used headers to find information but did _not_ > use the full standard pre-processor, which means that it didn't define _any_ > of the arch-specific macros. Do you remember any example ? calling cpp directly ? unifdef stuff ? > Are they more or less than those we have to fix now? I believe they are less since that is what multilib-portage is doing. OTOH I don't think the wrapper solution is a long term solution: for the freetype case they could very well switch to stdint.h (and maybe advertise a c99tomsvc processor ala libav/ffmpeg for building with msvc). That's why I prefer to ask people to declare what needs wrapping with a variable so that it is clear that it is discouraged (instead of doing it automagically ala multilib-portage). > I have no idea, tbh, > but at least this way we have a quick way to find the issue of the problem > and fix it. I'm not keen to debug the other issue if it happens in front of > me again. Breaking standard include paths that lots of packages rely on is IMHO a much bigger problem here. It is a 20+ years standard after all. Also, you can very well imagine a library installing a header #including freetype's headers but not providing a .pc file. In short: Not using pkg-config is *not* a bug!
(In reply to comment #3) > > Are they more or less than those we have to fix now? I have no idea, tbh, > but at least this way we have a quick way to find the issue of the problem > and fix it. I'm not keen to debug the other issue if it happens in front of > me again. Those issues are quick to find, but not quick to fix. E.g. cmake mostly ignores "Requires" from pkgconfig files. It is unlikely that cmake upstream will support our hacks, so if we are lucky they will add HINTS for their own search logic based on pkgconfig reported include dirs. But that's really all. We will still be left with bug 459862 and I doubt that's the only one of it's kind. append-libs and such stuff often refuses to work with cmake bases build systems too... And these issues will keep popping up randomly, because we are the only ones doing this. As for the header-wrapping solution I think that will be much less trouble, since we won't do that globally anyway if we integrate an eclass solution and afais multilib-portage does not have _massive_ problems like that with their solution, but tommy should rather comment on this.
(In reply to comment #3) > We used to have this for glibc, and I remember hitting at least a couple of > packages at the time, that used headers to find information but did _not_ > use the full standard pre-processor, which means that it didn't define _any_ > of the arch-specific macros. > > Are they more or less than those we have to fix now? I have no idea, tbh, > but at least this way we have a quick way to find the issue of the problem > and fix it. I'm not keen to debug the other issue if it happens in front of > me again. Initially, multilib-portage did wrap all headers (meaning moving the headers into an abi-specific subdir and placing a wrapper, which includes the real header per ABI). This behaviour did catch some packages directly accessing headers files. After improving the code to only wrap the headers, that differ per-abi, i have not yet seen any such report, so the number of packages to fix should be pretty low (excluding the core system with gcc and glibc, which have their own multilib setup not touched by multilib-portage). (In reply to comment #4) > (In reply to comment #3) > > We used to have this for glibc, and I remember hitting at least a couple of > > packages at the time, that used headers to find information but did _not_ > > use the full standard pre-processor, which means that it didn't define _any_ > > of the arch-specific macros. > > Do you remember any example ? calling cpp directly ? unifdef stuff ? > > > Are they more or less than those we have to fix now? > > I believe they are less since that is what multilib-portage is doing. > OTOH I don't think the wrapper solution is a long term solution: for the > freetype case they could very well switch to stdint.h (and maybe advertise a > c99tomsvc processor ala libav/ffmpeg for building with msvc). That's why I > prefer to ask people to declare what needs wrapping with a variable so that > it is clear that it is discouraged (instead of doing it automagically ala > multilib-portage). We should always diff the headers and then can do one of the following: a) always wrap headers, that differ per ABI b) by default wrap headers, that differ per ABI with OPT-OUT c) default die, when different headers per ABI are detected and do the wrapping with an OPTS-IN Since i already have 35 headers, that differ per ABI even on my small desktop system, i suggest going with a) or b) for now. Once the number has been reduced to a small set of packages, we can still talk about option c). As a reference and example, following is an example wrapper for the one header file of freetype, that differs per ABI: <quote> /* Autogenerated by portage FEATURE auto-multilib */ #ifdef __x86_64__ # include <gentoo-multilib/amd64/freetype2/freetype/config/ftconfig.h> #endif /* __x86_64__ */ #ifdef __i386__ # include <gentoo-multilib/x86/freetype2/freetype/config/ftconfig.h> #endif /* __i386__ */ </quote> If anyone wants to look into the code, it is in the multilib branch of the portage git repo inside bin/auto-multilib.sh in the _finalize_abi_install() function
I'm suggesting an intermediate solution: installing the headers into the usual place for native ABI, and into libdir for other ABIs. This will require the multilib packages to use pkg-config (as they are required now) without breaking non-multilib packages. AFAICS gcc prefers -I over the standard include paths so that should work. Still, I believe that the relevant bugs should be kept open until all upstreams are able to find freetype properly.
(In reply to comment #7) I clearly prefer the header wrapping solution: We do not need to have all the headers twice just because a couple of them differ between abis.
(In reply to comment #8) > (In reply to comment #7) > > I clearly prefer the header wrapping solution: We do not need to have all > the headers twice just because a couple of them differ between abis. You mean wrapping just the one header that needs it? Getting that to work gives me headache. The major point in having them all twice was to have them working with a simple change to ebuild until upstream fixes the specific macros.
(In reply to comment #9) > (In reply to comment #8) > > (In reply to comment #7) > > > > I clearly prefer the header wrapping solution: We do not need to have all > > the headers twice just because a couple of them differ between abis. > > You mean wrapping just the one header that needs it? Getting that to work > gives me headache. Yes - this seems the sanest and simplest solution to me. I can do it if you want but since this is quite a bit of work I want to be sure this is where we want to go first :)
(In reply to comment #9) > (In reply to comment #8) > > (In reply to comment #7) > > > > I clearly prefer the header wrapping solution: We do not need to have all > > the headers twice just because a couple of them differ between abis. > > You mean wrapping just the one header that needs it? Getting that to work > gives me headache. > > The major point in having them all twice was to have them working with a > simple change to ebuild until upstream fixes the specific macros. Huh? headachhe? How and why? You already have a function, that does a diff for the headers, so the only thing left is to wrap the headers that differ and to just install the rest as always. No need to have an additional copy of an identical file in a different location without any benefit. I dont expect you to have issues writing the needed lines for that, but if you have, you can always copy the code from multilib-portage, which has been doing this for a long time.
(In reply to comment #11) > (In reply to comment #9) > > (In reply to comment #8) > > > (In reply to comment #7) > > > > > > I clearly prefer the header wrapping solution: We do not need to have all > > > the headers twice just because a couple of them differ between abis. > > > > You mean wrapping just the one header that needs it? Getting that to work > > gives me headache. > > > > The major point in having them all twice was to have them working with a > > simple change to ebuild until upstream fixes the specific macros. > > Huh? headachhe? How and why? You already have a function, that does a diff > for the headers, so the only thing left is to wrap the headers that differ > and to just install the rest as always. No need to have an additional copy > of an identical file in a different location without any benefit. The checking function just does a checksum after each install run. It just throws an error since the headers were overwritten already. So we'd basically need to either: 1. install & rename for abi1, 2. install & rename for abi2, 3. ..., 4. write a wrapper header. which is a bit ugly, to be honest. Alternative is to install to dedicated locations and then move files. Which is hard to achieve and due to PMS stupidity, can't be done without all ebuilds explicitly being adjusted to do so.
(In reply to comment #12) > (In reply to comment #11) > > (In reply to comment #9) > > > (In reply to comment #8) > > > > (In reply to comment #7) > > > > > > > > I clearly prefer the header wrapping solution: We do not need to have all > > > > the headers twice just because a couple of them differ between abis. > > > > > > You mean wrapping just the one header that needs it? Getting that to work > > > gives me headache. > > > > > > The major point in having them all twice was to have them working with a > > > simple change to ebuild until upstream fixes the specific macros. > > > > Huh? headachhe? How and why? You already have a function, that does a diff > > for the headers, so the only thing left is to wrap the headers that differ > > and to just install the rest as always. No need to have an additional copy > > of an identical file in a different location without any benefit. > > The checking function just does a checksum after each install run. It just > throws an error since the headers were overwritten already. > > So we'd basically need to either: > 1. install & rename for abi1, > 2. install & rename for abi2, > 3. ..., > 4. write a wrapper header. > > which is a bit ugly, to be honest. I guess, you are just lazy, since you simply do the default ABI as last one letting src_install overwrite anything from other ABIs. If you do it properly with a different install location for each ABI (or moving the old away before installing the new one), you can simply do a diff between the 2 locations, save headers with differences into abi-specific locations and create a wrapper. And if you are lazy, you just copy the code from multilib-portage, i already wrote in comment 6, where to find that. > > Alternative is to install to dedicated locations and then move files. Which > is hard to achieve and due to PMS stupidity, can't be done without all > ebuilds explicitly being adjusted to do so. I dont see, how PMS does get in your way, in worst case, you just use subdirs inside the image dir (nothing prevents you from using a different DESTDIR variable, right?) and merge them after the install phase.
(In reply to comment #13) > (In reply to comment #12) > > The checking function just does a checksum after each install run. It just > > throws an error since the headers were overwritten already. > > > > So we'd basically need to either: > > 1. install & rename for abi1, > > 2. install & rename for abi2, > > 3. ..., > > 4. write a wrapper header. > > > > which is a bit ugly, to be honest. > > I guess, you are just lazy, since you simply do the default ABI as last one > letting src_install overwrite anything from other ABIs. If you do it > properly with a different install location for each ABI (or moving the old > away before installing the new one), you can simply do a diff between the 2 > locations, save headers with differences into abi-specific locations and > create a wrapper. I'd appreciate if you showed a little more respect or at least checked before saying something. You may not be aware of that since your whole project was done behind the back of PMS, but PMS doesn't allow us to override install location. Installing into a different location would therefore require every single ebuild to explicitly specify DESTDIR, and break multilib-minimal at least. And the major reason for not doing any magic like you're suggesting was to save user's time and space. There's no point in keeping N copies of the same files during install just so one corner case could be handled automagically and implicitly. > > Alternative is to install to dedicated locations and then move files. Which > > is hard to achieve and due to PMS stupidity, can't be done without all > > ebuilds explicitly being adjusted to do so. > > I dont see, how PMS does get in your way, in worst case, you just use > subdirs inside the image dir (nothing prevents you from using a different > DESTDIR variable, right?) and merge them after the install phase. And I've just answered that above.
wait, its much simpler than all this IMHO: we have a variable containing the list of headers to wrap, declared in the ebuild. for every header in that list: after installing for every abi, move the header that landed there to $(dirname of_it)/$abi/ and remove it from the headers checksum file. after finishing installing for every abi, install the wrapper where the original header would have landed that #include <$(dirname of_it)/$abi/$header> depending on the abi am I missing something there ? A header differing between abis will cause multilib_check_header to die, which is the preferred behavior IMHO. (NB: in the above, $(dirname of_it) is the relative dir path from /usr/include) I don't see why pms or overriding install location would matter there. It can even be done within multilib_check_headers.
(In reply to comment #15) > wait, its much simpler than all this IMHO: > > we have a variable containing the list of headers to wrap, declared in the > ebuild. > > for every header in that list: > after installing for every abi, move the header that landed there to > $(dirname of_it)/$abi/ and remove it from the headers checksum file. > after finishing installing for every abi, install the wrapper where the > original header would have landed that #include <$(dirname > of_it)/$abi/$header> depending on the abi > > am I missing something there ? That's what I described, basically. I'm not sure about using a variable there. I feel like we're going to add more and more variables to the generic framework to support more and more corner cases. I'd prefer doing a function which is called by the ebuild but that would probably require plugging it in two places which is even worse.
I've got one more idea. It's more of a temporary solution to not have to mask the packages until we prepare a more generic solution. $ diff -dupr /usr/lib*/freetype2/include/ diff -dupr /usr/lib32/freetype2/include/freetype2/freetype/config/ftconfig.h /usr/lib64/freetype2/include/freetype2/freetype/config/ftconfig.h --- /usr/lib32/freetype2/include/freetype2/freetype/config/ftconfig.h 2013-03-07 19:56:38.000000000 +0100 +++ /usr/lib64/freetype2/include/freetype2/freetype/config/ftconfig.h 2013-03-07 19:56:39.000000000 +0100 @@ -80,7 +80,7 @@ FT_BEGIN_HEADER #ifdef FT_USE_AUTOCONF_SIZEOF_TYPES #define SIZEOF_INT 4 -#define SIZEOF_LONG 4 +#define SIZEOF_LONG 8 #define FT_SIZEOF_INT SIZEOF_INT #define FT_SIZEOF_LONG SIZEOF_LONG Maybe we could just sed the headers with 'sizeof(long)' for now? I think that should satisfy gcc. Will attach patches. If you like it, I'll commit them ASAP as -r2.
Created attachment 341610 [details, diff] freetype: use sizeof() operator rather than hard-coded sizes
Created attachment 341612 [details, diff] Patch to the ebuild
(In reply to comment #18) > Created attachment 341610 [details, diff] [details, diff] > freetype: use sizeof() operator rather than hard-coded sizes did you suggest this fix upstream?
(In reply to comment #19) > Created attachment 341612 [details, diff] [details, diff] > Patch to the ebuild Ok,that was stupid. Well, my last idea: use the configure switch which enables multiarch support and uses the magic hacks for int/long sizes. Plus invent some kind of variable to disable the header check for this difference.
Created attachment 341748 [details, diff] Patch to the ebuild This one uses --enable-biarch-config which could be considered the upstream-blessed way.
Created attachment 341750 [details, diff] Multilib safety check patch And this one mostly makes sure that --enable-biarch-config is respected. it also helps with the header check.
Since Diego doesn't do tinderboxing for us anymore, I've tested the last version with a few random packages from the tracker list and confirmed that they build fine with the patch.
Please dont again do a quick shot with a custom workaround after some short local tests with some random packages. 1. If you want to help users, which have issues due to USE-dependencies with the new multilib eclasses, then just package.mask all packages using that eclass and the user systems will be working fine again 2. after doing step 1, there is no need to hurry, so you can implement some proper solution within the eclass (header wrapping), which can be re-used in other packages and then you can ask other devs to test it (still everything masked) 3. After a proper general solution has been created and tested, we can talk about unmasking eclass using packages again, so users wont have to re-compile packages more often than already required due to issues related to the introduction of the multilib eclasses There have been enough issues for users due to the overlay test and quick introduction into the main tree, so please dont repeat that mistake.
After consulting the patch with lu_zero and testing on a bunch of reverse dependencies, I've committed it and unmasked all the involved packages.
