I kept thinking we had a proper bug for this, but we didn't AFAIK. I shorehorned it into bug 866422 for a while. With static libraries and -flto, bitcode is included in the static archive (.a) and then when an application links using them later on, they're glued together and treated as the whole program. It allows more optimisation, but it's very brittle. The archives aren't compatible across GCC versions, even minor versions (depends on if a commit which changes certain interals gets backported). The interface version isn't always updated for such commits which causes a segfault/ICE instead of a nicer error for mismatched versions. We don't generally bother with static libraries and we can take the loss on optimising them, especially given it also often blows up memory usage with LTO too.
Fedora has two approaches: 1) For GCC, it checks that -ffat-lto-objects was used (they warn if not - https://fedoraproject.org/wiki/Changes/LTOBuildImprovements), and strips the bitcode from the static libraries using https://src.fedoraproject.org/rpms/redhat-rpm-config/blob/rawhide/f/brp-strip-lto 2) For LLVM, they try to convert the bitcode to ELF (https://src.fedoraproject.org/rpms/redhat-rpm-config/blob/rawhide/f/brp-llvm-compile-lto-elf).
So naively, a paired set of functions? In src_configure: lto-guarantee-fat() { if tc-is-lto; then append-cflags $(test-flags-CC -ffat-lto-objects) append-cxxflags $(test-flags-CXX -ffat-lto-objects) fi } in src_install, manually strip with -R .gnu.lto_* -R .gnu.debuglto_* -N __gnu_lto_v1 Or hook this up to portage's global stripping routine. See bug 920745.
openSUSE does https://en.opensuse.org/openSUSE:LTO#Static_libraries (https://github.com/openSUSE/brp-check-suse/blob/master/brp-15-strip-debug).
(In reply to Eli Schwartz from comment #2) I suppose this isn't that bad... especially given not _that_ many packages even install static libraries. We need to see if we can make the same approach work for LLVM too, ideally. I wouldn't want to have to do two approaches...
I was and potentially am still partial to post-processing static libraries by compiling them (or their contents) with '-x lto'. I haven't experimented with this at all, however, so it is entirely possible that it's not possible for some reason. The reason for me holding on to it is because it'd be less error-prone (maintainers wouldn't have to know ahead of time whether static libraries are being installed) and maybe faster. If someone plays around with this, please consider it.
(In reply to Arsen Arsenović from comment #5) > I was and potentially am still partial to post-processing static libraries > by compiling them (or their contents) with '-x lto'. > [...] > The reason for me holding on to it is because it'd be less error-prone > (maintainers wouldn't have to know ahead of time whether static libraries > are being installed) and maybe faster. I don't think it's less error prone because if we were to do it, it'd have to be in Portage (so PM-specific) and also figure out a way for ebuilds to control it... > > If someone plays around with this, please consider it. But I haven't given up on the idea and I will keep your preference in mind, of course
(In reply to Sam James from comment #6) > (In reply to Arsen Arsenović from comment #5) > > I was and potentially am still partial to post-processing static libraries > > by compiling them (or their contents) with '-x lto'. > > [...] > > > The reason for me holding on to it is because it'd be less error-prone > > (maintainers wouldn't have to know ahead of time whether static libraries > > are being installed) and maybe faster. > > I don't think it's less error prone because if we were to do it, it'd have > to be in Portage (so PM-specific) and also figure out a way for ebuilds to > control it... But the default is, IMO, far better. Packages that fail to build in that way are likely exceptions rather than the rule. But yes, the rest stands.. > > > > If someone plays around with this, please consider it. > > But I haven't given up on the idea and I will keep your preference in mind, > of course Thanks :-)
This came up during the 23.0 profile migration as well. The global USE default now enables zstd, which therefore toggles the gcc IUSE and enables zstd-compressed LTO bytecode. GCC promptly emits an ICE when linking against existing static archives that were built with a GCC that was configured to use zlib instead of zstd. This is a general problem if you rebuild the compiler with changed USE...
find /usr/lib* -name '*.a' | xargs equery belongs | sort -u 38 packages on my system. It is not a pleasant idea when doing a profile migration and --emptytree if you have to build all of those with LTO disabled before migrating, then rebuild them again with LTO to taste.
jakub has also suggested the fat-lto-object approach. maybe we should make a 'dot-a.eclass' for adding + processing installed lto objects and object archives, plus a QA check that detects packages forgetting to use them. this seems PM-independent but still solid enough
(In reply to Arsen Arsenović from comment #10) > jakub has also suggested the fat-lto-object approach. maybe we should make > a 'dot-a.eclass' for adding + processing installed lto objects and object > archives, plus a QA check that detects packages forgetting to use them. > this seems PM-independent but still solid enough Using fat-lto-objects just make it that if there's code that fails to understand static libs built with lto, it will try to use the part of the static lib that is normal. Is it bad? Yes, because it's saying "we don't care about performance gain from compiling libs with lto" . AFAIK shared libraries don't have lto at all, so static libs are the only approach for lto gain AFAIU. (In reply to Eli Schwartz from comment #8) > This came up during the 23.0 profile migration as well. > > The global USE default now enables zstd, which therefore toggles the gcc > IUSE and enables zstd-compressed LTO bytecode. > > GCC promptly emits an ICE when linking against existing static archives that > were built with a GCC that was configured to use zlib instead of zstd. > > This is a general problem if you rebuild the compiler with changed USE... I believe it's a problem of an end user. If they want, they use or zlib, or zstd, or they set `-flto-compression-level=0` . (In reply to Sam James from comment #0) > I kept thinking we had a proper bug for this, but we didn't AFAIK. I > shorehorned it into bug 866422 for a while. > The archives aren't compatible across GCC versions, even minor versions > (depends on if a commit which changes certain interals gets backported). The > interface version isn't always updated for such commits which causes a > segfault/ICE instead of a nicer error for mismatched versions. > Maybe create a set of packages that creates static libs ( or add those to @preserved_rebuild ), or get any mechanism to rebuild all static libs if gcc version is updated or changed the gcc's ---zlib--- zstd use flag > We don't generally bother with static libraries and we can take the loss on > optimising them, especially given it also often blows up memory usage with > LTO too. I would like not to take the loss, if it's possible. About memory usage: there's a lot of notes about sometimes massive RAM usage for when -flto , it's the problem of an end user, they understand what drawbacks are and agree to them. Actually, the only big problem with massive RAM usage for me was QEMU ( https://bugs.gentoo.org/883419 )( a guy at discord (Zen) with 512 GB of RAM said he could compile it with static-user and support for aarch64 ) and chromium packages ( require at least 128GB of RAM if lto + -ggdb3 ) . Nowadays, for all other packages the problem with big RAM usage is insignificant. The actual problems are: 1. Recently I've seen a guy at gentoo's discord which was trying to use static libs and was on llvm profile. He compiled libcxx with static libs. He got both static libs and shared libs of libcxx. Somehow when compiling clang after this he got a lot of complains from compiler about adding `-fPIC` . I guess it should be investigated because it's maybe problem not the packages that create LTO static libs, but the packages that use the LTO static libs. 2. GCC's static libs built with LTO are not understandable by Clang and vice versa: Clang's static libs build with LTO or LTO-thin are not understandable by GCC. I remember when I tried llvm profile one year ago there were enough of such problems. I recall at least next bug: https://bugs.gentoo.org/913040
(In reply to Arniii from comment #11) > (In reply to Sam James from comment #0) > > I kept thinking we had a proper bug for this, but we didn't AFAIK. I > > shorehorned it into bug 866422 for a while. > > > The archives aren't compatible across GCC versions, even minor versions > > (depends on if a commit which changes certain interals gets backported). The > > interface version isn't always updated for such commits which causes a > > segfault/ICE instead of a nicer error for mismatched versions. > > > > > Maybe create a set of packages that creates static libs ( or add those to > @preserved_rebuild ), or get any mechanism to rebuild all static libs if gcc > version is updated or changed the gcc's ---zlib--- zstd use flag > > > The actual problems are: > 1. Recently I've seen a guy at gentoo's discord which was trying to use > static libs and was on llvm profile. He compiled libcxx with static libs. He > got both static libs and shared libs of libcxx. Somehow when compiling clang > after this he got a lot of complains from compiler about adding `-fPIC` . I > guess it should be investigated because it's maybe problem not the packages > that create LTO static libs, but the packages that use the LTO static libs. > 2. GCC's static libs built with LTO are not understandable by Clang and vice > versa: Clang's static libs build with LTO or LTO-thin are not understandable > by GCC. I remember when I tried llvm profile one year ago there were enough > of such problems. I recall at least next bug: https://bugs.gentoo.org/913040 3. Figure out how to account what packages creates static libs and what packages depend on them, and if triggered ( gcc is updated or enabled zstd USE flag for gcc ), make mechanism of how to rebuild static libs and then rebuild packages that used them.
(In reply to Arniii from comment #12) . > > 2. GCC's static libs built with LTO are not understandable by Clang and vice > > versa: Clang's static libs build with LTO or LTO-thin are not understandable > > by GCC. I remember when I tried llvm profile one year ago there were enough > > of such problems. I recall at least next bug: https://bugs.gentoo.org/913040 I believe this can be solved by adding a metadata entry like if there was USE flag `static-libs` , there's going to be "STATIC_LIBS_LTO_COMPILER: gcc -llvm" or something like. Then we could figure out what packages should be rebuild if there was lto static lib built with gcc or llvm and make high-level entry, if it uses gcc and static libs, expect all packages that expected to have static libs to be compiled like "STATIC_LIBS_LTO_COMPILER: -gcc -llvm" or "STATIC_LIBS_LTO_COMPILER: gcc -llvm"
(In reply to Arniii from comment #13) > (In reply to Arniii from comment #12) > . > > > 2. GCC's static libs built with LTO are not understandable by Clang and vice > > > versa: Clang's static libs build with LTO or LTO-thin are not understandable > > > by GCC. I remember when I tried llvm profile one year ago there were enough > > > of such problems. I recall at least next bug: https://bugs.gentoo.org/913040 > > I believe this can be solved by adding a metadata entry like if there was > USE flag `static-libs` , there's going to be "STATIC_LIBS_LTO_COMPILER: gcc > -llvm" or something like. Then we could figure out what packages should be > rebuild if there was lto static lib built with gcc or llvm and make > high-level entry, if it uses gcc and static libs, expect all packages that > expected to have static libs to be compiled like "STATIC_LIBS_LTO_COMPILER: > -gcc -llvm" or "STATIC_LIBS_LTO_COMPILER: gcc -llvm" I believe we need such approach because only in a distant future there's going to be gcc-llvm lto compatibility or interoperability, so we need such approach.
(In reply to Arniii from comment #11) > (In reply to Arsen Arsenović from comment #10) > > jakub has also suggested the fat-lto-object approach. maybe we should make > > a 'dot-a.eclass' for adding + processing installed lto objects and object > > archives, plus a QA check that detects packages forgetting to use them. > > this seems PM-independent but still solid enough > > Using fat-lto-objects just make it that if there's code that fails to > understand static libs built with lto, it will try to use the part of the > static lib that is normal. > > Is it bad? Yes, because it's saying "we don't care about performance gain > from compiling libs with lto" . No, we don't care about compiling _static_ libraries with LTO. We very much care about performance gain from LTO in various other bits of code. I suspect most library interfaces wouldn't get much benefit from IPA over them anyway, though (but I have no data to back that up). > AFAIK shared libraries don't have lto at all, so static libs are the only > approach for lto gain AFAIU. They do, internally. > (In reply to Eli Schwartz from comment #8) > > This came up during the 23.0 profile migration as well. > > > > The global USE default now enables zstd, which therefore toggles the gcc > > IUSE and enables zstd-compressed LTO bytecode. > > > > GCC promptly emits an ICE when linking against existing static archives that > > were built with a GCC that was configured to use zlib instead of zstd. > > > > This is a general problem if you rebuild the compiler with changed USE... > > I believe it's a problem of an end user. If they want, they use or zlib, or > zstd, or they set `-flto-compression-level=0` . I disagree with that mentality. There are two valid solutions, and neither involve ICEs when compiling stuff that uses static libraries: 1) Strip LTO code from static libraries 2) Teach portage to track this dependency (1) seems far more favorable because there are so few static libraries, but (2) is useful for other things also. > (In reply to Sam James from comment #0) > > I kept thinking we had a proper bug for this, but we didn't AFAIK. I > > shorehorned it into bug 866422 for a while. > > > The archives aren't compatible across GCC versions, even minor versions > > (depends on if a commit which changes certain interals gets backported). The > > interface version isn't always updated for such commits which causes a > > segfault/ICE instead of a nicer error for mismatched versions. > > > > > Maybe create a set of packages that creates static libs ( or add those to > @preserved_rebuild ), or get any mechanism to rebuild all static libs if gcc > version is updated or changed the gcc's ---zlib--- zstd use flag We can't express that dependency today, I think. > > We don't generally bother with static libraries and we can take the loss on > > optimising them, especially given it also often blows up memory usage with > > LTO too. > > I would like not to take the loss, if it's possible. > About memory usage: there's a lot of notes about sometimes massive RAM usage > for when -flto , it's the problem of an end user, they understand what > drawbacks are and agree to them. > Actually, the only big problem with massive RAM usage for me was QEMU ( > https://bugs.gentoo.org/883419 )( a guy at discord (Zen) with 512 GB of RAM > said he could compile it with static-user and support for aarch64 ) and > chromium packages ( require at least 128GB of RAM if lto + -ggdb3 ) . > Nowadays, for all other packages the problem with big RAM usage is > insignificant. It is possible that failed because the build system (which is ninja) ran too many processes because it lacks job servers. I don't recall the details of when I ran into this also. > The actual problems are: > 1. Recently I've seen a guy at gentoo's discord which was trying to use > static libs and was on llvm profile. He compiled libcxx with static libs. He > got both static libs and shared libs of libcxx. Somehow when compiling clang > after this he got a lot of complains from compiler about adding `-fPIC` . I > guess it should be investigated because it's maybe problem not the packages > that create LTO static libs, but the packages that use the LTO static libs. We don't have a Discord. I've heard there's a Gentoo-themed community on Discord though. Usage of PIC is unrelated to LTO. Now, I don't know what the error was, but it was likely indicative of not using PIC but trying to build a shared library, which fails for obvious reasons. > 2. GCC's static libs built with LTO are not understandable by Clang and vice > versa: Clang's static libs build with LTO or LTO-thin are not understandable > by GCC. I remember when I tried llvm profile one year ago there were enough > of such problems. I recall at least next bug: https://bugs.gentoo.org/913040 Indeed, more reasons not to LTO static libs. (In reply to Arniii from comment #12) > 3. Figure out how to account what packages creates static libs and what > packages depend on them, and if triggered ( gcc is updated or enabled zstd > USE flag for gcc ), make mechanism of how to rebuild static libs and then > rebuild packages that used them. That'd work, but someone has to implement it. (In reply to Arniii from comment #13) > (In reply to Arniii from comment #12) > I believe this can be solved by adding a metadata entry like if there was > USE flag `static-libs` , there's going to be "STATIC_LIBS_LTO_COMPILER: gcc > -llvm" or something like. Then we could figure out what packages should be > rebuild if there was lto static lib built with gcc or llvm and make > high-level entry, if it uses gcc and static libs, expect all packages that > expected to have static libs to be compiled like "STATIC_LIBS_LTO_COMPILER: > -gcc -llvm" or "STATIC_LIBS_LTO_COMPILER: gcc -llvm" It'd be better not to special-case this instance, we have other examples (GHC, perl maybe?, emacs native-comp) where we'd want/need to propagate "computed" compatibility information
(In reply to Arniii from comment #14) > I believe we need such approach because only in a distant future there's > going to be gcc-llvm lto compatibility or interoperability, so we need such > approach. LTO bitcode is compiler internals, so that's not happening.
(In reply to Arsen Arsenović from comment #16) > (In reply to Arniii from comment #14) > > I believe we need such approach because only in a distant future there's > > going to be gcc-llvm lto compatibility or interoperability, so we need such > > approach. > > LTO bitcode is compiler internals, so that's not happening. That's why if compatibility or interoperability will be added, that will be only in a distant future. (In reply to Arsen Arsenović from comment #15) > (In reply to Arniii from comment #11) > > (In reply to Arsen Arsenović from comment #10) > > > jakub has also suggested the fat-lto-object approach. maybe we should make > > > a 'dot-a.eclass' for adding + processing installed lto objects and object > > > archives, plus a QA check that detects packages forgetting to use them. > > > this seems PM-independent but still solid enough > > > > Using fat-lto-objects just make it that if there's code that fails to > > understand static libs built with lto, it will try to use the part of the > > static lib that is normal. > > > > Is it bad? Yes, because it's saying "we don't care about performance gain > > from compiling libs with lto" . > No, we don't care about compiling _static_ libraries with LTO. We very much > care about performance gain from LTO in various other bits of code. If it gives any performance gain in most situation, we should still care about lto static libs. > I suspect most library interfaces wouldn't get much benefit from IPA over them anyway, though (but I have no data to back that up). Let's benchmark somehow to get the truth about this. > > AFAIK shared libraries don't have lto at all, so static libs are the only > > approach for lto gain AFAIU. > They do, internally. > > > (In reply to Eli Schwartz from comment #8) > > > This came up during the 23.0 profile migration as well. > > > > > > The global USE default now enables zstd, which therefore toggles the gcc > > > IUSE and enables zstd-compressed LTO bytecode. > > > > > > GCC promptly emits an ICE when linking against existing static archives that > > > were built with a GCC that was configured to use zlib instead of zstd. > > > > > > This is a general problem if you rebuild the compiler with changed USE... > > > > I believe it's a problem of an end user. If they want, they use or zlib, or > > zstd, or they set `-flto-compression-level=0` . > I disagree with that mentality. There are two valid solutions, and neither > involve ICEs when compiling stuff that uses static libraries: > > 1) Strip LTO code from static libraries > 2) Teach portage to track this dependency > > (1) seems far more favorable because there are so few static libraries, but > (2) is useful for other things also. About compression: it's a problem of an end user. We can make trigger rebuild of lto static libs and their reverse dependencies if compression gcc[zstd] USE flag is changed. 1) Usually those few static libraries are important like libcxx. > > > (In reply to Sam James from comment #0) > > > I kept thinking we had a proper bug for this, but we didn't AFAIK. I > > > shorehorned it into bug 866422 for a while. > > > > > The archives aren't compatible across GCC versions, even minor versions > > > (depends on if a commit which changes certain interals gets backported). The > > > interface version isn't always updated for such commits which causes a > > > segfault/ICE instead of a nicer error for mismatched versions. > > > > > > > > > Maybe create a set of packages that creates static libs ( or add those to > > @preserved_rebuild ), or get any mechanism to rebuild all static libs if gcc > > version is updated or changed the gcc's ---zlib--- zstd use flag > We can't express that dependency today, I think. > Or maybe we should somehow figure out how to do that. > > > We don't generally bother with static libraries and we can take the loss on > > > optimising them, especially given it also often blows up memory usage with > > > LTO too. > > > > I would like not to take the loss, if it's possible. > > About memory usage: there's a lot of notes about sometimes massive RAM usage > > for when -flto , it's the problem of an end user, they understand what > > drawbacks are and agree to them. > > Actually, the only big problem with massive RAM usage for me was QEMU ( > > https://bugs.gentoo.org/883419 )( a guy at discord (Zen) with 512 GB of RAM > > said he could compile it with static-user and support for aarch64 ) and > > chromium packages ( require at least 128GB of RAM if lto + -ggdb3 ) . > > Nowadays, for all other packages the problem with big RAM usage is > > insignificant. > > It is possible that failed because the build system (which is ninja) ran too > many processes because it lacks job servers. I don't recall the details of > when I ran into this also. > Not sure I understand about what you are talking about here. The problem is that the LTO there uses one thread per one resulting LTO linker's product, and the usage of RAM is massive because it's LTO and an end user agreed to try this package with LTO and they accept to solve on their own the problem with RAM or disabling LTO for this package. > > The actual problems are: > > 1. Recently I've seen a guy at gentoo's discord which was trying to use > > static libs and was on llvm profile. He compiled libcxx with static libs. He > > got both static libs and shared libs of libcxx. Somehow when compiling clang > > after this he got a lot of complains from compiler about adding `-fPIC` . I > > guess it should be investigated because it's maybe problem not the packages > > that create LTO static libs, but the packages that use the LTO static libs. > We don't have a Discord. I've heard there's a Gentoo-themed community on > Discord though. The search inside of discord for "gentoo" : https://discord.gg/gentoolinux . Though I believe it's not as official as plenty of IRC channels, but there's enough people. > > Usage of PIC is unrelated to LTO. Now, I don't know what the error was, but > it was likely indicative of not using PIC but trying to build a shared > library, which fails for obvious reasons. Maybe you are right about this. Though, here's details: from the gentoo's discord : https://discord.com/channels/249111029668249601/1265605402435911710 from there: (sorry for logs not being sent here as files): ``` emerge --info: https://bpa.st/S4DA make.conf: https://bpa.st/ZYLA clang fails which is quite problematic. clang log: https://0x0.st/Xpx2.log wgetpaste -s 0x0 -c 'equery u libcxx' : https://0x0.st/XpxF.txt wgetpaste -s 0x0 -c 'equery f libcxx' : https://0x0.st/Xp3X.txt ``` > > > 2. GCC's static libs built with LTO are not understandable by Clang and vice > > versa: Clang's static libs build with LTO or LTO-thin are not understandable > > by GCC. I remember when I tried llvm profile one year ago there were enough > > of such problems. I recall at least next bug: https://bugs.gentoo.org/913040 > Indeed, more reasons not to LTO static libs. Or maybe indeed it's the reason we should admit this is a problem ? > > (In reply to Arniii from comment #12) > > 3. Figure out how to account what packages creates static libs and what > > packages depend on them, and if triggered ( gcc is updated or enabled zstd > > USE flag for gcc ), make mechanism of how to rebuild static libs and then > > rebuild packages that used them. > > That'd work, but someone has to implement it. Yep . I'm not so proficient to help, though I agree to test. > > (In reply to Arniii from comment #13) > > (In reply to Arniii from comment #12) > > I believe this can be solved by adding a metadata entry like if there was > > USE flag `static-libs` , there's going to be "STATIC_LIBS_LTO_COMPILER: gcc > > -llvm" or something like. Then we could figure out what packages should be > > rebuild if there was lto static lib built with gcc or llvm and make > > high-level entry, if it uses gcc and static libs, expect all packages that > > expected to have static libs to be compiled like "STATIC_LIBS_LTO_COMPILER: > > -gcc -llvm" or "STATIC_LIBS_LTO_COMPILER: gcc -llvm" > It'd be better not to special-case this instance, we have other examples > (GHC, perl maybe?, emacs native-comp) where we'd want/need to propagate > "computed" compatibility information IDK then how to propagate this info.
(In reply to Arniii from comment #11) > Is it bad? Yes, because it's saying "we don't care about performance gain > from compiling libs with lto" . > > AFAIK shared libraries don't have lto at all, so static libs are the only > approach for lto gain AFAIU. They have intra-artifact LTO, but not inter-artifact LTO. (In reply to Arniii from comment #11) > I would like not to take the loss, if it's possible. We are not declaring that fundamental problems are just "the user did something wrong", based purely on the fact that a *single* user would rather keep the bug in order to gain highly dubious LTO benefits. As far as I am concerned there are exactly two valid options here: - get future versions of portage to handle static libraries by stripping LTO for safety and QA reasons. - offer a portage option to disable this safety feature, so that the future version of portage behaves exactly like the current version of portage in the event that people prefer the status quo and don't want to "take the loss".
(In reply to Eli Schwartz from comment #18) > As far as I am concerned there are exactly two valid options here: > > - get future versions of portage to handle static libraries by stripping LTO > for safety and QA reasons. > > - offer a portage option to disable this safety feature, so that the future > version of portage behaves exactly like the current version of portage in > the event that people prefer the status quo and don't want to "take the > loss". or third option: try to mess with it to solve it.
I don't want this discussion to be derailed by abstract pipedreams, sorry. We already consider static libraries to be a hazard and their use is discouraged. The issue here is essentially about working around the damage they cause when needed. Any discussion of broadly supporting static linking at large should be done separately and is off-topic here. Right now, we're discussing blockers for practically using LTO globally, and the static library situation is one.
> Yep . I'm not so proficient to help, though I agree to test. While I appreciate your enthusiasm, we're trying to focus on practical fixes which will work now, not possible pipedream ideas that could work provided someone will do the implementation work (which nobody is offering to do for those).
re libcxx, see bug 922202. re the general problem here, this is _NOT_ solely about helping people whose RAM use explodes (although it's part of it), it's also about ebuilds randomly failing because of the incompatibility in the IR between versions. And I strongly suspect neither GCC nor LLVM will ever guarantee compat even for themselves, let alone the insane prospect of it between them.
*** Bug 940540 has been marked as a duplicate of this bug. ***
*** Bug 940538 has been marked as a duplicate of this bug. ***
*** Bug 944810 has been marked as a duplicate of this bug. ***
