| Summary: | dev-qt/qtwebengine-bin package | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Pavel Kozlov <pkozlov.vrn> |
| Component: | New packages | Assignee: | Default Assignee for New Packages <maintainer-wanted> |
| Status: | RESOLVED WONTFIX | ||
| Severity: | normal | CC: | anrock623, binhost, brenosilveira1, bugs, cangming.liu, chain, cruzki123, email200202, flexibeast, flyingwaffle, FRadiatio, gentoo.2019, gentoo.org, gentoo, gentoo, gentoo, gerion.entrup, gpopac, iansilvester, ilmostro7, ionen, ivan, jlj.babydragon, joaompssantos, krinpaus, leho, leo, leonard, marc_heimann, melnikovsky, netbox253, newchief, nrndda, o.freyermuth, orodruinlair, pacho, phobosk, pulkitsukhija09, qt, rene.wilhelm, roberto.castagnola, rushhourresources, sam, simon.vanderveldt+gentoo, snowy.mail, spiderx, steils, thomas, tl, ulm, vanilla.sources, voron1, waebbl-gentoo, yellowhat46 |
| Priority: | Normal | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
oh yes please a bin package. this is becoming a pain . It will rise earth temperature by another 1°c by itself. +1 compilation times are skyrocketing here on my laptop with core 2 duo / 4GiB RAM: 2019-01-07T21:53:17 >>> dev-qt/qtwebengine-5.11.1: 7 hours, 29 minutes, 17 seconds 2019-06-05T07:06:21 >>> dev-qt/qtwebengine-5.12.3: 16 hours, 34 minutes, 2 seconds current merge (5.12.5) is running for 18 hours now and is not finished yet. Yes please provide a binary package! I have masked this package on my system because of the ridiculous long build time. This project includes a large part of the chromium code base. I need to install nextcloud-client, but am not able because of this package. :-( Maybe as a quick fix somebody can point me to a public binhost? For everyone with such long build times, please look at USE=jumbo-build, build will be much, much faster. Make sure your MAKEOPTS value is not unreasonable though taking into account available RAM. Demand can go up to 2GB per process. We'd already have a binary package if it was a simple thing to provide while also doing it right. With regards to the build time, I'll add that it might be worth checking if swapping and/or thermal throttling is happening. Either will very likely slow down the build considerably. I'll also add that it looks like jumbo-build will be going away at around Chromium 80 or some time after, but only time will really tell. I have found this flag: --no-keep-memory; in case the system runs out of memory when processing ld at the end, when using jumbo-build. Putting it on /portage/env could be a good idea in such case Till Schäfer It's still over 4:30 hours with my nice 8 core and 16GB. Never touches swap, though. +1 for this request, this package is a pain. (In reply to Chiitoo from comment #5) > We'd already have a binary package if it was a simple thing to provide while > also doing it right. > > With regards to the build time, I'll add that it might be worth checking if > swapping and/or thermal throttling is happening. Either will very likely > slow down the build considerably. > > I'll also add that it looks like jumbo-build will be going away at around > Chromium 80 or some time after, but only time will really tell. What are the challenges/what needs to be done to make a binary package for this possible? My guess is the choice of USE flags. I'd go for: alsa bindist -designer geolocation -kerberos pulseaudio +system-ffmpeg +system-icu widgets which is pretty much the defaults apart from removing designer. what I ve got (alsa bindist jumbo-build pulseaudio system-ffmpeg system-icu widgets -debug -designer -geolocation -kerberos -test) Having +system-icu and +system-ffmpeg would cause a rebuild in case either of these libraries changed their ABI. In my opinion, it might be best if these flags were off for the *binary* package of qtwebengine to reduce any external dependencies. Just like when both icu and libreoffice-bin get bumped together, because the former depends on the latter. (In reply to Vasilis Lourdas from comment #11) > Having +system-icu and +system-ffmpeg would cause a rebuild in case either > of these libraries changed their ABI. In my opinion, it might be best if > these flags were off for the *binary* package of qtwebengine to reduce any > external dependencies. Just like when both icu and libreoffice-bin get > bumped together, because the former depends on the latter. Agreed. -system-icu -system-ffmpeg This makes sense to me for a binary package and -debug -test +alsa +bindist what about theses? +pulseaudio (I am for it I use it) -designer (don't think many people use it) -geolocation -kerberos (In reply to jms from comment #12) > Agreed. > -system-icu -system-ffmpeg > This makes sense to me for a binary package > and > -debug -test > +alsa +bindist > what about theses? > +pulseaudio (I am for it I use it) > -designer (don't think many people use it) > -geolocation > -kerberos Yes, these make sense, but I also think +widgets is needed too. (In reply to Vasilis Lourdas from comment #13) > (In reply to jms from comment #12) > > Agreed. > > -system-icu -system-ffmpeg > > This makes sense to me for a binary package > > and > > -debug -test > > +alsa +bindist > > what about theses? > > +pulseaudio (I am for it I use it) > > -designer (don't think many people use it) > > -geolocation > > -kerberos > > Yes, these make sense, but I also think +widgets is needed too. yes I forgot (In reply to Simon from comment #8) > What are the challenges/what needs to be done to make a binary package for > this possible? Mainly probably reverse dependencies, and bundling the most annoying ones while also keeping them up-to-date especially with regards to security updates and such. (In reply to Chiitoo from comment #15) > (In reply to Simon from comment #8) > > What are the challenges/what needs to be done to make a binary package for > > this possible? > > Mainly probably reverse dependencies, and bundling the most annoying ones > while also keeping them up-to-date especially with regards to security > updates and such. OK, thanks for the clarification. That seems doable. Is there already something in place within gentoo's infrastructure to build and distribute binary packages? Or would this need to be done by a dev on his/her own account? I assume we could use a binpkg for this? definitely +1...this package has become a huge deal when updating my laptop. Though it's really old (11-years old, dual core, 4GB RAM), updating everything else every now and then is fine...but this one: > 14 hours +1 for the bin package. This would relieve my aging laptops a bit. Newer qtwebengine package. Still hours to build, especially when you maintain 4 Gentoo installations. Is there hope to finally see a binary version of this package? Vasilis, if your 4 installations share a common CPU architecture, you should be able to build a binary package on just one (the most powerful), and install that on the other 3. For added value, you might be able to use DISTCC to spread the compilation across all the machines. See the wiki for details of both. That said, I'd still like to see a qtwebengine-bin package! Ok, this is ridiculous. Today I got a libjpeg-turbo update which causes a qtwebengine rebuild. In my (relatively strong) PC, it takes an hour to compile. In my 2-core laptop, it took more than 3 hours to compile (even with the jumbo-build USE flag). And just because libjpeg-turbo was updated, another rebuild is needed. Today marks exactly 2 years after this bug was opened. Is someone willing to provide a binary package for this time-consuming package? +1 for a much needed qtwebengine-bin There are some packages I don't emerge and some USE flags I don't use so that they don't trigger the qtwebengine monster. Another day, another qtwebengine rebuild. And as I thought Zoom needed it (and in fact it did at first, but not now), the latest digikam-7.0 requires it. Today dev-libs/re2 causes a rebuild of this freaking package... +1 for the bin package. It takes 3 hours on my 8cores/16G computer. (In reply to email200202 from comment #24) > +1 for the bin package. > > It takes 3 hours on my 8cores/16G computer. You should enable the jumbo-build USE flag for this package, for your system, it shouldn't take more than an hour. (In reply to Vasilis Lourdas from comment #25) > (In reply to email200202 from comment #24) > > +1 for the bin package. > > > > It takes 3 hours on my 8cores/16G computer. > > You should enable the jumbo-build USE flag for this package, for your > system, it shouldn't take more than an hour. It takes 3 hours with jumbo-build USE flag enabled. In the latest version dev-qt/qtwebengine-5.15.2, jumbo-build USE flag was removed! # cat /usr/portage/dev-qt/qtwebengine/qtwebengine-5.15.2.ebuild .... IUSE="alsa bindist designer geolocation kerberos pulseaudio +system-ffmpeg +system-icu widgets" .... src_prepare() { if use ppc64; then eapply "${WORKDIR}/${PN}-ppc64" fi # QTBUG-88657 - jumbo-build is broken #if ! use jumbo-build; then sed -i -e 's|use_jumbo_build=true|use_jumbo_build=false|' \ src/buildtools/config/common.pri || die #fi dev-qt/qtwebengine-5.15.2 merge took 6 hours 43 minutes!
# genlop -t dev-qt/qtwebengine
Wed May 20 20:30:20 2020 >>> dev-qt/qtwebengine-5.14.2
merge time: 2 hours, 50 minutes and 28 seconds.
Wed May 27 18:48:10 2020 >>> dev-qt/qtwebengine-5.15.0
merge time: 3 hours, 9 minutes and 3 seconds.
Wed Jul 15 17:01:58 2020 >>> dev-qt/qtwebengine-5.15.0
merge time: 3 hours, 1 minute and 21 seconds.
Fri Sep 11 18:40:38 2020 >>> dev-qt/qtwebengine-5.15.1
merge time: 3 hours, 9 minutes.
Mon Oct 19 15:41:27 2020 >>> dev-qt/qtwebengine-5.15.1
merge time: 2 hours, 59 minutes and 23 seconds.
Thu Nov 5 03:37:44 2020 >>> dev-qt/qtwebengine-5.15.1
merge time: 2 hours, 59 minutes and 29 seconds.
Fri Nov 27 05:03:23 2020 >>> dev-qt/qtwebengine-5.15.2
merge time: 6 hours, 43 minutes and 28 seconds.
*** Bug 758341 has been marked as a duplicate of this bug. *** This really needs to be addressed; I'm on a decently new/good specced laptop and even after I started using the -jumbo-build flag in October or so you can see from genlop output how much that has NOT helped: Sat Jan 11 21:32:07 2020 >>> dev-qt/qtwebengine-5.14.0 merge time: 5 hours, 14 minutes and 48 seconds. Mon Feb 3 01:53:42 2020 >>> dev-qt/qtwebengine-5.14.1 merge time: 9 hours, 9 minutes and 53 seconds. Sun Feb 16 05:14:32 2020 >>> dev-qt/qtwebengine-5.14.1 merge time: 9 hours, 46 minutes and 27 seconds. Sun Mar 22 01:07:15 2020 >>> dev-qt/qtwebengine-5.14.1 merge time: 7 hours, 36 minutes and 29 seconds. Sun Apr 5 08:53:14 2020 >>> dev-qt/qtwebengine-5.14.1 merge time: 8 hours, 41 minutes and 20 seconds. Sun Apr 12 02:44:46 2020 >>> dev-qt/qtwebengine-5.14.2 merge time: 8 hours, 34 minutes and 8 seconds. Fri May 1 19:35:51 2020 >>> dev-qt/qtwebengine-5.14.2 merge time: 6 hours, 6 minutes and 27 seconds. Sun May 10 09:02:10 2020 >>> dev-qt/qtwebengine-5.14.2 merge time: 8 hours and 18 seconds. Sun May 31 02:55:19 2020 >>> dev-qt/qtwebengine-5.15.0 merge time: 9 hours, 57 minutes and 12 seconds. Wed Jul 15 15:20:16 2020 >>> dev-qt/qtwebengine-5.15.0 merge time: 6 hours, 59 minutes and 19 seconds. Fri Sep 11 16:15:06 2020 >>> dev-qt/qtwebengine-5.15.1 merge time: 5 hours, 31 minutes and 28 seconds. Sat Oct 31 16:44:17 2020 >>> dev-qt/qtwebengine-5.15.1 merge time: 4 hours, 12 minutes and 21 seconds. Wed Nov 4 16:13:45 2020 >>> dev-qt/qtwebengine-5.15.1 merge time: 4 hours, 26 minutes and 38 seconds. Fri Nov 27 23:15:26 2020 >>> dev-qt/qtwebengine-5.15.2 merge time: 9 hours, 33 minutes and 18 seconds. Tue Dec 8 22:32:36 2020 >>> dev-qt/qtwebengine-5.15.2 merge time: 11 hours, 45 minutes and 28 seconds. CPU : Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz RAM : 4Gb dev-qt/qtwebengine-5.15.2: 19 hours, 58 minutes, 22 seconds average for 1 merge dev-qt/qtwebengine-5.14.1: 15 hours, 20 minutes, 50 seconds average for 1 merge USE flags : dev-qt/qtwebengine-5.15.2 -alsa -bindist -debug -designer -geolocation -kerberos -pulseaudio -system-ffmpeg -system-icu -test widgets Shrug... Worst of this : it is required only for KmyMoney (which I will simply download the AppImage in the future) ... Previous "jumbo-build" USE flag looks to have saved like 4 hours... (In reply to Tiger from comment #30) > CPU : Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz > RAM : 4Gb > > dev-qt/qtwebengine-5.15.2: 19 hours, 58 minutes, 22 seconds average for 1 > merge > dev-qt/qtwebengine-5.14.1: 15 hours, 20 minutes, 50 seconds average for 1 > merge > > USE flags : > dev-qt/qtwebengine-5.15.2 -alsa -bindist -debug -designer -geolocation > -kerberos -pulseaudio -system-ffmpeg -system-icu -test widgets > > Shrug... > > Worst of this : it is required only for KmyMoney (which I will simply > download the AppImage in the future) ... > > Previous "jumbo-build" USE flag looks to have saved like 4 hours... Today I synced and I also got the stable 5.15.2 ebuild without the jumbo-build flag. I copied the ebuild to my local overlay, re-enabled the jumbo-build flag since I don't use GCC 10 and I will compile it. Let's hope that in 2021 the developers will decide to provide a binary package. another +1 this gets troubling to build on my haswell i7-4810MQ system 2020-11-20T02:53:17 >>> dev-qt/qtwebengine-5.15.1: 2 hours, 18 minutes, 14 seconds 2020-11-26T15:58:09 >>> dev-qt/qtwebengine-5.15.2: 5 hours, 6 minutes, 46 seconds There won't be any progress here if all that people do is posting additional +1 comments. Someone has to step up and write an ebuild, and be prepared to maintain the package (e.g., rebuild it against new versions of dev-libs/icu). I'm sorry for yet another +1, but this package is a potential death sentence on Gentoo on many of my desktop machines. Ulrich, is it possible to release a binpkg binary as an ebuild? The discussions earlier seem to imply a binary package along the lines of say firefox, which can optionally contain specific versions of dependencies; I understand why that can make sense for such packages, but the need here is simply (for certain values of "simple") avoiding the mega-compilation. IIUC, two things need to be developed: something in catalyst to build and package qtwebengine for various platforms, and an ebuild that installs the result. I do something different on my home setup - I compile it on my ryzen-9 desktop machine (in less than an hour), and install the binpkg on my laptop (where it would probably take a day). But AFAIK ebuilds don't do that. Or can they? It is also worth mentioning that 4GB of RAM with all non-critical processes turned off is insufficient for compiling the newest qtwebengine. I confirm that the lack of it forces me to switching to a different distro till the *-bin package appears. (In reply to Marek Kozlowski from comment #36) > It is also worth mentioning that 4GB of RAM with all non-critical processes > turned off is insufficient for compiling the newest qtwebengine. I confirm > that the lack of it forces me to switching to a different distro till the > *-bin package appears. I've just checked with 6GB - is not enough either! (In reply to Marek Kozlowski from comment #37) > (In reply to Marek Kozlowski from comment #36) > > It is also worth mentioning that 4GB of RAM with all non-critical processes > > turned off is insufficient for compiling the newest qtwebengine. I confirm > > that the lack of it forces me to switching to a different distro till the > > *-bin package appears. > > I've just checked with 6GB - is not enough either! The same with 8GB! Do I really need a supercomputer to just install calibre (a required dependency)? Please don't spam this bug. Tune your -j value to a realistic number for your machinery, and definitely *less* than total GB of RAM/2. (In reply to Andreas Sturmlechner from comment #39) > Please don't spam this bug. Tune your -j value to a realistic number for > your machinery, and definitely *less* than total GB of RAM/2. -j is jest to 3 ( https://wiki.gentoo.org/wiki/Lenovo_ThinkPad_T400 ). 8GB of RAM. 3 < 8/2. It still fails. I'm running Intel i9 @3.60GHz, 10 cores, 20 threads; 64 GB RAM. dev-qt/qtwebengine is failing for me too, even with -j1 and jumbo-build. (Tried emerging today, several times). qtwebengine's emerge is awful. Count me as a +1-er. While building on a dual socket AMD Epyc 64 cores I noticed a peak RAM usage of 150 GB. I think the problem should be addressed upstream :(. (In reply to Vasco Gervasi from comment #42) > While building on a dual socket AMD Epyc 64 cores I noticed a peak RAM usage > of 150 GB. I think the problem should be addressed upstream :(. If you have a lot of RAM it will try to utilize most of it, which is why you might see a couple hundred MB being used while idle on a system with 2 GB of RAM, but on a system with 8 GB it'll be several times more (take that with a grain of salt though, I'm not an expert). Today I've removed qtwebengine (and all of the software depending on it) from all of my Gentoo machines. Enough was enough. Check also bug #793935. Ok, so I am willing to donate a small amount to any developer that spends time to create a ebuild for a binary package. I know I can't do it myself, if I did, I would have already tried it. If anyone here is willing to do the same, would this help? Does anyone know if there's a standardized way within the gentoo org of building and providing binary packages? I'd assume something like that would already exist. Then it would just be as simple as adding qtwebengine to the list of packages to build and add a -bin ebuild to the tree to use the built artifact. Because everybody's dependencies are frequently different, its difficult/impossible to build a 'generic' binary. This is why compiling from source works, because the build system can detect and adapt to what's present/etc. Most Gentoo binary packages come from upstream, where they can take care of compatibility testing, which Gentoo doesn't really have the capacity to. (In reply to Michael 'veremitz' Everitt from comment #48) > Because everybody's dependencies are frequently different, its > difficult/impossible to build a 'generic' binary. This is why compiling from > source works, because the build system can detect and adapt to what's > present/etc. > > Most Gentoo binary packages come from upstream, where they can take care of > compatibility testing, which Gentoo doesn't really have the capacity to. Allow me to disagree. The binary version of Libreoffice is built by Gentoo and is not the upstream binary version. For such a complex package (consider that it is a user application with UI etc.), I tend to think it's "relatively" easy to provide a binary package for a Qt library. well, that is the first I've seen of that, indeed tamiko is building and hosting that himself (evident from SRC_URI). I merge dev-qt/qtwebengine and other large packages with --buildpkg flag on one computer and then I use the binary packages on other computers. The binary package is not large: # ls -lh /usr/portage/packages/dev-qt/qtwebengine/qtwebengine-5.15.2_p20210521-2.xpak -rw-r--r-- 1 root portage 56M May 31 04:10 /usr/portage/packages/dev-qt/qtwebengine/qtwebengine-5.15.2_p20210521-2.xpak The 'emerge' tool checks for dependencies and use flags before accepting the binary package. Any conflict will cause 'emerge' to compile from the sources instead. If we define a new use flag say "binary" (or other mechanism) to direct 'emerge' to download the binary package from a gentoo server and to invoke the binary install functions, we will have a binary install without the need for a different ebuild. dev-qt/qtwebengine is not the only package which takes unreasonable time to merge. So we need a solution which does not need different *-bin package and a maintainer for it. Just my two cents. (In reply to Michael 'veremitz' Everitt from comment #48) > Because everybody's dependencies are frequently different, its > difficult/impossible to build a 'generic' binary. This is why compiling from > source works, because the build system can detect and adapt to what's > present/etc. > > Most Gentoo binary packages come from upstream, where they can take care of > compatibility testing, which Gentoo doesn't really have the capacity to. No? All a binary is, is code that's compiled to machine code. Compiling from source allows you to select the features you want from a program and the ability to customize optimization options. Unless you're using --native or something, it is very easy to create a generic binary. Compatibility testing should not be a problem as the vast majority of distributed binaries are compiled with -O2, so any "incompatibility" that happens is on the fault of the developer. Perhaps the solution isn't to distribute *-bin packages at all. Instead gentoo.org or some other trusted authority could host binpkgs for the larger packages on architectures in which they are stable. I would host it myself (as I've already built many of these packages), but it's probably not a good idea to have people installing binaries from some random guy's website. Another alternative would be to distribute completely pre-configured images like sakaki's rasperry pi images. Just as a bit meta discussion: NixOS has exactly that. The package manager tries to install an (officially) prebuilded package and if that fails it compiles it by hand. The whole point why this works is because in Nix the inputs and outputs are determined. With this, one can check if there is a package for exactly "this particular input configuration". Gentoo's approach with bin-packages currently is to take a source package, apply a very specific use flag combination/compiler version/compiler flags, and use that to provide an extra binary package. In my understanding, the Nix approach should be possible for Gentoo, too, if the inputs can be specified exactly: 1. dependencies are already specified in the ebuild (and part of the input set) 2. use flags are already specified in the ebuild (and part of the input set) 3. compiler version and flags are not part of the ebuild (are there a necessary input? Compiler are normally interoperable, just like CFLAGS except march=) I'm not exactly sure but quite a time ago there was a discussion about a binary package projekt on the Gentoo mailing list: https://archives.gentoo.org/gentoo-dev/message/e2eaa90b45a8a342d58231a06cf18137 I can't say for all... I have just stop using qtwebengine with rebuilding my kde set with -webengine use flag. Hi, a quick update, I was able to compile dev-qt/qtwebengine 5.15.2_p20210824 on my i7-4810QM with 16 GB of RAM in 3 hours and 23 mins. A great improvement from not being able to do it and the 5 hours of 5.15.2. <5.15.2 had IUSE=jumbo-build (so default disabled) 5.15.2 did not contain IUSE=jumbo-build because it was broken. 5.15.2_p* contains IUSE=+jumbo-build (so default enabled) Someone pointed me to this link: https://gentoo.osuosl.org/experimental/amd64/binpkg/default/linux/17.1/x86-64/dev-qt/qtwebengine/ Contains xpak binaries (?) for the latest version. I synced today and it built 5.15.2_p20211019 in 1 hour and 40' in my PC, while the previous builds were finished in a hour. I will certainly try these binaries out. On a sidenote, this freaking package was *one* of the reasons I switched from Gentoo to OpenSUSE Tumbleweed in my 7th Gen i5 laptop... (In reply to Vasilis Lourdas from comment #58) > Someone pointed me to this link: > https://gentoo.osuosl.org/experimental/amd64/binpkg/default/linux/17.1/x86- > 64/dev-qt/qtwebengine/ > > Contains xpak binaries (?) for the latest version. I synced today and it > built 5.15.2_p20211019 in 1 hour and 40' in my PC, while the previous builds > were finished in a hour. > > I will certainly try these binaries out. More info here: https://dilfridge.blogspot.com/2021/09/experimental-binary-gentoo-package.html. > > On a sidenote, this freaking package was *one* of the reasons I switched > from Gentoo to OpenSUSE Tumbleweed in my 7th Gen i5 laptop... Maybe we can get you back then ;) (In reply to Vasilis Lourdas from comment #58) > On a sidenote, this freaking package was *one* of the reasons I switched > from Gentoo to OpenSUSE Tumbleweed in my 7th Gen i5 laptop... These sidenotes are irrelevant in bugs. >On a sidenote, this freaking package was *one* of the reasons I switched from Gentoo to OpenSUSE Tumbleweed in my 7th Gen i5 laptop...
+1. I have not switched from Gentoo just yet, but I'm almost done: this freaking package is the gentoo's killer for me.
For those who are struggling with compiling qtwebengine, you can try building it with clang. It uses 30-50% the RAM comparing to gcc, when compiling chromium/qtwebengine, which might allow you to bump up MAKEOPTS, which in turn might make it less painful. Hang in there, guys. :-) There are plenty of bugs filed by people trying to build qtwebengine with clang, not because of resource exhaustion, but regular build errors. (In reply to Andreas Sturmlechner from comment #63) > There are plenty of bugs filed by people trying to build qtwebengine with > clang, not because of resource exhaustion, but regular build errors. Yeah, I know. Nothing comes easy with qtwebengine, and it takes some fiddling. But it's entirely possible to do: Wed Dec 8 21:22:31 2021 >>> dev-qt/qtwebengine-5.15.2_p20211207 merge time: 1 hour, 29 minutes and 16 seconds. And the ram usage is so much lower that it might be worth it in certain situations. I remember someone told me on #gentoo-chat that they were able to lower the build time from 8 to 2 hours. They had a decent CPU, but only 8 gigs of ram, so that's a bit specific, but still. (In reply to Michael from comment #64) > I remember someone told me on #gentoo-chat that they were able > to lower the build time from 8 to 2 hours. That's just misconfiguration. GCC may as well take just 2 hours in that scenario without overly optimistic MAKEOPTS settings leading to swapping. See also comment #4: > Make sure your MAKEOPTS value is not unreasonable though taking into account > available RAM. Demand can go up to 2GB per process. If I remember correctly, he had a ryzen 3600, 6 cores, 12 threads, and 8GB is of course grossly underspecced, it needs at very least 3 times as much. But the point is GCC was chugging along with -j3 or -j4, because when building qtwebengine it can easily gobble up to 2.5-3GB per thread. And with clang he was able to bump it up to -j10 or something like that, which of course sped it up significantly. And of course it is misconfigured, in the sense that ryzen 5 needs more ram, otherwise it'll not be fully utilized. That particular anecdote is of course just that. An anecdote. It was a low hanging fruit for speeding things up. For me switching to clang didn't change the compile time at all, because my system is not quite as grossly underspecced in terms of RAM. I'm just mentioning all that, because maybe there's someone else out there bottlenecking on RAM. (In reply to Michael from comment #66) > And of course it is misconfigured, in the sense that ryzen 5 needs more ram, > otherwise it'll not be fully utilized. It will never take a Ryzen 5 3600 eight hours to build qtwebengine even with only 8 GB RAM, with aptly adjusted MAKEOPTS, without swapping. Even an old i7-4790K will do it in 2 hours, with 8 GB RAM. So first and foremost we will always tell people to set their MAKEOPTS to realistic values. The reason I am objecting are all the other issues that switching to clang systemwide brings. I would never recommend it as a quick fix; it takes some commitment. (In reply to Andreas Sturmlechner from comment #67) > The reason I am objecting are all the other issues that switching to clang > systemwide brings. I would never recommend it as a quick fix; it takes some > commitment. Just for the record, I was not at all suggesting systemwide clang. Only for anything with chromium in it. Which is basically chromium and qtwebengine. (In reply to Andreas Sturmlechner from comment #67) > It will never take a Ryzen 5 3600 eight hours to build qtwebengine even with > only 8 GB RAM, with aptly adjusted MAKEOPTS, without swapping. Even an old > i7-4790K will do it in 2 hours, with 8 GB RAM. So first and foremost we will > always tell people to set their MAKEOPTS to realistic values. I have an old i7-2600K with 8GB RAM, and I have used MAKEOPTS="-j4" in the past. These are the last build times for this package: Sat Nov 2 00:49:44 2019 >>> dev-qt/qtwebengine-5.12.5 merge time: 3 hours, 35 minutes and 18 seconds. Sat Apr 4 12:20:30 2020 >>> dev-qt/qtwebengine-5.14.1 merge time: 4 hours, 6 minutes and 8 seconds. Thu May 21 19:34:34 2020 >>> dev-qt/qtwebengine-5.14.2 merge time: 4 hours, 33 minutes and 49 seconds. Thu Oct 29 15:14:33 2020 >>> dev-qt/qtwebengine-5.15.1 merge time: 17 hours, 45 minutes and 20 seconds. Thu Dec 3 03:36:11 2020 >>> dev-qt/qtwebengine-5.15.2 merge time: 16 hours, 10 minutes and 7 seconds. Tue Feb 2 16:18:39 2021 >>> dev-qt/qtwebengine-5.15.2 merge time: 16 hours, 12 minutes and 5 seconds. The big difference was due to the use-flag jumbo-build that was broken for 5.15 at the beginning; when it was available again, re-enabling it the build failed with both -j4 and -j3. I was able to build it a bit faster only enabling jumbo-build and using -j4 at the beginning; every time the build failed, I have changed the option reducing the amount of thread directly in Makefile and restarted the build with command 'ebuild', ending with -j2 to be able to complete the build. PS: the build is started with nothing else running and swap disabled; I run 'openrc boot' to keep the max resource available only for the build I will try to use clang only for this package (I have to figure out how to do it); the only other solution will be the binary package. Further, is there a way to define a different PORTAGE_TMPDIR for a single package? I tryed to add it into /etc/portage/env/dev-qt/qtwebengine but it didn't work. (In reply to Roberto Castagnola from comment #69) > Further, is there a way to define a different PORTAGE_TMPDIR for a single > package? > I tryed to add it into /etc/portage/env/dev-qt/qtwebengine but it didn't > work. Yes, there is a way to do that, and you are on the right track. But in addition you need a package.env file with something like `dev-qt/qtwebengine tmpdir.conf` and a /etc/portage/env/tmpdir.conf with `PORTAGE_TMPDIR=....` in it. You'll have to figure out how to use package.env if you want to set the compiler on a per package basis. I'm fully expecting you to be able to go to at very least -j4 with clang. (In reply to Michael from comment #71) > I'm fully expecting you to be able to go to at very least -j4 with clang. with MAKEOPTS="-j5" Mon Dec 13 05:03:35 2021 >>> dev-qt/qtwebengine-5.15.2_p20211207 merge time: 7 hours, 37 minutes and 1 second. still a lot of time but better than before. I needed few workaround to compile it with clang: bug #813249 : I have defined EXTRA_GN as suggested bug #828099 : I have used the patch www-client/chromium/files/chromium-glibc-2.34.patch (adapted for qtwebengine code) Further, almost at the end, the build failed because it tried to run clang++ without the full path. Not sure if it is an issue of qtwebengine build or of clang. I have tried to make a link for clang++ in /usr/bin, but in this way the build fails at the beginning. My workauround was to define CC and CXX with full path. (In reply to Roberto Castagnola from comment #72) > I needed few workaround to compile it with clang Build system in qtwebengine is notoriously flaky, so not unexpected. > I have tried to make a link for clang++ in /usr/bin, but in this way the > build fails at the beginning. > My workauround was to define CC and CXX with full path. I haven't hit that one yet, but thanks for the heads-up. :-) +1 for a much needed qtwebengine-bin On my i7 (7th gen) laptop with 32 GB of RAM it takes around 3 hours to merge qtwebengine, which is quite long for a package that got merged, on average, twice a month in the last year. My MAKEOPTS is set to "-j6". I wonder if disabling the system-ffmpeg and system-icu USE flags could reduce the amount of rebuilds that are needed? I build qtwebengine like twice a week on my Ryzen 5 with 12 threads and 32 GB RAM. It still utterly sucks. So I think cheaping out building tiny packages (compared with qtwebengine) like ffmpeg and icu won't help much here. On my system (Quad Core, 8 GB), switching to clang reduced the compilation time from ~16h to ~5h (with jumbo-build). (In reply to Krzysiek from comment #77) > On my system (Quad Core, 8 GB), switching to clang reduced the compilation > time from ~16h to ~5h (with jumbo-build). Any special changes you did for building with clang? (In reply to Pulkit Sukhija from comment #78) > (In reply to Krzysiek from comment #77) > > On my system (Quad Core, 8 GB), switching to clang reduced the compilation > > time from ~16h to ~5h (with jumbo-build). > > Any special changes you did for building with clang? No, I just edited files: /etc/portage/env/compiler-clang /etc/portage/package.env according to this manual: https://wiki.gentoo.org/wiki/Clang#Clang_environments (In reply to Krzysiek from comment #79) > (In reply to Pulkit Sukhija from comment #78) > > (In reply to Krzysiek from comment #77) > > > On my system (Quad Core, 8 GB), switching to clang reduced the compilation > > > time from ~16h to ~5h (with jumbo-build). > > > > Any special changes you did for building with clang? > > No, I just edited files: > > /etc/portage/env/compiler-clang > /etc/portage/package.env > > according to this manual: > https://wiki.gentoo.org/wiki/Clang#Clang_environments Yeah the same for me. qtwebengine builds without any problems using CLang nowadays. (In reply to Krzysiek from comment #79) > (In reply to Pulkit Sukhija from comment #78) > > (In reply to Krzysiek from comment #77) > > > On my system (Quad Core, 8 GB), switching to clang reduced the compilation > > > time from ~16h to ~5h (with jumbo-build). > > > > Any special changes you did for building with clang? > > No, I just edited files: > > /etc/portage/env/compiler-clang > /etc/portage/package.env > > according to this manual: > https://wiki.gentoo.org/wiki/Clang#Clang_environments Oh, alright, 'cause for me it failed at somewhat 2800 tars, maybe i didn't edit the confs the correct way sys-kernel/gentoo-kernel-bin is good, but a qtwebengine-bin will be a real redemption for gentoo. I had it disabled for many years, but last seasons I started using www-client/falkon and it hurts so much to wait for hours and hours for it to compile... Even with 16GB of RAM the build may fail so a binary package will be great (I see chromium-bin was added recently). --- /usr/lib/gcc/x86_64-pc-linux-gnu/11.3.0/include/g++-v11/bits/unique_ptr.h:155:63: internal compiler error: Segmentation fault --- (In reply to sebaro from comment #83) > Even with 16GB of RAM the build may fail so a binary package will be great > (I see chromium-bin was added recently). Configure your system accordingly so this does not happen. chromium(-bin) is a leaf package. qtwebengine is not. Is it possible to build it with the "keepwork" feature?
I tried with:
PORTAGE_TMPDIR="/home/Apps/Gentoo/emerge" FEATURES="${FEATURES} keepwork" emerge qtwebengine
On "emerge --resume" I get this:
/bin/sh: line 1: /home/Apps/Gentoo/emerge/portage/dev-qt/qtwebengine-5.15.7_p20221122/temp/python3.10/bin/python: No such file or directory
There is a binary package in Gentoo Experimental: https://gentoo.osuosl.org/experimental/amd64/binpkg/default/linux/17.1/x86-64/dev-qt/qtwebengine/ It can be installed with "emerge -G qtwebengine" by using one of these two options: /etc/portage/make.conf --- PORTAGE_BINHOST="hhttps://gentoo.osuosl.org/experimental/amd64/binpkg/default/linux/17.1/x86-64/" --- /etc/portage/binrepos.conf --- [gentoo-binpkg] priority = 9999 sync-uri = https://gentoo.osuosl.org/experimental/amd64/binpkg/default/linux/17.1/x86-64/ --- emerge qtwebengine is really painful, it takes longer and longer each time, and I really need a bin package (In reply to cnfczn from comment #87) > emerge qtwebengine is really painful, it takes longer and longer each time, > and I really need a bin package So have you tried the experimental -bin package mentioned in the comment immediately before yours? (In reply to Alexis from comment #88) > (In reply to cnfczn from comment #87) > > emerge qtwebengine is really painful, it takes longer and longer each time, > > and I really need a bin package > So have you tried the experimental -bin package mentioned in the comment > immediately before yours? The experimental -bin package is requiring a systemd profile/package. End of try for all openrc users. (In reply to Fab from comment #89) > The experimental -bin package is requiring a systemd profile/package. End of > try for all openrc users. The first problem is qtnetwork which was built with networkmanager USE flag and net-misc/networkmanager was built with systemd USE flag enabled. Emerge qtnetwork without networkmanager or networkmanager without systemd. Second problem, qtwebengine was built with pulseaudio USE flag, which pulls media-libs/libpulse built with systemd, so emerge media-libs/libpulse without systemd. So any update on this issue? I have a 16 core CPU, compilation time takes up to 4 hours, and it takes 32G memory during this progress, that is too crazy. (In reply to flex from comment #91) > So any update on this issue? I have a 16 core CPU, compilation time takes up > to 4 hours, and it takes 32G memory during this progress, that is too crazy. Probably time to just close this not to give expectations, nobody in qt@ seems to have interest and at this point would rather it not be a thing (even if someone wants to handle it) given it's pretty messy with sync'ing with releases and subslot rebuilds. Not that this means you will never get a binary package, there is on-going work to provide a binhost for (at least) popular stable packages and that includes qtwebengine:5 (and will likely have :6 eventually). Aka it just won't be coming as a qtwebengine-bin but rather a .gpkg.tar There will be a formal announcement (gentoo news) when that's ready. (In reply to flex from comment #91) > So any update on this issue? I have a 16 core CPU, compilation time takes up > to 4 hours, and it takes 32G memory during this progress, that is too crazy. Also, you don't seem to have a weak machine, and building qtwebengine should not be *that* bad, just use lower MAKEOPTS' -j appropriate for how much RAM you have. On my subpar i7-8700 (6 core, 12 threads) with 16GB I keep it down to -j6 and it takes a reasonable ~2 hours for qtwebengine:6. Different story for others with far weaker hardware where it can take over 24h. (In reply to Ionen Wolkens from comment #92) > There will be a formal announcement (gentoo news) when that's ready. To clarify, meant as something official -- this is the rework of the former experimental binhost offering. No idea on ETA though. New binhost location: https://distfiles.gentoo.org/releases/amd64/binpackages/17.1/x86-64/ > emerge -Gv qtwebengine > [binary U ] dev-qt/qtwebengine-5.15.10_p20230815-8:5/5.15::gentoo > [5.15.8_p20230112:5/5.15::gentoo] USE="alsa bindist jumbo-build pulseaudio* > screencast* system-icu widgets -debug -designer -geolocation -kerberos > -test (-system-ffmpeg%*)" 56,470 KiB Hello, is there any chance that binpkg for qtwebengine[-pulseaudio] will be provided? I've no strong knowledge of gentoo ebuilds, nor C++, but if it isn't that complicated to add this option, I'd try to help. OFC I understand that having another combination of USE flags brings a lot of (not only) computation time that has to be paid. Thank you. (In reply to Ondřej Kajínek from comment #95) > Hello, > > is there any chance that binpkg for qtwebengine[-pulseaudio] will be > provided? Not really, unlikely anyone is going to make multiple builds of this to provide binpkgs for every configurations, esp. not just to skip a tiny library like libpulse (which doesn't even install the pulseaudio daemon itself). And then pulseaudio is a default on the plasma profile (gnome too) so it makes sense for the binpkg to have it too. If really want to change typical profile defaults, you'll have to compile it yourself. > which doesn't even install the pulseaudio daemon itself
My bad, I should've tried before asking. Thanks for an advice, this will help me a lot with system updates.
Let me go the other way around :) I have to compile qtwebengine:6 now by myself due to the vaapi useflag which is not activated in the binary ebuild (I guess vaapi support is fairly new). Do you plan to make it default on amd64 systems? I guess, the majority of AMD64 CPUs is VAAPI capable right now and would benefit from it. |
Hello! dev-qt/qtwebengine takes really huge time to compile - and it became worse after every next version. Last time it took more than 8 hours to build on my laptop! Of course it is not new, but still core-i5 with 6 gigabytes of memory... Sat Oct 20 15:05:28 2018 >>> dev-qt/qtwebengine-5.11.1 merge time: 8 hours, 9 minutes and 46 seconds. I know that gentoo has some bin versions of packages which were created in intention to reduce compile-time which doesn't make really much sense - like firefox-bin or libreoffice-bin. In fact, firefox takes just about 1.5 hour to compile in my laptop, which is acceptable, I don't use firefox-bin, I use firefox package - it is OK for me. But why qtwebengine takes so long? Can it be improved? I am OK to use -bin version for it, is this possible? emerge --info (first part with processor and memory) Portage 2.3.49 (python 3.6.5-final-0, default/linux/amd64/17.1/desktop/plasma, gcc-7.3.0, glibc-2.26-r7, 4.18.9-gentoo x86_64) ================================================================= System uname: Linux-4.18.9-gentoo-x86_64-Intel-R-_Core-TM-_i5-3230M_CPU_@_2.60GHz-with-gentoo-2.4.1 KiB Mem: 5909444 total, 1578476 free KiB Swap: 1048572 total, 851256 free ...