| Summary: | net-libs/webkit-gtk: requesting a binary package | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Peter Harpending <pharpend2> |
| Component: | Current packages | Assignee: | Gentoo Linux Gnome Desktop Team <gnome> |
| Status: | UNCONFIRMED --- | ||
| Severity: | enhancement | CC: | alex_y_xu, jdavid.ibp, np-hardass, powerman-asdf, simon.vanderveldt+gentoo, steffen.weber, tl, vostorga, yesi |
| Priority: | Normal | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Attachments: | webkit-gtk-2.20.ebuild | ||
|
Description
Peter Harpending
2014-10-27 01:11:03 UTC
the binaries are upstream. Reopening.
There are many packages that depend on webkit-gtk (~50 in {,R}DEPEND). For example, GNOME and MATE desktop systems have help menus in their applications which only function with gnome-extra/yelp installed (which requires webkit-gtk). net-libs/webkit-gtk-2.10.9 for instance requires 18 GB space for compilation (in addition to the time constraints).
Requesting that this be put back for consideration, especially since the GNOME team didn't weigh in on this (at least in bugzilla) last time.
This package indeed takes unbelievably long time to build, rivalling chromium. The fact it has three slots (all of which are used on my system) doesn't make things any better. So providing a binary package as an option makes perfect sense. It's not exactly trivial to provide binary packages for something like this. It is VERY different from firefox-bin (and libreoffice-bin). It provides a stable API and ABI set of libraries for other stuff to use, not just some binary to run a single browser. All the deps would have to accept the -bin too somehow, the -bin would have to provide all the stuff necessary to build things against it, etc etc... This is particularly comlicated also due to the many USE flag options, which other packages USE depend upon - while nothing depends on firefox-bin or libreoffice-bin. So no, we have not seriously considered providing a -bin.. Note that upcoming webkit-gtk-2.20 (in March or so) will build twice as fast as 2.18, but might take more memory during build. Created attachment 526972 [details]
webkit-gtk-2.20.ebuild
can confirm the 2.20 build is almost 2x faster than previous
compile time dropped from 40-50min to ~25min
seems to work fine w/ the 1 prog i use it for...
attached my ebuild in case anyone wants to test - just added a cmake option for "-DUSE-WOFF2=OFF" (the changelog mentioned ubuntu and debian don't have it, so i assume it was off before 2.19)
(In reply to zlice from comment #6) > can confirm the 2.20 build is almost 2x faster than previous > compile time dropped from 40-50min to ~25min > > seems to work fine w/ the 1 prog i use it for... > > attached my ebuild in case anyone wants to test - just added a cmake option > for "-DUSE-WOFF2=OFF" (the changelog mentioned ubuntu and debian don't have > it, so i assume it was off before 2.19) Did you observe any trouble with RAM during compilation or whatnot? chromium[jumbo-build] ate my machine to swap hell with 8GB RAM and less than CPU count of MAKEOPTS -j (around 4 on 6-core I think I tried that), though for a small subset of the build, which otoh due to that took a lot of time percentage just due to lack of RAM. Any similar potential with the Apple's cmake based implementation here that webkit-gtk is using now? (and yes, I know webkit-gtk-2.20 needs bumped in tree, there's a security bug for it too now; ETA within a week or so, need to handle the woff2 and other stuff indeed; and research the build implications to our poor users if possible - hence the question here to whom already played and felt to comment on this bug) (In reply to Mart Raudsepp from comment #7) > Did you observe any trouble with RAM during compilation or whatnot? > chromium[jumbo-build] ate my machine to swap hell with 8GB RAM and less than > CPU count of MAKEOPTS -j (around 4 on 6-core I think I tried that), though > for a small subset of the build, which otoh due to that took a lot of time > percentage just due to lack of RAM. Any similar potential with the Apple's > cmake based implementation here that webkit-gtk is using now? > > (and yes, I know webkit-gtk-2.20 needs bumped in tree, there's a security > bug for it too now; ETA within a week or so, need to handle the woff2 and > other stuff indeed; and research the build implications to our poor users if > possible - hence the question here to whom already played and felt to > comment on this bug) nice on the version bump, i'll keep an eye out no, didnt really realize anything while compiling. i use 'vimb' with webkit-gtk and that's about it. vimb takes a few seconds to compile (after the 20+min webkit) so that's nothing like chromium. i've got 4G tmpfs on /var/tmp/portage and 2G on /tmp i've got 16GB total. any commands that find logged mem usage during emerge? 6700k (4core/8thread) MAKEOPTS="-j10" i think when i ran it o.0? if not "-j8" *** Bug 680294 has been marked as a duplicate of this bug. *** Lately, some bug reports about build failures arised. I think this was due to the maching having not enough RAM. I personally experienced multiple build failures. On some of my machines, I could build webkit-gtk with "-j1" makeopts set, but not on all. On old and/or thin hardware, it's almost impossible to build webkit-gtk nowadays. Alas, some packages I need do need it. So it would either be nice to have the possibility to do a minimalistic build with less hardware requirements or -- however this may be accomplished -- have a binary webkit-gtk. I think what's a real benefit of Gentoo is I can build it on my 14 years old notbeook or on my Raspberry Pi 1 as well as on my Xeon server. But this package is a real problem considering using Gentoo for desktop on old hardware. webkit-gtk-2.22 always used unified builds, which is basically the same as chromium[jumbo-build] as mentioned above. This is sure to take more peak memory. webkit-gtk-2.24, which I just pushed, supports disabling that, making it optional (albeit a private option and so disabling it is not really tested upstream). I've put that behind a IUSE=jumbo-build as well, instead of some new unified-build USE flag, as people are familiar with the former already. Because of upstream defaults, and it really not causing problems to people with 2-4G+ RAM, and it being on some machines even 2 times quicker build, that USE flag is default-enabled. Those machines that struggle with lower MAKEOPTS value on OOM issue may be able to successfully build again by disabling jumbo-build USE flag on it. Those machines are likely to not benefit that much from jumbo-build for build time either, and with limited memory it might actually end up being faster due to not swapping around in swap so much from big unified sources. If you have 4G or more RAM, I definitely suggest to keep jumbo-build and maybe reduce -j a bit per-package (e.g -j5 on a six-core if -j6 is too much or something), as it seems to be faster still (compared to non-unified builds with high -j), even more so if it's really hyper-threading, not full cores. Though that is old limited experience based on the change from 2.22 to 2.24, not toggling the USE flag with 2.24. I don't have the free cycles to test this much right now. Relatedly I'm curious if it unifies the build into the same amount of compilation units for everyone, or it depends on MAKEOPTS or CPU somewhat. For me jumbo-build of webkit-gtk-2.24.0 has 2814 compilation units with FEATURES=test. This can be seen during the compilation in build.log as its progress hint, e.g. [2490/2814] Without unified build it was something over 7k; that should be same for everyone. The non-unified build option had to be temporary removed again due to bug 680382 It's a joy to cross-compile webkit-gtk-3.22.x with cross-emerge, and to emerge the binary on the target later on. This however doesn't work with introspection enabled, simply because dev-libs/gobject-introspection is a mountain to climb when only attempting to cross-compile it to the rootfs needed to cross-compile webkit-gtk against it later on. It has been done in the past for 1.46.0, but if you read through the recipe you'll notice to what an extend this package is cross-compile unfriendly: http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-gnome/gobject-introspection/gobject-introspection_1.46.0.bb?h=krogoth There might be a way to compile the libs seperatly, but gobject-introspection itself acts as a compiler for typlib files (binary/library, no idea?) and therefore it is most likely a waste of time since the cross-compiled binary won't run on the host. Cross-compiling with g-ir-compiler seems to be unheard of, too. Just wanna report something to share the pain… Today I got OOM with LA above 100 because all my 24GB RAM was used (I've no swap). It turns out it happens because of building net-libs/webkit-gtk-2.24.1. Before I've started emerge I had 10GB RAM free, but looks like 10GB isn't enough today for compiling this package. Right now I've tried to build it once again, make sure there are about 20GB RAM free before starting emerge once again. This time it was build successfully. I've checked `free` output every 1 minute and noticed it has used 6GB RAM for sure, but probably once per minute isn't enough to notice peak usage, so I believe it may occasionally goes up to 9-10GB at some moments. Several months ago I had similar issue on (much weaker) my parent's computer, which was unable to build firefox because it literally take days to compile because it had not enough RAM and used a lot of swap. This. Is. Not. Acceptable. Gentoo is supposed to be able to emerge on usual workstations, but this way you guys are just killing Gentoo. You really should try to find a way around this, and I hope it won't be a binary packages - both because it's Gentoo, and because e.g. firefox-bin require pulseaudio, which isn't acceptable for many people. (In reply to Alex Efros from comment #16) > Gentoo is supposed to be able to emerge on usual workstations, but this way > you guys are just killing Gentoo. You really should try to find a way around > this, and I hope it won't be a binary packages So what do you want us to do? Become hardcore compiler developers and work on its memory usage?? The higher MAKEOPTS -j you have, the more memory it will take, but you aren't sharing those settings in your comment. I assume you are using something like -j16 to get problems with 20GB free RAM, and maybe even compiling webkit-gtk with PORTAGE_TMPDIR in tmpfs too. (In reply to Mart Raudsepp from comment #17) > The higher MAKEOPTS -j you have, the more memory it will take, but you > aren't sharing those settings in your comment. > I assume you are using something like -j16 to get problems with 20GB free RAM, MAKEOPTS="-j8" > and maybe even compiling webkit-gtk with PORTAGE_TMPDIR in tmpfs too. Yes, I'm using tmpfs for /var/tmp/portage. > So what do you want us to do? Maybe force -j4 or -j3 maximum for such packages if that helps. Maybe design /var/tmp/portage in such a way to make some packages build using another directory which shouldn't be on tmpfs. Maybe there are some other options to control emerge memory usage. Maybe change internal package testing process to run emerge with restricted amount of available memory to make sure all packages are able to build with such constraint. Then probably make it possible to speed up builds on machines with a lot of RAM using either auto-detection of amount of available RAM or option in make.conf. The problem is not how emerge handles the build, but the project itself. This is an upstream issue. I mean, what should the Gentoo Devs do? Witchcraft?! I also ran into this problem, even with -j1 set on an old notebook. Look at QtWebEngine. This one builds a huge library which is also a HTML rendering engine and it's no problem on an old/thin machine. Just takes it time. So the ones to blame are the webkit-gtk guys, not the Gentoo Devs. I think they can do nothing about the memory usage of webkit-gtk. (In reply to Alex Efros from comment #18) > (In reply to Mart Raudsepp from comment #17) > > The higher MAKEOPTS -j you have, the more memory it will take, but you > > aren't sharing those settings in your comment. > > I assume you are using something like -j16 to get problems with 20GB free RAM, > > MAKEOPTS="-j8" And CXXFLAGS? > > and maybe even compiling webkit-gtk with PORTAGE_TMPDIR in tmpfs too. > > Yes, I'm using tmpfs for /var/tmp/portage. Well, don't do that. You are taking a huge amount of RAM away from the compilation process by storing huge intermediate object files in RAM that are only used once again at the very end of the process, with no option to have them swap out due to no swap space. If you have -g, -ggdb or similar in CXXFLAGS, this may very well take 15GB on its own. > > So what do you want us to do? > > Maybe force -j4 or -j3 maximum for such packages if that helps. > Maybe design /var/tmp/portage in such a way to make some packages build > using another directory which shouldn't be on tmpfs. /var/tmp/portage is not a tmpfs unless you make it so. You get to deal with your choices. That is the result of having a choice. > Maybe there are some other options to control emerge memory usage. > Maybe change internal package testing process to run emerge with restricted > amount of available memory to make sure all packages are able to build with > such constraint. > Then probably make it possible to speed up builds on machines with a lot of > RAM using either auto-detection of amount of available RAM or option in > make.conf. How do you imagine that? But nevermine, this is not a topic for this bug, as this isn't a webkit-gtk matter (alone). (In reply to Mart Raudsepp from comment #20) > And CXXFLAGS? CFLAGS="-march=native -O2 -pipe" CXXFLAGS="-march=native -O2 -pipe" FCFLAGS="-march=native -O2 -pipe" FFLAGS="-march=native -O2 -pipe" LDFLAGS="-Wl,-O1 -Wl,--as-needed" > > Yes, I'm using tmpfs for /var/tmp/portage. > > Well, don't do that. You are taking a huge amount of RAM away from the > compilation process by storing huge intermediate object files in RAM that > are only used once again at the very end of the process, with no option to > have them swap out due to no swap space. How is it "huge"? Even large packages use no more than 2-3 GB in /var/tmp/portage, so with 16+ GB RAM it never became a problem, because IMO emerge should be able to compile any package using no more than 3-4GB RAM, that's the point. > /var/tmp/portage is not a tmpfs unless you make it so. You get to deal with > your choices. That is the result of having a choice. Sure. But if a couple packages from thousands we've in portage use unusual amount of space in /var/tmp/portage then it may make sense to take this in account in their ebuilds and provide some workaround, because many people use tmpfs on /var/tmp/portage. Actually in last couple years I get used to run on my parent's workstation: umount /var/tmp/portage && emerge firefox && mount /var/tmp/portage && emerge -uDN world if I see firefox is going to update today in `emerge -uDNav world` output, because, yes, this workstation doesn't have that much RAM to compile firefox with tmpfs on /var/tmp/portage - but all other packages compile this way just fine. I didn't whine about this case just because that workstation has only 4GB RAM and can be honestly considered not powerful enough to use tmpfs (but, once again, it's fine for everything else than firefox), but current case with 24GB RAM is different, isn't it? there *is* a workaround, it's check-reqs. package.env is a thing. |
