I'm going to take a wild guess, and assume this has been discussed already, if only informally. If a patch comes out for say, kspread for example (see bug # 11058), it's a drag to have to recompile all of koffice just for 10-15 lines of patched code in one component of koffice. I'm thinking we (the royal we) should take advantage of portage here at it's fullest and split the koffice (and other kde ports) out to smaller ports, so that when patches are available for specific apps, all the user has to do is update that port, not the whole koffice package. This also has the secondary benefit of giving the user the ability to omit certain applications they don't want. Like for example, I don't really care about karbon, kontour, or kugar. Don't need 'em, don't want 'em... but meta-ports could exist for people who just want to install everything. Of course this means there would be a lot more ports to maintain, but I think it's ultimately the better solution for users. What do you think?
This has been discussed before. Formally if you want to call it that - on the mailing lists at least. There are two ways of implementing this. Under the current portage architecture, we'd need to use separate, autonomous ebuilds for evry 'subpackage'. This however raises a problem: A kde configure script takes as much as a minute or more to run. Today when you emerge all of kde you run ~17 configure scripts, i.e. as much as 20 minutes goes there (of course everything depends on the speed of the machine). If all kde-base packages are split into separate subpackage ebuilds, you'll get hundreds of subpackags (200+). That means an extra 200 minutes -- 3.5 hours -- added to the compilation time of all of KDE. And most people do want the whole of kde. This is unacceptable. There were various proposals of caching the configuration info, or sharing it somewhere in /var/tmp. But all that is complex, nonelegant and unpleasant to implement and maintain. The other possibility is having a single ebuild provide real 'subpackages' any combination of which could be emerged by request. However there would only be a sngle workdir, so unless you deleted /var/tmp/portage in the middle of it, 'emerge kde' would run configure a minimal amount of times. However this requires new portage features: fex. keeping in /var/db/pkg the info of which subpackages are istalled, and recognizing a variables/use flag for that purpose. Someday I hope this can be done.
Dan: Thanks for the detailed reply. Perhaps I could suggest implementing a simple solution by a "subpackage" USE flag. 1. If the ebuild supports the subpackage flag, a developer would issue a "doselect" command inside of pkg_setup. "doselect" will look for an "all_packages" var, consisting of a series of targets, space-separated. 2. "doselect" prompts the user via stdin to pick the packages they want, or, if the "subpackage" flag doesn't exist, it defaults to all packages. 3. $packages gets assigned with the packages the user selected/was assigned. 4. The developer handles the compile of each package however it's needed in src_compile(). Example: pkg_setup { all_packages="kword kspread kpresenter kivio \ kontour krita kugar kchart kformula" doselect } src_compile { econf for i in $packages ; do make ${i} done } Crude example, but you get the idea. The only problems I see (which aren't insignificant) would be that it'd likely break kportage and portagemaster, due to the user input.
How about use flags for an application group that are tied to the group name, eg: Group application is koffice, specific use flages for koffice are: koffice_kword koffice_kspread koffice_kpresenter koffice_kivio koffice_kontour koffice_krita koffice_kchart koffice_kformula koffice_kugar In the ebuild if any of the koffice_* use flags are set then only those are built, if none are set then the entire application group is built. This could be extended to other ebuilds eg: ALSA or Mozilla
I'm not sure that the solution is to add a bunch of USE flags. I believe that method may be out of scope for what USE flags were intended for. Correct me if I'm wrong but I believe USE should guide which features get built in an application, not which binaries get built. Applied broadly (beyond koffice), this same method with other package bundles might result in inordinate amounts of USE flags down the road...
Here's a thought. Short-term kludge, random idea. We dep on 'dialog'... somehow check for a tty, and if it exists, present the user with a simple menu of sub-apps to select or deselect. Deselected apps will get tossed into the DO_NOT_COMPILE var, and the rest takes care of itself. Users of kportage or portagemaster will by default install everything, because they can't be interactive during the ebuild...
This looks like it needs to be recategorized into Future Portage Development...
How about this... configure each package with: --cache-file=/var/db/pkg/kde-base/config.cache And make the used KDE eclass check that /var/db/pkg/kde-base/config.cache isn't more than 24 hours old. If it is, delete it so it's regenerated. Now if the configuration process is still too slow, spend some time modifying KDE's configuration process to cache more / use more cached data. This is how Apache and other applications that have multiple configure files handle it.
It is not a problem to make an ebuild compile a subset of all available subpackages. In fact you can already do this now: export KDE_REMOVE_DIR=<list of dirs> in emerge's environment to remove those dirs from compilation. Fex. export KDE_REMOVE_DIR=kmail to get a kdenetork without kmail. This is however _not_ something intended for end-user use, but rather for an ebuild to set where appropriate. As a consequence those ebuilds that set it don't extend it (ie they overwrite your setting), and it doesn't support dirs that aren't at top level (fex. you can't use it to remove one of the plugins in kdeaddons). Portage doesn't really 'know' about it either. Again, you are **not** supposed to use it, I'm only telling you in case you really, really need a modified kde package for some reason ;-) Now the best solution IMHO would be to imlpement 'subpackages' as i described before. An ebuld could then divide the files it installs into a number of 'subpackages'. If the user requested merging certain subpackages, the ebuild would (should) be smart enough to take only the minimal action required to compile those subpackages. And if the user requested everything (as in an ordinary emerge command), the ebuild would be smart enough not to repeat actions (ie not to run configure more than once). This obviously needs portag-side support and so must come after the present (pre-1.4-release) feature freeze ends. I'll try to add such support then, meanwhie I'm dealing with bugs only...
I would be interested in seeing some, but not all, programs split up into separate packages. The concept of 200+ separate ebuilds is absurd, of course. But it is conceivable for people to want to only install konqueror. Personally I use konsole and nothing else. It would be great to only have to build konsole + depends instead of all of kdebase. So what about splitting out the most popular apps, and/or splitting kdebase (or other large bundles) into a handful of smaller bundles. For example: kdebase1 has the most popular apps, kdebase2 less so, kdebase3 contains more obscure stuff, etc. The intended goal is for each bundle to take about the same amount of time to compile. With this arrangement, I can only install the bundle that contains konsole and cut my compile time by 1/3rd.
I believe I've answered that in my previous comments. portage's housekeeping of subpackages and such could be made transparent to you (they would all live in a single .ebuild file, but provide several packages), and here would then be no performance loss.
The subpackage idea sounds good, but I'd appreciate an easy way of installing all of KDE minus apps I don't want as quickly as doing "emerge kde", so that I can specify, in a config file or environment variable, the components not to compile, or perhaps the components to compile, and then every time I do an "emerge kde" it checks this variable. Could this be done simply by extending the subpackage idea a little?
An enviroment variabile like SELECTED_APPS wouldn't be enough? if is not set, you just compile the whole package, if not you pass the proper option to the configure script to compile, or maybe the enviroment should be UNSELECTED_APPS cause usually are more the apps you want to install than the ones you don't want and then do the opposite, tell the configure script don't compile the unselcted apps
*** Bug 21041 has been marked as a duplicate of this bug. ***
Everyone is invited to join the testing at http://kde-metaebuilds.berlios.de/ - finally ebuilds that do this properly. They aren't supported by the g.o kde team, just by me and Simone (motaboy@g.o), so don't file bugreports here; use the mailing list at the berlios site. As to the various objections discussed here: the ebuilds are perfectly normal ebuilds, so they incur the overhead of running configure etc. multiple times. Things do get faster with time - not just improvements in hardware, but the configure scripts themselves seem to get a bit better; so this may be OK at some point. Also there's already an experimental patch to portage called confcache that manages central caching for configure scripts - http://dev.gentoo.org/~danarmak/ Also, if unsermake becomes stable, it'll give us a noticeable speed boost in the compile stage.
I'm glad to see this was finally realized. In retrospect, I've been using DO_NOT_COMPILE and I've been happy with it... although KDE always adds more stuff upstream between releases, so it isn't foolproof. I'd be interested to see what the difference in merge time is between the way it is now, and the ~300 ebuilds. From the standpoint of building, I'd imagine it's measurably higher, but maybe not significantly so. So the main benefit to this approach that I can see (now) is: allows the user to selectively unmerge things, without unmerging everything in a given KDE module. Also, huge benefit for somebody who maybe wants K3B or Konqueror but doesn't want anything else. I'm not sure how many people fit that use case.
The main benefit is simply that the separate packages are brought under real portage version control. You can not just install and uninstall individual apps, but specify deps between them and on them, and we can issue updates to specific packages, so that testing fixes etc becomes easier. And the demand to install individual apps is bigger than you perhaps think. A _lot_ of people want either to install only a few KDE apps, or to exclude everything they don't need just for speed's sake. Emerging kde-meta is slower than emerging kde, yes, but if you exclude the 100 or 200 programs you probably will never once use (there is a total of >300 apps in kde), it's a lot faster.
*** Bug 30215 has been marked as a duplicate of this bug. ***
> _lot_ of people want either to install only a few KDE apps, or to exclude I'm one of them, I just emerged cervisia (KDE's cvs 'browser' which I grew to love when I was using fedora and kde). I had begun emerging kdesdk, but I aborted it when I remember reading about the KDE split in a GWN (GWN 15 Nov 2004). So I fetched kde-metaebuilds-7.tar.bz2 and unpacked it in my portage overlay dir. I then simply emerged cervisia (cervisia-3.3.1, the rest of the kde ebuilds emerge is 3.3.0 atm) and it seems to work, though haven't yet tested it much yet, still got to setup my cvs on this machine (btw I'm on an amd64, it seems cervisia compiles and at least starts and exists cleanly under amd64 - should I report if it continues to work with out errors? (if so where, should I just make a regular bug-which-isn't-really-a-bug here on bugs.gentoo.org to tell it's working on amd64?) ). How about solving the split like this: (not sure if anyone has suggested this already) add an ebuild which is named something like "kde-split" which sets up an portage overlay with the splitted-kde ebuilds and then adds the PORTDIR_OVERLAY to /etc/make.conf (which should really via the "file protection magic" create /etc/._cfg000_make.conf (or something like that) and then let the user update the config as ususal, maybe make some notice on how to handle multiple portage overlay dirs, which to my uderstanding is done by PORTDIR_OVERLAY="/path/dir1 /path/dir2"). The advantages of above is that the kde-split overlay can be updated, and when the user does an "emerge sync" updates to the kde-split ebuilds will be avalible by emerging/upgrading kde-split. One dis-advantage could be possible conflicts with the kde-split ebuilds and other ebuilds in the 'main ebuild tree' - with some 300 apps I'd say Murpys law is proof alone there will be at least one name conflict. Also, this way, wouldn't it be possible to still keep the old all-of-kde ebuilds for those who want it all? >If a patch comes out for say, kspread for example (see bug # 11058), >it's a drag to have to recompile all of koffice just for 10-15 lines of >patched code in one component of koffice. > >I'm thinking we (the royal we) should take advantage of portage here at it's >fullest and split the koffice (and other kde ports) out to smaller ports, so >that when patches are available for specific apps, all the user has to do is >update that port, not the whole koffice package. this makes me think, maybe "kde-split" could contain ebuilds for that (kspread, etc and the koffice ebuild works something like fortune-mod-all), but if emerge with the use flag "minimal" eg koffice would be one big ebuild, else it could be splitted to all of it's subcomponents, or maybe that would be too hard to implement. (I must say I think the idea about "smallest upgrade only" is good)
re: part 1 of comment #18: that's possible, but the ebuilds should enter portag soon-ish anyway, so I'm not going to do it. For now the best yu can do is svn checkout our berlios.de repository and upadte once in a while. We don't increase ebuild revisions in that repo though - too lazy :-/ And yes we _are_ keeping the monolithic ebuilds in portage for now, until everyone's satisfied with the new ones.
Committed, announced, cf. http://dev.gentoo.org/~danarmak/kde-split-ebuilds.html - and I'm very proud to be able to close this bug again :-)
