Bugzilla Bug 11123

I'm going to take a wild guess, and assume this has been discussed
already, if only informally.

If a patch comes out for say, kspread for example (see bug # 11058),
it's a drag to have to recompile all of koffice just for 10-15 lines of
patched code in one component of koffice.

I'm thinking we (the royal we) should take advantage of portage here at it's
fullest and split the koffice (and other kde ports) out to smaller ports, so
that when patches are available for specific apps, all the user has to do is
update that port, not the whole koffice package.

This also has the secondary benefit of giving the user the ability to omit
certain applications they don't want.  Like for example, I don't really care
about karbon, kontour, or kugar.  Don't need 'em, don't want 'em...  but
meta-ports could exist for people who just want to install everything.

Of course this means there would be a lot more ports to maintain, but I
think it's ultimately the better solution for users.

What do you think?

This has been discussed before. Formally if you want to call it that - on the
mailing 
lists at least. 

There are two ways of implementing this. 

Under the current portage architecture, we'd need to use separate, autonomous 
ebuilds for evry 'subpackage'. This however raises a problem: 

 A kde configure script takes as much as a minute or more to run. Today when
you 
emerge all of kde you run ~17 configure scripts, i.e. as much as 20 minutes
goes 
there (of course everything depends on the speed of the machine). 
If all kde-base packages are split into separate subpackage ebuilds, you'll get 
hundreds of subpackags (200+). That means an extra 200 minutes -- 3.5 hours -- 
added to the compilation time of all of KDE. And most people do want the whole
of 
kde. This is unacceptable. 

There were various proposals of caching the configuration info, or sharing it 
somewhere in /var/tmp. But all that is complex, nonelegant and unpleasant to 
implement and maintain. 

The other possibility is having a single ebuild provide real 'subpackages' any 
combination of which could be emerged by request. However there would only be a 
sngle workdir, so unless you deleted /var/tmp/portage in the middle of it,
'emerge kde' 
would run configure a minimal amount of times. 

However this requires new portage features: fex. keeping in /var/db/pkg the
info of 
which subpackages are istalled, and recognizing a variables/use flag for that 
purpose. 

Someday I hope this can be done. 

Dan:  Thanks for the detailed reply.     
     
Perhaps I could suggest implementing a simple solution by a "subpackage" USE 
flag.  
 
1.  If the ebuild supports the subpackage flag, a developer would issue a 
"doselect" command inside of pkg_setup.  "doselect" will look for an 
"all_packages" var, consisting of a series of targets, space-separated. 
2.  "doselect" prompts the user via stdin to pick the packages they want, or, 
if the "subpackage" flag doesn't exist, it defaults to all packages. 
3.  $packages gets assigned with the packages the user selected/was assigned. 
4.  The developer handles the compile of each package however it's needed in 
src_compile().  
 
Example: 
  
pkg_setup {  
	all_packages="kword kspread kpresenter kivio \ 
	kontour krita kugar kchart kformula" 
	doselect 
}     
 
src_compile { 
	econf 
	for i in $packages ; do 
		make ${i} 
	done 
} 
 
Crude example, but you get the idea.  The only problems I see (which aren't 
insignificant) would be that it'd likely break kportage and portagemaster, due 
to the user input. 

How about use flags for an application group that are tied to the group name,
eg:

Group application is koffice, specific use flages for koffice are:
   koffice_kword       koffice_kspread 
   koffice_kpresenter  koffice_kivio
   koffice_kontour     koffice_krita
   koffice_kchart      koffice_kformula
   koffice_kugar

In the ebuild if any of the koffice_* use flags are set then only those are
built, if none are set then the entire application group is built.

This could be extended to other ebuilds eg: ALSA or Mozilla

I'm not sure that the solution is to add a bunch of USE flags.  I believe that 
method may be out of scope for what USE flags were intended for.  Correct me 
if I'm wrong but I believe USE should guide which features get built in an 
application, not which binaries get built. 
 
Applied broadly (beyond koffice), this same method with other package bundles 
might result in inordinate amounts of USE flags down the road... 
 
 
 
 

Here's a thought.  Short-term kludge, random idea. 
 
We dep on 'dialog'... somehow check for a tty, and if it exists, present the user with 
a simple menu of sub-apps to select or deselect.  Deselected apps will get tossed 
into the DO_NOT_COMPILE var, and the rest takes care of itself. 
 
Users of kportage or portagemaster will by default install everything, because they 
can't be interactive during the ebuild... 

This looks like it needs to be recategorized into Future Portage Development...

How about this...   configure each package with:

  --cache-file=/var/db/pkg/kde-base/config.cache

And make the used KDE eclass check that /var/db/pkg/kde-base/config.cache isn't
more than 24 hours old.  If it is, delete it so it's regenerated.

Now if the configuration process is still too slow, spend some time modifying
KDE's configuration process to cache more / use more cached data.  This is how
Apache and other applications that have multiple configure files handle it.

It is not a problem to make an ebuild compile a subset of all available
subpackages. In fact you 
can already do this now: export KDE_REMOVE_DIR=<list of dirs> in emerge's
environment to 
remove those dirs from compilation. Fex. export KDE_REMOVE_DIR=kmail to get a
kdenetork 
without kmail. 

This is however _not_ something intended for end-user use, but rather for an
ebuild to set 
where appropriate. As a consequence those ebuilds that set it don't extend it
(ie they overwrite 
your setting), and it doesn't support dirs that aren't at top level (fex. you
can't use it to 
remove one of the plugins in kdeaddons). Portage doesn't really 'know' about it
either. Again, 
you are **not** supposed to use it, I'm only telling you in case you really,
really need a 
modified kde package for some reason ;-) 

Now the best solution IMHO would be to imlpement 'subpackages' as i described
before. An 
ebuld could then divide the files it installs into a number of 'subpackages'.
If the user requested 
merging certain subpackages, the ebuild would (should) be smart enough to take
only the 
minimal action required to compile those subpackages. And if the user requested
everything 
(as in an ordinary emerge command), the ebuild would be smart enough not to
repeat actions 
(ie not to run configure more than once). 

This obviously needs portag-side support and so must come after the present 
(pre-1.4-release) feature freeze ends. I'll try to add such support then,
meanwhie I'm dealing 
with bugs only... 

I would be interested in seeing some, but not all, programs split up into
separate packages.  The concept of 200+ separate ebuilds is absurd, of course. 
But it is conceivable for people to want to only install konqueror.  Personally
I use konsole and nothing else.  It would be great to only have to build
konsole + depends instead of all of kdebase.

So what about splitting out the most popular apps, and/or splitting kdebase (or
other large bundles) into a handful of smaller bundles.  For example: kdebase1
has the most popular apps, kdebase2 less so, kdebase3 contains more obscure
stuff, etc.  The intended goal is for each bundle to take about the same amount
of time to compile.

With this arrangement, I can only install the bundle that contains konsole and
cut my compile time by 1/3rd.

I believe I've answered that in my previous comments. portage's housekeeping of 
subpackages and such could be made transparent to you (they would all live in a
single .ebuild 
file, but provide several packages), and here would then be no performance
loss. 

The subpackage idea sounds good, but I'd appreciate an easy way of 
installing all of KDE minus apps I don't want as quickly as doing "emerge 
kde", so that I can specify, in a config file or environment variable, the 
components not to compile, or perhaps the components to compile, and 
then every time I do an "emerge kde" it checks this variable. Could this be 
done simply by extending the subpackage idea a little? 

An enviroment variabile like SELECTED_APPS wouldn't be enough? if is not set, 
you just compile the whole package, if not you pass the proper option to the 
configure script to compile, or maybe the enviroment should be 
UNSELECTED_APPS cause usually are more the apps you want to install than the 
ones you don't want and then do the opposite, tell the configure script don't compile 
the unselcted apps 

*** Bug 21041 has been marked as a duplicate of this bug. ***

Everyone is invited to join the testing at http://kde-metaebuilds.berlios.de/ 
- finally ebuilds that do this properly.
They aren't supported by the g.o kde team, just by me and Simone (motaboy@g.o),
so don't file bugreports here; use the mailing list at the berlios site.
As to the various objections discussed here: the ebuilds are perfectly normal
ebuilds, so they incur the overhead of running configure etc. multiple times.
Things do get faster with time - not just improvements in hardware, but the
configure scripts themselves seem to get a bit better; so this may be OK at
some point. Also there's already an experimental patch to portage called
confcache that manages central caching for configure scripts - 
http://dev.gentoo.org/~danarmak/ Also, if unsermake becomes stable, it'll 
give us a noticeable speed boost in the compile stage.

I'm glad to see this was finally realized.  In retrospect, I've been using
DO_NOT_COMPILE and I've been happy with it... although KDE always adds more
stuff upstream between releases, so it isn't foolproof.

I'd be interested to see what the difference in merge time is between the way
it is now, and the ~300 ebuilds.  From the standpoint of building, I'd imagine
it's measurably higher, but maybe not significantly so.

So the main benefit to this approach that I can see (now) is:  allows the user
to selectively unmerge things, without unmerging everything in a given KDE
module.  Also, huge benefit for somebody who maybe wants K3B or Konqueror but
doesn't want anything else.  I'm not sure how many people fit that use case.

The main benefit is simply that the separate packages are brought under real
portage version control. You can not just install and uninstall individual
apps, but specify deps between them and on them, and we can issue updates
to specific packages, so that testing fixes etc becomes easier.
And the demand to install individual apps is bigger than you perhaps think.
A _lot_ of people want either to install only a few KDE apps, or to exclude
everything they don't need just for speed's sake. Emerging kde-meta is slower
than emerging kde, yes, but if you exclude the 100 or 200 programs you probably
will never once use (there is a total of >300 apps in kde), it's a lot faster.

*** Bug 30215 has been marked as a duplicate of this bug. ***

> _lot_ of people want either to install only a few KDE apps, or to exclude

I'm one of them, I just emerged cervisia (KDE's cvs 'browser' which I grew to love when I was using fedora and kde). I had begun emerging kdesdk, but I aborted it when I remember reading about the KDE split in a GWN (GWN 15 Nov 2004). So I fetched kde-metaebuilds-7.tar.bz2 and unpacked it in my portage overlay dir. I then simply emerged cervisia (cervisia-3.3.1, the rest of the kde ebuilds emerge is 3.3.0 atm) and it seems to work, though haven't yet tested it much yet, still got to setup my cvs on this machine (btw I'm on an amd64, it seems cervisia compiles and at least starts and exists cleanly under amd64 - should I report if it continues to work with out errors? (if so where, should I just make a regular bug-which-isn't-really-a-bug here on bugs.gentoo.org to tell it's working on amd64?) ).

How about solving the split like this: (not sure if anyone has suggested this already)
add an ebuild which is named something like "kde-split" which sets up an portage overlay with the splitted-kde ebuilds and then adds the PORTDIR_OVERLAY to /etc/make.conf (which should really via the "file protection magic" create /etc/._cfg000_make.conf (or something like that) and then let the user update the config as ususal, maybe make some notice on how to handle multiple portage overlay dirs, which to my uderstanding is done by PORTDIR_OVERLAY="/path/dir1 /path/dir2").
The advantages of above is that the kde-split overlay can be updated, and when the user does an "emerge sync" updates to the kde-split ebuilds will be avalible by emerging/upgrading kde-split. 
One dis-advantage could be possible conflicts with the kde-split ebuilds and other ebuilds in the 'main ebuild tree' - with some 300 apps I'd say Murpys law is proof alone there will be at least one name conflict.
Also, this way, wouldn't it be possible to still keep the old all-of-kde ebuilds for those who want it all?


>If a patch comes out for say, kspread for example (see bug # 11058),
>it's a drag to have to recompile all of koffice just for 10-15 lines of
>patched code in one component of koffice.
>
>I'm thinking we (the royal we) should take advantage of portage here at it's
>fullest and split the koffice (and other kde ports) out to smaller ports, so
>that when patches are available for specific apps, all the user has to do is
>update that port, not the whole koffice package.

this makes me think, maybe "kde-split" could contain ebuilds for that (kspread, etc and the koffice ebuild works something like fortune-mod-all), but if emerge with the use flag "minimal" eg koffice would be one big ebuild, else it could be splitted to all of it's subcomponents, or maybe that would be too hard to implement. (I must say I think the idea about "smallest upgrade only" is good)

re: part 1 of comment #18: that's possible, but the ebuilds should enter portag
soon-ish anyway, so I'm not going to do it. For now the best yu can do is svn
checkout our berlios.de repository and upadte once in a while. We don't
increase ebuild revisions in that repo though - too lazy :-/

And yes we _are_ keeping the monolithic ebuilds in portage for now, until
everyone's satisfied with the new ones.

Committed, announced, cf.
http://dev.gentoo.org/~danarmak/kde-split-ebuilds.html - and I'm very proud to
be able to close this bug again :-)