Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 43760 - There is a relative easy way to break apart KDE apps
Summary: There is a relative easy way to break apart KDE apps
Status: VERIFIED INVALID
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: New packages (show other bugs)
Hardware: All All
: High enhancement (vote)
Assignee: Gentoo KDE team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2004-03-04 17:49 UTC by Charles Phoenix
Modified: 2004-04-13 05:27 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Charles Phoenix 2004-03-04 17:49:00 UTC
The task is easy the depth is not - the addition of about 100 ebuilds.

The magic program is cvs2dist which is part of kdesdk. From a regular tarball distribution it will create a self-contained app tarball with documentation. There is a file called 'subdirs' in the main directory of most KDE tarballs that lists the sub-directories. It should be easy to script the whole thing, less any customization (e.g., patches). -doc flag support will be be more involved; I did not explore this.

When using cvs2dist, I have learned that it is better to use the same name as the app directory during creation since certain configuration files relies on it. (during testing libkdegames would not compile under another name)

During testing I was able to break apart kbattleship and libkdegames and have them install separately.

There could be also be meta(?) ebuilds, like Arcade, Board, & Strategy.


FYI, the creation of a text file called 'inst-apps' in main source directory allows you to specify which directories (apps) to compile. It is possible to compile only the KBattleship directory. The problem is the make files has to be patched since references to libkdegames is internal not external. cvs2dist takes care of such conflicts.
Comment 1 Paul de Vrieze (RETIRED) gentoo-dev 2004-03-05 02:13:28 UTC
How does one determine automatically the dependencies between those packages. I don't say that we will do this, but it would be very helpfull if the ebuilds can be autogenerated from the sources.
Comment 2 Caleb Tennis (RETIRED) gentoo-dev 2004-03-05 04:53:42 UTC
And again, the biggest showstopper we've had for this breakup of packages is that you have to run the same configure script over and over again, substantially adding to the length of time of the ebuild.
Comment 3 Charles Phoenix 2004-03-05 07:52:53 UTC
Concerning dependenies: already made an inquiry to find out if that information is simple; i.e., like the subdirs text file. I could not find anything like that.

Another way to obtain that information is to parse the files in the debian subfolder; it has a complete breakdown for Debian setup. Especially the 'control' file.

The only thing I have not figured out is a simple way to incorporate patches.


--------------

Concerning repeating the configuring: humorous to hear that on Gentoo. Is it not the nature of the beast? My first suggestion was to give maximum flexiblity allowing custom installs. Now I find it humorous since any seperate KDE app has to go through a similar configuration step. (e.g., juk before it was added the KDE distribution). Now the difference is I am *forced* to compile juk. If users selectively block the apps they don't want (at least 50% on my system) would that not balance out the extra configuration time? Of course an article explaining this would have to be written.

To counter your complaint - user decides. I was thinking of having a seperate independant "branch" of KDE ebuilds for app builds. That is, kdegames-kbattleship (of course with a blocker to kdegames-3.2). It would have a different "root" meta ebuild - kde-apps. It would still call packages like kdelibs, no need to be that redundant. This way if I want to have removable builds for kdegames I install kdegames-apps instead of kdegames. Or I can remove kdegames and replace it with kdegames-apps. Or if I just want KBattleship, I install kdegames-kbattleship. The negative is the semi-redundant nature of adding the new sources. My second suggestion, modifying inst-apps file, elminates that issue.

The naming scheme would use a same base name to remain consistent with current KDE ebuilds and KDE modules.


Another solution/option is to have a kde-minimal meta ebuild which would build only what is necessary. This way people can add what they need - the Gentoo philosophy. There is always the option to go meta ebuild crazy: kdegames-card, kdegames-boardgames, kdegames-strategy, kdenetwork-dialup, and so on. I believe there should be some way to distinguish a meta ebuild from a regular ebuild. For example, kdegames-meta-card or kdegames-card-apps.


Another idea is to duplicate the Debian directory and have one compile broken down into numerous packages. The only problem is that would create a remove only option.
Comment 4 Charles Phoenix 2004-03-05 10:19:36 UTC
Update:

cvs2dist allows for the creation a seperate app directory.

This elminates the need to create and maintain seperate tarballs and now it is just creating ebuilds. During testing I realized that creating seperate tarballs adds about 1M of overhead to each file therefore it is more economical to download the original module tarball.

As for patches, they would be applied before using cvs2dist. Automation can be as simple as a two item text file... patchx, appx

As for the -doc flag support, it is simple... it is in the /doc subdirectory of the app directory.


What I lack is an understanding of Gentoo. 
This could be a simple template file. 
Grab module name and version from ebuild name 
	(kdegames-kbattleship-3.2 becomes kdegames-3.2)
Extract it
Patch it, if necessary
create temp dir
execute cvs2dist to extract app (kbattleship)
move to app dir (kbattleship)
configure/make/install
apply -doc flag.


There is no way to elminate the time used by configure step.

So the only problem is automating dependancies.
Comment 5 Dan Armak (RETIRED) gentoo-dev 2004-03-06 07:17:13 UTC
This is something I always wanted to do, but couldn't find any reasonable easy way to accomplish... If someone demonstrates it's possible, I'll join in the work :-)

There's another problem not yet raised here. We probably don't want to mix components from different minor versions of KDE. There might be assumptions in the code about a uniform version, at least across components of the same meta-package. It might give rise to some odd bugreports and the kde people upstream probably wouldn't be too happy about it.

Portage doesn't check for reverse dependencies, so even if X depends on =Y-1.0 you can still upgrade to Y-1.1 (or downgrade to 0.9!) without complaints. People will end up with wierdly constructed systems that way.

As for meta-ebuilds, I'd be a lot happier to have portage support for them (lists of packages, much like profiles), because they muddy up deps a lot. If kdebase-4.0 is an empty ebuild that depends on many small ebuilds for konqueror etc., then you could upgrade or unmerge those small ebuilds and portage would still consider kdebase to be installed. So if we do introduce meta-ebuilds, other ebuilds probably shouldn't depend on them...
Comment 6 Charles Phoenix 2004-03-06 10:44:08 UTC
Update 2: Possible solution to dependency issue.

Discovered that makefile.am have dependency info, not sure if it is complete

Example:
COMPILE_FIRST = libkdegames
COMPILE_AFTER_libksirtet = ksirtet kfouleggs klickety

Also there is a COMPILE_BEFORE variable in other modules

There is compiled dependency on libraries being in the default directories.
Comment 7 Charles Phoenix 2004-03-12 15:47:01 UTC
Compile times solved in a two path solution for same ebuild.

path one, caching: ~30 secs overhead - give over 50% under best circumstances
path two, skipping: ~5 secs overhead - give < 10%

most of the work will be done by an eclass and minor modifications to source

What I need is someone who knows emerge to allow emerge to call itself. Anyone interested?

Closing since only one person is showed serious interest.
Comment 8 Dan Armak (RETIRED) gentoo-dev 2004-03-13 01:16:09 UTC
Er, that one person was apparently me ;-)

So if you want more info on what Charles and I might come to do about this, contact either of us. Right now we're wrangling the issue of email, if/when we finish that I'll post a summary of anything interesting here.
Comment 9 Charles Phoenix 2004-04-13 05:27:58 UTC
For the record:

My intial statements were a tad zealous. I have almost created a solution but the problems are beyond my knowledge and progress stops at this point.

I solved the issue of time by creating a one compile-install many solution. The 'module' ebuild (1st "half") builds but does not install; control is handed over to the 'apps' ebuild (2nd "half"); it then calls the individual app ebuilds and they install their associated files and docs by referencing the module ebuild (Path 1); and finally the 'apps' ebuild cleans up. Overhead time <10%. The result is each app is now referenced separately and can be uninstalled separately.

The app ebuilds also allow for individual installs using the regular  configure-compile-install method (Path 2). This two path execution is achieved thorough a "skip" variable. It is possible to incorporate granular execution control directly into emerge through the use of control flags (e.g., skip_src_install). This would give eclass control over execution and make the ebuild transparent.

The majority of the functionality is accomplished inside eclasses that are controlled by variables.

A partial solution to configure times is the use of caching, but the problem of maintaining cache integrity has not been solved. If this idea could ever be solved it could be applied to Gentoo and improve compile time system wide.


The major problem with Path 2 is even thou breaking up the modules can be done at the moment it requires patching and regenerating the Makefiles. If "externalization(1)" could be achieved there is a possiblity of it being incoporated upstream thus removing the associated maintainence costs. Most modules can be "externalized" except for a few that reference headers in unusual way. This has caused a roadblock, especially the kdepim module (see kalarm Makefile). There are alternatives but they require sections of code to be recompiled for each app that deps on it.

Two other issues: Since gentoo does not have a package list option, re-installing and unmerging behave just like the kde meta-ebuild does now. That is, they do not work. Second, it is possible to use the original tarballs  (option 1) or break the sources into tarball sections (option 2). Option 1 requires little maintainence but requires a full download, even if only one app is to be compiled. Patching would limit what is complied. Option 2 allows less download time but then requires the creation those tarballs. Binary tarballs can be created completely or partially to increase compile times.

So yes it is possible. Yes it is possible to do a full granular module install with little additional time. Yes it is possible to do seperate installs/uninstalls of individual apps. The only major issue is "externalization" of certain modules.



(1) Exteralization is a word I use to describe the seperation of sections (subdirs) so they can be called externally. This would be done by installing to and then referencing (headers and library) from ${kdedir}.