| Summary: | CPAN in Gentoo? | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Seemant Kulleen (RETIRED) <seemant> |
| Component: | Current packages | Assignee: | Nicholas Jones (RETIRED) <carpaski> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | acme, alain, baz, brianr, drobbins, jrray, kain, liquidx, lordvan, marko, Markus.Lidel, mcummings, mkeadle, pdcawley, rajiv, rendhalver, vapier |
| Priority: | High | ||
| Version: | 1.1a | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Bug Depends on: | |||
| Bug Blocks: | 4919 | ||
| Attachments: |
ebuilder.pl
Simple generated ebuild: YAML-0.35.ebuild Complicated generated ebuild: AxKit-XSP-Util-1.6.ebuild VERY PRIMITIVE NOT WORKING EXAMPLE g-cpan.pl Update to jrray's model emerges on the fly cleaned up even more (thanks baz) good version Further amends to the script [mostly aesthetic] g-cpan.pl Patch against my previous attachment. |
||
|
Description
Seemant Kulleen (RETIRED)
2002-06-06 06:41:54 UTC
mcummings, let's carry out the analysis discussion on here, so it's better documented.
This came up on the -user list this week, and it got me thinking about the
the ramifications of ebuild vs cpan. The below is a pasted email I sent to
Seemant
---*snip*---
"Problem" (not exactly, but it could get painfull after a while): When a
perl module is required for an installation, currently it requires that
a seperate ebuild file be built to support it. That ebuild file has a
finite life span since it points only to the direct download location
(generally at the primary cpan site). For perl users, this can
eventually lead to problems when modules are updated. In addition, many
modules have other dependancies, either to other perl modules or in some
cases to external programs/libraries (any of the image handling modules,
archive handling modules (tar, zip, arj, etc.), all of the secure socket
modules, etc.).
My take on a solution: Avoid the ebuild completely. Perl is
distributed with a core set of modules, including the CPAN module. The
CPAN module is an interactive and command line accessible module that
allows the user to install the current stable version of a module,
and is configurable to follow all dependancies and download them as
well. It's like portage for perl. Actually, it's a ton like portage for
perl, and when I first started using gentoo my thought was "wow, this is
like the cpan module...only no interactive shell...". The cpan module
can be invoked to start a shell environment to install modules, or it
can be a one liner install. Depending on how you have configured cpan,
i.e. whether you told it to ask or to follow dependancies, whether to
perform tests, etc., it can do everything for you without requiring
monitoring/interaction. If an ebuild has a perl component that requires
a particular module, you would send that module name (i.e.,
Bundle::LWP,Net::FTP, etc.) to the cpan module
as
perl -MCPAN -e 'install Net::FTP'
The beauty of using the CPAN module is that it will make sure your
perl installation has all of the dependancies, and it will download from
mirrors automatically. You can also use the cpan module to confirm not only
that a version of the module is installed, but which version(in case your
ebuild has a dependancy on foo-bar.1.4, not foo-
bar<=1.3).
The caveats: the cpan module must be configured at least once (it can always
be reconfigured later). This configuration includes your target mirrors to use,
favorite pager, ftp program, etc. It is easy to determine whether it has been
configured yet or not (it creates a .cpan directory in your (root's generally)
home directory where it performs all if its builds and stores module listings),
and it should be possible to create a generic, written on the fly from local
environment variables, config file (the actual config file is Configure.pm and
is located in /usr/lib/perl5/5.6.1/CPAN/).
Another caveat, and I throw this out only because it is possible, not
because I would anticipate any ebuild having a dependancy on the latest version
of cpan.pm, is that if you tell it to upgrade cpan.pm, it will also upgrade
your perl version to the latest release (and bypass the emerge -u sys-
devel/perl completely). I am only aware of it upgrading perl during an update
to the cpan module, however.
Does that make any more sense? From what I know, it should be pretty
straightforward/simple to set up an eclass to call perl -MCPAN and install the
required module, but it will require that the cpan module be configured. Would
it be better to take the java config route, i.e. inform the user that they must
perform the configuration themselves, preferably sometime just after installing
perl, or would it be better to come up with a schema to create a generic cpan
config, tell them about it, remind them that it would work better if they
reconfig'd, and then went on our merry way?
---*snip*---
One more caveat is that I am not familiar enough with the inner mechs of the
build process to address keeping portage up to date with installed
modules...though as I type this I realize you could add to the (potential)
perl_cpan eclass a test fetaure to see what the current install is...
Leon, no worries :) I'd like to hear some of your thoughts about this, in fact. CPAN in gentoo is something I've been thinking about for about 2.5 months now, but I dunno nearly enough about perl to know how to proceed. So, from what I understand from Leon's e-mail, is that CPAN does not know how to handle non-perl-module dependencies. That correct? To recap. The idea is to have a few perl modules in dev-perl (like sdl-perl and gtk-perl) and possibly anything else with a strict exernal dependencies, and then for the rest, use CPAN. Question is, how can we have CPAN interact nicely with portage? Maybe my experience has just been fortunate, but I have seen CPAN grab *non* perl dependancies as well when the dependancy in question is a C library. If you attempt to install the perl module for ssh via cpan, one of the requirements is a C math library (yeah, names escape me right now...math-pari? something like that) - CPAN.pm will include that as part of the installation. It *won't* grab gtk if you are attempting to instal perl-gtk, however - it assumes you've gotten that far already (and portage fits in nicely right there). My experience with cpan.pm has been that it will fulfill dependancies if they are either other perl modules, or internal/core libraries, but not if they are actually external programs. Hope that helps, Mike Right. I've been thinking over dependencies recently and the process where modules with strict dependencies are ebuilds and everything else is managed "magically" by CPAN seems as if it might work. Little note: I don't really like CPAN.pm. A replacement has been coded which has much better internals and is intended to have an easily-programmable backend, which we'd probably have to plug into. http://www.perl.com/pub/a/2002/03/26/cpanplus.html http://cpanplus.sourceforge.net/ http://search.cpan.org/search?dist=CPANPLUS CPANPLUS still needs some configuration when it is first installed, but I'm working with the developers to make it assume sensible defaults (they're currently on a road trip but I'll meet up with them at the O'Reilly Open Source Conference). So basically, all we need is a mechanism for portage so that when it sees a dependency like dev-perl/Acme-Buffy and it doesn't actually know about Acme::Buffy, for it to run: use strict; use CPANPLUS::Backend; my $cp = CPANPLUS::Backend->new(); $cp->install(modules => ['Acme::Buffy']); I imagine this might require some portage hacking, but I'm afraid I don't know much about that. I'm very interested in helping out. If we can get this to work nicely it'd be great. [Note that some module installs are interactive, which sucks. There isn't even a standard "please accept decent defaults". I might have to hound module authors...] So, where do I have to look to find out more about portage? ;-) I didn't include CPANPLUS in my post for only one reason: cpan.pm is part of
the stable build of perl (or, unstable ;) ), and cpanplus is an additional
module...I know, with portage et al., grabbing and installing cpanplus isn't a
biggie, was just looking at ways to do this with a minimal of extra work on the
users (or DEPEND's) part.
Other than that, cpanplus has advantages, and AFAIK (and tested) cpanplus
accepts the same command line implementation as cpan (with the bonus of being
able to turn of verbosity), i.e.
perl -MCPANPLUS -e 'install egg::foo'
Here's the requirements: 1. Anything that cpan installs must be recorded properly in the portage db, so that it may be uninstalled if desired. 2. Emerge -u world should also update installed perl packages, even those that were installed as deps by cpan. 3. ebuild perl-pkg package still needs to work. My gut instinct is that coupling cpan to portage is going to be such a pain that it isn't going to be worth while, but I could well be wrong. Ok, here's my take on CPAN. Whatever is decided upon for the perl modules and CPAN should be a nice user friendly idea. From what I've seen most gentoo users are barely adept enough to run linux, let alone gentoo, and I do include myself in that category, but I'm trying hard to learn. I have CPAN setup and have used it to install some things. How bout having ebuilds for the perl modules, but in the ebuild have it error out and tell users to use CPAN instead, kinda like a few of the other ebuild that error out and tell u to dl the files and then copy them to where they belong. Well, that my opinion. Have fun feloow gentoo-ers I thought about this on the train home and I have the odd feeling that Grant's gut instinct is right. Perhaps a better approach is to autogenerate lots of ebuild files. I'm going to try and get a program to generate correct ebuilds (with accurate dependencies) for every CPAN distribution beginning with A. This should give us a fairly good idea of how hard this is. A couple of notes I scribbled on the back of my book: there's a difference between modules and distributions (for example List::Util and Scalar::Util are both in Scalar-List-Utils). The gentoo name should be the distribution, so dev-perl/Scalar-List-Utils. Also, a lot of Perl modules come with tests and we should run the tests - they're there for a reason. Anyway, I'll let you know if generating ebuild files is feasible. Leon,
Yeah, still stuck on using perl to do the actual update, so bear with
me...what if the ebuild file pointed to cpan/cpanplus/whatever for the actual
build execution? passing the ${P} name to cpan, yada?
A is going to be huge...if it gets too big, maybe try Z (so few z's).
Just a random thought.
Mike
Leon,
Didn't think of this until I was in the car on the way home. What aboout
using the modules list file to use for comparing version and current releases?
It's all cpan does. It can be found at your favorite cpan mirror
under /pub/CPAN/*module*.tar.gz (exact name escapes me). Off the cuff, could
the virtual ebuild approach work here? Compare installed module to mod list, do
a virtual ebuild if necessary? Just a thought, will play with this in a bit,
Mike
You guys are going to love this. Right, I've just finished up a ebuilder.pl file that goes through and creates ebuilds for every module on CPAN beginning with Y. You were right: there are a lot of A's. I've tested it with the As and it takes a while ;-) OK, I've commented this but the script still has a couple of dependencies, which should be obvious from the "use"-es at the top. I've attached the script and an example dead-simple ebuild and a more complicated ebuild, all autogenerated. If you run the script it'll write out ebuilds to a directory called "ebuilds" in your current directory (for all the Ys by default, as there aren't that many). It does this by: downloading the module, extracting it, running 'perl Makefile.PL', looking in the generated Makefile and writing out an ebuild in the right place and with the correct text (hey, even a description) and dependencies. The generated ebuilds work for me. Are they in the right style? Note the DEPENDS hash, where we can add dependencies. This way, simple external library dependencies can be satisfied. I'm pretty impressed by how far this quick script got. It's generating proper ebuilds (with correct dependencies) for about 80% of the modules on CPAN. As with all magic tools, there are a couple of caveats: a) It doesn't know about external library dependencies. But we can get around this with DEPENDS: 'XML::LibXML' => 'dev-libs/libxml2' b) Some module authors get their dependencies wrong. We should tell them to fix it, but we can work around it by putting it in DEPENDS c) Some modules have interactive installs. These are very annoying. Some of them have options which use sensible defaults - we can probably hack ebuilder to use this, but otherwise we have to do the ebuild by hand (or patch the module) Cool or what? ;-) Created attachment 2089 [details]
ebuilder.pl
Created attachment 2090 [details]
Simple generated ebuild: YAML-0.35.ebuild
Created attachment 2091 [details]
Complicated generated ebuild: AxKit-XSP-Util-1.6.ebuild
Would it be possible to get a day or so to compare this to my simplistic test model (still being written)? I don't know that this is a must push out today issue (could be wrong!!!). A little concerned (perhaps unwarranted) with staticly creating ebuilds for *all* perl modules, mostly in the sense of overwhelming rsync's with a perl module list that outmatches the rest of the /usr/portage tree (unless I miss my guess, there are thousands of modules). Thanks, Mike It occurs to me that a hybrid scheme might well be in order so that if one were to do 'emerge dev-perl/some-module' and there was no ebuild file available, then portage would use ebuilder.pl (or its moral equivalent) to generate an ebuild file, which would then be used to build the module. Of course, there will probably be some 'maverick' modules that have other dependencies, these can be handled either with a custom ebuild (in the case of a 'pathological' module), or by a module specific config file for ebuilder.pl. Given that the majority of packages on CPAN build very straightforwardly, this should allow the auto building of anything from CPAN without having to rsync vast amounts of, essentially identical, ebuilds... The large number of ebuilds generated does vaguely worry me, but it's necessary to have portage handle all the CPAN modules. Otherwise, we'd get a situation like other distributions have where users install some modules using portage and then install other ones using CPAN.pm or CPANPLUS. This'd lose the advantage of using portage properly, and would probably fail Grant's requirements. I'll play some more and see if I think this will work. My take would be (and what I'm working in my oh so luxurious free time) that a hybrid would be better - ebuilds for those that absolutely demand it, but otherwise doing an "ebuild on the fly." In addition to grant's requirements, and in order to help make sure that portage is used instead of cpan directly, would be to make the portage solution "easier" than cpan/cpanplus.pm. Not only so that modules can be easily included as dependancies, but so that it is a simple matter of "emerge dev-perl/modname" and letting it go. If the module ebuild exists, build, otherwise do a virtual ebuild, and this is where that spurious note about the mirror listing would come in handy, because so long as that is maintained somewhat, you would have in a few files a listing of all the modules available (grep -v or equiv for protected modules, i.e. modules that have specific ebuilds) and available cpan mirrors to connect to (the path I gave yesterday was off, should have been mirror:/pub/CPAN/modules/*.gz). At its core, the cpan/cpanplus modules are no more complicated than your average ebuild. It grabs the file, runs the make, test, install, and tells you its done. Maintenance on a G-CPAN (making it up as I go) would be relatively low level - make sure the ./files/ has the current listings available for rsync, and tada. Gasp, this is starting to sound easier and feasible the more I write. I should stop now. Scrolling down, I realize that I've pretty much said the same thing you did in your last post, Leon ;) Update: I've been working on this during the evenings, so I thought I'd let you know what I'm up to. I've gone through the whole of CPAN and generated 2879 ebuilds and am currently in the process of QA. Unfortunately this means, for each ebuild: a) install ebuild to see if the dependencies are ok b) remove all newly installed ebuilds. This is going to take a while to run, and I've had to skip lots of Perl modules on the way which were too tricky to figure out, but at least I'm getting somewhere. Once I have a tested set of ebuilds I'll let you know. Leon Leon, all I have to say is "wow!" I've been working on this evenings as well, hope to have samples to post soon. I've taken a different approach than Leon - I'm trying to get the builds to occur on the fly so that there doesn't need to be an individual ebuild for each module, but instead a single cach all module that I'm calling gcpan for now (gentoo-cpan). Since the all the cpan script really does is perform a wget against a list of mirrors, it would seem to fit into the ebuild schema pretty well. The only thing I'm not certain about is that I think this would require a change to the emerge code for emerge requests against the dev-perl/ tree. Created attachment 2276 [details]
VERY PRIMITIVE NOT WORKING EXAMPLE
This is a very early, primitive, NON-FUNCTIONAL (I think) sample of where I am
thinking. Many bugs yet to work out, but you can get an idea from here of what
I am thinking. If you untar this and type ./g-cpan* yaml (for instance), I echo
a few of the variables before exiting (and not actually downloading and
installing). Still very preliminary/primitive (can I stress that enough?), but
I wanted to have something to show. Opinions and feedback welcome,
Mike (and yes, I know I am missing keywords like KEYWORDS, SLOT, etc. - this
isn't even close to a submission yet)
<MaxPerl> I don't know enough about the ebuild system to say. don't know why perl modules can't just be translated through to cpan install MaxPerl> emerge CGI::Cookie would translate to "perl -e'install CGI::Cookie' -MCPAN" without any other involvement We can't just translate emerge calls to CPAN.pm calls as we want proper package management (updates, unmerge etc.) BTW I'm at the O'Reilly Open Source conference next week so my work on this will be delayed for a little bit, although I'd love to meet up with anyone who will be there. Ok... here's what I had intended to do once I got around to this... Parts: 1. CPAN ebuild 1.a. Modify CPAN to sandbox itself by editing CPAN itself 2. Integrate /etc/portage/ 2.a. /etc/portage/on-update/ <-- Scripts for on-update 2.b. Add configuration/tracking of module installation into /etc/portage/perl CPAN would be setup to use Portage's directories as setup by the admin. CPAN provides variables for doing this. CPAN would sandbox itself by executing the perl makefiles inside of the sandbox, and would install into a sandboxed directory... Might requires some codeplay. Upon completion of building, sandboxed code would be md5sum'd and appended to the current perl var/db CONTENTS files so that it will be delete upon unmerge of existing versions. The module built would also get it's own CONTENTS file in the var/db tree so that individual upgrades could be dealt with. Module names would be stored into the /etc/portage/perl CPAN config/tracker for upgrade/rebuilds. The /etc/portage/on-update scripts would have to be integrated into portage to allow for CPAN to execute proper upgrades when a new CPAN version was not present. Otherwise upgrades would never occur. CPAN's codifications would also have to preclude certain modules from being updated via CPAN due to their effect of sandboxing and portage-integration. I may have more... But I'd have to update as I go... NOTE ON OTHER ATTEMPTS: 800K files in the portage tree are NOT acceptable. If you wish to continue with downloading the module info files, do that as a SRC_URI, not as a $FILES. Also, if you use an automatic generation, remember that QC on every update will need to be performed by someone. I'm not sure how the sandboxed install directory would work unless you made all of /usr/lib/perl5 a sandbox. Depending on the module, it can install to different portions of that tree. And in order to be usable (without unnecessary coding by the users), the module must be installed and 'notice' (i.e., added to perl's autoload table) sent to perl itself for it to be usable. Me thinks. As to modifying the cpan module distribution itself, that was what my (rather poor) attempt was aiming for, sort of. The cpan module requires a complete configuration before it can be used. My primitive example was to try and get the functionality of cpan without relying on the actual perl module involved. All cpan sites maintain a core of key, updated files that contain a list of valid mirrors, authors, current module packages, and their paths, that might be exploitable to some degree. Of course, I completely failed to take into consideration QC (honest mistake) for each package. What I was hoping for, naively, was a way to build perl modules on the fly, like cpan does, but with the ability to add those additions to the portage db so that gentoo hard core perl folk could have that extra bit of functionality. Of course, when I first looked into this, I was also thinking of an easier way for ebuilds that have a dep on a perl module to grab those modules - at least from a basic find/grep, none exist (yet). One caveat, no matter what the approach is, is that there would need to be a way to mask certain modules from ever being updateable via a simple ebuild. For instance, if you update the cpan module to the latest, it will go ahead and update your perl version as well, which without post-install checking of modules installed vs modules in the db, would be lost to -u. In my rather poor example code above, I have a sub that will display all installed perl modules on your system (classic textbook example). My thought was that this could be used in coordination with portage to determine versions installed in relation to available updates. I do like the idea of a hacked cpan module for gentoo, something i didn't attain though an inspiration for my attempt, but it would still require some configuration, somewhere along the line, to meet the user's needs. By placing an ebuild script around the calls to that script, I was hoping to make some of the tasks easier, not sure I did that. I think I'm babbling at this point. Feedback? CPAN provides build and install prefixing variables. Those can be exploited to sandbox CPAN. Where it installs in unimportant as we just want _what_ it installs; prefix it to the image directory after it builds. I have no idea what you mean by 'notice'. I guess I'll have to look into that at some point. >> a way to mask certain modules from ever being updateable via a simple >> ebuild. For instance, if you update the cpan module to the latest, it >> will go ahead and update your perl version as well, which without >> post-install checking of modules installed vs modules in the db, would >> be lost to -u. As I had mentioned... CPAN would not be allowed to update itself. This would be explicitly defined in the code, or perhaps optionally in the config/module file that cpan would use. >> I do like the idea of a hacked cpan module for gentoo, something i didn't >> attain though an inspiration for my attempt, but it would still require >> some configuration, somewhere along the line, to meet the user's needs. By >> placing an ebuild script around the calls to that script, I was hoping to >> make some of the tasks easier, not sure I did that. Any way you look at it, you'd have to force-build it everytime to get the updates. Portage would have no way to determine if a module needed to be rebuilt. (One exception would be the bazillion-ebuild-no-cpan method.) 'notice' was a poor choice of words on my part. What I meant was "Each time a module is installed on your system, it appends information like the following to a file called perllocal.pod which can be found in /usr/local/lib/perl5/version number/architecture/ or something akin to that. The path for your specific installation is in your @INC which you can divine with perl -V." (http://www.cpan.org/misc/cpan-faq.html#How_installed_modules) From everything that I have read so far, and all my research, here are my conclusions. Most, if not all of this, is regurgitated info... just a summary of my thoughts on the issues involved. To my understanding... A package-system (portage,rpm,deb,tgz... pick) needs to have a package built to satisfy its dependencies. However, these packages often (and quickly) fall behind their CPAN counterparts. When a developer updates a module via CPAN, it obsoletes the packaging-system info about said module. I looked at how other distributions deal with this issue. I hate having to re-invent the wheel when I can avoid it. Unfortunately, none of their solutions are elegant. In fact, they are all pretty much the same. All have the same issue we are experiencing. CPAN will always know about CPAN updates before the package-management system. We don't own it, nor were it's developers kind enough to provide call-backs for us to use. However, that doesn't stop us from mimicking it's behavior / doing it's job. Hacking CPAN to use portage, or more labor intensive (but worth it), writing our own drop-in replacement (here in called GPAN) is the only practical solution to get CPAN to play nicely with portage. Michael Cummings (mcummings@gentoo.org) in comment #22 stated he's "trying to get the builds to occur on the fly so that there doesn't need to be an individual ebuild for each module." This will avoid having 1000's of ebuilds and will create /var/db/pkg/* entries when someone updates modules via GPAN. This will allow 'CPAN' to function as it expects, and assist portage so portage can behave as we expect / need. CPAN/MyConfig.pm might be able to be used in some manner. (refer to /usr/lib/perl5/5.6.1/CPAN.pm) Everything that CPAN expects for a configuration is listed here: cpan_home, keep_source_where, build_dir, build_cache, scan_cache, index_expire, gzip, tar, unzip, make, pager, makepl_arg, make_arg, make_install_arg, urllist, inhibit_startup_message, ftp_proxy, http_proxy, no_proxy, prerequisites_policy, cache_metadata We should modify portage so when dev-cpan/*.ebuild is called, it uses GPAN to check if said module exists, and if so, check if we already have an ebuild. If we have an ebuild, use it, else generate one. We can compare the CPAN files list to the /var/db/pkg/dev-perl/* to see if an update is even necessary. GPAN can convert CPAN 'external' dependencies to 'real' ebuilds. This will take care of CPAN dependencies that are not CPAN modules. CPAN modules that have broken dependencies can be put in dev-cpan until we can get the authors of the modules to fix it. Once fixed, let GPAN deal with it if possible. Nicholas Jones (carpaski@gentoo.org) in comment #26 stated he intends to get CPAN to behave nicely with sandboxing. We can take this functionality and use it in GPAN, as it would solve that issue. Futher comments exist on this topic in comment #28. NOTE: ebuilder's functionality would be the main difference between CPAN and GPAN ::> CAVEATS "Some modules have interactive installs." This is where dev-cpan/ comes in. Create sensible defaults if possile. ebuilder.pl can not generate ebuilds for about 20% of all CPAN modules. The best we can do for now is either, enhance ebuilder.pl or create those ebuilds by hand. "CPAN has the ability to update perl itself, bypassing portage." We can not make this system fool-proof, but we sure as hell can try. There will always be a way around portage. We just have to accept this. Did I miss anything? If GPAN is the way to go, Mr. Cummings, Mr. Jones, anyone else who's interested, let me know and I'll assist you, if possible. I hate to be the one to say this (see thread if you doubt that), but I am beginning to wonder at the advantages to having a cpan interface in gentoo. I broke down and searched through all of the current ebuilds (unmasked, at least), and not one outside of the perl tree has a dependancy on any perl modules inside the tree. This is beginning to feel like a lot of hacks (old fashioned definition) for something that would only be a little bit valuable, and could potentially bloat up the portage db even more (even virtual ebuilds would need to be listed...). In the end, it would seem like the only folks that would find this good are those of that use perl daily, and we already know how to run the config tool for cpan and do a one liner install. I'd be happy to work with anyone interested in pursuing this, and would welcome encouraging words, but that's where my market shares (all two cents worth) lie right now. Michael, is it perhaps worth then to just collect a fairly current list of CPAN modules with external dependencies (ie non-perl-modules dependencies) and ebuild them into the tree? By the same token, is it worth removing the ebuilds currently in portage which do depend solely on other perl modules? Not entirely sure of the policy regarding pkg_setup() in ebuilds but if policy allows it, perhaps, just a call to cpan to actually install the perl-module dependencies that each might have. In case that is unclear, let me make a small example. Let's say we have the foo-perl module, which depends on GTK+ and also on Net::Timer and My::PerlModule. But, Net::Timer only depends on Net::Clocks, and My::PerlModule only depends on Your::PerlModule. Then, foo-perl's ebuild would just ask cpan to install My::PerlModule (thereby pulling in Your::PerlModule) and Net::Timer (pulling in Net::Clock). On the other hand, if Net::Clock or Your::PerlModule have external dependencies, we're in a web again. Bleh, I've dizzied myself. Can anyone with free time (any version of perl will do) check out the following eclass files - http://cvs.gentoo.org/~mcummings/new_perl_eclass.tgz - these are to replace the current perl-modules and perl-post eclasses and address issues with perllocal.pod getting misset or wiped of all local info. The folks on this bug list seemed the best group to target given the subject :) Mike Automatic dependency resolving by portage is very much wanted. I was thiking of installing SlashCode for some testing, ended up generating and testing a bunch of ebuilds for Perl-Modules. About the issue of bloating the portage tree: Why not generate symlinks for all identical ebuilds after rsync? Only problem beeing the $header: line. While generating a small number of ebuild scripts I ran into some problems. 1. A dependency tricked me into generating an ebuild for File:Spec unmerge of this ebuild would brake perl and module would install. 2. Modules would get installed in diffrent places breaking the Gentoo "FHS". /usr/lib/Site_Perl/<Module> was popular, however /usr/lib/Site_Perl is not included int @INC as /usr/lib/perl5/Site_Perl is the official Gentoo location (my assumption). 3. As already stated some packages required interaction during install. Very annoying. DBIx::Password in this case. It would be impossible to supply sensible defaults to this package. John *** Bug 3540 has been marked as a duplicate of this bug. *** *** Bug 9658 has been marked as a duplicate of this bug. *** I was gonna read this thread before adding this but JEEZ. I want to see my (future) wife (whom I haven't met yet) sometime! Webmin has a perl module installer that downloads the tarballs from CPAN into /tmp, builds and test them, and then cleans up after itself on install/failure. It's all dynamic, even down to perl module selection. The list of modules is grabbed from CPAN each time you use it (or you can enter the module name in directly). I've had admins use it several times with a very high success rate. Now, it doesn't keep track of the actuall files installed, so we'd need some kind of wrapper for that. But I bet there is something in there we could take a hint from! Created attachment 5443 [details] g-cpan.pl Here's yet-another-script I threw together. It leverages the CPAN.pm module. The upside is CPAN takes care of doing all the dirty work. The downside is it does a lot of dirty work. This script generates ebuilds for perl modules that don't already exist, on demand. In my test case, I want to install the Xmms::Perl module. As root you run "./g-cpan.pl Xmms::Remote". First, the script has to find out what dist contains the Xmms::Perl module, it uses CPAN to discover this can be found in "Xmms-Perl-0.12.tar.gz". Then, it uses the CPAN module to 'make' this dist. Here's the ugly part: CPAN will fetch the file and compile it. This is necessary to make CPAN figure out what this module's dependencies are. But it's convenient because now we have the file and can create a digest for it. We could also move the file to DISTFILES but I didn't do that. CPAN tells us that Xmms-Perl has a number of depends. One is 'MPEG::MP3Info', but the script figures out this is MP3-Info, and an ebuild already exists for that, so it moves on. The script moves along, recursively making ebuilds for all the depends. It ends up making an ebuild for 'Term-ReadLine-Perl', 'ANSIColor', and finally 'Xmms-Perl'. It puts these in your portage overlay directory. Finally, it helpfully suggests you 'emerge Xmms-Perl' to really install the perl module. When the script writes out the ebuilds, it fills in the DEPEND variable appropriately. Here is the ebuild it makes for Xmms-Perl: Xmms-Perl-0.12.ebuild ========================================================================= # Copyright 1999-2002 Gentoo Technologies, Inc. # Distributed under the terms of the GNU General Public License v2 inherit perl-module S=${WORKDIR}/${P} DESCRIPTION="Perl interface to the xmms_remote_* API" SRC_URI="http://www.cpan.org/modules/by-authors/id/D/DO/DOUGM/${P}.tar.gz" HOMEPAGE="http://www.cpan.org/modules/by-authors/id/D/DO/DOUGM/${P}.readme" SLOT="0" LICENSE="Artistic | GPL-2" KEYWORDS="x86 ppc alpha" DEPEND="dev-perl/MP3-Info dev-perl/Term-ReadLine-Perl dev-perl/ANSIColor" ========================================================================= I was able to emerge this successfully. Now, the script falls apart on certain modules that have elaborate config scripts, one such example is Net::SSH::Perl, it starts prompting the user to make decisions. A package like this is going to need someone to make an ebuild manually, I guess. We want to avoid a script that merely generates more ebuilds - we are trying to get away from a dev-perl tree that is huge, in favor of being able to install perl modules "on the fly". The issue at this point (as I see it) isn't with the coding of such a tool - CPAN, CPANPLUS, homebrew, webmin, what have you, all are servicable ways to do this. IMO (won't say humble, but definitely IMO) the issue that needs to be resolve first is integration with the emerge system itself. I have a bug open to discuss this, will make this bug a dep of it so you all can see it. Mike, one guy, one opinion, one track mind =:) OK, so Spanky closed the bug I was thinking of (9658, see above for link) as a duplicate of this, which sort of defeats the purpose. I could've sworn I was asked to make such a tool. We obviously don't want an ebuild for every perl module, but the advantage to using something like my script over just using CPAN is that the modules you install get in the portage db and can be uninstalled later. Sorry jrray =:) Eh, what do I know? The original thinking was to avoid extra
ebuilds - i.e., consolidate the ebuilds that aren't specific (exceptions
including the DBI modules), that really are just a perl Makefile.PL -args,
make, make install and be able to do them from a single source.
I wonder if a hack of what you did (some type of clean up afterwards to remove
the new ebuild?) would work? Presumably, once your version was done the new
perl module would be in portage's db, at which point we could do whatever we
wanted with the temporary/new ebuild that was generated.
Would still need some kind of "hook" to be able to search against the possible
modules (modules list from cpan, updated regularly?), but what you've done
would eliminate a lot of steps.
I'm also wondering (top of my head, just thought of this, kind of contradicts
the need for searches and extra hooks) if the ability to install these perl
modules shouldn't be some type of function in the emerge process that kicks off
something like your script, but that isn't an ebuild itself, kind of like an
embedded ebuild that is autogenerated as you go. For example, ebuild foo needs
perl module bar. inside that ebuild, there is a line that says something like
perl_module("bar") which calls a section of code like the one you wrote, runs
the ebuild, then proceeds with the rest of the parent ebuild.
Part of my reasoning is that my understanding as inheritor of the dev-perl tree
is that we will only add ebuilds for perl modules that satisfy deps (even if
that satisfaction is sixteen layers deep, techincally all ebuilds in dev-perl
should satisfy a dep elsewhere). So we (from this perspecitve at least) don't
need to be concerned with being able to emerge module-fooie, just that emerge
pakg-barrie can meet its perl module deps.
Just a thought. Way too much thinking on my part without enough coffee this
morning. Good to see this bug getting activity again! =:)
My understanding is that we want to ship ebuilds for perl modules that have non-perl module dependencies. An example would be the GD module that has a dep on media-libs/libgd. Currently, we would need an ebuild for the situation you describe, when an external package has a dep on a perl module. Can you come up with an example for that? Imagemagick probably should be an example, but the ebuild for that just installs the perl module itself. Michael mentioned above that he couldn't find any ebuilds that have a perl module as a dependency. An interesting way that my script could be used would be a way for an ebuild to specify a perl module dependency without having to provide the ebuild for it. Portage could generate the ebuild on demand. I don't think the ebuilds should be considered temporary and deleted after the merge. After you run my script you can 'emerge -s' for them, even if you didn't install the modules. As for providing a searchable list of perl modules, the files you can find in ~root/.cpan/sources/modules will help. 03modlist.data.gz is a list of modules and descriptions. In true geek fashion I go out for drinks and end up thinking about perl and Gentoo. Anyway, I bear my thoughts nekkid for all to see </me beats his chest>: As devs and maintainers, part of our duties include making things available and easy for end users (within boundries, of course). Ebuilds, though they serve other purposes too, are essentially automatic package installion script. A lot of the work we de involves automation. I often fall into a mindset after dealing with many ebuilds that if something can be automated it should be automated. I would be suprised if I'm the only one who slips into this. So what does this mean in reference to CPAN/Gentoo? Looking at other distros, you mostly find a static set of available/supported packages. Take Red Hat as an example: The distribution that is Red Het Linux 8.0 is a kernel and base system, some Red Het specific stuff, and pretty much just a large collection of applications. Perl modules themselves are not applications, but you'll still find rpms of many perl modules in the distribution. They are there because they are dependancies or because they are popular and widely used. Outside of that, there is so far no tool that I know of that any distro includes that provides a way to obtain extra perl modules and still contain them within it's base packaging system, rpm in the case of Red Hat. If a user wants perl modules that are not included it is up to them to look for 3rd party rpms or install them through the CPAN module directly. This has become accepted as "the way it is" with Red Hat. Gentoo, I believe, could benefit from a simple policy. If a perl module is a dependancy of an external application, there should an automated (ebuild, if not otherwise) way to install it. If it is a popular, well used module, there should also be an automated process. Outside of this it should be left up to the user. This is where my first main point comes in. It seems that we're feeling pressured, and perhaps obligated, to overextend ourselves and automate the entire CPAN archive more than is realistic. From comment #20 we see "I've gone through the whole of CPAN and generated 2879 ebuilds". How many of those would actually get used, or, more importantly, *not* used? And, as the comment also states, QA of those ebuilds is a nightmare. The aforementioned policy is meant to be a set of guidelines for us as much as it is for the end user. I do not believe there is any sustainable, maintainable, and predictable way for us to automate the whole of CPAN and place it into the confines of portage. Having said that, I'm still a believer in progress towards a worthy goal. So here is my take on a solution. First, it's based on letting the CPAN module do most of the work, and letting go of the idea that portage itself will keep track of every installed module. Second, it's built around the idea that, even if the system isn't perfect, providing the user with as much information as possible can make up for a lot. Third, I am not a programmer, at least as much as this would require, so feel free to be critical of the work involved. So here we go! Choose your interface, either ncurses, GTK, or whichever. A userspace program that lets the user see which perl modules are currently installed as well as a list of currently available modules from CPAN. The CPAN list can be called in realtime directly from CPAN while online and possibly cached to local media, letting the user choose to refresh the list at their will. Users could be able to select modules from the CPAN list that they would like to install. Once selected the program would be able to discover if the selected modules are available as ebuilds and then take them directly from portage, easily handled by externally calling emerge. If the modules are not available from portage the program would externally call perl -MCPAN -e 'install <modulename>'. Also, currently installed modules should be able to be selected for removal. It would also be desireable to show if an installed module was emerged or brought in through CPAN. As devs, our routine would not have to change much, if at all. If we add a new ebuild for a perl module, the program should be able to detect that and act accordingly. As users, they would get a pretty face and a nice way to keep track of installed modules. As Gentoo, it would be the first distribution that I know of to have a custom means to aquire allmost any perl module available. Now, as has already been discussed, not every perl module can be automated. Also, there would be little in the way of preventing users from removing important perl modules. I believe both of these points are moot. If a module is important to include in Gentoo, we, as devs, will find a way. As far as users removing important pieces of a system, there's nothing to prevent a user from unmerging the baselayout program either. At some point the user has to be held accountable, and we cannot be responsible for quick and poor judgement on their part. We are not their "Guide to Linux". 4:00 am and I need a refill on this Coke and a ciggy. So pick this appart. I'm not suggesting a best way to handle this, just another idea. Two notes - and ammendment for jrray - while when I wrote that original post there were no deps on dev-perl items outside of dev-perl, that has since changed. There are now a handful or so packages, written perl, outside of dev-perl, that do dep on dev-perl items. mkeadle - I think I agree with you. (would favor CPANPLUS over CPAN only because CPAN always crapped out on me when it attempted LWP transfers whereas CPANPLUS seems to be more intelligent about its transfers). In fact, I agree enough that if I find some free time in the next day or so I'm going to play with the curses and tk modules for perl to see if I can get a draft app together. My $.02 -- FreeBSD does this quite successfully with FreeBSD ports, they use CPAN without user interaction for both install and uninstall, to resolve dependencies on perl modules. Might be worthwhile to look at those scripts they use and see if they are adaptable. To Matt in #44: A situation I don't think you covered in your message is when you go to install a perl module with no ebuild, and that perl module depends on other perl modules that do have ebuilds. My little script handles this case, and this case needs to be handled when a perl module such as GD has external dependencies to C libraries. If you let CPAN have its merry way it will upgrade installed modules, potentially breaking compatiblity with installed libraries. I've had this happen to me with libgd. The approach I take is to use CPAN to discover available packages and discover their dependencies, but not use it to install anything. I think it is better to leave the installing part to Portage. Has anyone looked at or tried my script? It isn't very long. Gentlemen, please test jrray's script. It is almost exactly what we want. jrray,
Just a few comments/notes. These are based on using your script in
production vs as a backend perl module ebuild maker.
* In addition to checking the portage dir for an existing ebuild, the
script needs to check /var/db/pkg/dev-perl to see if it has generated/installed
an ebuild on the fly for this module before
* In addition to checking portage in general, needs to do a ExtUtils call
to see if the module has been manually installed before, and if so that the
version is >= the required version.
* Rather than writing the module to PORTAGE_OVERLAY, why not reset OVERLAY
for the duration of the script? That way, A) We can clean it out once the ebuild
is done running and B) it doesn't matter if they have one set or not :)
Obviously, my comments are geared towards using this in production vs as a
dev tool. I think I had another thought, but this is what I remember from the
drive home. Have to run now, but basicaly my line of thought was using your
script to create ebuilds that we use to satisfy a dep, then erase when we're
done - but since they were added via an ebuild, portage has a record of them
(and presumably records some stuff in /var/db/pkg).
I'll try and code a revision along these lines tonight/tomorrow if I can
and you don't object. Can't we source their make.conf and use the $ENV vars
created to set some of this? just a random thought as I hit commit...
Created attachment 6171 [details]
Update to jrray's model
Not completed. I've added checks for /var/db/pkg/dev-perl for modules installed
on the fly (the future for this script?) and for manually installed modules. I
tried using ExtUtils::Installed properly, but that was returning only partial
results, so I had to resort to a eval 'use module;'. Not pretty, but efficient.
I also commented out the overlay_dir checks and replaced them with a mapping to
/tmp/perl-modules. The idea being (per conversation on IRC) that the ebuilds
generated would be installed, then wiped. On a TO-DO list still (either when I
have a second or someone else does):
* Version checking (I have code for that, just need to see how to incorporate
it) so that if necessary, we can assure that version X of the module is
installed
* installing the module from within the script
* other stuff my brain is big enough to remember
* Clean up
Created attachment 6252 [details]
emerges on the fly
OK, this is a tested, working model. I installed a module we don't have in
portage (CGI::Application) - it autocreated the ebuild, installed the deps for
it that were in portage, then installed the module. The module shows up in a
search of portage =:) and shows up in perl as valid module =:D
As far as I know, only two things remain. I need help/time to get it to clean
up the tmp directory it makes for the ebuilds, and we need to see if this is
portable into mainstream. Oh yeah, and tons more testing, and I could really
use a code review on this because I know my coding isn't that great. Thanks
all!
Created attachment 6274 [details]
cleaned up even more (thanks baz)
Ok, this version has some added functionality. It will emerge an ebuild if you
already have one, as well as emerging it's temp ebuilds (one stop shopping).
Added in Digest::MD5 to minimize non-perl calls. Added a section for source
make.conf since there were a few other important vars in there that the script
needed to be aware of.
ToDO is still post install clean up. Anyone want to take that on? =;) and
system("rm -rf tmp_overlay_dir") is not preferred.
I'll get to it, but I wanted to post an update for those interested.
Created attachment 6286 [details]
good version
Final update for the day. This version cleans up after itself and contains a
few syntax changes. Ready for steady input folks!
Realized there is still one more "feature" to add. Ideally, this will (hopefully?) be incorporated main stream so that it can be used by ebuilds to fulfill any perl deps they have. Currently, it requires a configured CPAN on your box to run. Thinking it shouldn't be to hard to check if CPAN is configured already, and if not either auto-config it with the no option, or provide a generic config for it to use. Created attachment 6352 [details]
Further amends to the script [mostly aesthetic]
Just tidied up a little - made code (hopefully) more readable, removed some
surplus code, standardised coding methods. I'm not even a gentoo developer, so
feel free to flame me if appropriate!
Since it's a slow day on this end, I'm going to see if I can break the dep on cpan. Ideally this will be used non-interactively, which is defeated if at any point the user needs to interact with cpan for configuration. Even supplying a "no" doesn't cut it - we have users in all manner of network setups, including behind firewalls, proxies, etc. We really only use CPAN for one or two parts - getting the current verison of the ebuild (duplicable), and checking deps (also duplicable). I'm going to work based on bas's copy, so if anyone else is working on an update ping me please. Created attachment 7081 [details]
g-cpan.pl
Poking my head back in this bug, the current incarnation wasn't working for me,
so I did some tweaking and a little cleanup. Forgive me for that ugly python
bit, but it is probably the easiest way to get that done.
Absorbed into portage for 2.0.47-r1 Anybody got some docs? Quick sanity check:
> use File::Spec;
> use File::Path;
> use CPAN;
> use Digest::MD5;
These mean that we'll still need at least four perl module ebuilds in portage,
when we can get rid of the dependency on Digest::MD5 by just using eg.
$md5 = `md5sum $file_to_get_sum_for`;
or equivalent.
I'm assuming md5sum is included in the base system, if not then please ignore -
I don't have a gentoo system to hand on which to check at the moment :)
Likewise, there are methods to make this work without using the File:: routines.
Should we be removing the dependancy on these ebuilds too?
Just my
Quick sanity check:
> use File::Spec;
> use File::Path;
> use CPAN;
> use Digest::MD5;
These mean that we'll still need at least four perl module ebuilds in portage,
when we can get rid of the dependency on Digest::MD5 by just using eg.
$md5 = `md5sum $file_to_get_sum_for`;
or equivalent.
I'm assuming md5sum is included in the base system, if not then please ignore -
I don't have a gentoo system to hand on which to check at the moment :)
Likewise, there are methods to make this work without using the File:: routines.
Should we be removing the dependancy on these ebuilds too?
Just my £0.02...
The 4 modules you mention ship with Perl 5.8. Digest::MD5 doesn't ship with 5.6.1 so I can see where avoiding that may be a good idea. I'll post a patch. Created attachment 7115 [details, diff]
Patch against my previous attachment.
Can someone sanity check this please?
Dealing with the libperl_rebuilder, I get errors about modules I installed with g-cpan.pl because the ebuilds don't exist, but the script wants to re-emerge these modules. So to make the script succeed, I have to unmerge these "bad" modules, run the script, and then merge them again with g-cpan. I still question the real benefit of not leaving the generated ebuild files in a real overlay tree for later use. Otherwise portage is always going to treat these packages as orphaned. What about a compromise? If they have a portdir_overlay set, use it for the generated ebuilds, otherwise, not. What is the future of this tool, though? If it's goal is to replace the need for a dev-perl tree (as much as possible, granted) then won't this issue resolve itself at that date? If it's not, then maybe we should consider wrapping this up as an ebuild soon and just letting it go. The future is folding this functionality into Portage. First off, some combination of the source files CPAN uses as its backend database (ftp://ftp.perl.org/pub/CPAN/authors/01mailrc.txt.gz, ftp://ftp.perl.org/pub/CPAN/modules/02packages.details.txt.gz, ftp://ftp.perl.org/pub/CPAN/modules/03modlist.data.gz) will be in our portage tree somewhere. This will make all available perl modules visible to Portage, show up in emerge -[sS], etc. I think these modules should appear in a virtual category like perl-cpan/... or something. emerge can take a special arg to allow modules to be called via their module name (i.e. Some::Thing) rather than their package name (i.e. perl-cpan/some-thing). During an emerge of a virtual perl module package, things like what happen in the g-cpan.pl script will happen. I feel it wouldn't be a hefty step to avoid using CPAN.pm at this point. I used it just so I could resolve 'Some::Thing' to a package name without having to fetch and parse the aforementioned backend files. The dependency information is very easily parseable out of the Makefile.PL files. The steps taken would then be: Fetch the package, place the tarball in $(DISTFILES). Extract Makefile.PL from the tarball and parse the depends. Generate temporary ebuild and merge. This is of course a recursive process. Now we don't need to keep the temporary ebuild around because we have the database in the portage tree. Also, now portage can detect upgraded versions of modules that are installed. Portage knows about all the modules without having to generate ebuilds for them all. The only sticky part is making the modules we create by hand in dev-perl/* to mask the ones in the module database. Perhaps this is something we can do on the server side when we turn the 3 CPAN database files into something that Portage will parse. *** Bug 5474 has been marked as a duplicate of this bug. *** root@lhotse:/home/sethbc/oggasm-1.4.0# /usr/lib/portage/bin/g-cpan.pl
Bareword found where operator expected at /usr/lib/portage/bin/g-cpan.pl line 296, near ")
PORTAGE_DIR"
(Missing operator before PORTAGE_DIR?)
Can't modify constant item in scalar assignment at /usr/lib/portage/bin/g-cpan.pl line 296, near "portageq portdir_overlay)
"
syntax error at /usr/lib/portage/bin/g-cpan.pl line 296, near "portageq portdir_overlay)
"
syntax error at /usr/lib/portage/bin/g-cpan.pl line 311, near "}"
Execution of /usr/lib/portage/bin/g-cpan.pl aborted due to compilation errors.
root@lhotse:/home/sethbc/oggasm-1.4.0#
****************************
well...someone needs to brush up on perl i guess...me included :-)
Ok. Fixed in cvs for 47-r9 Please make a new bug... Reference this one. Don't reopen it. This one is way too big. |