| Summary: | ebuilds for twisted 2.0 | ||
|---|---|---|---|
| Product: | Gentoo Linux | Reporter: | Marien Zwart (RETIRED) <marienz> |
| Component: | [OLD] Development | Assignee: | Python Gentoo Team <python> |
| Status: | RESOLVED FIXED | ||
| Severity: | enhancement | CC: | brianpankey, cmoad, farcepest, ferringb, gentoo, humpback, ppc, ste |
| Priority: | High | ||
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Bug Depends on: | 80087, 84631 | ||
| Bug Blocks: | 83182 | ||
| Attachments: | Small patch | ||
|
Description
Marien Zwart (RETIRED)
2005-02-03 14:46:48 UTC
New version available at http://mzz.mine.nu/twisted-tarballs-2.tar and http://mzz.mine.nu/twisted-overlay-2.tar.gz with the following changes: - updated svn checkout with some bugfixes. - zope-interface removed from the overlay, as it is in portage now. - licenses/Twisted added to the overlay - all ebuilds properly tabified according to gentoo standards (I hope) - minor cleanups to the twisted eclass and the main twisted ebuild - tests for twisted core and twisted-conch disabled with a RESTRICT=test because they don't work properly. New version available at http://mzz.mine.nu/twisted-overlay-3.tar.gz with the following changes: Now installs the official prerelease instead of a svn checkout. To do this I changed SRC_URI in twisted.eclass, which means this really shouldn't go into the tree like this. Not that anyone was planning on adding this prerelease to the tree anyway, right? :) Added twisted-svn.eclass and svn ebuilds for all packages. These should work, the main thing I'm not sure about is what to call them. Currently they're called twisted-web-666.ebuild (same package as 'normal' twisted-web, higher version). An alternative would be twisted-web-svn-666 (new package name). The advantage of the current scheme is you can easily install some parts of twisted from svn, while using released versions for the rest of it (with the -svn in the package name that'd mean ugly "|| ( twisted-web twisted-web-svn )" for everything depending on twisted). The disadvantage is there are not really enough masking keywords. If the live svn stuff ever makes it into the tree like this it would probably need a "stronger" mask than ~arch, which would mean -arch or package.mask, and I don't think those are "supposed" to be used for this. Comments/suggestions on this are most welcome. Some minor test-related cleanups (tests are now skipped if a needed package is missing). Some dep changes because the twisted folks fixed stuff. Thanks, twisted folks :) twisted-pair now depends on eunuchs (see bug 80087 for an ebuild). All subpackages now depend on >=dev-python/twisted-1.9 instead of twisted-2 to be able to install the "official" prerelease. Probably other things I've missed. =========================== END OF CHANGES ==================================== If you want to test this, you should probably mask the -666 live svn versions of the ebuilds in /etc/portage/package.mask. For your convenience http://mzz.mine.nu/package.mask does that (sorry for the html content-type, I should really fix that...) If you want to run the unit tests, you'll automagically run all of them with USE=test FEATURES=test *except* for twisted-conch and twisted core. The conch tests fail (hang) for me, I'm not entirely sure why but I think it's related to the weird setup the tests run in from the ebuild, so I've disabled them. The core tests rely on twisted-web, so to run those you'll need to run trial -R twisted.test yourself after installing twisted-web. I have the feeling I'm the only one using this stuff, I can't believe it's working so well nobody has anything to say :) New overlay at http://mzz.mine.nu/twisted-overlay-4.tar.gz Installs the a4 prerelease (or live svn if you use the -666 ebuilds) Now includes the eunuchs build from bug 84631, for your convenience. All tests work now, including core and conch tests. Core works because upstream fixed the web dep, conch (probably) works because I run the tests in a different way now. This means FEATURES=test emerge -va twisted-sumo now tests the entire suite. Ripped out the subpackages USE flags because they're imho not worth it. If you really like them and want them back, let me know. web2 is not packaged because it's not going to be released. svn ebuild is still there though. Probably other stuff I forgot. twisted 2.0 has been released, ebuilds are at http://mzz.mine.nu/twisted-overlay-5.tar.gz No big changes since the previous overlay, just removed some cruft from the ebuilds, replaced all >=twisted-1.9 DEPENDs with >=twisted-2, and updated the descriptions. Known issues: the core tests fail if the core is already installed. I think this is because python finds some stuff from the installed plugins directory, but this doesn't match the contents of the directory it's running tests from. I haven't thought of a good way around that one yet. So the testing procedure for this is currently: - make sure everything twisted is uninstalled - FEATURES=test emerge twisted-sumo Apart from that test issue, the only todo I know of is the missing support for the zsh completion system that was added shortly before the release. I don't understand why you marked them as version 666? The point version number is for them to accurately chronologically out-date themselves. I made my own full-package twisted package and installed with that, but if I wanted to use these ebuilds, I am only creating a nuissance for myself. When the real ebuilds get added to portage, they will precede my overlay, so I will never notice they got added/fixed/updated. It would seem more appropriate to call them twisted*-2.0.0 and let things take their natural course. Ok, my apologies, I misunderstood because I had my own ebuilds, didn't realize what the 666 deal was until I read through them. It has always been the portage way to have a *-cvs package, so it would seem most appropriate to have them be twisted-*-svn. You should do your overlay package to be like that, there is no reason people should have to mask off the svn releases. re comment #6: heh, that's what I get for writing huge unreadable gazillions of comments. Let me repeat the relevant section, with additional line breaks :) """ Currently they're called twisted-web-666.ebuild (same package as 'normal' twisted-web, higher version). An alternative would be twisted-web-svn-666 (new package name). The advantage of the current scheme is you can easily install some parts of twisted from svn, while using released versions for the rest of it (with the -svn in the package name that'd mean ugly "|| ( twisted-web twisted-web-svn )" for everything depending on twisted). The disadvantage is there are not really enough masking keywords. If the live svn stuff ever makes it into the tree like this it would probably need a "stronger" mask than ~arch, which would mean -arch or package.mask, and I don't think those are "supposed" to be used for this. Comments/suggestions on this are most welcome. """ Also don't call the alternative "the portage way" too quickly, find /usr/portage -name "*999*.ebuild" finds some more that use the "insanely high version" approach (mostly enlightenment stuff), probably for the same reason I had: to make it easier to depend on them. My current opinion is the live svn ebuilds shouldn't go into the tree at all, or if they do, they should be package.mask'd. That way they won't be "in the way" for the people that don't use them, and there's no need to change the other ebuilds to work with the live svn stuff for people that do want to use it. Comments still welcome :) Marien, I'm looking at the 2.0 release and it only mentions net-zope/zopeinterface-3.0.1 as a requirement. Can you tell me where the eunuchs requirement comes from? Also, it installs fine with just "inherit distutils", so are you still suggesting breaking twisted into separate packages, and if so why? Thanks Heh, seems to be somewhat undocumented. If you fetch the twisted pair package (from http://twistedmatrix.com/projects/pair/) and look at tuntap.py you'll see the following: """ You need Eunuchs for twisted.pair.tuntap to work. """ Actaully, half of pair works without eunuchs, so if this is really unwanted the dep can be left out. What are you installing with "inherit distutils"? The point is it's not really me who's splitting things, it's upstream. If you look at my twisted-2.0.0.ebuild you'll see it's essentially just "inherit distutils" with some sprinkles on top (for tests, documentation, and an init.d script), installing the "core" tarball from http://twistedmatrix.com/projects/core/ . *but* if you install just that you get a lot less than you'd get from twisted-1.3.0, since that included all the now-split-out packages. I guess you could also use the TwistedSumo tarball from that same page, but imho that would be a bit odd: why force people to install all subpackages or none at all if it's not necessary? It would also become annoying in the future, since the release cycles for the twisted subprojects/packages are now supposed to be decoupled, so if you offered just a single twisted-2.0.0.ebuild you'd have to revbump it whenever one of the subpackages was bumped. That sumo tarball is really "intended" for people installing manually without a package manager, afaik. I have the feeling I'm not making myself clear, so feel free to poke me for more information either here or on irc (marienz on freenode). commited twisted, twisted-web, twisted-conch, twisted-lore,.. already more to come. Created attachment 55711 [details]
Small patch
I had to change the ebuild to get it compile on my machine. Twisted depends on
zopeinterface and gives a strange error when emerging...
Just for the record, zopeinterface is really zope.interface, which is one of the components of zopex3. However, zopex3 does not currently install as a Python site-package, but instead gets it's own installation directory (/usr/lib/zopex3-3.0.0/lib/python) which is not on the normal PYTHONPATH. zopeinterface appears to install as a site-package, judging from the ebuild. In theory, zopex3 would also satisfy the dependency, but twisted can't find it. Since the design of zopex3 is to have separate components, it may make more sense to install it as a site-package. Otherwise, it could be added to the PYTHONPATH using an entry in /etc/env.d/, i.e. PYTHONPATH=/usr/lib/zopex3-3.0.0/lib/python. Good to see this make its way into portage. The current ebuilds have a number of minor issues though:
- the doc installation for the twisted subpackages doesn't work. It tries to get docs from the dir "docs", which doesn't exist (it's called "doc"). It doesn't use doman to install the man pages into the system-wide man dirs. It tries to install the docs to /usr/share/doc/${P} while the gentoo standard seems to be ${PF}. The difference is that if there's ever a twisted-words-0.1.0-r1 ${P} will be twisted-words-0.1.0 while ${PF} is twisted-words-0.1.0-r1, which means the README and friends installed by distutils_src_install will end up in a different dir than the twisted-specific docs. You won't notice for versions without -rsomething, but I think it's a good idea to stick to ${PF} for consistency.
A src_install that should work for the subpackages:
src_install() {
distutils_src_install
if [[ -d doc/man ]]; then
doman doc/man/*
fi
if use doc && [[ -d doc ]]; then
insinto /usr/share/doc/${PF}
doins -r $(find doc -mindepth 1 -maxdepth 1 -not -name man)
fi
}
- although there's no separate doc tarball and no api docs, twisted core does have docs (and manpages) that are not currently installed. A src_install that takes care of this:
src_install() {
distutils_src_install
# weird pattern to avoid installing the index.xhtml page
doman doc/man/*.?
if use doc; then
insinto /usr/share/doc/${PF}
doins -r $(find doc -mindepth 1 -maxdepth 1 -not -name man)
fi
}
where the doc/man/*.? pattern is needed because there's a doc/man/index.xhtml that shouldn't be installed as a manpage.
- some of the subpackages have dependencies on each other:
- twisted-words needs twisted-xish for twisted.words.protocols.jabber and twisted-web for twisted.words.protocols.msn and "mktap words" to work. I'm not sure if these should be "hard" dependencies or jabber and msn USE-flag dependant.
- twisted-lore depends on twisted-web.
- twisted-news depends on twisted-mail if you want to use twisted.news.database or twisted.news.tap or run the unit tests. Only twisted.news.nntp and twisted.news.news seem to work without it.
- twisted-mail "sometimes" (quote from the README) depends on twisted-names. "sometimes" seems to mean twisted.mail.relaymanager, twisted.mail.tap, and the unit tests. I can't think of an obvious USE flag to put this under.
- twisted core doesn't depend on pycrypto, only twisted-conch does.
I checked these deps by running the unit tests (trial -R twisted.something) with only "something" installed, looking at the README for the subpackages, looking at http://twistedmatrix.com/~tv/2.0-BEWARE/build-depends.dot.png (Tv is a twisted dev and debian maintainer of the twisted packages), and some grepping.
- personally I'd like it if the pyserial dependency from core was USE=serial only, since most of twisted works just fine without it. Same for pyopenssl and USE=crypt.
- since twisted-runner can be used as an inetd (I'm using it as one), would it make sense to add PROVIDE=virtual/inetd to the ebuild? Likewise for twisted-conch and virtual/ssh.
- seems like I wasn't entirely correct about pair depending on eunuchs. Although twisted.pair.tuntap does depend on eunuchs, this is not documented in the README and the unit tests don't test this. I think the dep can be left out unless upstream decides to at least document this.
If wanted I'll make patches against the current in-tree ebuilds that make the above changes.
A bunch of other ebuilds in the tree that depend on twisted should be modified because they won't work with just twisted-2. I haven't really tested these, only looked at what they try to import.
- twibber should probably depend on || ( twisted-xish <twisted-2 )
- freevo should probably depend on || ( twisted-web <twisted-2 )
- flumotion should probably depend on || ( twisted-web <twisted-2 )
- quotient needs at least twisted-web, twisted-mail, twisted-names and either twisted-runner or a patch (procutils was moved out of runner into twisted.python, so either quotient needs to look in the new place if it's using twisted-2 or it needs runner so it also shows up in the old places)
Should bugs be filed against the above-mentioned packages that they should test with the new twisted and adjust their dependencies if needed?
Some things I missed: - the license was changed, twisted and subpackages are now under the MIT license. - twisted core includes zsh completion. To install this, the following should be added to src_install: insinto /usr/share/zsh/site-functions/ doins twisted/python/_twisted_zsh_stub If needed I'll happily provide patches that make one or both of these changes. I will have a look as soon as I get some time. I intend to maybe write an eclass for twisted since it's more than one package now. Fixed the dep on zope.interface (i knew about it but forgot completely to commit it.. thanks :)) amd64, ppc and alpha devs please check if zope.interface works on your platforms and mark it ~arch + add ~arch to twisted. thanks My ebuilds in http://mzz.mine.nu/twisted-overlay-5.tar.gz use an eclass to make the twisted-subpackage ebuilds as small as possible. Twisted core doesn't use the eclass. Those ebuilds should have all the modifications from comment 13, but not the right license and zsh stuff from comment 14. If there's interest I'll put up a new tarball with those two minor things fixed. amd64 done Thomas, need a hand with any of this? In regard to naming, I think "twisted-svn-666.ebuild" and "twisted-web-svn-666.ebuild" are better than "twisted-666.ebuild". I'm writing an ebuild that depends on twisted svn. If I make it depend on >=twisted-666 later when the svn code is released I'll have to change it to >=twisted-2 Which will be making it go down several hundred versions. This seems quite illogical, I'f much rather have it change from: twisted-svn to: >=twisted-2.1 Is 666 the svn revision number? Or is it some play on 999 + antichrist or something? (In reply to comment #19) > In regard to naming, I think "twisted-svn-666.ebuild" > and "twisted-web-svn-666.ebuild" are better than "twisted-666.ebuild". My rationale for this is in comment #7. Attempt at summary: It is very common to have something (non-twisted) depend on ">=dev-python/twisted-x", and the svn ebuilds are supposed to fill that kind of dep. With the current version number scheme, this works. If you call the ebuilds "twisted-svn" then everything *depending* on twisted would have to depend on "|| ( >=dev-python/twisted-x dev-python/twisted-svn )", which would be annoying. Since twisted and twisted-svn would also have to block each other (they live in the same location) this would mean that if some ebuild "forgot" the || in their deps then that package and twisted-svn can't be installed at the same time. But this is pretty much moot, since I don't think it's likely live svn stuff like this will go in the "official" tree, and you're free to rename stuff in your overlay :) > Is 666 the svn revision number? Or is it some play on 999 + antichrist or something? It was just a (bad) joke, I should've used the number "9999" which seems to be common for this kind of thing in the tree. Oh, and if anyone wants ebuilds for twisted 2.0.1 I can roll a new tarball and/or provide patches to the last tarball (or change MY_PV yourself). http://mzz.mine.nu/twisted-overlay-6.tar.gz is up, changes: - the svn ebuilds are no longer included. I don't use them, and if nobody else does I don't intend to maintain them. If someone wants them back, let me know. - license changed from Twisted to MIT - zsh completion is installed. - twisted version bumped to 2.0.1 (and stupid hardcoded version removed from the ebuild) - twisted patches twisted.spread.jelly to include the new-style classes support that is already in svn. This shouldn't hurt anything and might help with bug 95410. - twisted-pair no longer depends on eunuchs, for reasons mentioned above. Known issue: the plugin system messes up the tests. I mostly know what's going on, but I can't think of a nice way around this. If there are other issue, please let me know. http://mzz.mine.nu/twisted-overlay-7.tar.gz is up, changes: - added ebuild for web2, which was released yesterday. - added web2 to sumo - make FEATURES=test emerge twisted work if a twisted subpackage is already installed. Also fix the Manifest in the process (forgot to update that after adding the jelly patch...) - remove eunuchs and nevow from the tar, which slipped in in the previous version. Known issues: It might be a good idea to regenerate the dropin.cache after installing a twisted subpackage. Need to figure out if this is a good idea and if it is, how to do it. I'll have a look at your latest overlay as soon as I have some more time. twisted-2.0.0 and zopeinterface are ~alpha. Re-add us if you need anything else I probably missed. Thanks, Ferdy taking over this bug as it is quite important and much requested. http://mzz.mine.nu/twisted-overlay-8.tar.gz is up, changes: - add twisted-2.1.0, with patches to make the tests run and to make zsh completion not yell about a deprecated -1 argument to head. The tests patches are from upstream svn, I've mentioned the zsh completion thing to upstream but it's not in svn (yet). - add twisted-conch-0.6.0, twisted-mail-0.2.0, twisted-names-0.2.0 and twisted-words-0.3.0. - add patches to make twisted-web2-0.1 and twisted-xish-0.1.0 tests pass with trial from twisted-2.1.0. This only affects the tests, and they still pass with 2.0.1 as well. - dropped patch for jellying new-style classes with twisted-2.0.1. Use twisted-2.1.0 if you need this. - remove hardcoded 2.0/2.1 from SRC_URI - remove gtk2 from IUSE, gtk USE flag now pulls in gtk2 - remove doc USE flag, always install docs. If someone wants this USE-based again let me know, and I'll try to figure out what the policy on this flag is. - handle dropin.cache (the plugins cache) better. This file is now regenerated after every twisted or twisted subpackage merge, and removed when twisted is removed. Should work with FEATURES=collision-protect, and should not cause attempts to rewrite the cache in normal use. Testing of this is appreciated. - lots of extra variable quoting in the ebuild and eclasses. - removed RESTRICT=primaryuri from twisted.eclass Known issues: - twisted-conch-0.5.0's tests fail with twisted-2.1.0. The "proper" way to fix this touches non-test code, and I don't think that's worth it just to make the tests pass. Use conch 0.6.0 with twisted 2.1.0, or don't run the tests. No blocking DEPEND since it only affects the tests. - twisted from svn, and therefore the next release, removes the -R option from trial, breaking twisted.eclass's src_test. Will have to add some versiondetect magic there before the next release. - when I just tried the web2 tests with twisted-2.0.1 they stalled. I'm pretty sure they worked before, even with the new patch that shouldn't affect the place where they got stuck anyway. - the way the non-subpackage twisted ebuilds deal with dropin.cache is not pretty. Other stuff: I've recreated live svn ebuilds for my own use. Can package those up if requested, although they shouldn't go in the tree. Please test and let me know if anything breaks. I've been using Marien's overlay for months, on numerous machines and architectures (x86 and amd64), without the slightest hint of trouble. First: thanks Marien for putting these together. Just upgraded to 2.1.0/.8, works great. Second: Hey portage herd, how about getting these into the official tree? version 8 of the overlay is now in the main tree, with some small changes: temporary directories used for the tests are cleaned up properly, the virtual/ssh PROVIDE in conch was removed because I'm not sure it is appropriate, and the twisted-sumo meta was removed. I'll be removing the previous ebuilds in the near future. The dependency issues from comment 13 are on my todo list. Closing this bug, please open a new one (and assign it to me) if I missed something or something doesn't work. |
