Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 342041 - dev-python/celery (New package)
Summary: dev-python/celery (New package)
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: [OLD] Library (show other bugs)
Hardware: All Linux
: High normal with 2 votes (vote)
Assignee: Default Assignee for New Packages
URL: http://celeryq.org/
Whiteboard:
Keywords:
Depends on:
Blocks: 342043
  Show dependency tree
 
Reported: 2010-10-21 16:46 UTC by Petteri Räty (RETIRED)
Modified: 2012-05-30 06:49 UTC (History)
7 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
celery ebuild (celery-2.0.1.ebuild,2.20 KB, text/plain)
2010-10-21 18:06 UTC, Matt Summers (RETIRED)
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Petteri Räty (RETIRED) gentoo-dev 2010-10-21 16:46:51 UTC
Celery is an open source asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.
Comment 1 Matt Summers (RETIRED) gentoo-dev 2010-10-21 18:06:44 UTC
Created attachment 251475 [details]
celery ebuild
Comment 2 Matt Summers (RETIRED) gentoo-dev 2010-10-21 18:09:00 UTC
I have the init and conf for this, however they are not terribly pretty
Comment 3 Matt Summers (RETIRED) gentoo-dev 2010-10-21 18:12:16 UTC
Further, this and django-celery have several dependencies, like amqplib, carrot, importlib (if <python-2.7), pyparsing, anyjson, and django-picklefield. I have basic ebuilds for these as well.
Comment 4 Matt Summers (RETIRED) gentoo-dev 2010-10-21 18:17:42 UTC
Further, I believe that celery itself belongs in net-libs (like zeromq) or met-misc (like rabbitmq), not dev-python. </$0.02>
Comment 5 Petteri Räty (RETIRED) gentoo-dev 2010-10-21 20:16:53 UTC
(In reply to comment #4)
> Further, I believe that celery itself belongs in net-libs (like zeromq) or
> met-misc (like rabbitmq), not dev-python. </$0.02>
> 

I didn't put much thought into the selection of category so net-libs is as well fine by me.
Comment 6 Johan Bergström 2010-10-21 20:50:13 UTC
How about adding these ebuilds to an overlay somewhere so its easier to test out? 
Comment 7 Matt Summers (RETIRED) gentoo-dev 2010-10-21 21:54:05 UTC
Ok, here you go. All the deps are in here, as far as celery and django-celery
goes. We currently use the amqp backend with great success.

Please note the init.d and conf.d scripts are a mess, and not what we are using
in production. I need to get those into the repo soon. That and clean up the
celery ebuild, its a bit messy too.

http://git.overlays.gentoo.org/gitweb/?p=dev/quantumsummers.git
Comment 8 Matt Summers (RETIRED) gentoo-dev 2010-10-26 15:53:37 UTC
Updated celery to 2.1.1 in dev overlay (also carrot dep has been bumped)
Comment 9 Randy Barlow 2011-03-10 23:55:41 UTC
I'd love to see this get into portage!
Comment 10 Dirkjan Ochtman gentoo-dev 2011-09-10 13:55:48 UTC
Matt, have those cleaner init and conf files made it into the overlay yet?
Comment 11 Matt Summers (RETIRED) gentoo-dev 2011-09-13 17:03:25 UTC
I suppose I would like some feedback on what I have been doing for myself here in regards to the init.d script and associated conf.d file AND what I would like to do moving forward.

Currently, I use a slight variation on what is in the overlay for init and conf, to launch a single celeryd instance (which generally has many processes). This instance is tied to a specific invocation via django's manage.py (since that is my use case). The invocation is specified in the conf.d file, so there is little to ease the fact that its explicit (and verbose) what is being called here. When I want to run multiple instances of celeryd for different django projects (proper), I essentially copy the conf.d and init.d scripts to something like "celeryd.projectname" so that I can manage them separately. I do this simply because I do not like what the old provided example debian multiple instance script does, which now is deprecated anyway.

What I would like to know is if anyone has any bright ideas to make this multi-instance setup cleaner. What I would consider proposing is something along the lines of how the network init scripts work. So, we would have a single conf.d file where we would define each celeryd service (or instance) and init.d files symlinked to the main one. This route may just complicate things for people that really only want to run a single instance of celeryd.

Finally, I would appreciate any feedback on this. For now, I think the easy way to get this into the portage tree would be to fix up the init script for baselayout-2 (minor changes for posix compliance) and forget about the fancy multiple instance use case. For those of us that want to run more than one celeryd, I can write a little bit about how I do it, which is to simply copy things.

TIA for the feedback. I am happy to answer any questions posed either here, on IRC or on the soon to be active gentoo-python ML.
Comment 12 Corentin Chary (RETIRED) gentoo-dev 2011-12-06 07:14:15 UTC
As Matt Summers know, but other may note, I've got a set of updated ebuilds available in my tree http://git.iksaif.net/?p=portage.git;a=tree.

It comes with an enhanced init script that allow you to do that:

/etc/conf.d/celery.yourpoject:
  CELERY_USER="user"
  CELERY_GROUP="user"
  CELERY_DJANGO="yes"
  CELERY_PROJDIR="/var/lib/yourproject"
  CELERYD_ENABLED="yes"
  CELERYD_NODES="worker1 worker2"
  CELERYEV_ENABLED="yes"
  CELERYBEAT_ENABLED="yes"

And CELERY_DJANGO will do the magic to make the whole thing work with django (using manage.py, setting CELERYEV_CAM to djcelery.snapshot.Camera, etc..). It also works for non-django project of course.
Comment 13 Leho Kraav (:macmaN @lkraav) 2012-03-17 20:27:59 UTC
Corentin, I'm emerging celery-2.5.0 using your ebuild and running into python-dateutil:python-3 forcing a pull of python-3 with python-dateutil-2.0. Why are both python version dateutils in RDEPEND? Removing python-dateutil:python-3 line lets me roll on with python-2.7.
Comment 14 Corentin Chary (RETIRED) gentoo-dev 2012-03-18 21:30:11 UTC
(In reply to comment #13)
> Corentin, I'm emerging celery-2.5.0 using your ebuild and running into
> python-dateutil:python-3 forcing a pull of python-3 with
> python-dateutil-2.0. Why are both python version dateutils in RDEPEND?
> Removing python-dateutil:python-3 line lets me roll on with python-2.7.

Well, when using celery and python-3 together you need python-dateutil-2.0, when using it with python-2.7 you need python-dateutil-1.5. What's the proper way to add a dependency only if a specific version of python is installed ?
Comment 15 Leho Kraav (:macmaN @lkraav) 2012-03-19 12:15:25 UTC
If I knew I would've already proposed a solution... :> Hopefully the other bright minds here got something.
Comment 16 Corentin Chary (RETIRED) gentoo-dev 2012-04-17 09:49:26 UTC
python-dateutil issue has been solved using the new version that is compatible with both python 2 and python 3. I pushed celery 2.5.3 in my overlay yesterday.
Comment 17 Corentin Chary (RETIRED) gentoo-dev 2012-05-30 06:49:33 UTC
Ebuild commited in tree.