Celery is an open source asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.
Created attachment 251475 [details] celery ebuild
I have the init and conf for this, however they are not terribly pretty
Further, this and django-celery have several dependencies, like amqplib, carrot, importlib (if <python-2.7), pyparsing, anyjson, and django-picklefield. I have basic ebuilds for these as well.
Further, I believe that celery itself belongs in net-libs (like zeromq) or met-misc (like rabbitmq), not dev-python. </$0.02>
(In reply to comment #4) > Further, I believe that celery itself belongs in net-libs (like zeromq) or > met-misc (like rabbitmq), not dev-python. </$0.02> > I didn't put much thought into the selection of category so net-libs is as well fine by me.
How about adding these ebuilds to an overlay somewhere so its easier to test out?
Ok, here you go. All the deps are in here, as far as celery and django-celery goes. We currently use the amqp backend with great success. Please note the init.d and conf.d scripts are a mess, and not what we are using in production. I need to get those into the repo soon. That and clean up the celery ebuild, its a bit messy too. http://git.overlays.gentoo.org/gitweb/?p=dev/quantumsummers.git
Updated celery to 2.1.1 in dev overlay (also carrot dep has been bumped)
I'd love to see this get into portage!
Matt, have those cleaner init and conf files made it into the overlay yet?
I suppose I would like some feedback on what I have been doing for myself here in regards to the init.d script and associated conf.d file AND what I would like to do moving forward. Currently, I use a slight variation on what is in the overlay for init and conf, to launch a single celeryd instance (which generally has many processes). This instance is tied to a specific invocation via django's manage.py (since that is my use case). The invocation is specified in the conf.d file, so there is little to ease the fact that its explicit (and verbose) what is being called here. When I want to run multiple instances of celeryd for different django projects (proper), I essentially copy the conf.d and init.d scripts to something like "celeryd.projectname" so that I can manage them separately. I do this simply because I do not like what the old provided example debian multiple instance script does, which now is deprecated anyway. What I would like to know is if anyone has any bright ideas to make this multi-instance setup cleaner. What I would consider proposing is something along the lines of how the network init scripts work. So, we would have a single conf.d file where we would define each celeryd service (or instance) and init.d files symlinked to the main one. This route may just complicate things for people that really only want to run a single instance of celeryd. Finally, I would appreciate any feedback on this. For now, I think the easy way to get this into the portage tree would be to fix up the init script for baselayout-2 (minor changes for posix compliance) and forget about the fancy multiple instance use case. For those of us that want to run more than one celeryd, I can write a little bit about how I do it, which is to simply copy things. TIA for the feedback. I am happy to answer any questions posed either here, on IRC or on the soon to be active gentoo-python ML.
As Matt Summers know, but other may note, I've got a set of updated ebuilds available in my tree http://git.iksaif.net/?p=portage.git;a=tree. It comes with an enhanced init script that allow you to do that: /etc/conf.d/celery.yourpoject: CELERY_USER="user" CELERY_GROUP="user" CELERY_DJANGO="yes" CELERY_PROJDIR="/var/lib/yourproject" CELERYD_ENABLED="yes" CELERYD_NODES="worker1 worker2" CELERYEV_ENABLED="yes" CELERYBEAT_ENABLED="yes" And CELERY_DJANGO will do the magic to make the whole thing work with django (using manage.py, setting CELERYEV_CAM to djcelery.snapshot.Camera, etc..). It also works for non-django project of course.
Corentin, I'm emerging celery-2.5.0 using your ebuild and running into python-dateutil:python-3 forcing a pull of python-3 with python-dateutil-2.0. Why are both python version dateutils in RDEPEND? Removing python-dateutil:python-3 line lets me roll on with python-2.7.
(In reply to comment #13) > Corentin, I'm emerging celery-2.5.0 using your ebuild and running into > python-dateutil:python-3 forcing a pull of python-3 with > python-dateutil-2.0. Why are both python version dateutils in RDEPEND? > Removing python-dateutil:python-3 line lets me roll on with python-2.7. Well, when using celery and python-3 together you need python-dateutil-2.0, when using it with python-2.7 you need python-dateutil-1.5. What's the proper way to add a dependency only if a specific version of python is installed ?
If I knew I would've already proposed a solution... :> Hopefully the other bright minds here got something.
python-dateutil issue has been solved using the new version that is compatible with both python 2 and python 3. I pushed celery 2.5.3 in my overlay yesterday.
Ebuild commited in tree.
