 |
| ||||
It was initially created for Django, but is now pure Python-friendly.
It can also operate with other languages via HTTP+JSON.
Celery is used for executing tasks asynchronously, routed to one or more worker servers, running concurrently using multiprocessing.
The tool is designed to solve certain problems related to running websites demanding high-availability and performance.
It is perfect for filling caches, posting updates to twitter, mass downloading data like syndication feeds or web scraping. Use-cases are plentiful. Implementing these features asynchronously using celery is easy and fun, and the performance improvements can make it more than worthwhile.
Here are some key features of "Celery":
· Uses AMQP messaging (RabbitMQ, ZeroMQ, Qpid) to route tasks to the worker servers. Experimental support for STOMP (ActiveMQ) is also available.
· Can run as many worker servers as you want, and still be guaranteed that the task is only executed once.
· Tasks are executed concurrently using the Python 2.6 `multiprocessing module (also available as a back-port to older python versions)
· Supports periodic tasks, which makes it a (better) replacement for cronjobs.
· When a task has been executed, the return value can be stored using either a MySQL/Oracle/PostgreSQL/SQLite database, Memcached, MongoDB, Redis or Tokyo Tyrant back-end. For high-performance you can also use AMQP messages to publish results.
· If the task raises an exception, the exception instance is stored, instead of the return value.
· All tasks has a Universally Unique Identifier (UUID), which is the task id, used for querying task status and return values.
· Tasks can be retried if they fail, with a configurable maximum number of retries.
· Tasks can be configured to run at a specific time and date in the future (ETA) or you can set a countdown in seconds for when the task should be executed.
· Supports task-sets, which is a task consisting of several sub-tasks. You can find out how many, or if all of the sub-tasks has been executed. Excellent for progress-bar like functionality.
· Has a map like function that uses tasks, called celery.task.dmap.
· However, you rarely want to wait for these results in a web-environment. You'd rather want to use Ajax to poll the task status, which is available from a URL like celery//status/. This view returns a JSON-serialized data structure containing the task status, and the return value if completed, or exception on failure.
· The worker can collect statistics, like, how many tasks has been executed by type, and the time it took to process them. Very useful for monitoring and profiling.
· Pool workers are supervised, so if for some reason a worker crashes it is automatically replaced by a new worker.
· Can be configured to send e-mails to the administrators when a task fails.
What's New in This Release: [ read full changelog ]
· The worker is now thread-less, giving great performance improvements.
· The new "Canvas" makes it easy to define complex workflows.
· New API.
· All of Celery's command line programs are now available from a single celery umbrella command.
· This is the last version to support Python 2.5.
Via: Celery 3.0.3
0 Comment:
Post a Comment