Mastering Asynchronous Tasks: Celery with Django Rest Framework

We will explore the benefits of Celery with Django Rest Framework for efficiently managing tasks. Discover how to handle time-consuming processes, schedule tasks, and seamlessly integrate Celery into your Django projects to boost performance. Learn about Celery’s integration with Django and practical implementation steps to enhance your development skills. Join us on this journey to unlock the potential of building high-performance APIs with Celery’s capabilities.

Why do we need Celery?

1. Offloading Tasks

To serve the incoming requests our application server needs to be available. Consider, that if our application server is also processing the time-consuming tasks Synchronously(wait until ready), it becomes unavailable to serve more incoming requests. To overcome this we can offload these tasks by moving them outside the main request-response cycle.

Offloading the time-consuming tasks and executing them asynchronously, shorter the response time which eventually makes the user experience better.
The task that is not required to be completed before serving the response of the request is eligible or a candidate to be offloaded.

Some examples of commonly offloaded tasks are,

1. Sending emails
2. Processing images and videos
3. Data Analysis
4. Report Generation
5. Making Third-party requests

2. Running Tasks at Schedule

Consider we have to run some code/task every day or every hour which is not included in the request-response cycle. Here Celery can make our life easy by making these tasks work at regular intervals.

These tasks are executed by celery workers asynchronously(in the background) at the scheduled time.

Some examples of scheduling tasks at particular intervals are,

1. Send out trial period ending emails.
2. Regularly scrape a website and store results in the database.
3. Process a batch of data every night.
4. Report generation (creating PDF files).

🔹 Asynchronous what? Have you noticed in both cases asynchronous was common? Let’s deep dive into the concept of Asynchronous Processing.

Asynchronous Processing

• Synchronous processing is the opposite of Synchronous processing, as the client does not have to wait for a response after a request is made, and can continue other forms of processing.

• In synchronous processing, our client-server seamlessly executes the APIs in a synchronous HTTP request-response cycle. This request-response cycle is suitable for faster tasks that can be finished within milliseconds.

• However, some time-consuming tasks take more than one or two seconds to execute and are ultimately too slow for synchronous execution.
• Also as I mentioned above(Running tasks at Schedule) other tasks need to be executed in the future that can’t be included in synchronous processing.
• So, the best course of action is to move these tasks outside the execution request-response cycle.
• Simply we need to make sure that our web server notifies another program that certain tasks need to be processed at a later time.

• Now, instead of the time-consuming tasks running as a part of the actual web response, the processing runs separately so that the web application can respond quickly to the request.
• To achieve asynchronous processing, a middleman needs to be inserted between the client and the processing server to allow multiple separate processes to pass information to one another.
• The middle man can be called a message queue, which ingests the request messages and distributes them to the server based on the load the processing server can handle at any given moment.

Message Queue

• Message Queues manage the tasks that must be executed outside the usual HTTP request-response cycle.
• The message is the data to be sent from producer to consumer. Message queues don’t process messages and simply store them.
• We can consider the producer as a client application that produces messages based on user interactions and the consumers are the services that can process the arriving messages.

• So with the message queues, producers don’t have to wait for the consumers to become available and can add messages to the queue.

Also, consumers can process messages whenever they are available.

Eventually, there will be no overhead in waiting.

Till now we have enough knowledge about Asynchronous Processing and all the related stuff. Now let’s deep dive into the concept of Celery.

What is Celery?

Celery is a simple, flexible, and reliable distributed system to processes vast amounts of messages while providing operations with the tools required to maintain such a system.

It’s a task queue with a focus on real-time processing, while also supporting task scheduling.

Task queues are used as a mechanism to distribute work across threads or machines.

A Celery system can consist of multiple workers and brokers, giving way to high availability and horizontal scaling.

Celery allows you to execute background tasks asynchronously, which is especially useful for long-running or time-consuming operations that shouldn’t block the main application.

Let’s see its internal workings and workflow.

Related read: Exploring Running Background Tasks with Django Celery Using Redis

Internal Working

The internal working of Celery can be stated as the Producer/Consumer model as we discussed above.

• Producer: Producers are web services that handle web requests.

During the request processing, the time-taking tasks or the periodic tasks are pushed into the message queue.

• Queue: Celery communicates via messages, usually using a broker to mediate between clients and workers. To initiate a task the client adds a message to the queue(the broker) and then delivers that message to a worker. Here the queue can be referred to as Redis, RabbitMQ, or many other services supported by Celery.
• Consumers: Consumers are the worker nodes. The main difference between the web and worker is that web nodes receive requests from the internet and place tasks to be processed asynchronously in the queue and worker nodes are the machines that pick these tasks and execute them.

Celery-Django Workflow

In the above image, we can see the workflow of Celery and Django when the user signs up on a particular website and receives an onboarding email.

Project Description – Sending an email to everyone simultaneously.

I think you are already familiar with django and django rest framework.

My Project File Structure:

We are directly starting from celery installation.

Step 1: Celery & Redis Installation

• Install the celery package using the pip package manager.

pip3 install celery

• Verify the installed celery version by executing the following command.

celery --version

• Install Redis using the apt package manager.

sudo apt install redis -y

• Start redis-server using the following command.

sudo systemctl start redis-server.service

• Enable redis-server. This command helps to start the Redis server automatically as soon as the server is rebooted.

sudo systemctl enable redis-server.service

• Check the status of the Redis server, by running the below command.

redis-cli ping

If Redis is running successfully, it will return a PONG message.

Step 2: Celery Integration with Django

Now it’s time to integrate celery with our Django project.
Navigate to the Django core app directory.
Create a file named celery.py with the following snippet.

In the above code, we just created the Celery instance called app, and to use Celery within our project we simply import this Celery instance.

Add the below code in the __init.py__ file present in the current directory.

And run the following command to test the above configurations in the Django app directory.

celery -A myproject worker --loglevel=info

Now you need to add this in settings.py.

Step 3: Create Tasks and API

• Now we are going to create the tasks that are going to be run asynchronously.
• Tasks can be created within any app of the project under the filename tasks.py (so that celery can auto-discover them)
• Navigate to the Django project directory and create a Django app named bulkemail.
• Add the app name “bulkemail” to the settings.py file.
• Create a file named tasks.py and add the following code.

• Open models.py file and add the following model.

And run makemigrations and migrate commands.

• Now open the views.py file and add the following code.

• Open the urls.py in the current directory (multimedia app) and add the following code.

• Open the urls.py file in the Django core app directory and add the following path.

path(‘api/’, include(‘bulkemail.urls’)).

Step 3: Test API

Now run the celery worker using following command,

celery -A myproject worker --loglevel=info

After this start the django server And, then open the Postman Application and enter the request URL and choose POST Method as the request method, in the body choose JSON and add the following data into the body and hit api.

After hitting the API you will get the response but mail will be sent after 1 minute.

Related read: Leveraging Apigee Management APIs For Your Web Based Applications