Long Running Task in AWS Serverless

Everyone is loving Serverless. It supports more or less all the use cases, be it creating Web based application, Exposing Rest APIs with Rate limit, Event based Asynchronous processing and many more. But one thing that generally force people to think about is — Executing long running tasks in Serverless?

In almost all the Enterprise applications, there is a high probability of having a use case which needs long running tasks. For example —

1. Doing ETL on large amount of data
2. Video Processing
3. Batch Jobs
4. Log processing/scanning
5. Long running Cron Jobs, and many more

With the limitations we have with the Lambda compute service in terms of Resource and Execution Duration, what options do we have? In other words, my ask is —

1. I don’t want to manage the Infrastructure and Operational overhead
2. I want to pay only for the resources I consume
3. And important one — I want to run process for a long time, OK…more than Lambda limit of 15 minutes 😏.

Well, AWS Fargate to the rescue. AWS Fargate is a serverless compute engine for containers that works with both Amazon Elastic Container Service (ECS) and Amazon Elastic Kubernetes Service (EKS).

AWS Fargate | Serverless Compute Engine | Amazon Web Services

There is another option or I should say a workaround as well — using Lambda in a recursion way. May not be the cleanest approach but if you want to stick to the Lambda world for whatever reason, then you can give a try 😏.

Here is how it works —

1. Have your AWS Lambda function created with the business logic you want to execute as part of long running task?
2. Configure it for maximum timeout of 15 minutes (Assuming that your task will always go beyond 15 minutes)
3. Configure the trigger for the function.
4. Now the important one — Enhance your Lambda function logic such that it periodically checks the time left before the execution times out using getRemainingTimeInMillis() on the AWS Lambda context object for NodeJS. And if the task cannot be finished in the left-over time then invoke itself again passing the metadata of the task. Check this link for details — https://docs.aws.amazon.com/lambda/latest/dg/nodejs-context.html
5. Make use of S3 or EFS to store the in-process/transient data if it needs to be shared across Lambda invocations.

Do ensure that you have some kind of check in place to ensure that your Lambda function running the task does terminate eventually after running the task and does not get into an infinite loop. What damage it may cause? Well, not much, it can drain out your money apart from the business impact depending on the type of task you are executing 😏

Let’s look at a hypothetical scenario where we utilize both Lambda and Fargate to deliver the User Experience to the end users —

1. End user wants to download a list of files from S3. The files to be zipped in a single file and should be available for download.
2. User selects the files to be downloaded using the Web User Interface of the Custom Application and submits the request.
3. Backend system processes the request and sends a link to the user to the registered Email ID once the zip file is created.
4. User can select a maximum of 30 files in one request.
5. Size of a file can vary in the range of 100 MB to 500 MB.

To implement the above use case, we can have a Serverless based Architecture that looks as below —

AWS Serverless Architecture to Support Long Running Tasks

And the flow goes something like this -

1. User access the Web Application and Authenticates against the AWS Cognito Service.
2. Selects the files on the Web User Interface and submits a request for downloading the list of files.
3. API Gateway handles the request, invokes the AWS Lambda to store the data of the request in the DynamoDB and launches Fargate Task.
4. User gets response immediately that the request has been accepted.
5. While launching the Fargate Task, AWS Lambda sets the Bucket name and Request ID from the DynamoDB (which has the metadata of the request) in the Environment variables of the Task.
6. Fargate Task reads the list of files to be downloaded from the DynamoDB, gets the files from S3, saves them temporarily in the EFS file and then create a zip file.
7. Fargate task uploads the zip file back to the S3, generates a S3 signed URL and sends it the end user Email via SES service.

I know what you are thinking. This is not a full proof solution — anyone with S3 signed URL can download the zip file. Or what if the Signed URL gets expire, etc. etc. Yes, you are right, that needs to be handled. In-fact, with the Business requirement we have, Lambda configured with EFS can do the trick. We really don’t need Fargate Task. I agree, but just assume that EFS feature for Lambda is not available as of writing this blog 😏. The idea here was to show how Lambda and Fargate Task can complement each other and need not always compete.

Well, that’s it for this one. Hope you liked it.

Cheers !!!