Long Running Task in AWS Serverless

Everyone is loving Serverless. It supports more or less all the use cases, be it creating Web based application, Exposing Rest APIs with Rate limit, Event based Asynchronous processing and many more. But one thing that generally force people to think about is — Executing long running tasks in Serverless?

In almost all the Enterprise applications, there is a high probability of having a use case which needs long running tasks. For example —

1. Doing ETL on large amount of data
2. Video Processing
3. Batch Jobs
4. Log processing/scanning
5. Long running Cron Jobs, and many more

With the limitations we have with the Lambda compute service in terms of Resource and Execution Duration, what options do we have? In other words, my ask is —

1. I don’t want to manage the Infrastructure and Operational overhead
2. I want to pay only for the resources I consume
3. And important one — I want to run process for a long time, OK…more than Lambda limit of 15 minutes 😏.

Well, AWS Fargate to the rescue. AWS Fargate is a serverless compute engine for containers that works with both Amazon Elastic Container Service (ECS) and Amazon Elastic Kubernetes Service (EKS).