Long Running Task in AWS Serverless

  1. Doing ETL on large amount of data
  2. Video Processing
  3. Batch Jobs
  4. Log processing/scanning
  5. Long running Cron Jobs, and many more
  1. I don’t want to manage the Infrastructure and Operational overhead
  2. I want to pay only for the resources I consume
  3. And important one — I want to run process for a long time, OK…more than Lambda limit of 15 minutes 😏.

Well, AWS Fargate to the rescue. AWS Fargate is a serverless compute engine for containers that works with both Amazon Elastic Container Service (ECS) and Amazon Elastic Kubernetes Service (EKS).

  1. Have your AWS Lambda function created with the business logic you want to execute as part of long running task?
  2. Configure it for maximum timeout of 15 minutes (Assuming that your task will always go beyond 15 minutes)
  3. Configure the trigger for the function.
  4. Now the important one — Enhance your Lambda function logic such that it periodically checks the time left before the execution times out using getRemainingTimeInMillis() on the AWS Lambda context object for NodeJS. And if the task cannot be finished in the left-over time then invoke itself again passing the metadata of the task. Check this link for details — https://docs.aws.amazon.com/lambda/latest/dg/nodejs-context.html
  5. Make use of S3 or EFS to store the in-process/transient data if it needs to be shared across Lambda invocations.
  1. End user wants to download a list of files from S3. The files to be zipped in a single file and should be available for download.
  2. User selects the files to be downloaded using the Web User Interface of the Custom Application and submits the request.
  3. Backend system processes the request and sends a link to the user to the registered Email ID once the zip file is created.
  4. User can select a maximum of 30 files in one request.
  5. Size of a file can vary in the range of 100 MB to 500 MB.
AWS Serverless Architecture to Support Long Running Tasks
  1. User access the Web Application and Authenticates against the AWS Cognito Service.
  2. Selects the files on the Web User Interface and submits a request for downloading the list of files.
  3. API Gateway handles the request, invokes the AWS Lambda to store the data of the request in the DynamoDB and launches Fargate Task.
  4. User gets response immediately that the request has been accepted.
  5. While launching the Fargate Task, AWS Lambda sets the Bucket name and Request ID from the DynamoDB (which has the metadata of the request) in the Environment variables of the Task.
  6. Fargate Task reads the list of files to be downloaded from the DynamoDB, gets the files from S3, saves them temporarily in the EFS file and then create a zip file.
  7. Fargate task uploads the zip file back to the S3, generates a S3 signed URL and sends it the end user Email via SES service.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Naresh Waswani

Naresh Waswani

#AWS #CloudArchitect #CloudMigration #Microservices #Mobility #IoT