AWS ECS

Find the video instructions on how to start an AWS free trial using ECS deployment option:

What is Fargate

AWS Fargate is a technology that you can use with Amazon ECS to run containers without having to manage servers or clusters of Amazon EC2 instances.

When you run your Amazon ECS tasks and services with the Fargate launch type or a Fargate capacity provider, you package your application in containers, specify the Operating System, CPU and memory requirements, define networking and IAM policies, and launch the application. Each Fargate task has its own isolation boundary and does not share the underlying kernel, CPU resources, memory resources, or elastic network interface with another task.

See more details here

ECS task execution can be triggered manually, from other AWS Services, e.g. AWS Step Functions, or via external orchestrators, e.g. Apache Airflow

Requirements

To run a TDK task on AWS ECS Fargate, you need to set up an ECS cluster that is able to run Fargate tasks.

The TDK ECS Task must have access to resources from AWS ECR, AWS S3, AWS Secret Manager, AWS Cloudwatch. Please make sure that the ECS Cluster network is configured properly.

Create resources using Cloudformation Template

For each transformation, you can create the ECS Task Definition using the provided AWS CloudFormation Template.

In order to perform several transformation, you can create several task definitions.

This can be done either via AWS Console or the AWS CLI

After the stack is created, the ECS Task Definition and the IAM Role are created.

Create stack using AWS Console

To create stack using AWS Console, please follow this guide

Quick link to create stack in the us-east-1 region.

Create stack using AWS CLI

To create stack using AWS CLI, please follow this guide

The example of console command:

aws cloudformation create-stack \
  --stack-name TdkTaskDefinition \
  --template-url https://synthesized-tdk-assets.s3.amazonaws.com/ecs-task-definition.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameters ParameterKey=DockerImage,ParameterValue={Your Docker Image} \
               ParameterKey=ConfigPath,ParameterValue={Your User Config Path} \
               ParameterKey=InputJdbcUrl,ParameterValue={Your Input JDBC URL} \
               ParameterKey=InputUsername,ParameterValue={Your Input DB Username} \
               ParameterKey=InputPassword,ParameterValue={Your Input DB Password} \
               ParameterKey=OutputJdbcUrl,ParameterValue={Your Output JDBC URL} \
               ParameterKey=OutputUsername,ParameterValue={Your Output DB Username} \
               ParameterKey=OutputPassword,ParameterValue={Your Output DB Password}

Run ECS Task

To run the created task via AWS Console, follow the instructions at this page

Alternatively, you can use the following console command:

aws ecs run-task  \
  --cluster {Your ECS Cluster name} \
  --task-definition {Your Task Definition Family}:1 \
  --launch-type="FARGATE" \
  --network-configuration "awsvpcConfiguration={subnets=[{Your Subnets}],securityGroups=[{Your Security groups}]}"