AWS/Terraform Workshop #6: EC2 Container Service, AWS Lambda

This post is part of our AWS/Terraform Workshops series that explores our vision for Service Oriented Architecture (SOA), and closely examines AWS Simple Storage Service, Terraform Remote State, and Identity Access Management. To learn more, check out our introductory workshop and new posts at Smartling Engineering Blog.

Prerequisites

• Go through workshops #1, #2, #3, #5
• Docker Introduction
• AWS EC2 Container Service Introduction

Preface

AWS EC2 Container Service (Amazon ECS)

Amazon ECS is a highly scalable, fast, container management service that makes it easy to run, stop, and manage Docker containers on a cluster of Amazon EC2 instances. It lets you launch and stop container-based applications with simple API calls, allows you to get the state of your cluster from a centralized service, and gives you access to many familiar Amazon EC2 features.

• Cluster: A logical grouping of container instances on which you can place tasks.
• Container instance: EC2 instance that is running the Amazon ECS agent and has been registered into a cluster.
• Task definition: A description of an application that contains one or more container definitions.
• Task: An instantiation of a task definition that is running on a container instance.
• Service: ECS service allows you to run a specified number of tasks simultaneously. If any of the tasks fails or stops for any reason (e.g. OOM), ECS service scheduler re-launches the failed task.

It’s possible to attach Elastic Load Balancer (ELB) to ECS service to distribute traffic across the multiple instances of the service.

You can update a running service to change the number of tasks (desired count) and/or definition for the tasks in ECS service.

Read more:

• AWS ECS Clusters
• AWS ECS Container Instances
• AWS ECS Services

AWS Lambda for synchronization of Auto Scale group and ECS

In this workshop, we will setup AWS Lambda function that will sync a desired count of tasks in ECS service depending on the number of EC2 instances in the Auto Scaling group. This is now natively supported by AWS, therefore, mentioning the Lambda function may be used for learning purposes.

Note: We recommend using AWS ECS service Auto Scaling in production environments instead of the Lambda function presented in this workshop.

Hands On

1. Create EC2 instance with IAM role with full access to AWS.

Note: there are no intentional mistakes or confusions at this step. But next sections in this workshop still contain some so you'll need to fix them.
a. Go to w6/workshop_part_1 directory in cloned Smartling/aws-terraform-workshops git repository.
b. Configure terraform to create IAM role, security group and EC2 instance.
c. Make sure that user data for EC2 instance contains your SSH key.
d. Apply terraform configuration:
$ terraform plan
$ terraform apply
e. Login to newly created EC2 instance via SSH.
f. Checkout git repository with terraform workshops.
$ git clone https://github.com/Smartling/aws-terraform-workshops
$ cd w6/workshop_part_2

2. Create ASG and ECS cluster with EC2 instance registered in it.

ECS cluster is a logical grouping of container instances on which you can place tasks.

Container instance: EC2 instance that is running the Amazon ECS agent and has been registered into a cluster. Instances of different types can be included into a cluster. Container instance can be a member of one cluster at a time. Type of container instance determines resources available within the cluster.

ECS agent makes API calls to ECS service on your behalf, therefore container instances must be created in an instance profile that opens the required permissions.

Note: Credentials for Terraform are not required for these steps as “terraform apply” will be executed on EC2 instance which has full access to AWS.
a. Go to w6/workshop_part_2 directory at newly created EC2 instance and configure terraform to create ECS cluster. You may use mcedit, vim or nano command line text editors.
b. Apply terraform configuration:
$ terraform plan
$ terraform apply
Note: At this step you should run terraform in w6/workshop_part_2 directory at newly created EC2 instance.
c. Go to AWS ECS console to see newly created ECS cluster.
d. Configure terraform to create Auto Scaling Group and instances registered in this ECS cluster.
e. Check AWS ECS web console to see new container instance.

3. Create ECS service with ELB attached to it.

ECS service allows you to simultaneously run and maintain a specified number of instances of a task definition simultaneously. If any of the tasks fails or stops for any reason (e.g. crash, OOM), ECS service scheduler launches another instance of task definition to replace it and maintain the desired count of tasks in the service. Number of tasks running in ECS service is specified in its parameter desired_count.

If attached to ECS service Elastic Load Balancer (ELB), it distributes traffic across the tasks that are associated with ECS service. ECS and ELB take care of adding and removing container instances into/from balancing, without downtime for the application running in containers. ECS can have one ELB attached.

In order to attach the ELB to ECS service you must specify IAM role that will allow ECS service to communicate with ELB on your behalf.

After ECS service is created, container port, container name, ELB specified in ECS service configuration cannot be changed. In order to change them you must re-create ECS service.

ECS task definition contains information about containers which should be started at container instances. This information includes: docker images, CPU, and memory available for containers, links between containers, volumes shared (or not) by containers, ports mapping between containers and container instance, and container configuration environment variables. Task definition structure:

{
 “family”: “myservice”, 
 “containerDefinitions”: [
 {
   “name”: “myservice-container-one”, 
   “image”: “myservice-image:latest”, 
   “cpu”: 100, 
   “memory”: 512, 
   “portMappings”: [
   { “containerPort”: 8080, “hostPort”: 80, “protocol”: “tcp” }
   …
   ],
   “environment”: [ { “name”: “variable1”, “value”: “value1” } ],
   …
  }
  …
}

ignore_changes is meta-parameter in Terraform that customizes how diffs are evaluated for resources, allowing individual attributes to be ignored through changes. For example, terraform will not apply desired_count specified in ECS service configuration if ECS service already exists.

a. Configure terraform to create sample ECS service with ELB.
  i. Notice ignore_changes configuration in ECS service definition.
  ii. Apply Terraform configuration.
$ terraform plan
$ terraform apply
b. Use ELB DNS name to open sample service in browser: check events for ECS service in AWS web console to see why tasks aren’t starting at container instance.
c. Adjust terraform configuration and apply it to fix the issue.

4. Initiate deployment to ECS service.

You can update a running ECS service to change the number of tasks that are maintained by a service (desired_count) or decide which task definition is used by the tasks.

ECS service deployment configuration is an optional parameter for ECS service that controls how many tasks run during the deployment and the ordering of stopping and starting tasks: minimumHealthyPercent (the lower limit of the number of running tasks that must remain running in a service during a deployment) and maximumPercent (the upper limit of the number of running tasks that must remain running in a service during a deployment).

Default ECS deployment configuration has minimumHealthyPercent = 100% that means that your ECS cluster capacity must have enough container instances to start a new version of the application while still running the old version.

a. Change docker image name from anosulchik/workshop-sample-application:v1.0 to anosulchik/workshop-sample-application:v2.0 in containers.txt that defines containers in ECS task.
b. Apply Terraform configuration.
$ terraform plan
$ terraform apply
c. Use ELB DNS name to open sample service in browser: You should see the new version v2.0 of the sample application here.

5. Create Lambda function to sync ASG desired EC2 instances capacity with ECS desired tasks count.

a. Configure terraform to bind SNS topic used by ASG to send notifications to trigger Lambda function.
b. Apply Terraform configuration.
$ terraform plan
$ terraform apply
c. Uncomment resource in lambda.tf and apply terraform configuration.
d. Change min_size = max_size = 2 in ASG configuration.
e. Check Lambda function’s logs and ECS service desired count value in web console (you should see desired_count = 2 here applied by Lambda).
f. Change min_size = max_size = 1 in ASG configuration.

6. Destroy AWS resources created in this workshop.

Note: Order is extremely important here — run Terraform destroy at EC2 instance first, then locally.
a. Run “terraform destroy” command on EC2 instance where you logged into via SSH, in the w6/workshop_part_1 directory.
b. Run “terraform destroy” on your computer in w6/workshop_part_2 directory.

Introduction:

DevOps as a Culture — set of Smartling workshops

Workshop #1:

Getting started with Terraform and AWS EC2, first steps. Workshop #1

Workshop #2:

AWS/Terraform Workshop #2: EC2 Networking, Autoscaling Groups, CloudWatch

Workshop #3:

AWS/Terraform Workshop #3: ELB, SNS, AutoScaling

Workshop #4:

AWS/Terraform Workshop #4: S3, IAM, Terraform remote state, Jenkins

Workshop #5:

AWS/Terraform Workshop #5: AWS Lambda Functions