Containers & Docker

Quick intro?

Containers

chroot on steroids

• provide isolation between workloads: CPU, memory, disk, network, IO
• containers share host kernel
• lightweight, do not use virtualiziation

Docker

"Docker allows you to package an application with all of its dependencies into a standardized unit for software development."

• tools and specification how to create and share container images
• runtime for container execution and management of network and data resources
• large ecosystem

Key Components of ECS

• Container Instance
• Cluster
• Container
• Task Definition
• Task
• Scheduler

Container Instance

Any EC2 instance running Docker and the ECS Agent

Provides resources like CPU, RAM and disk

Cluster

Resource pool, grouping of Container Instances

Consolidates resources like CPU, RAM and disk across multiple servers

Container

Runs your code in a Docker container

Scheduled on a Container Instance in a specific cluster, consumes CPU, RAM, disk

Task Definition

Blueprint for Tasks, similar to docker-compose.yml

{
    "family": "f_producer", 
    "containerDefinitions": [
        {
            "name": "producer",
            "image": "noeding/ecs-test", 
            "cpu": 50,
            "memory": 50, 
            "essential": true, 
            "environment": [{"name": "MODE", "value": "producer"}]
}]}
                        

Defines how many containers to run, what resources they need, how to link them and which Docker image to use

Task

Unit of work, either batch job or service

Batch Tasks

one-off jobs, tasks terminate when container terminates

Service Tasks

long running, terminated containers are restarted as necessary

Scheduler

Decides which container instance executes a container

By default random placement, though custom schedulers can be integrated (e.g., Mesos)

Working with ECS

From Docker image to running container

Create and upload Docker Image

Skipping this here for brevity

Upload ecs.config to private S3 bucket

Set cluster name and grant access to private docker registry

ECS_CLUSTER=default
ECS_ENGINE_AUTH_TYPE=dockercfg
ECS_ENGINE_AUTH_DATA={"https://index.docker.io/v1/": {"auth": "...", "email": "..."}}
                        

Create IAM role for Container Instances

Grant the container instances access to AWS resources

• ECS API
• private S3 bucket with ecs.config file
• ...

Create ECS cluster

aws ecs create-cluster --cluster-name default

Launch Container Instance(s)

Use EC2 user data to configure instance

#!/usr/bin/env/python
import boto.ec2

user_data = '''#!/bin/bash

yum install -y aws-cli
aws s3 cp s3://noeding-ecs-test/ecs.config /etc/ecs/ecs.config
'''

d = {
    'image_id': 'ami-ecd5e884', # Amazon Linux for ECS (us-east-1)
    'instance_profile_name': 'noeding-ecs-test',
    # ...
}

conn = boto.ec2.connect_to_region('us-east-1')
reservation = conn.run_instances(**d)
# ...
                        

Create Task Definition

aws ecs register-task-definition --generate-cli-skeletion > taskdef.json

{
    "family": "f_producer", 
    "containerDefinitions": [
        {
            "name": "producer",
            "image": "noeding/ecs-test", 
            "cpu": 50,
            "memory": 50, 
            "essential": true, 
            "environment": [{"name": "MODE", "value": "producer"}]
}]}
                        

aws ecs register-task-definition --cli-input-json file://taskdef.json

Working with Tasks

# check if container instances are ready
aws ecs list-container-instances --cluster default

# launch batch task, using random placement
aws ecs run-task --cluster default --task-definition f_producer:2

# list running tasks
aws ecs list-tasks --cluster default

# describe task details
aws ecs describe-tasks --cluster default --tasks ${ID}

# stop task
aws ecs stop-task --cluster default --task ${ID}