Do you have docker containers running in an AWS ECS cluster and worried about how to get notified when any container in production got killed by any reason, Then you are on right place.

Just think, Your production container is down and you never know about it immediately, Because you are playing pubg in your mobile 😉

Haha its a serious problem right, “ OFCOURSE DUDE“

What if I had a alert mechanism to my AWS Infrastructure which could mail me whenever my containers got stopped , like “Hey Bala your service ABC container got stopped due to this reason “. Then it would be great to know about the situation and act on it accordingly.

How To achieve this??????

AWS provides you Cloudwatch Rules to monitor your ECS task state change in an ECS cluster.

To create a rule that triggers on an event:

Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.
In the navigation pane, choose Events, Create rule.
Choose event pattern
choose Service name as Elastic Container Service (ECS)
Event type as State change.
Choose Any cluster option to monitor all your ECS clusters or choose Specific cluster option to monitor your specific ECS cluster.

It will look like below image.

This will track the change in state of your docker containers .

It will have 3 states:

RUNNING
PENDING
STOPPED

By default all three states will be tracked by cloudwatch rules, but if you wish to track only STOPPED state and get notified, then you can add the below code to Event Pattern Preview section. (You can actually add multiple states or another state other than STOPPED aswell)

{

“source”: [

“aws.ecs”

“detail-type”: [

“ECS Task State Change”

“detail”: {

“clusterArn”: [

“arn:aws:ecs:<add your region here> :<account-id>:cluster/<cluster-name>”

“lastStatus”: [

“STOPPED”

]

}

Then on the right side , there will be Targets section, choose target as SNS topic(if you dont have SNS topic, create an SNS topic and subscribe your email address to that SNS topic and confirm the subscription) .

Here comes the interesting part, now if you choose matched event under configure input section(Which is default option) , you will receive a long json object which I feel will be difficult to read in a mail about all the details together. So to avoid this we can customise the email which we get.

Choose Input Transformer :

In the first text area, add the following lines:

{“stoppedReason”:”$.detail.stoppedReason”,”lastStatus”:”$.detail.lastStatus”,”group”:”$.detail.group”}

In the second text area, add the following lines:

“The container <group> in production got <lastStatus> due to <stoppedReason> , Kindly check “

It will look like the below image :

NEXT ON STEP 2 PAGE:

Give an appropriate name and description to your Cloudwatch rule. and click on that tick mark to enable the rule.

You can actually enable and disable this rule using that tick mark. (like during code deployments you can disable and enable when deployment is done.)

HURRAY!!!!! THATS ALL YOU NEED TO DO.

Whenever any task(container) in ECS cluster gets killed:

$.detail.group will track the name of the container that is stopped.

$.detail.lastStatus will track that it is stopped.

$.detail.stoppedReason will track the reason why the container is stopped.

and will send an email like

“The container service:web-container in production got STOPPED due to OutOfMemoryError, Kindly check “

Thats it 🙂

Thanks for your Time.

Balakrishna Cherukuri

Working as a DevOps Engineer with 2+ years of experience. Certified from AWS with AWS CERTIFIED SOLUTIONS ARCHITECT- ASSOCIATE certificate.