In this comprehensive guide, we will walk you through the process of setting up an AWS ECS (Elastic Container Service) environment with essential components such as CloudWatch Logs, CloudTrail, S3 Buckets, and KMS (Key Management Service) integration. By following this step-by-step tutorial, you will be able to configure a robust ECS setup that incorporates advanced logging, auditing, secure storage, and encryption capabilities.

Prerequisites

1. Install or update AWS CLI to the latest version by following the guide here.
2. Verify that the AWS CLI is installed correctly by running aws --version in your terminal.

Let’s Go…

To set up ECS Exec with your current ECS setup using Fargate, you'll need to perform the following steps:

1. Install and Configure the SSM Agent: ECS Exec requires the AWS Systems Manager (SSM) agent to be installed on the ECS instances. The SSM agent allows you to run commands against the instances using the ECS Exec functionality. You can install and configure the SSM agent manually on your ECS instances or use an automation tool like AWS Systems Manager Run Command or AWS Systems Manager State Manager.

   1. Verify that the SSM plugin is installed correctly by running aws ssm help in your terminal.

      Install the Session Manager plugin for the AWS CLI - AWS Systems Manager

2. Set up KMS: ECS Exec requires an AWS Key Management Service (KMS) key for encryption. If you don't have one already, create a KMS key through the AWS Management Console or using the AWS CLI.

   1. Note: It may be helpful to create a user and role for your KMS keys with no policies at this point.

   2. KMS Key Configuration: Make sure you have a KMS key created. If you haven't created one yet, follow the AWS documentation to create a KMS key. Once created, note down the KMS key ARN.

      1. To add a KMS key to your ECS cluster, you need to update the task execution role associated (ecsTaskExecutionRole) with your cluster. Here's how you can do it:

         1. Open the AWS Management Console and navigate to the Amazon ECS service.

         2. Select your ECS cluster from the list.

         3. In the left navigation pane, click on "Settings".

         4. Under the "Permissions" section, locate the "Task execution IAM role" and click on the role name.

         5. This will take you to the IAM console, specifically to the IAM role associated with your task execution role.

         6. In the IAM role details page, click on "Add inline policy" under the "Permissions" tab.

         7. On the "Create Policy" page, select the "JSON" tab and enter the following JSON policy, replacing your-kms-key-arn with the ARN of your KMS key:

            {
              "Version": "2012-10-17",
              "Statement": [
                {
                  "Effect": "Allow",
                  "Action": ["kms:Decrypt", "kms:Encrypt", "kms:GenerateDataKey"],
                  "Resource": "your-kms-key-arn"
                }
              ]
            }
            

3. Update Task Definitions: Update your task definitions to include the necessary configurations for ECS Exec. You'll need to modify the task definition to add the FargatePlatformConfiguration section and set the platformVersion to 1.4.0 or later. Also, ensure that your task execution role has the necessary permissions to interact with S3, CloudTrail, and KMS.

   1. Add Task role for execution on container (must be combined with elastic cache or other AWS programmatic access permissions).

      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": [
              "ssmmessages:CreateControlChannel",
              "ssmmessages:CreateDataChannel",
              "ssmmessages:OpenControlChannel",
              "ssmmessages:OpenDataChannel"
            ],
            "Resource": "*"
          }
        ]
      }
      

      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": ["kms:Decrypt"],
            "Resource": "your-kms-key-arn"
          }
        ]
      }
      

      				{
                  "Effect": "Allow",
                  "Action": "s3:PutObject",
                  "Resource": "arn:aws:s3:::<ECS_EXEC_BUCKET_NAME>/*"
              },
              {
                  "Effect": "Allow",
                  "Action": "s3:GetEncryptionConfiguration",
                  "Resource": "arn:aws:s3:::<ECS_EXEC_BUCKET_NAME>"
              }
      

      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": ["logs:DescribeLogGroups"],
            "Resource": "*"
          },
          {
            "Effect": "Allow",
            "Action": ["logs:CreateLogStream", "logs:DescribeLogStreams", "logs:PutLogEvents"],
            "Resource": "arn:aws:logs:<region>:<account-id>:log-group:<path>:*"
          }
        ]
      }
      

4. Configure Audit Logging: ECS Exec requires audit logs to be enabled in CloudTrail. If you haven't done so already, enable CloudTrail and ensure it's configured to log API events for ECS.

   1. Audit Logging Configuration: Enable audit logging for your ECS cluster using CloudTrail. Ensure that CloudTrail is properly set up and configured to capture ECS API events.

      1. This policy allows the CloudTrail service (cloudtrail.amazonaws.com) to perform the kms:GenerateDataKey and kms:Decrypt actions on the specified KMS key (your-kms-key-arn).

      2. Remember to review and adjust the policy based on your specific requirements and resource configurations. Additionally, you may need to include other statements or conditions depending on your use case and desired access control.

      3. Might need to add s3 bucket to Key Policy

      4. Add the following to the Key Policy for you new KMS key

         {
           "Version": "2012-10-17",
           "Id": "key-policy-id",
           "Statement": [
             {
               "Sid": "Allow CloudTrail to GenerateDataKey and Decrypt",
               "Effect": "Allow",
               "Principal": {
                 "Service": "cloudtrail.amazonaws.com"
               },
               "Action": ["kms:GenerateDataKey", "kms:Decrypt"],
               "Resource": "your-kms-key-arn"
             }
           ]
         }
         

5. Create an S3 Bucket for Container Logs: Set up an S3 bucket to store the container logs generated by ECS Exec. You can create a new bucket using the AWS Management Console or the AWS CLI. Make sure to configure appropriate permissions and access controls for the bucket.

6. Launch or Update Your ECS Cluster: Launch or update your ECS cluster with the updated task definitions. This will ensure that new tasks launched on your cluster have the necessary configurations for ECS Exec.

Once you've completed these steps, you should be able to use ECS Exec locally by following the official documentation provided by AWS. This typically involves configuring the AWS CLI, setting up session tokens, and running the ecs exec command to establish a session with the container running in your ECS task.

Remember to refer to the AWS documentation for detailed instructions on setting up ECS Exec and configuring the necessary components for your specific environment.

Run https://github.com/aws-containers/amazon-ecs-exec-checker to validate setup (may need to run brew install jq on mac)

https://github.com/aws-containers/amazon-ecs-exec-checker

Please note that it may take some time for the ECS service to propagate the configuration changes and for the amazon-ecs-exec-checker tool to reflect the updated status.

Update task to enable service, example below:

aws ecs update-service \
    --cluster <cluster> \
    --service <service> \
    --task-definition <task-definition> \
    --enable-execute-command \
    --force-new-deployment

Testing the exec command.

We can use either sh or bash in most Docker instances, provided we are running Linux, to test our exec command. If you are logged into the container shell, it worked!

Note: Container name can be found in the AWS ECS cluster via the task section.

aws ecs execute-command \
    --cluster <cluster> \
    --task <task-id> \
    --container <container> \
    --interactive \
    --command "sh"

Initial Success! But we still need to update our configuration to use the KMS key.

Update Task Role Permissions

Update the cluster to allow for the new logging config with KMS

aws ecs update-cluster \
    --cluster <cluster> \
    --region <region> \
    --configuration executeCommandConfiguration="{logging=OVERRIDE,\
        kmsKeyId='<your-kms-key-arn>',\
        logConfiguration={cloudWatchLogGroupName="/ecs/<path>",\
                        s3BucketName='<S3-bucket>',\
                        s3KeyPrefix=exec-output}}"

Configuring CloudTrail

1. Go to → Dashboard → Create Trail
2. Add a trail name
3. Use the existing S3 bucket that you created in the previous step to store logs
4. Keep Log file encryption on using KMS
5. Select the existing KMS key used for your set up
6. Enable cloudwatch logs
   1. Use an existing log group name of /ECS/<container-name>
   2. Create a new role for your cloud trail CloudTrailRoleForCloudWatchLogs-<container-name>
7. Create logger

For reference the cloud trail role for cloud watch logs policy is below

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AWSCloudTrailCreateLogStream2014110",
      "Effect": "Allow",
      "Action": ["logs:CreateLogStream"],
      "Resource": [
        "arn:aws:logs:<REGION>:<ACCOUNT-ID>:log-group:/ecs/<PROJECT>:log-stream:<CLOUDTRAIL-RESOURCE>*"
      ]
    },
    {
      "Sid": "AWSCloudTrailPutLogEvents20141101",
      "Effect": "Allow",
      "Action": ["logs:PutLogEvents"],
      "Resource": [
        "arn:aws:logs:<REGION>:<ACCOUNT-ID>:log-group:/ecs/<PROJECT>:log-stream:<CLOUDTRAIL-RESOURCE>*"
      ]
    }
  ]
}

Sweet Success!

By following this guide, you have successfully set up an AWS ECS environment with comprehensive logging using CloudWatch Logs, auditing capabilities with CloudTrail, secure storage using S3 buckets, and encryption through KMS integration. This setup ensures enhanced visibility, compliance, and security for your ECS containerized applications. Feel free to explore additional features and advanced configurations to further optimize your ECS deployment.

Remember to refer to the official AWS documentation for detailed instructions and best practices, as well as to keep your setup up to date with the latest security recommendations and enhancements.

Guides / More info

Using Amazon ECS Exec for debugging - Amazon Elastic Container Service

Configure AWS KMS key policies for CloudTrail - AWS CloudTrail

Getting Started with ECS Exec for ECS Fargate.

Use AWS ECS execute command to access a container on AWS Fargate and EC2

Using the Amazon ECS command line interface - Amazon Elastic Container Service