Skip to content

FND324 - Secure interactive access to instances using Session Manager

In todays world of modern application development and immutable infrastructure deployed as code, there remains a number of reasons why engineers require shell-level access to their instances on occasion. They might need to kill runaway processes, debug problems on a live machine, or fine-tune configurations during development, all while maintaining a strong security profile.

In this session, you will configure AWS Systems Manager Session Manager to provide secure interactive access to your managed instances without the need to expose inbound ports, manage bastion hosts, or manage SSH keys. You will learn how Session Manager works by default and will progressively increase the security posture of your environment by enabling enhanced session encryption, configuring session logging and reducing default permissions.

Scenario

You have been tasked with replacing the legacy bastion infrastructure at your organization with an alternative interactive shell-level access solution. You have been given a few key requirements and must develop a proof of concept that demonstrates the ability of Session Manager to address each:

  • Secure Access: The solution must communicate over a secure encrypted channel for all control and session data. The solution must not require inbound ports to be authorized (e.g. TCP 22 or TCP 5985/5986).

  • Access Control: Users must be able to authenticate using IAM security principals (e.g. users and roles) and must not be required to leverage host-level authentication methods (e.g. public-key, password, etc.).

  • Auditing: All session activity must be tracked and logged to include all command input and output.

  • Cross-Platform Interactivity: The solution should provide synchronous execution of commands across both Windows and Linux platforms.

Prerequisites

Install Session Manager CLI plugin (Optional)

If you want to use the AWS CLI to start your sessions (instead of using the AWS Systems Manager console), version 1.16.12 or later of the CLI must be installed on your local machine.

You can call aws --version to check the version installed on your machine. If you need to install or upgrade the CLI, see Installing the AWS Command Line Interface in the AWS Command Line Interface User Guide.

In addition, to use the CLI to manage your instances with Session Manager, you must first install the Session Manager plugin on your local machine. For information, see (Optional) Install the Session Manager Plugin for the AWS CLI.

Step 1 - Deploy Proof of Concept Environment

The first thing we need for our proof of concept is a standardized VPC environment where we can launch EC2 instances. The CloudFormation template below creates a standardized VPC environment, network components, S3 buckets, (2) Windows Server 2019 EC2 instances, (2) Amazon Linux 2 EC2 instances and IAM security principals. Note: This step will deploy resource in the us-east-2 region.

Click here to deploy POC environment and resources into your account.

NOTE: This CloudFormation template contains IAM resources. You will be required to acknowledge this to authorize the AWS::IAM::InstanceProfile and AWS::IAM::Role capabilities.

Acknowledge Capabilities

Once the stack deployment is complete, browse to the CloudFormation console, locate the session-manager-demo stack and browse to the Outputs tab. Take note of the S3 bucket and VPC details provided.

Step 2 - Evaluate Default Session Manager Configuration

To begin our evaluation, we need to examine the default configuration and behavior of Session Manager and determine what is needed to meet the requirements given to us for this project.

Evaluate cross-platform behavior, security context and default privilege levels.
  1. Browse to the AWS Systems Manager Managed Instances console.

  2. Select the session-manager-linux-prod instance, click Actions, click Start Session.

  3. Now that we have established an interactive shell to the instance, let's determine our user context on the instance and evaluate our privilege level. Execute whoami to determine what user context we're running under. Execute sudo su - and cat /etc/sudoers.d/ssm-agent-users to evaluate the privilege level assigned to the user ssm-user.

  4. We have been told the AWS Systems Manager agent establishes the Session Manager channel by initiating outbound communications to the service. Execute netstat -nputw | grep -i ssm to validate this behavior and verify communications are actually being sent over a secure channel. Click Terminate to terminate the session.

  5. Select the session-manager-windows-prod instance, click Actions, click Start Session.

  6. Since this is a Windows instance, notice we've established an interactive PowerShell session. Execute whoami to determine what user context we're running under. Execute net user ssm-user to determine the local group memberships assigned to the local user ssm-user. Click Terminate to terminate the session.

Evaluate the default auditing and logging configuration for Session Manager.
  1. Browse to the Session Manager console.

  2. Select the Session history tab and review the session history details available. Note that no information is presented under the Output location field.

Evaluate the port requirements and default permissions of managed IAM policies.
  1. Browse to the EC2 Instances console

  2. Review the inbound and outbound rules in the session-manager-demo security group associated with the instances. Note that no inbound ports have been authorized and all outbound traffic is authorized.

  3. Review the permissions granted by the managed IAM policy AmazonEC2RoleforSSM. This managed policy is attached to the session-manager-demo-default IAM role currently associated with our managed instances.

Step 3 - Configure Session Logging

As we observed during our initial evaluation, our activity within a session is not yet being logged. In this step, we are going to configure Session Manager to store session log data in a specified Amazon S3 bucket for auditing purposes. The default option is for logs to be sent to an encrypted S3 bucket. Encryption is performed using the key specified for the bucket.

Enable Session Logging for Session Manager
  1. Browse to the preferences tab on the Session Manager console.

  2. Click Edit, under the Write session output to an Amazon S3 bucket heading, select S3 bucket.

  3. Leave Encrypt log data selected. In the S3 bucket name field, select the bucket created for you in Step 1. This bucket will follow the naming convention session-manager-demo-[ACCOUNT_NUMBER]-us-east-2. In the S3 key prefix field, enter Session-Manager/. Click Save

  4. Browse to the Session Manager console, select a Linux instance and click Start session.

  5. Execute some basic commands like ps -ef | grep -i ssm or sudo ls -l /etc to demonstrate session logging is working as expected. Type some characters AAFFHHJJ and then backspace until all characters are gone. Execute whoami. Click Terminate to terminate the session.

  6. Browse to the Session History tab and locate our last session. Wait for session Status to move from Terminating to Terminated. In the Output location column, click Amazon S3 to view the session log.

  7. Observe the data captured in the session log includes all input and output of the commands we entered. Take note that even the characters we typed and erased (AAFFHHJJ) without executing a command were logged.

Note: The S3 bucket created for you is configured to use S3 Default Encryption with AWS S3-managed keys (SSE-S3). All session log data will be encrypted by default but you can also choose to use your own KMS Customer Master Key (SSE-KMS).

Step 4 - Configure KMS Session Encryption

By default, Session Manager exchanges data between a client and a managed instance over a secure channel that is encrypted using TLS 1.2. Session Manager also enables you to use your own AWS KMS-managed key to encrypt session traffic between a browser or CLI client and an instance. Encrypting session data with your key also enables sessions to handle confidential data interactions, such as password resets, and further improves your security posture when using Systems Manager Session Manager.

Create KMS Customer Managed Key (CMK)
  1. Browse to the KMS console. Click Create key.

  2. In the Alias field, enter session-manager-demo-encryption. Click Next twice.

  3. Under Key administrators, select your current IAM user. Leave Allow key administrators to delete this key selected. Click Next.

  4. Under Define key usage permissions, select the session-manager-demo-user IAM user, and the session-manager-demo-default and session-manager-demo-limited IAM roles. Click Next. Click Finish.

  5. Record the ARN of the CMK we just created for future use.

Configure KMS Session Encryption
  1. Browse to the Session Manager preferences console. Click Edit.

  2. Under the Enable KMS encryption for active session data heading, select Key Management Service (KMS).

  3. In the KMS key field, select the alias/session-manager-demo-encryption CMK created in the previous steps. Click Save.

  4. Browse to the Session Manager console and attempt to establish a Session Manager session to any of the Linux managed instances launched from our CloudFormation template.

*Note: You receive an error indicating the version of SSM Agent on the instance doesn't support KMS encryption. To use KMS session encryption, you must be running agent version 2.3.539.0 or later.

Session Manager KMS Error

Update Systems Manager Agent
  1. Browse to the Run Command console.

  2. In the Command document field, select the AWS-UpdateSSMAgent document. Under Targets, select Manually selecting instances. Select all four session-manager-* instances we launched from our CloudFormation template.

  3. Click Run and monitor the status of the command until the status of all four targets changes from In Progress to Success.

  4. Browse to the Session Manager console and attempt to establish a Session Manager session to any of the Linux managed instances launched from our CloudFormation template. You should see a message indicating This session is encrypted using AWS KMS.

KMS Encrypted Session

Step 5 - Reduce Scope of IAM Instance Profile Permissions to only Session Manager

Now that we've evaluated Session Manager against managed instances with the default IAM permissions granted by the AmazonEC2RoleforSSM managed IAM policy, we want to evaluate our ability to reduce the permissions assigned to our managed instances.

The following steps will grant the necessary permissions for Session Manager only.

  1. Browse to the session-manager-demo-limited role created by our CloudFormation template.

  2. This role has no permissions attached at this point. Click Add inline policy. Select the JSON tab.

  3. Replace all text in the editor with the following sample policy, updating ARNs for your S3 bucket and KMS key where indicated:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssmmessages:CreateControlChannel",
                "ssmmessages:OpenControlChannel",
                "ssmmessages:CreateDataChannel",
                "ssmmessages:OpenDataChannel",
                "ssm:UpdateInstanceInformation"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts",
                "s3:ListBucketMultipartUploads"
            ],
            "Resource": "<S3_BUCKET_ARN>/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetEncryptionConfiguration"
            ],
            "Resource": "<S3_BUCKET_ARN>"
        },
        {
            "Effect": "Allow",
            "Action": [
                "kms:GenerateDataKey",
                "kms:Decrypt"
            ],
            "Resource": "<KMS_ARN>"
        }
    ]
}
  1. Click Review Policy. In the Name field, enter Session-Manager-Limited. Click Create policy.

  2. Browse to the EC2 Console.

  3. Select the session-manager-linux-prod instance. Click Actions, open Instance Settings, select Attach/Replace IAM role.

  4. In the IAM role field, select session-manager-demo-limited. Click Apply. Click Close.

  5. Repeat steps 6 & 7 for the remaining EC2 instances launched by our CloudFormation template.

  6. Browse to the Session Manager console. Select an instance and click Start session. Verify the session starts successfully and indicates the session is encrypted with KMS.

Step 6 - Create End User Policy for Session Manager

Now that we've evaluated Session Manager from an administrative perspective, we want to test the ability to authorize users to only start sessions and only against an approved set of instances.

  1. Browse to the IAM console and locate the session-manager-limited group.

  2. Edit the session-manager-user-policy policy.

  3. Find REPLACE-WITH-KEY-ID in the policy and replace with the KMS key ID created in previous steps. Click Apply Policy.

  4. Browse to the session-manager-demo-user. Next to Console password, click Manage.

  5. Under Console access, select Enable. Under Set password, leave Autogenerated password enabled. Click Apply. Record the password and click Close.

  6. Log into the AWS Management Console as session-manager-demo-user.

  7. Browse to the Session Manager console. Select session-manager-linux-prod, click Start session. Observe that we get an authorization error.

  8. Browse to the Session Manager console. Select session-manager-linux-stage, click Start session. Observe that we are successfully connected.

  9. Log into the AWS Management Console as your administrative user.

Note: We can use conditional statements to further restrict which resources users can access or restrict them to accessing resources only during certain time windows. Review sample IAM policies for more user access examples.

Step 7 (Optional) - Configure VPC Endpoints

You can further improve the security posture of your managed instances by configuring AWS Systems Manager to use an interface VPC endpoint which restricts all network traffic between your managed instances, Systems Manager, and Amazon EC2 to the Amazon network.

  1. Browse to the VPC console and click Endpoints from the left navigation menu. Click Create Endpoint.

  2. Under Service Name, select com.amazonaws.us-east-2.ssm. Under VPC, select the session-manager-demo VPC.

  3. Under Subnets, select session-manager-demo-Private-A and session-manager-demo-Private-B. Under Enable Private DNS Name, ensure Enable for this endpoint is selected.

  4. Under Security group, select the session-manager-demo-endpoints security group. Click Create endpoint.

  5. Repeat steps 1-4, replacing the endpoint service name with the following endpoint services:

  6. com.amazonaws.us-east-2.ec2messages
  7. com.amazonaws.us-east-2.ec2
  8. com.amazonaws.us-east-2.ssmmessages
  9. com.amazonaws.us-east-2.kms

  10. Browse to the VPC console and click Endpoints from the left navigation menu. Click Create Endpoint.

  11. Under Service Name, select com.amazonaws.us-east-2.s3. Under VPC, select the session-manager-demo VPC.

  12. Under Configure route tables, select all route tables. Under Policy, leave Full Access selected. Click Create endpoint.

  13. Browse to the VPC console and verify the status of all endpoints show available.

  14. Browse to the Session Manager console. Select a managed instance and click Start session to verify you are able to access instances successfully.

Step 8 (Optional) - Reduce Privilege of Session Manager User

When a version of SSM Agent that supports Session Manager starts on an instance, it creates a user account with root or administrator privileges called ssm-user. On Linux machines, the account is added to /etc/sudoers. On Windows machines, it is added to the Administrators group. Sessions are launched using the credentials of this user account.

If you want to prevent Session Manager users from running administrative commands on an instance, you can update its ssm-user permissions. You can also restore these permissions after they have been removed.

  1. Browse to the Session Manager console, select session-manager-linux-stage. Click Start session.

  2. Elevate to root privilege by executing sudo su -. Edit /etc/sudoers.d/ssm-agent-users and remove ssm-user ALL=(ALL) NOPASSWD:ALL from the file. Save and close the file. Click Terminate to terminate the session.

  3. Select session-manager-linux-stage. Click Start session.

  4. Attempt to elevate privilege by executing sudo su -. Observe that you are prompted for a password and no longer have elevated privileges.

Note: To restore elevated permissions, you should normally use Run Command to execute the AWS-RunShellScript document with the following command: cd /etc/sudoers.d && echo "ssm-user ALL=(ALL) NOPASSWD:ALL" > ssm-agent-users. In Step 5 however, we reduced the permissions attached to the instance profiles so that they can only use Session Manager. If you wish to restore permissions, switch the IAM role back to session-manager-demo-default.

Cleanup

  1. Browse to the session-manager-demo-user user in the IAM console. Under Console password, select Manage. Under Console access, select Disable.

  2. Browse to the session-manager-demo-limited role in the IAM console. Next to the Session-Manager-Limited inline policy, click the X to delete the policy. Click Remove.

  3. Browse to the VPC Endpoints console. Select the first VPC endpoint created in previous steps. Click Actions, click Delete Endpoint. Click Yes, Delete. Repeat for all VPC endpoints created in previous steps.

  4. Browse to the S3 console. Select the S3 bucket that was created for you session-manager-demo--us-east-2. Click Empty, confirm the bucket name and click Confirm.

  5. Browse to the Session Manager preferences console. Click Edit. Disable Enable KMS encryption for active session data, click Confirm. Disable Write session output to an Amazon S3 bucket. Click Save.

  6. Browse to the session-manager-demo stack in the CloudFormation console. Click Delete, click Delete stack.