Skip to content

GRC320-R - Build an enterprise compliance management & remediation system on AWS

In this builder session, we show you how to build a fleet-wide enterprise compliance management and remediation system using AWS Systems Manager and Amazon CloudWatch. In addition, we provide compliance stakeholders visibility into the performance of the compliance system by using Amazon QuickSight and Amazon Athena for reporting.

Building Configuration Compliance with AWS Systems Manager

Security of your systems and your customer's data is paramount, therefore you need to be able to define what constitutes a system secure. The objective of this section is to define a policy, continously monitor it, and ensure the sytem remains compliant to the desired configuration according to the policy defined. We will use AWS Systems Manager to monitor and remediate Linux and Windows instances, using common industry standard Domain Specific Languagues (DSLs), PowerShell DSC and Ansible, respectively. We will first begin with Linux and Ansible, then we can move to Windows and PowerShell DSC, if needed. The AWS Systems Manager service also supports Chef Inspec, so that automation platform can also be leveraged in the same manner to meet compliance, security, and policy requirements.

Scenario 1 - Linux and Ansible with AWS Systems Manager

Ansible is a configuration platform that allows you to defined your systems for security. In this first example, we will use an Ansible playbook to define a configuration and optionally remediate the configuration non-conpliance. In this section, we will also leverage AWS Systems Manager Session Manager, which allow us to remotely manage a system, without needing network connectivity or needing to manage SSH keys. We will use a similar configuration platform for Microsoft Windows, which will go an extra step and ensure the services required to remotely manage an instance are disabled.

Launching an Ubuntu EC2 Instance

  1. Login to your AWS account.
  2. Once you’re logged in, go to the EC2 Console.
  3. Launch an Ubuntu Server EC2 Instance. Get to AWS Config Console
  4. Select a General purpose, t2.micro instance, and lick “Next: Configure Instance Details”. Select EC2 instance
  5. On the next screen, we will select all the defaults except the “IAM role” section. In this section, we will click on the “Create new IAM role” link. Create IAM Role
  6. On the “Roles” page, we will click on “Create role”. Refresh new IAM Role
  7. In the “Select type of trusted identity” section, we will select “AWS service”. We will select “EC2” in the “Choose the service that will use this role” section. Then, click “Next: Permissions”. Role and Policy associations
  8. In the “Attach permissions policies page”, we will select the “AmazonEC2RoleforSSM” policy and click on the “Next: Tags” button. Permissions policy
  9. We will click “Next: Review” on the “Add tags” page.
  10. In the “Review” page we will type “reInforce2019” in the “Role name”, and click on “Create role”. Role Name
  11. Once the role is created, we will go back to the “Configure Instance Details” page and finish creating our instance. Before moving forward, we will refresh the “IAM role” section and select the role we just created.
    Role selection in EC2 instance deployment
  12. Then, click on “Next: Add Storage”.
  13. In the “Add Storage” page, we will go with the default settings and click “Next: Add Tags”.
  14. In the “Add Tags” page, we will enter “Name” as the “Key” and “re:Inforce Linux” in the “Value” sections.
    EC2 tags
  15. Then click on “Review and Launch”.
  16. On the “Review Instance Launch” page, click on “Launch”.
  17. Then we will “Proceed without a key pair” and click “Launch Instances”. No key at deployment

Security Compliance

Once the instance is launched, we will use AWS Systems Manager to maintain security compliance.

  1. Go to the “AWS Systems Manager” Console.
  2. We will then use “Session Manager” to install the Ansible components, which will make sure our systems remain security compliant.
    Session Mgr
  3. Then, we will click on the “Start session” button to connect to the EC2 instance we just created. Start Session
  4. From the “Start a session” page, click on the EC2 instance we just created and click on the “Start Session” button.
  5. Once a new tab is open and the session with the EC2 instance has been established, run the following commands to install Ansible.
sudo apt-get install software-properties-common -y
sudo apt-add-repository ppa:ansible/ansible -y
sudo apt-get update -y
sudo apt-get install ansible -y
  1. When prompted, select “Yes”. Configuring Ansible
  2. Once all components are installed, we will click on “State Manager”.
    State Manager Navigation
  3. From that page, we will click on the “Create association” button.
    Assons
  4. In the “Name” field, we will type “reInforceAnsible” or “reInforcePowerShellDSC”.
    Label
  5. In the “Document” section, we will select the “AWS-RunAnsiblePlaybook” document. RunAnsiblePlayPlaybook
  6. In the “Parameters” section, we will type following URL in the “Playbookurl” section, and select “True” from the “Check” drop down. The playbook used in this example installs a web service on the EC2 instance; however, we will not be installing the service and only use the playbook to flag the instance as being non-compliant. Copy and paste the location of the .yml definition file: https://s3.amazonaws.com/www.awsmanagementweek.com/ansible.yml
    Parame Content of the ansible.yml file:
- hosts: localhost
  tasks:
    - name: stopping ssh service
      service:
       name: ssh
       state: stopped
       enabled: no
  become: yes
  1. In the “Targets” section, we will select the “Specifying tags” radial button and “Name” and “re:Inforce Linux” for the key/value pair.
    Target
  2. Leave the defaults in the “Specify schedule” section. Note: Although the association runs every 30 minutes, any new instances matching the resource tags, will have the desired state configuration in the Ansible playbook applied within a few minutes.
  3. In the “Advanced options” setting, select “Critical” for the “Compliance severity”. This will help us easily identify any EC2 instances that are not within our compliance requirements in the Compliance capability of Systems Manager.
    Advancd options
  4. All other settings can be left default. Click on the “Create Association” button.
  5. Once the “State Manager” association is created, we will check whether our EC2 instance is within compliance or not. To do this, we will go to “Compliance” within the Systems Manager Console.
    Comlian
  6. Once there, we will focus on the “Compliance resources summary”. In our scenario, since we configured Ansible to alert of anything out of compliance (i.e. Check only), then we will see anything out of compliance in this dashboard. Complian Dashboard
  7. Don’t mind other associations depicted, in this case we only care about the “Critical resources” compliance alert that we configured in previous steps.
  8. If we click on the “Critical resources” alert, we can see the instance we created is the one that is not compliant. If we wanted to automatically remediate this compliance issue, we would change the Association settings to execute the Ansible playbook, based on the .yml definition, instead of just reporting it like we have done here.

Objective

In this scenario we setup a desired state configuration for an Ubuntu Linux instance using Ansible. Ansible Playbook are able to run natively on AWS Systems Manager because the service has the runtime engine required to execute the instructions. The benefit for using this approach is as a customeer you no longer need to worry about the undeferentiated heavy lifting that goes into managing an Ansible server infrastructure.

Scenario 2 - Windows and PowerShellDSC (Desired State Configuration)

Now let's assume you decided to disable Remote Desktop Protocol (RDP) service on their EC2 Windows instances, because of a recent RDP vulnerability has been found. To address this issue, you decide to implemeent a solution using AWS Systems Manager to disable RDP on any EC2 instance with a tag key "Name" and tag value “re:Inform Windows”. The policy we create will be reapplied and automatically remediate a system, within 30 minutes of when RDP is enabled on a new or existing instance. We will also use the Compliance dashboard to easily identify non-compliant systems.

Launching a Windows EC2 Instance

  1. We will first go to the EC2 Console and click on the “Launch Instance”.
  2. Select the Microsoft Windows Server 2019 Base image.
    EC2 instance
  3. Select a General purpose, t2.micro instance, and lick “Next: Configure Instance Details”.
    IAM Role
  4. On the next screen, we will select all the defaults except the “IAM role” section. In this section, we will click on the “Create new IAM role” link. Create Role
  5. On the “Roles” page, we will click on “Create role”. Create Role2
  6. In the “Select type of trusted identity” section, we will select “AWS service”. We will select “EC2” in the “Choose the service that will use this role” section. Then, click “Next: Permissions”. Role policy
  7. In the “Attach permissions policies page”, we will select the “AmazonEC2RoleforSSM” policy and click on the “Next: Tags” button. Create policy
  8. We will click “Next: Review” on the “Add tags” page.
  9. In the “Review” page we will type “reInforce2019” in the “Role name”, and click on “Create role”. Review policy
  10. Once the role is created, we will go back to the “Configure Instance Details” page and finish creating our instance. Before moving forward, we will refresh the “IAM role” section and select the role we just created.
    Select IAM Role
  11. Then, click on “Next: Add Storage”.
  12. In the “Add Storage” page, we will go with the default settings and click “Next: Add Tags”.
  13. In the “Add Tags” page, we will enter “Name” as the “Key” and “re:Inforce Linux” in the “Value” sections.
    Tags
  14. Then click on “Review and Launch”.
  15. On the “Review Instance Launch” page, click on “Launch”.
  16. Then we will “Proceed without a key pair” and click “Launch Instances”.
    Key pair

Security Compliance

Once the instance is launched, we will use AWS Systems Manager to maintain security compliance.

  1. Once the instance is launched, we will go to the “AWS Systems Manager” Console.
  2. We will then use “Session Manager” to install the Ansible components, which will make sure our systems remain security compliant.
    Sess Manager
  3. Then, we will click on the “Start session” button to connect to the EC2 instance we just created. St Session
  4. From the “Start a session” page, click on the EC2 instance we just created and click on the “Start Session” button.
  5. Once the session is running, we will run the following command.
    netstat -ab | findstr 3389
  1. Now we will locate “State Manager” in the navigation.
    State Mager
  2. From that page, we will click on the “Create association” button.
    Create Aociation
  3. In the “Name” field, we will type “reInforcePowerShellDSC”.
    Associion name
  4. In the “Document” section, we will select “AWS-ApplyDSCMofs”. Unlike the Ansible playbook which can run without needing to be compiled, PowerShell DSC requires the configuration to be compiled into a .MOF file extension file.
    Docs
    For reference, the .MOF file was compiled with the via PowerShell DSC using the following commands.
Configuration DisableRDP
{
    Import-DscResource -Module xRemoteDesktopAdmin, NetworkingDsc

    Node ('localhost')
    {        
        xRemoteDesktopAdmin RemoteDesktopSettings
        {
           Ensure = 'Absent'
           UserAuthentication = 'Secure'
        }

         Firewall DisableRDPRule
        {
            Name                  = 'RemoteDesktop-UserMode-In-TCP'
            Group                 = 'Remote Desktop'
            Ensure                = 'Present'
            Enabled               = 'False'
        }
    }
}
DisableRDP
  1. To avoid having to compile the .MOF file, we have uploaded the file to our site (https://s3.amazonaws.com/www.awsmanagementweek.com/localhost.mof), which we can use in the “Parameters” section under “Mofs To Apply”. MOF url
  2. We will select “Apply” in the Mof Operation Mode” section. Selecting “ReportOnly” will not automatically remediate the compliance issues identified.
    Apply MOF
  3. In the “Compliance Type” section, we will customize the compliance label to identify “RDPCompliance” as the identifier.
    Compliance
  4. In the “Targets” section, we select the “Specifying tags” radial button and type “Name” and “re:Inforce Windows” in the key/value pair, respectively.
    Tags
  5. Leave the defaults in the “Specify schedule” section. Note: Although the association runs every 30 minutes, any new instances matching the resource tags, will have the desired state configuration in the Ansible playbook applied within a few minutes.
  6. In the “Advanced options” setting, select “Critical” for the “Compliance severity”. This will help us easily identify any EC2 instances that are not within our compliance requirements in the Compliance capability of Systems Manager.
    Critical
  7. All other settings can be left default. Click on the “Create Association” button.
  8. Once the “State Manager” association is created, we will check whether our EC2 instance is within compliance or not. To do this, we will go to “Compliance” within the Systems Manager Console.
    Compliance
  9. Once there, we will focus on the “Compliance resources summary”. In our scenario, since we configured PowerShellDSC to alert of anything out of compliance (i.e. Check only), then we will see anything out of compliance in this dashboard.
  10. Once there, we can locate the custom compliance type we created, labeled as “Custom:RDPCompliance”.
    RDPCompliance
  11. We will click on the “Non-Compliant resources”, which will depict the instance that has not gotten the desired configuration yet, since we just created the association (see below). Instance compliance
  12. Once the State Manager association has been correctly applied and assuming there are no issues with the .MOF file executing as expected, the instance we created or any instances with the key/value pair of “Name” and “re:Inforce Windows” will have the configuration automatically applied. This is particularly useful when you have AutoScaling Groups with elastic workloads.
  13. While we wait, we will go back to the Session Manager session and rerun the netstat to check whether the RDP port is open.
    netstat -ab | findstr 3389
  1. The expected output should be the following. Closed ports

Objective

In this scenario we setup a desired state configuration for an Windows EC2 instance using PowerShellDSC. PowerShell DSC MOF files are able to run natively on AWS Systems Manager because the service has the runtime engine required to execute the instructions. The benefit for using this approach is as a customeer you no longer need to worry about the undeferentiated heavy lifting that goes into managing the PowerShellDSC infrastructure.

Managing Compliance with AWS Systems Management and AWS Config

In this section, we will use AWS Systems Manager Inventory and AWS Config to catalog all applications installed on an EC2 instance and then blacklist an application, which has been deemed not secure. Note: The application used is only a common sample application, we are not making any statements regarding the security of the application.

Creating a Non-Compliant EC2 Instance

  1. We will first go to the EC2 Console and click on the “Launch Instance”.
  2. Select the Microsoft Windows Server 2019 Base image. AMI
  3. Select a General purpose, t2.micro instance, and lick “Next: Configure Instance Details”. Instance Size
  4. On the next screen, we will select all the defaults except the “IAM role” section. In this section, we will click on the “Create new IAM role” link.
    Role
  5. On the “Roles” page, we will click on “Create role”. Create role
  6. In the “Select type of trusted identity” section, we will select “AWS service”. We will select “EC2” in the “Choose the service that will use this role” section. Then, click “Next: Permissions”.
    Policy
  7. In the “Attach permissions policies page”, we will select the “AmazonEC2RoleforSSM” policy and click on the “Next: Tags” button.
    Inline policy
  8. We will click “Next: Review” on the “Add tags” page.
  9. In the “Review” page we will type “reInforce2019” in the “Role name”, and click on “Create role”. Create role
  10. Once the role is created, we will go back to the “Configure Instance Details” page and finish creating our instance. Before moving forward, we will refresh the “IAM role” section and select the role we just created.
    Role label
  11. We will then click on “Advanced Details” and enter the following in “User data”.
    User data
![User data](/reinforce/GRC320RAssets/Config9.png)
<powershell>
$url = "https://javadl.oracle.com/webapps/download/AutoDL?BundleId=238698_478a62b7d4e34b78b671c754eaaf38ab"
$output = "c:\Windows\Temp\jre-8u211-windows-i586-iftw.exe"
$wc = New-Object System.Net.WebClient
$wc.DownloadFile($url, $output)
start-sleep 20
c:\Windows\Temp\jre-8u211-windows-i586-iftw.exe /s
</powershell>
  1. Then, click on “Next: Add Storage”.
  2. In the “Add Storage” page, we will go with the default settings and click “Next: Add Tags”.
  3. In the “Add Tags” page, we will enter “Name” as the “Key” and “re:Inforce Java” in the “Value” sections.
    Tags
  4. Then click on “Review and Launch”.
  5. On the “Review Instance Launch” page, click on “Launch”.
  6. Then we will “Proceed without a key pair” and click “Launch Instances”.
    Key pair

Configuring AWS Systems Manager Inventory

  1. Go to the AWS Systems Manager Console.
  2. Then, click on “Managed Instances” in the navigation.
    Managed Instances
  3. Once there, click on the “Setup Inventory” link.
    Configure Inventory
  4. In the “Name” field, we will enter “reinforce-Association”.
    Inventory Name
  5. In the “Targets” section, we will specify the tags we used when launching the EC2 instance. Inventory Targets
  6. Everything else will be default. Then, we will click on the “Setup Inventory” button. Complete Inventory
  7. Once that is configured, we will click on the “Actions” drop down and select “Edit AWS Config recording”.
    Config Recording
  8. Once on the “Settings” page, we will make sure recording is on.
    Recording on

Setting up Config and Config Rules

  1. From there, we will click on “Rules” from the navigation.
    Config Rules
  2. Then, we will click on “Add rule”.
    Add Rules
  3. In the “Add rule” page, we will type “blacklist” in the search bar.
    Blacklist
  4. We will then select the predefined rule with the name “ec2-managedinstance-applications-blacklisted”.
  5. Now, we will open a new browser tab to the “Managed Instances” section of the Systems Manager Console.
  6. Then, we will locate the EC2 instance with the tag “re:Inforce Java” and click on the “Instance ID” link.
  7. From the instance details page, we will click on the “Inventory” tab the get the software installed on the instance.
    Managed Instance
  8. Copy the exact name of the Java application (i.e. Java 8 Update 211). We will use this when creating our Config Rules.
  9. We will configure the “Trigger” to “All changes” for the “Scope of changes”.
    Trigger
  10. We will then enter the appropriate “Java 8 Update 211” version in the “Rule parameters” configuration.
  11. We will then set the “Remediation action” to “AWS-StopEC2Instance”.
    Rule Parameter
  12. Then we will click on the “Save” button. Rule Save
  13. Once the rule is created, it will take some time for the evaluation to complete.
    Evaluation
  14. Once the evaluation is complete, we should see 1 noncompliant resource.
    Non-compliant
  15. If we click on the rule name (i.e. ec2-managedinstance-applications-blacklisted) we will be able to see additional details, including the resource that is not compliant.
  16. From the “Rule details” page, we will select the instance that needs remediation and click on the “Remediate” button.
    Rule details
  17. At this point, we will see the remediation action being executed.
    Remediate
  18. To ensure the action worked, we will go to the EC2 Console and check whether our instance is shut down or not.
    EC2 verification

Objective

The objective of this section is to show you how to identify any systems that are not following security compliance policies. By leveraging both desired state configuration and AWS Config and AWS Config Rules, we can setup a multi-layered approach to ensure our security policies are being followed. Although in this labe we used EC2 instances, because of their simplicity to manage and to also show the before and after, the capabilities highlighted can be used to interact with other services.

Reporting with Amazon Athena and QuickSight

In order to use Athena and QuickSight, we need to complete some additional pre-requisite operations. This includes creating an S3 bucket and create a policy to allow Systems Manager to connect to the bucket.

  1. Go to the S3 Console and create a bucket. Alternatively, you can use an existing bucket.
  2. Then we will apply the following S3 “Bucket Policy” to allow Systems Manager to access S3.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SSMBucketPermissionsCheck",
            "Effect": "Allow",
            "Principal": {
                "Service": "ssm.amazonaws.com"
            },
            "Action": "s3:GetBucketAcl",
            "Resource": "arn:aws:s3:::bucket-name"
        },
        {
            "Sid": " SSMBucketDelivery",
            "Effect": "Allow",
            "Principal": {
                "Service": "ssm.amazonaws.com"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::bucketname/*/accountid=AccountID/*",
            "Condition": {
                "StringEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control"
                }
            }
        }
    ]
}
  1. Once the policy has been applied to the bucket, we will configure the inventory data sync.

Config Inventory Data Sync

  1. Go to the AWS Systems Manager Console, from the navigation pane select “Inventory”.
    Report1
  2. Then, we will click on the “Resource Data Syncs” button.
    Report2
  3. Once there, we will click on “Create resource data sync” button.
    Report3
  4. We will name type a name for the “Sync name”, provide the bucket name of the bucket we previously created.
    Report4
  5. Source: https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-inventory-datasync.html

Using Amazon Athena

In order to access the inventory data from S3, we must first create a schema using Athena. Once the schema is created, we will be able to use it to visualize the data using QuickSight.

  1. Open the Amazon Athena Console.
  2. If you have never used Athena, we will click on the “Get Started” button.
  3. We will then create a database for our compliance data. We will use the PowerShell DSC data. Top do so, we will run the following query.
    Report5
CREATE DATABASE dsc
  1. Once the database is created it, we will populate it with the following query.
    Report6
CREATE EXTERNAL TABLE IF NOT EXISTS dsc.status (
`ServicePath` string,
`InstanceId` string,
`ComputerName` string,
`AppliedMofs` string,
`Status` string,
`Compliant` string,
`NotCompliant` string,
`SetCount` string,
`LastRunId` string,
`LastRunUTC` string
)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
WITH SERDEPROPERTIES (
'serialization.format' = '1'
) LOCATION 's3://ssmbucketfordemo/default/'
TBLPROPERTIES ('has_encrypted_data'='false');
  1. When that is complete, we will run the following query.
    Report7
SELECT computername, instanceid, appliedmofs from dsc.status WHERE status = 'Compliant'
  1. If you don’t see any results, we will change the query to look for status = “NotCompliant”. You should now see similar results.
    Report8

Creating Compliance Reports

  1. To visualize the Inventory and Compliance data, we will use Amazon QuickSight. Let’s open the QuickSight Console.
  2. Then, we will click on the “Connect to another data source or upload a file”.
    Report9
  3. Then, we will select “Athena”.
    Report10
  4. We will type a “Data source name” and select “Crate data source”.
    Report11
  5. We will then select the “dsc” database and the “status” for the data to visualize. Click on the “Select” button.
    Report12
  6. We will use the “Spice” engine for visualization.
    Report13
  7. You can now begin visualizing your data.
    Report14

Further Reading:

Running Ansible Playbooks using Systems Manager Run Command and State Manager: https://aws.amazon.com/blogs/mt/running-ansible-playbooks-using-ec2-systems-manager-run-command-and-state-manager/

Run compliance enforcement and view compliant and non-compliant instances using AWS Systems Manager and PowerShell DSC: https://aws.amazon.com/blogs/mt/run-compliance-enforcement-and-view-compliant-and-non-compliant-instances-using-aws-systems-manager-and-powershell-dsc/

Using AWS Systems Manager to orchestrate PowerShell DSC: https://aws.amazon.com/blogs/infrastructure-and-automation/using-aws-systems-manager-to-orchestrate-powershell-dsc/

Using Amazon QuickSight to report on AWS Systems Manager Inventory data: https://aws.amazon.com/blogs/aws/category/amazon-quicksight/