In the first part of this series Ingesting Non-Microsoft Cloud Security Data into Microsoft Sentinel for Government & DIB Customers we looked into the compliance levels of the clouds and the architecture of the AWS connector. In part 2 Ingesting Non-Microsoft Cloud Security Data into Microsoft Sentinel for Gov & DIB customers part 2 we walked thru setting up the connectors from AWS GovCloud and AWS into Microsoft Sentinel in Azure Government. In this blog we’ll discuss connecting AWS GovCloud into Microsoft Sentinel in Azure Commercial. We recognize that many Government and DIB customers use Azure Commercial and AWS GovCloud, so we’ve included the ability to monitor and respond to those cloud environments. Most of the same steps will apply, this will be very similar to part 2 so I’ve added more trouble shooting information in this part.
Installing the solution in Sentinel from Content Hub
First, we will open the Content Hub in Sentinel and search for “Amazon” to find the solution. The solution includes the S3 connector, 57 Analytic Rules, 36 Hunting Queries & 2 Workbooks.
Click install and the solution will install the connector and other parts of the solution. If you have already installed the Amazon Web Services solution, click on manage. Once the solution is installed You will be able to select the S3 Connector
The connector uses Windows PowerShell and AWS CLI to automate the setup required in AWS.
Configuring the installation environment
Prior to installing the connector using the PS1 script, you must properly configure the system you are using. If you have already done this skip forward to Running the ConfigueAwsConnector.ps1 Script section.
When running the script, ensure that you are using Windows PowerShell as there are Differences between Windows PowerShell and PowerShell 7.x - PowerShell.
Here is the link for installing PowerShell Install PowerShell on Windows, Linux, and macOS - PowerShell
AWS CLI installation is required by the configuration script to create the connections in AWS. Here is the link from Amazon to setup AWS CLI, AWS CLI Installation instructions.
The configuration script will confirm these settings when it runs. If this process returns any errors, check the *_accessKeys.csv file that you downloaded from AWS when you set up the access key, and ensure that the region is correct. In the AWS Console, the region displays US-Gov-West. You will need to update this to us-gov-west-1 when you run the AWS configure command or the script will throw an error.
Verified AWS access configuration
You will need to run the ConfigureAwsConnector.ps1 script. Depending on your organization’s policies, you may not be able to run the PowerShell script without setting the Windows PowerShell execution policy (using Set-ExecutionPolicy) to allow running unsigned scripts. You can check your policies by running.
PS C:\> Get-ExecutionPolicy –List
If you need to change the policy run this command to allow running unsigned scripts
PS C:\> Set-ExecutionPolicy -ExecutionPolicy Unrestricted
Once you are done installing the connections, you should set the execution policy back. More information about setting execution policies can be found on this page Execution Policies - PowerShell
Running the ConfigueAwsConnector.ps1 Script
To get started, download / extract the PowerShell script.
Make note of the log analytics workspace ID, this will be required when running the ConfigureAwsConnector.ps1 script. To further simplify this process, a Cloud Formation Template is in development. That information will be added to this blog series upon release.
In this example we are connecting VPC Flow logs.
The script will ask for the following information in order to setup the OpenID Connection ((OIDC):(
- S3 bucket and will create one if needed.
- Azure Workspace ID
- SQS queue to be created.
Make a note of the information listed under “Role Arn”, “Sqs Queue”, and “Destination Table” as you will need this to configure the connection in Sentinel. On the connector page in Sentinel paste in the information
Enter the Role ARN, SQS URL & Destination table on the connector blade in Sentinel
Then select add connection. The first time you connect, you may see an error (ID25).
This is due to a test message the new SQS queue sends on first connection. The test message does not have the correct record that the connector is expecting and throws the error. Microsoft is working on either clarifying the error message or suppressing the error. When you click connect a second time and then the SQS queue sends the correct message, and the connection will be successful.
You can repeat this process for the other data types in AWS GovCloud. The connection process is the same for each data type. Guard Duty & Cloud Trail data types will ask if you want to use KMS. I used a different KMS for each data type just to keep things organized. Here is an example of what I used in my lab
Once you have all the connections setup you will start seeing data flowing. It might take a few minutes for data to come into Sentinel. This depends on amount of data AWS generates and when Sentinel communicates with the SQS queue. You should see a list of the connection on the Amazon S3 connector page in Sentinel.
Unique Scenario – Sending AWS Guard Duty data into both Azure Gov and Azure Commercial.
I found the scenario for ingesting one AWS account into two Sentinels due to the way my test lab and accounts were configured. I don’t think many customers will encounter this scenario, but I wanted to cover it just in case.
Since Guard Duty Settings it can only use a single bucket / SQS queue, it is not possible to send Guard Duty logs to 2 instances of Sentinel “out-of-the-box”. If you want to do this, you would have to add another Sentinel Workspace and possibly API path to the Trust Relationship Policy.
You will also have to find a way to handle the messages in the SQS queue. Since each Sentinel instance will pick up the messages and the other Sentinel instance may miss the message to get the logs.
If you want to send single AWS account data into 2 Sentinel instances this will require using something like a message bus and custom code. That discussion is beyond the scope of this blog.
Troubleshooting the connector –
In the second blog entry Ingesting Non-Microsoft Cloud Security Data into Microsoft Sentinel for Gov & DIB customers part 2 we investigated verifying data flow and querying it once it has been imported to Microsoft Sentinel. In this section, we will tackle troubleshooting unexpected issues. There are several places that are helpful to check in AWS while troubleshooting or to ensure that the script ran correctly:
- S3 Bucket – logs and event notifications
- VPC – Flow Logs
- SQS queues
- Cloud Trail
- Guard Duty
S3 Buckets - Here, we will be looking for logs and event notifications. In the bucket you should see a folder called AWSLogs. Under that folder look for the folder with your account ID - you may also see a folder that starts with o-****. The folder that with o-*** is where organization logs may be. Look for the *.gz files, these are the actual logs.
Also check the Event Notifications & SQS queues on the bucket to ensure they were created
Check the Bucket Policy and check the OIDC Role
VPC Flow Logs – verify the Destination Name is the correct S3 bucket in the VPC configuration on the AWS console. Then you can verify that the folder mentioned above actually has *.gz files.
SQS Queues – Look on the monitoring tab and verify traffic on the monitor tab of the queue(s). If you don’t see any traffic here, then there might be an issue with the event notifications.
Verify the Access Policy has the correct OIDC role and permissions
Cloud Trail – Verify Trail settings in the Cloud Trail console-
- Trail Log location under General Details. and the IAM Role on the Trail page.
- AWS KMS key is correct
- IAM Role in the CloudWatch Logs
Guard Duty – verify the S3 Protection is on
the correct S3 bucket is set correctly in the console- Guard Duty Settings
Summary
In this part of this series, we’ve walked through setting up the CLI and scripts required to establish the connection from AWS GovCloud into Microsoft Sentinel in Azure commercial and how can verify that the script setup the AWS resources correctly. You're encouraged to explore the workbooks, analytics and hunting queries that are installed with the AWS Solution. These concepts can be explored in-depth with the helpful links below:
- Visualize your data using workbooks in Microsoft Sentinel | Microsoft Learn.
- Detect threats with analytics rule templates in Microsoft Sentinel | Microsoft Learn
- Hunting capabilities in Microsoft Sentinel | Microsoft Learn
We just released the Multi Cloud Attack Essentials solution in the Sentinel Content Hub & GitHub. Here is the link to the repository Azure-Sentinel/Solutions/Multi Cloud Attack Coverage Essentials - Resource Abuse at master · Azure/A...
In our next blog, we will cover Google Cloud Platform (GCP) integration and the new Cloud Formation installation option for AWS.