Skip to main content

CollectorsCloud Collectors Administration Guide

AWS CloudTrail (via S3) Cloud Collector

Migrate the AWS CloudTrail (via S3) Cloud Collector

The AWS CloudTrail (via S3) Cloud Collector was first introduced for early access on the Exabeam Cloud Connectors platform.

With Exabeam Cloud Collectors, the AWS CloudTrail (via S3) Cloud Collector is now available for the Exabeam Security Operations Platform. The new cloud collector enables you to ingest logs into the Exabeam Security Operations Platform and use the Exabeam Search to find specific events in those logs. Migration to the new app is recommended.

AWS-MultiTenant Cloud Connector's cloud-trail endpoint is mapped to the new Cloud Collector AWS CloudTrail (via S3) and the new AWS account. If you previously used the early access AWS CloudTrail (via S3) Cloud Connector, and want to take advantage of the new Cloud Collectors service, you must first migrate the SaaS cloud connector. Before you migrate, consider the following:

  • License Requirements: No additional license is required. The Cloud Collectors app is included with your existing license.

  • SaaS Cloud Connectors Support: Both the SaaS Cloud Connectors and the new Cloud Collectors environments can run in parallel.

The following table displays the AWS-MultiTenant Cloud Connector's endpoints that are mapped to the new Cloud Collector AWS CloudTrail (via S3).

Legacy Connector

Legacy Endpoint

New Collector

New Account

AWS-MultiTenant

Cloud-trail

AWS CloudTrail (via S3)

AWS

When you are ready to migrate:

  1. Stop the AWS account for AWS CloudTrail (via S3) Cloud Collector.

  2. Check the account status. If the account status displays the status as running, wait until the tasks that are running are complete, and the status changes to 'complete'.

  3. Follow the steps to Configure the AWS CloudTrail (via S3) Cloud Collector.

Prerequisites to Configure the AWS CloudTrails (via S3) Cloud Collector

Before you set up the cloud collector, review the following prerequisites:

Set up AWS to Send CloudTrail Logs From Multiple AWS Accounts Into a Single Account

To monitor CloudTrail from multiple AWS accounts, you can set up AWS to forward the trails from the various accounts into a single account’s S3 Bucket. You can also set up a SQS queue that is notified when new objects are added to that bucket.

In this guide you will implement this architecture within AWS, including the creation of the collecting bucket, the SQS queue, the various accounts’ trails, all with air-tight permissions/policies, and finally you will create an AWS IAM User with permissions to orchestrate this setup, again, with air-tight policy.

For more information, refer to the AWS documentation for Receiving CloudTrail log files from multiple accounts and Amazon S3 Event Notifications.

  1. Collect all the account numbers that will be sending CloudTrail events.

    Find your Account # in My Account under Account Settings > Account ID. The account number is a 12 digit number, for example 111111111111.

  2. In the account you want to set up to receive the trails (for the duration of this procedure, we use Account A as an example of the account name), create an S3 bucket.

    Record the ARN for the bucket (see the properties tab).

  3. Create an SQS queue (Standard).

    Note

    Make sure the Queue is in the same region as the bucket.

    Record the URL.

  4. Set a policy for the SQS Queue to allow only the S3 bucket to send events to it:

    1. In SQS, select the queue and then Permissions.

    2. Click Edit Policy Document (Advanced).

    3. Paste the following policy and modify it for your environment.

       { 
              "Version": "2012-10-17", 
              "Id": "queue ARN/SQSDefaultPolicy", 
              "Statement": [ 
                  { 
                  "Effect": "Allow", 
                  "Principal": { 
                      "AWS": "*" 
                  }, 
                  "Action": ["SQS:SendMessage"], 
                  "Resource": "queue ARN", 
                  "Condition": { 
                      "ArnLike": { 
                      "aws:SourceArn": "bucket ARN" 
                      } 
                  } 
                  } 
              ] 
        } 
  5. Configure the S3 Bucket to notify the SQS Queue when files are added to it:

    1. In S3, select the Properties tab for the bucket.

    2. Select Events > Add notification.

    3. Enter a meaningful name (i.e. “Notify SQS new Files”).

    4. Select ObjectCreate (All).

    5. Under Send to, select SQS Queue.

    6. Select the queue and then Save your changes.

  6. Set a policy for the S3 Bucket to allow only the accounts for which you want to send CloudTrail events to this bucket to add objects to it:

    1. In S3, select the bucket > Permissions > Bucket Policy.

    2. Paste the following policy and modify it to add a row with the account ID for each source account:

      {
          "Version": "2012-10-17",
          "Statement": [
              {
              "Effect": "Allow",
              "Principal": {
                  "Service": "cloudtrail.amazonaws.com"
              },
              "Action": "s3:GetBucketAcl",
              "Resource": "Bucket ARN"
              },
              {
                  "Effect": "Allow",
                  "Principal": {
                      "Service": "cloudtrail.amazonaws.com"
                  },
                  "Action": "s3:PutObject",
                  "Resource": [
                      "Bucket ARN/[optional] myLogFilePrefix/AWSLogs/Account ID 1/*",
                      "Bucket ARN/[optional] myLogFilePrefix/AWSLogs/Account ID 2/*"
                  ],
                  "Condition": { 
                      "StringEquals": { 
                      "s3:x-amz-acl": "bucket-owner-full-control" 
                      }
                  }
              }
          ]
      }
      
  7. Configure a CloudTrail for each account to send its events to the bucket in Account A.

    1. Log in to the desired account.

    2. In CloudTrail, create a new trail (or re-configure an existing one).

    3. Under Storage location select No in Create new bucket and set the bucket name (not ARN) of Account A.

Configure the AWS CloudTrail (via S3) Cloud Collector

The AWS CloudTrail (via S3) Cloud Collector collects the data stored in S3 buckets to ingest data in the Exabeam Security Operations Platform.

To configure the AWS CloudTrail (via S3) Cloud Collector:

  1. Complete the prerequisites to configure the AWS CloudTrail (via S3) collector.

  2. Log in to the Exabeam Security Operations Platform with your registered credentials as an administrator.

  3. Navigate to Collectors > Cloud Collectors.

  4. Click New Collector.

    AWS_S3_collectors_page.png
  5. Click AWS CloudTrail (via S3).

    AWS_S3_1.png
  6. Specify a name for the Cloud Collector instance.

  7. Select the AWS account that you created.

    If you have not created an AWS account, in the Account list, select Create Account. For instructions, see Add Accounts for AWS Cloud Collectors.

  8. Enter the SQS URL that you obtained while completing the prerequisites The URL of the SQS is in use for the central audit. For example (not a valid value for use), https://sqs.us-west-2.amazonaws.com/123456789/new-trail-s3-file-from-snsS3

  9. Select the SQS region. For example, us-east-1.

  10. Select the S3 bucket region name, from the S3 bucket used for the central CloudTrail. For example, us-west-2.

  11. (Optional) SITE – Select an existing site or to create a new site with a unique ID, click manage your sites. Adding a site name helps you to ensure efficient management of environments with overlapping IP addresses.

    By entering a site name, you associate the logs with a specific independent site. A sitename metadata field is automatically added to all the events that are going to be ingested via this collector. For more information about Site Management, see Define a Unique Site Name.

  12. (Optional) TIMEZONE – Select a time zone applicable to you for accurate detections and event monitoring.

    By entering a time zone, you override the default log time zone. A timezone metadata field is automatically added to all events ingested through this collector.

    Timezone_sitename_site_management_1.png
  13. (Optional) Add filter conditions using regex syntax to include and exempt logs.

    Note

    This feature is available as a part of the early access program. To participate, see Sign Up for the Early Access Program.

    Azure_Event_Hub_Regex_Egress_filtering.png
    • In the Allowed Conditions section, add conditions to include logs to be sent to the Exabeam Security Operations Platform.

      For example, for EventCode that matches 100X, use the Allowed Condition EventCode=100[0-9].

      Raw log:

      LogName=Application EventCode=1001 EventType=4 ComputerName=windows-splunk-forwarder-vp-23 SourceName=Windows Error Reporting Type=Information RecordNumber=168946846 Keywords=Classic TaskCategory=None OpCode=Info Message=Fault bucket , type 0 Event Name: APPCRASH Response: Not available Cab Id: 0
    • In the Deny Conditions section, add conditions for the logs that you don't want to send to the Exabeam Security Operations Platform.

      For example, if you don't want to include eventType between 1 and 5, use the Deny Condition EventType=[1-5].

      Raw log:

      LogName=Application EventCode=1001 EventType=4 ComputerName=windows-splunk-forwarder-vp-23 SourceName=Windows Error Reporting Type=Information RecordNumber=168946846 Keywords=Classic TaskCategory=None OpCode=Info Message=Fault bucket , type 0 Event Name: APPCRASH Response: Not available Cab Id: 0
      Azure_Event_Hub_Regex_Egress_filtering2.png
  14. Click Check Filters to verify if the filters are providing the correct results. Then, add log files or paste raw logs in the right pane to verify if the filters that you set are working appropriately.

    Azure_Event_Hub_Regex_Egress_filtering_3_1.png
  15. Click Import.

    If you specify the Allowed and Deny conditions, the cloud collector processes logs based on your conditions. Ensure that the regex pattern that you specify is valid.

    Refer to the following screenshot as an example of the logs that are processed based on the Allowed and Deny conditions.

    Azure_Event_Hub_Regex_Egress_filtering_3.png

    Note

    Using egress filters affects the performance of the collector and decreases overall EPS. The more you set complex filters, the bigger impact you will observe on the collector performance. For example with three filters, overall EPS can decrease approximately by 15% to 20% based on the filter complexity.

  16. To verify that the cloud collector can communicate with the AWS service, click Test Connection.

    Note

    You can start the cloud collector via the user interface only if the test connection during cloud collector configuration is successful.

  17. Click Install.

    AWS_S3_2.png

    A confirmation message informs you that the new collector is created.