Skip to main content

CollectorsCloud Collectors Administration Guide

Table of Contents

Prerequisites to Configure the AWS Security Lake Cloud Collector

Before you configure the AWS Security Lake Cloud Collector, complete the following prerequisites:

  • Enable Security Lake. For more information, see Getting started with Amazon Security Lake in the Amazon AWS documentation.

  • Note the S3 bucket details after enabling Security Lake. The bucket name appears in the following format: aws-security-data-lake-<region>-<*****>.

  • Configure S3/SQS Integration that receives a notification when new objects are added to the S3 Bucket.

    • Create a user with access to AWS created s3 bucket for Security Lake.

    • Provide the user with S3 object read permission.

    • Provide the user with SQS/SNS message read and delete access.

    • Update the notification configuration for s3 bucket to publish notification for each .parquet file that is uploaded in the S3 bucket.

  • Obtain the SQS properties.

  • Add Accounts for AWS Cloud Collectors

Note

  • This guide assumes an existence of S3 bucket, and an SQS queue. Before you begin, identify the names for your S3 Bucket and the SQS queue ARNs. If you do not know the names for your S3 Bucket and the SQS queue ARNs, you can find them on the AWS Web Console or use the CLI. Configure the S3 bucket and the SQS queue such that whenever a new object is put into the S3 bucket, the SQS queue gets a notification.

  • Ensure that the SQS queue and the raw log data file in the S3 bucket contain at least one event, so the cloud collector can run a test connection during configuration and start collecting events after successful test.

Configure S3/SQS Integration

Use the following steps to configure the SQS queue that receives a notification when new objects are added to the S3 Bucket.

  1. In the AWS Web Console, navigate to the S3 bucket where the data is located.

  2. Under Properties, enable event notification for the SQS queue in the S3 bucket on which you want to receive logs.

    For more information, see Enable Event Notifications in the AWS documentation.

  3. Choose ObjectCreate (All) events to be notified.

    Refer to the following screenshot to add suffix as .gz.parquet in the suffix option.

    Suffix_config.png

  4. Allow the S3 Bucket to send events to the SQS Queue.

    Replace the access policy attached to the queue with the following policy (in the SQS console, you select the queue, and in the Permissions tab, click Edit Policy Document (Advanced).

    {
   "Version":"2012-10-17",
   "Id":"example-ID",
   "Statement":[
      {
         "Sid":"example-statement-ID",
         "Effect":"Allow",
         "Principal":{
            "AWS":"*"
         },
         "Action":[
            "SQS:SendMessage"
         ],
         "Resource":"SQS-queue-ARN",
         "Condition":{
             "StringEquals": {
               "aws:SourceAccount: "*********"
           },
            "ArnLike":{
               "aws:SourceArn":"arn:aws:s3:*:*:bucket-name"
            }
         }
      }
   ]
}

Obtain the SQS Properties

To obtain the SQS properties (SQS URL, SQS Region, and the SQS Message Origin), log in to your AWS account and refer to the following steps:

  • SQS URL – Go to the SQS service and check the SQS Queue used for the S3 events notifications. In the Details tab copy the URL value.

    360012776914-mceclip0.png

  • SQS Region – In the SQS Service page in the SQS Detail tab, you view the SQS region. It is the string between the sqs and amazonaws strings in the URL. In the example above, the region is: us-east-1.