Skip to main content

CollectorsCloud Collectors Administration Guide

Prerequisites to Configure the AWS S3 Cloud Collector

Before you configure the AWS S3 Cloud Collector, complete the following prerequisites:

Note

  • This guide assumes an existence of S3 bucket, and an SQS queue. Before you begin, identify the names for your S3 Bucket and the SQS queue ARNs. If you do not know the names for your S3 Bucket and the SQS queue ARNs, you can find them on the AWS Web Console or use the CLI. Configure the S3 bucket and the SQS queue such that whenever a new object is put into the S3 bucket, the SQS queue gets a notification.

  • Ensure that the SQS queue and the raw log data file in the S3 bucket contain at least one event, so the cloud collector can run a test connection during configuration and start collecting events after successful test.

Configure S3/SQS Integration

Use the following steps to configure the SQS queue that receives a notification when new objects are added to the S3 Bucket.

  1. In the AWS Web Console, navigate to the S3 bucket where the data is located.

  2. Under Properties, enable event notification for the SQS queue in the S3 bucket on which you want to receive logs.

    For more information, see Enable Event Notifications in the AWS documentation.

  3. Choose ObjectCreate (All) events to be notified.

    360012808673-mceclip0.png

  4. Allow the S3 Bucket to send events to the SQS Queue.

    1. Replace the access policy attached to the queue with the following policy (in the SQS console, you select the queue, and in the Permissions tab, click Edit Policy Document (Advanced).

      {
   "Version":"2012-10-17",
   "Id":"example-ID",
   "Statement":[
      {
         "Sid":"example-statement-ID",
         "Effect":"Allow",
         "Principal":{
            "AWS":"*"
         },
         "Action":[
            "SQS:SendMessage"
         ],
         "Resource":"SQS-queue-ARN",
         "Condition":{
             "StringEquals": {
               "aws:SourceAccount: "*********"
           },
            "ArnLike":{
               "aws:SourceArn":"arn:aws:s3:*:*:bucket-name"
            }
         }
      }
   ]
}

Obtain the SQS Properties

To obtain the SQS properties (SQS URL, SQS Region, and the SQS Message Origin), log in to your AWS account and refer to the following steps:

  • SQS URL – Go to the SQS service and check the SQS Queue used for the S3 events notifications. In the Details tab copy the URL value.

    360012776914-mceclip0.png

  • SQS Region – In the SQS Service page in the SQS Detail tab, you view the SQS region. It is the string between the sqs and amazonaws strings in the URL. In the example above, the region is: us-east-1.

  • SQS-Message-Origin – Based on the integration you are configuring; messages are put in the queue in different ways.

    • For the messages put in the queue by the above setup, in which S3 notifies the SQS directly, select S3_TO_SQS.

    • For the messages put in the queue using the above method, in which the messages are sent from S3 to SNS, and then configured in SNS to be sent to SQS, select S3_TO_SNS_TO_SQS.