Skip to main content

CollectorsCloud Collectors Administration Guide

Configure the AWS Security Lake Cloud Collector

Set up the AWS Security Lake Cloud Collector to continuously ingest security events from AWS S3 Security Lake bucket.

  1. Before you configure the AWS Security Lake Cloud Collector, ensure that you complete the prerequisites.

  2. Log in to the New-Scale Security Operations Platform with your registered credentials as an administrator.

  3. Navigate to Collectors > Cloud Collectors.

  4. Click New Collector.

  5. Click AWS Security Lake.

  6. Enter the following information for the cloud collector.

    AWS_Security_Lake.png

    • Name – Specify a name for the Cloud Collector instance.

    • Account – Select the AWS account you want to use with this Cloud Collector.

      If you have not yet created an AWS account, in the Account list, select Create Account. For instructions, see Add Accounts for AWS Cloud Collectors.

    • SQS-URL – Enter the SQS URL that you obtained while completing prerequisites.

    • SQS-Region – Select the SQS region. For example, us-east-1.

    • S3-Region – Select the S3 region. For example, us-west-1.

  7. (Optional) SITE – Select an existing site or to create a new site with a unique ID, click manage your sites. Adding a site name helps you to ensure efficient management of environments with overlapping IP addresses.

    By entering a site name, you associate the logs with a specific independent site. A sitename metadata field is automatically added to all the events that are going to be ingested via this collector. For more information about Site Management, see Define a Unique Site Name.

  8. (Optional) TIMEZONE – Select a time zone applicable to you for accurate detections and event monitoring.

    By entering a time zone, you override the default log time zone. A timezone metadata field is automatically added to all events ingested through this collector.

    Timezone_sitename_site_management_1.png

  9. (Optional) Add filter conditions using regex syntax to include and exempt logs.

    Azure_Event_Hub_Regex_Egress_filtering.png

    • In the Allowed Conditions section, add conditions to include logs to be sent to the New-Scale Security Operations Platform.

      For example, for EventCode that matches 100X, use the Allowed Condition EventCode=100[0-9].

      Raw log:

      LogName=Application EventCode=1001 EventType=4 ComputerName=windows-splunk-forwarder-vp-23 SourceName=Windows Error Reporting Type=Information RecordNumber=168946846 Keywords=Classic TaskCategory=None OpCode=Info Message=Fault bucket , type 0 Event Name: APPCRASH Response: Not available Cab Id: 0

    • In the Deny Conditions section, add conditions for the logs that you don't want to send to the New-Scale Security Operations Platform.

      For example, if you don't want to include eventType between 1 and 5, use the Deny Condition EventType=[1-5].

      Raw log:

      LogName=Application EventCode=1001 EventType=4 ComputerName=windows-splunk-forwarder-vp-23 SourceName=Windows Error Reporting Type=Information RecordNumber=168946846 Keywords=Classic TaskCategory=None OpCode=Info Message=Fault bucket , type 0 Event Name: APPCRASH Response: Not available Cab Id: 0
      Azure_Event_Hub_Regex_Egress_filtering2.png

      Caution

      • Enter a valid regex syntax. Do not enter .* or .+ at the start or at the end of the regex syntax.

      • Use .* or .+ only if required, in the middle of the regex syntax after adding ? next to each of these characters. For example, .*?.

      • Do not use these prefixes ^.*, ^.+,  ^.*?, ^.+? and suffixes .*$, .+$, .*?$, .+?$.

  10. Click Check Filters to verify if the filters are providing the correct results. Then, add log files or paste raw logs in the right pane to verify if the filters that you set are working appropriately.

    Azure_Event_Hub_Regex_Egress_filtering_3_1.png

  11. Click Import.

    If you specify the Allowed and Deny conditions, the cloud collector processes logs based on your conditions. Ensure that the regex pattern that you specify is valid.

    Refer to the following screenshot as an example of the logs that are processed based on the Allowed and Deny conditions.

    Azure_Event_Hub_Regex_Egress_filtering_3.png

    Note

    Using egress filters affects the performance of the collector and decreases overall EPS. The more you set complex filters, the bigger impact you will observe on the collector performance. For example with three filters, overall EPS can decrease approximately by 15% to 20% based on the filter complexity.

  12. To confirm that the New-Scale Security Operations Platform communicates with the service, click Test Connection. 

  13. Click Install.

    AWS_S3_2.png

    A confirmation message informs you that the new Cloud Collector is created.