Skip to main content

CollectorsCloud Collectors Administration Guide

Configure the Splunk Cloud Collector

Set up the Splunk Cloud Collector to continuously ingest security events from Splunk Cloud.

  1. Before you configure the Splunk Cloud Collector, ensure that you complete the Prerequisites to Configure the Splunk Cloud Collector.

  2. Log in to the New-Scale Security Operations Platform with your registered credentials as an administrator.

  3. Navigate to Collectors > Cloud Collectors.

  4. Click New Collector.

    Splunk_home_page.png

  5. Click Splunk.

  6. Enter the following information for the cloud collector:

    Splunk_Configuration_1.png

    • Name – Specify a name for the Cloud Collector instance.

    • Account – Select an account. If you have not created a Splunk account, click New Account to add a new Splunk account. You can use this account information across one or more Splunk Cloud Collectors. For more information, see Add Accounts for Splunk Cloud Collector.

    • Event Type – Select the format in which you want to receive event data. This is a format in which you want your Splunk query to respond.

      • Plain text – Use the plain text format to ingest a cloud log source that can forward logs in plain text format.

        Event_type__plain_Text2.png

      • JSON – Use the JSON format to ingest a cloud log source that can forward logs in JSON format, containing single or multiple objects.

        Event_type_Json_2.png

      • Windows Multiline – Use the Windows Multiline format to ingest a cloud log source that can forward logs in Windows Multiline format.

        Event_type_windows_Multiline_2.png

      Note

      The Splunk Cloud Collector fetches the Splunk metadata fields: _time, sourcetype, and host in addition to the _raw field.

    • Splunk Query – Enter the Splunk query that you use in the Splunk Cloud for fetching data, based on your requirement. Ensure that the Splunk query that you specify suits the event format that you set for receiving data. For example, a Splunk search query: search index=_internal.

    • Ingest From – Select the time and date to provide a threshold before which the collector will exclude events. If you want to select a threshold to include events to be ingested from the past, select a date previous to the present date. You can select a date which is backdated to 30 days.

      If you leave this field blank and do not provide a threshold, all data is ingested starting from when the collector started running. However, the Cloud Collector does not collect historical events.

      Note

      • After you complete the Cloud Collector configuration, you cannot modify the Ingest From date.

      • Collecting historical events is not supported with Advanced Analytics i62.x or i63.x. For more information see Supported Deployments.

  7. (Optional) SITE – Select an existing site or to create a new site with a unique ID, click manage your sites. Adding a site name helps you to ensure efficient management of environments with overlapping IP addresses.

    By entering a site name, you associate the logs with a specific independent site. A sitename metadata field is automatically added to all the events that are going to be ingested via this collector. For more information about Site Management, see Define a Unique Site Name.

  8. (Optional) TIMEZONE – Select a time zone applicable to you for accurate detections and event monitoring.

    By entering a time zone, you override the default log time zone. A timezone metadata field is automatically added to all events ingested through this collector.

    Timezone_sitename_site_management_1.png

  9. To confirm that the New-Scale Security Operations Platform communicates with the service, click Test Connection. 

  10. Click Install.

    AWS_S3_2.png

    A confirmation message informs you that the new Cloud Collector is created.

Troubleshoot Common Errors

Refer to the following examples to troubleshoot common errors.

Example 1– Error Message about Search Job Limit

Error – The following error indicates that the current searches are 50 and the search job limit is 50.  The Search Job Limit must be greater than the configured limits.

HTTP 503 – {"messages":[{"type":"WARN","text":"Search not executed: The maximum number of concurrent historical searches on this instance has been reached., concurrency_category=\"historical\", concurrency_context=\"instance-wide\", current_concurrency=50, concurrency_limit=50","help":""}]}

Resolution – In the Splunk, set the Search Job Limit to twice the total number of Splunk queries configured in the Splunk Cloud Collectors. For example, if you have three collectors configured—Windows, Linux, and Firewall—the recommended Search Job Limit should be six.

Example 2 – Error Message about Search Disk Cache Limit

Error – The following error indicates current disk usage is over the 500MB limit.

HTTP 503 -- {"messages":[{"type":"ERROR","text":"reason=\"Search not executed: The maximum disk usage quota for this user has been reached. Use the Job Manager to delete some of your saved search results.\", usage=686MB, quota=500MB, user=exabeam, concurrency_category=\"historical\", concurrency_context=\"user_instance-wide\"","help":""}]}

Resolution – In Splunk, it is best practice to increase the disk usage limit. Search results are stored for 10 minutes. Ensure that you have enough storage for all query results including possible spike in data.