Skip to main content

CollectorsCloud Collectors Administration Guide

Configure the Rest API Cloud Collector

The REST API Cloud Collector offers configurable data collection from REST API endpoints, streamlines log ingestion, and supports custom API integrations. The collector provides customizable parameters that include dynamic variables to allow you to tailor API requests. You can easily insert these parameters in request configurations to ensure compatibility with various vendor APIs. The multiple pagination strategies available for configuration enable you to choose the most suitable approach based on the vendor REST API.

Refer to the following sections for more details. 

Understand Key Terminologies

This section includes more information about important terms or concepts used within the collector configuration page. Learning the essential definitions, roles, and relationships of critical terms will help you to seamlessly configure the REST API collector.

  • Request Headers – Request headers provide metadata about the API request. The metadata includes information such as authentication details, content type, and caching preferences.

  • Request Parameters – In a REST API, these are values sent by the client to the API server as part of an HTTP request. These values are used for specifying or modifying the data being requested or submitted.

  • Request Methods (GET or POST) – These are the two most commonly used HTTP request methods. The GET request is used for retrieving data from the API server. The POST request is used for submitting data to the API server.

  • Pagination – It is the practice of dividing large datasets into smaller, manageable pages when retrieving data from an API. Pagination enables efficient data handling and provides a structured way to navigate between pages.

    • Response Body Attribute – It is used for extracting a value from the response body that identifies the next page of data to retrieve.

    • Response Header Attribute – Name(s) of attribute(s) in the response header that contain next-page information. It is used for extracting the next-page value from the response header.

    • Web-linking – Web-linking in response headers is commonly used for pagination by including Link headers with rel="next" and rel="prev". These headers help clients such as browsers, search engines, APIs to understand how to navigate between paginated resources efficiently. Refer to the following example of Pagination in Response Headers.

      HTTP/1.1 200 OKLink: <https://example.com/page2>;; rel="next",<https://example.com/page0>;; rel="prev"

  • Timestamp Format – The format for the timestamp depends on the API you want to work with. APIs require timestamp to synchronize data, ensure accurate event ordering, or implement request validation. For example, you can send a request with a timestamp to an API that processes time-sensitive data.

    Following are the widely used timestamp formats.

    • EPOCH MSEC: Represents the timestamp in epoch milliseconds. For example, 1738231737108 corresponds to Thursday, January 30, 2025 10:08:57.108 AM

    • EPOCH SEC: Represents the timestamp in epoch seconds. For example, 1738231737 corresponds to Thursday, January 30, 2025 10:08:57.000 AM

    • CUSTOM: Represents a user-defined timestamp format. For example, yyyy-MM-dd'T'HH:mm:ss.SSS'Z'; Thursday, January 30, 2025 10:08:57.108 AM would be represented as 2025-01-30T10:08:57.108Z.

  • Auth (Authentication Type) – Authentication type for authenticating the log source. The supported authentication types are: Basic, OAuth 2.0, and Static Token.

Use Dynamic Variables

The REST API Cloud Collector supports use of variables to configure the collector. Variables are placeholders for different values in the API request that need to be computed for every API request. The collector supports the following types of variables.

  • Timestamp variables

  • Predefined variables

  • Custom variables

Timestamp Variables

In time-series REST APIs, ${startTime} and ${endTime} are the dynamic parameters that are used for defining the time range for querying data. These placeholders are replaced with actual timestamps during API requests to retrieve specific data within a given period.

The timebound parameters ${startTime} and ${endTime} are predefined by Exabeam and supported for the REST API collector configuration.

Note

Ensure that you use the required dynamic variables ${startTime} and ${endTime} that facilitate time-bound data collection in your collector configuration.

Refer to the following examples of an API request.

Vendor API

Collector API Configuration

API Request

GET https://example.com/events?event_start_time=${startTime}&event_end_time=${endTime}

The collector is configured with the following parameters.

  • Request Method – GET

  • Request URL – https://example.com/events

  • Request Parameters – event_start_time: ${startTime}

    event_end_time: ${endTime}

The configuration corresponds to the following API request.

  • Request Method – GET

  • Request URL – https://example.com/events

  • Request Parameters – event_start_time: 2025-01-30T00:00:00Z

    event_end_time: 2025-01-30T23:59:59Z

Predefined Variables

While configuring the collector, you can use the variables that are are predefined by Exabeam. Refer to the following example to know more about the usage of predefined variables.

For the Static Token authentication type, click the copy icon to copy the variable reference of this field ${authentication.staticToken}. Use this value in Authorization Expression for every API request.

  • Static Token – f408018a-38c7-467c-9c7e-0ab77df12345

  • Authorization Expression – Bearer ${authentication.staticToken}

  • Value computed during evaluation – Bearer f408018a-38c7-467c-9c7e-0ab77df51495

Note

The copy icon next to each variable lets you to quickly copy its reference name to your clipboard for later use and helps you add variables effortlessly and minimize configuration errors.

Custom Variables

The custom variables are user-defined variables. These variables are evaluated for every API request and their values are determined based on the response of the previous API request.

Refer to the following screenshot for an example of custom variable.

Example_custom_variables.png

Note

The copy icon next to each variable lets you to quickly copy its reference name to your clipboard for later use and helps you add variables effortlessly and minimize configuration errors.

Refer to the following example of usage of custom variables in pagination.

Vendor API

Pagination Configuration

API Call

Base Request

GET https://example.com/events?event_start_time=${startTime}&event_end_time=${endTime}

Sample API Response

{
    "data": [
        {
            "event": "login",
            "timestamp": "2025-01-30T00:00:00Z",
            ...
        }
    ],
    "pagination": {
        "nextCursor": "fdjbrt45834h23ebfe23843583eeu",
        "totalItems": 1
    }
}

The vendor provides a cursor handle/pointer in the API response which you can use for fetching the next page of results from the API.

In the example, nextCursor is the cursor to get the next page of results.

For this example, assume that the Vendor API requires you to send the cursor as an HTTP Request Parameter called cursor in the API request to get the next set of results.

Refer to the Pagination configuration column for configuration details.

  • Request Method – GET

  • Request URL – https://example.com/events

  • Pagination – Response Body Attribute

  • Pagination Cursor Type – Next Token

  • Page Response Body Attributes –

    nextPageCursor: $.pagination.nextCursor

    In this example, nextPageCursor is the custom variable defined. Use the copy icon to copy its variable reference.

  • Request Parameters –

    • event_start_time: ${startTime}

    • event_end_time: ${endTime}

    • cursor: ${collect.pageResponseBodyAttribute.nextPageCursor}

      The custom variable references here. The value of this variable is computed during the next page API call.

With this pagination configuration, the API call initiated by the collector appears as follows.

  • Initial API Request –The initial API request would not have the cursor parameter to begin with, here the cursor parameter is not sent in the initial request.

    • Request Method – GET

    • Request URL – https://example.com/events

    • Request Parameters

      • event_start_time: 2025-01-30T00:00:00Z

      • event_end_time: 2025-01-30T23:59:59Z

  • Next Page API Request – Cursor is provided for the following request if it is available in the initial API response (Refer the sample API response above).

    • Request Method – GET

    • Request URL – https://example.com/events

    • Request Parameters –

      • event_start_time: 2025-01-30T00:00:00Z

      • event_end_time: 2025-01-30T23:59:59Z

      • cursor: fdjbrt45834h23ebfe23843583eeu

Supported Features

The following table displays the list of features that are supported by the REST API cloud collector.

Feature

Options

Supported

Not Supported

Request Method

GET

✓

POST

✓

Request Headers

✓

Request Parameters

✓

Response Body Attribute

Pagination Cursor Type

✓

Response Header Attribute

Response Header Attribute

✓

Web-linking (RFC 5988)

Next Page Link

✓

Ingest From

Historical Data Collection

✓

Timestamp Format

  • EPOCH MSEC

  • EPOCH SEC

  • CUSTOM

✓

Authentication

None

✓

Basic

✓

OAuth 2.0

✓

Static Token

✓

XAuth

✓

Collection Period

✓

Collection Interval

✓

Collection Timeout

✓

Maximum Historical Fetch Duration

✓

Set up the Rest API Cloud Collector

Set up the Rest API Cloud Collector to continuously ingest data from API that the you want to configure.

  1. Before you configure the REST API Cloud Collector, ensure that you complete the prerequisites.

  2. Log in to the New-Scale Security Operations Platform with your registered credentials as an administrator.

  3. Navigate to Collectors > Cloud Collectors.

  4. Click New Collector.

  5. Click REST API.

  6. Specify a name for the collector.

    REST1.png

  7. Proceed to configure request, pagination, authentication, processing, and settings.

    1. Configure the API Request 

    2. Configure Pagination

    3. Configure Other Request Parameters

    4. Configure Authentication 

    5. Configure Event Processing and Additional Settings  

  8. (Optional) SITE – Select an existing site or to create a new site with a unique ID, click manage your sites. Adding a site name helps you to ensure efficient management of environments with overlapping IP addresses.

    By entering a site name, you associate the logs with a specific independent site. A sitename metadata field is automatically added to all the events that are going to be ingested via this collector. For more information about Site Management, see Define a Unique Site Name.

  9. (Optional) TIMEZONE – Select a time zone applicable to you for accurate detections and event monitoring.

    By entering a time zone, you override the default log time zone. A timezone metadata field is automatically added to all events ingested through this collector.

    Timezone_sitename_site_management_1.png

  10. To confirm that the New-Scale Security Operations Platform communicates with the service, click Test Connection. 

  11. Click Install.

    AWS_S3_2.png

    A confirmation message informs you that the new Cloud Collector is created.

Configure the API Request

In the Request tab, specify details for request to be sent the REST API server.

  • REQUEST URL – Enter the URL expression for data collection. For example, https://dev-123456.okta.com/api/v1/logs.

  • REQUEST METHOD – Select the API request method: GET or POST to retrieve data from a server through an API.

    The following table describes each field associated with GET and POST request. 

    Field

    Description

    Request Headers 

    Click New and specify values for KEY and VALUE to provide context, metadata, or instructions to the REST API server. Key is a header name.

    Request Parameters 

    Click New and specify values for KEY and VALUE to send additional information to the server. Key is the is the parameter name.

    REQUEST BODY (only for POST)

    Specify the payload required for the API request. Refer to the vendor specific REST API documentation to determine the payload required. You can use one or more Predefined Dynamic Parameters.

The following table displays the list of features that are currently supported by the REST API cloud collector for configuring an API request.

Feature

Options

Supported

Not Supported

Request Method

GET

✓

POST

✓

Request Headers

✓

Request Parameters

✓

Configure Pagination

Select the pagination type: None, Response Body Attribute, Response Header Attribute, Web-linking (RFC 5988) for your integration. Refer to the vendor specific REST API documentation to determine the pagination type you need.

Refer to the following description for each of the pagination options.

None

Select None if you do not want to use pagination.

Response Body Attribute

Use Response Body Attribute if you want the collector to extract a value from the response body that identifies the next page of data to retrieve.

  • PAGINATION CURSOR TYPE – Select the pagination cursor type URL or NEXT TOKEN.

    • URL (PATH TO CURSOR KEY)– Select this option if the vendor REST API provides a URL in the API response body to retrieve the next set of results.

    • NEXT TOKEN (Page Response Body Attribute) – Select this option if the vendor REST API provides a token or a handle to use for fetching the next set of results. The key is a custom variable name of your choice and the value is the JSON path to the attribute.

      Note

      Click the copy icon next to each variable to quickly copy its reference name to your clipboard for later use. Using this icon you can add variables effortlessly and minimize configuration errors.

      Refer to the Configure Event Processing and Additional Settings  section to know how to specify a JSON path.

      Refer to the integration examples section for more details.

Response Header Attribute

Select Response Header Attribute if you want the collector to extract a particular URL or token from one of the response headers.

  • CURSOR HEADER NAME – Enter the name for response header which has the cursor for pagination to extract the next-page value from the response header.

  • PAGINATION CURSOR TYPE – Select the pagination cursor type URL or NEXT TOKEN.

    The Pagination Cursor Type depends on whether pagination comes in the response as URL for next API call or token that the collector needs to pass as parameter to the request.

    • URL – Cursor Key Format Regex – Enter a regex pattern to extract the cursor for pagination. For example, <([^>]+)>;\s*rel="next"

    • NEXT TOKEN – Page Response Header Attribute – Enter the key and value for Response Header Attribute to extract the next page attributes from the response header. The key is a custom variable name and the value is a Regular Expression (RegEx) to extract the value of the attribute from the header.

Web-linking (RFC 5988)

Select Web-linking (RFC 5988) if you want to include links such as next and previous to API responses to enable navigation for paginated data.

Next Page Link Regex Format – If you select Web-linking, enter a regex pattern to identify the next page link for the data collection phase. You can use this option for APIs that follow RFC 5988 conventions to specify the next-page link in a header.

For example, refer to Integration Example: Okta API Integration and Integration Example: ServiceNow API Integration.

Configure Other Request Parameters

Set the time and date for the collector to start ingesting REST API data and a format for the timestamp.

  • INGEST FROM – Select the time and date from which the collector must start ingesting REST API data. If you leave this field blank and do not provide a threshold, all logs are ingested.

  • TIMESTAMP FORMAT – Select a format for the timestamp to send a request to the API: EPOCH MSEC, EPOCH SEC, or CUSTOM. For more information about the timestamp format, refer to the Understand Key Terminologies section.

    Refer to the following table to select an option to determine the Custom date and time format supported by the log source API. For more information about convention to be followed, see Date and Time Patterns in Oracle documentation.

    Date and Time Pattern

    Result

    yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'

    2024-05-20T12:59:48.000495Z

    yyyy-MM-dd'T'HH:mm:ss.SSSX

    2024-05-20T12:59:48.495Z

    yyyy-MM-dd'T'HH:mm:ss.SSSZ

    2024-05-20T12:59:48.495+0000

    yyyy-MM-dd'T'HH:mm:ss.SSS

    2024-05-20T12:59:48.495

    yyMMddHHmmssZ

    240520125948+0000

    EEE, d MMM yyyy HH:mm:ss Z

    Mon, 20 May 2024 12:59:48 +0000

The following table displays the list of features that are currently supported by the REST API cloud collector for configuring pagination.

Feature

Options

Supported

Not Supported

Response Body Attribute

Pagination Cursor Type

✓

Response Header Attribute

Response Header Attribute

✓

Web-linking (RFC 5988)

Next Page Link

✓

Ingest From

Historical Data Collection

✓

Timestamp Format

  • EPOCH MSEC

  • EPOCH SEC

  • CUSTOM

✓

Configure Authentication

In the AUTH section, select one of the following authentication types based on what the API that you want to integrate supports.

Authentication Type

Description

None 

Retain the default option None if you do not want to use authentication.

Basic 

Select to enter HTTP Basic authentication username and password. Select this type if the API supports basic authentication.

OAuth 2.0 

Select to enter URL, headers and parameters for authentication via the OAuth 2.0 protocol.

  • LOGIN URL – Enter the endpoint for the OAuth API call with POST request method. This URL is the URL of the authentication server used for validation.

  • Authentication Headers – Enter key and Value for the parameter for each additional OAuth request parameter you want to send in the body of POST requests.

  • Authentication Body Parameters – Enter key and Value for the parameter for each additional OAuth request parameter you want to send in the body of POST requests.

  • PATH TO TOKEN – Enter the JSON path to the token attribute in the response. For example, $.access_token.

  • AUTHORIZATION HEADER – Enter the header name to pass during the collection phase. The header name is used in API call. For OAuth, pass the token that you get in the OAuth call, in the Authorization header for the REST API call.

  • AUTHORIZATION EXPRESSION – Enter the Value corresponding to the Authorization Header. For example, Bearer ${authentication.staticToken}

  • ACCESS TOKEN FORMAT – Select Plain Text or JSON to determine the format of the authentication request to authenticate the log source.

Static Token 

Select to enter values for static token, authorization header, and authorization expression for authentication via access token.

  • STATIC TOKEN – Enter the value of the access token that is used for authenticating the log source. For more information and examples, see .

  • AUTHORIZATION HEADER – Enter the header name for the authentication of the API.

  • AUTHORIZATION EXPRESSION – Enter the Value corresponding to the Authorization Header. For example, `Bearer ${authentication.staticToken}`

The following table displays the list of features that are currently supported by the REST API cloud collector for configuring authentication.

Feature

Options

Supported

Not Supported

Authentication

None

✓

Basic

✓

OAuth 2.0

✓

Static Token

✓

XAuth

✓

Configure Event Processing and Additional Settings 

In the Processing tab, specify the following details. 

PATH TO EVENT – Specify a JSON Path to individual events within the collection HTTP response. The JSON Path determines the property within the response body that contains events that you want the collector to collect.

The collector retrieves events from the Vendor API JSON response based on the JSON Path you specify. Ensure that the JSON Path directs to the desired events in the API response. Check the Events Extracted section in the Test Connection results to confirm that the JSON Path is fetching the correct events.

Refer to the following table for the examples of the JSON Path to be specified for different event formats.

#

API Response

JSON Path

Events extracted

Number of events extracted

1

{
  "status": "success",
  "metadata": {
    "request_time": "2025-01-30T23:59:59Z",
    "total_results": 2,
    "data_range": {
      "start": "2025-01-30T00:00:00Z",
      "end": "2025-01-30T23:59:59Z"
    },
    "pagination": {
      "page": 1,
      "per_page": 50
    }
  },
  "events": [
    {
      "event_id": "event-001",
      "timestamp": "2025-01-30T10:15:00Z",
      "action": "suspicious_login_attempt",
      "user": {
        "id": "user501",
        "username": "john_doe",
        "ip": "198.51.100.22",
        "location": "New York, USA"
      },
      "details": {
        "device_type": "mobile",
        "browser": "Chrome 89",
        "operating_system": "Android"
      },
      "severity": "medium",
      "log_type": "authentication"
    },
    {
      "event_id": "event-002",
      "timestamp": "2025-01-30T14:45:00Z",
      "action": "unauthorized_access",
      "user": {
        "id": "user502",
        "username": "alice_smith",
        "ip": "203.0.113.45",
        "location": "London, UK"
      },
      "details": {
        "target_system": "database_server",
        "access_type": "read",
        "method": "brute_force"
      },
      "severity": "high",
      "log_type": "access_control"
    }
  ]
}

$.events[*]

Event 1

{
  "event_id": "event-001",
  "timestamp": "2025-01-30T10:15:00Z",
  "action": "suspicious_login_attempt",
  "user": {
    "id": "user501",
    "username": "john_doe",
    "ip": "198.51.100.22",
    "location": "New York, USA"
  },
  "details": {
    "device_type": "mobile",
    "browser": "Chrome 89",
    "operating_system": "Android"
  },
  "severity": "medium",
  "log_type": "authentication"
}

Event 2

{
  "event_id": "event-002",
  "timestamp": "2025-01-30T14:45:00Z",
  "action": "unauthorized_access",
  "user": {
    "id": "user502",
    "username": "alice_smith",
    "ip": "203.0.113.45",
    "location": "London, UK"
  },
  "details": {
    "target_system": "database_server",
    "access_type": "read",
    "method": "brute_force"
  },
  "severity": "high",
  "log_type": "access_control"
}

2

2

[
    {
          "Time": "2024-07-17 09:51:51",
          "Username": "[email protected]",
          "IP_Address": "10.16.21.21",
          "Action": "Failed Login Attempt",
          "Data": ""
    },
    {
          "Time": "2024-07-17 09:51:56",
          "Username": "[email protected]",
          "IP_Address": "10.16.21.21",
          "Action": "Failed Login Attempt",
          "Data": ""
    }
}

$.[*]

Event 1

{
    "Time": "2024-07-17 09:51:51",
    "Username": "[email protected]",
    "IP_Address": "10.16.21.21",
    "Action": "Failed Login Attempt",
    "Data": ""
}

Event 2

{
    "Time": "2024-07-17 09:51:56",
    "Username": "[email protected]",
    "IP_Address": "10.16.21.21",
    "Action": "Failed Login Attempt",
    "Data": ""
}

2

3

{
    "status": "OK",
    "next": null,
    "data": {
        "Event1": {
            "Time": "2024-07-17 09:51:51",
            "Username": "[email protected]",
            "IP_Address": "10.16.21.21",
            "Action": "Failed Login Attempt",
            "Data": ""
        },
        "Event2": {
            "Time": "2024-07-17 09:51:56",
            "Username": "[email protected]",
            "IP_Address": "10.16.21.21",
            "Action": "Failed Login Attempt",
            "Data": ""
        }
    }
}

$.data.*

Event 1

{
    "Time": "2024-07-17 09:51:51",
    "Username": "[email protected]",
    "IP_Address": "10.16.21.21",
    "Action": "Failed Login Attempt",
    "Data": ""
}

Event 2

{
    "Time": "2024-07-17 09:51:56",
    "Username": "[email protected]",
    "IP_Address": "10.16.21.21",
    "Action": "Failed Login Attempt",
    "Data": ""
}

2

4

{
  "status": "success",
  "logs": [
    {
      "log_id": "log-003",
      "timestamp": "2025-01-30T22:10:00Z",
      "action": "unauthorized_login",
      "user": {
        "id": "user111",
        "username": "tomjones",
        "ip": "192.168.1.120"
      },
      "login_attempt": {
        "status": "failed",
        "reason": "wrong_password"
      },
      "severity": "medium",
      "log_type": "authentication"
    },
    {
      "log_id": "log-004",
      "timestamp": "2025-01-30T22:30:00Z",
      "action": "malicious_activity_detected",
      "source_ip": "192.168.2.90",
      "destination_ip": "192.168.1.120",
      "activity": {
        "type": "network_scan",
        "target_ports": [80, 443, 22]
      },
      "severity": "high",
      "log_type": "network_security"
    }
  ]
}

$.logs[*]

Event 1

{
  "log_id": "log-003",
  "timestamp": "2025-01-30T22:10:00Z",
  "action": "unauthorized_login",
  "user": {
    "id": "user111",
    "username": "tomjones",
    "ip": "192.168.1.120"
  },
  "login_attempt": {
    "status": "failed",
    "reason": "wrong_password"
  },
  "severity": "medium",
  "log_type": "authentication"
}

Event 2

{
  "log_id": "log-004",
  "timestamp": "2025-01-30T22:30:00Z",
  "action": "malicious_activity_detected",
  "source_ip": "192.168.2.90",
  "destination_ip": "192.168.1.120",
  "activity": {
    "type": "network_scan",
    "target_ports": [80, 443, 22]
  },
  "severity": "high",
  "log_type": "network_security"
}

2

5

{
  "status": "success",
  "incidents": [
    {
      "incident_id": "incident-1234",
      "timestamp": "2025-01-30T15:20:30Z",
      "user_account": {
        "id": "user105",
        "username": "sarahwhite",
        "ip": "192.168.10.25",
        "compromise_details": {
          "method": "credential_stuffing",
          "detected_by": "SIEM",
          "status": "compromised"
        },
        "affected_systems": [
          {
            "system_id": "server-06",
            "system_name": "file_server",
            "os": "Windows Server 2019",
            "files_accessed": [
              "/home/user105/private_document.pdf",
              "/home/user105/financial_report.xlsx"
            ]
          }
        ]
      },
      "severity": "critical",
      "event_type": "account_compromise_alert"
    },
    {
      "incident_id": "incident-5678",
      "timestamp": "2025-01-30T16:00:00Z",
      "network_activity": {
        "source_ip": "192.168.10.25",
        "destination_ip": "203.0.113.10",
        "protocol": "HTTP",
        "data_volume": "10GB",
        "activity_type": "high_traffic"
      },
      "severity": "high",
      "event_type": "data_exfiltration_alert"
    }
  ]
}

$.incidents[*]

Event 1

{
  "incident_id": "incident-1234",
  "timestamp": "2025-01-30T15:20:30Z",
  "user_account": {
    "id": "user105",
    "username": "sarahwhite",
    "ip": "192.168.10.25",
    "compromise_details": {
      "method": "credential_stuffing",
      "detected_by": "SIEM",
      "status": "compromised"
    },
    "affected_systems": [
      {
        "system_id": "server-06",
        "system_name": "file_server",
        "os": "Windows Server 2019",
        "files_accessed": [
          "/home/user105/private_document.pdf",
          "/home/user105/financial_report.xlsx"
        ]
      }
    ]
  },
  "severity": "critical",
  "event_type": "account_compromise_alert"
}

Event 2

{
  "incident_id": "incident-5678",
  "timestamp": "2025-01-30T16:00:00Z",
  "network_activity": {
    "source_ip": "192.168.10.25",
    "destination_ip": "203.0.113.10",
    "protocol": "HTTP",
    "data_volume": "10GB",
    "activity_type": "high_traffic"
  },
  "severity": "high",
  "event_type": "data_exfiltration_alert"
}

2

Some collector configuration fields require you to specify a JSON Path Attribute. This attribute is typically a pagination cursor or URL used for configuration of API pagination. The JSON Path Attribute refers to the JSON Path for extracting the desired attribute from the API response.

Refer to the following table for examples of examples of JSON Paths for extracting various attributes from the API response.

#

Description

API Response

JSON Path

Attribute Extracted

1

JSON Path to extract the next page cursor from the API Response Body.

{
    "data": [
        {
            "accountId": "12345",
            "accountName": "Technical",
            "activityType": 19,
            "activityUuid": "5580828a-1234-44c5-ae82-74ac30080000",
            "agentId": "197111854310549123454",
            "createdAt": "2024-12-10T08:44:18.446258Z",
            "id": "5884674675676335",
            "osFamily": null,
            "primaryDescription": "Threat with confidence level malicious detected: Unconfirmed 680044.crdownload",
            "secondaryDescription": "d27265074c9eac2e2122ed69294dbc4d7cce9141\nIP address: 10.1.0.10",
            "threatId": "43434343464342",
            "updatedAt": "2024-12-10T08:44:18.427511Z",
            "userId": null
        }
    ],
    "pagination": {
        "nextCursor": "eyJpZF9jb2x1bW4iOiAiQWN0aXZpdHkuaWQiLCAiaWRf",
        "totalItems": 50000
    }
}

$.pagination.nextCursor

eyJpZF9jb2x1bW4iOiAiQWN0aXZpdHkuaWQiLCAiaWRf

2

JSON Path to extract the next page URL from the API Response Body.

{
  "status": "success",
  "data": [
    {
      "event_id": "abc123xyz",
      "timestamp": "2025-02-04T12:34:56Z",
      "event_type": "unauthorized_access",
      "source_ip": "192.0.2.1",
      "destination_ip": "203.0.113.10",
      "user_id": "user_98765",
      "severity": "high",
      "description": "Multiple failed login attempts detected."
    },
    {
      "event_id": "def456uvw",
      "timestamp": "2025-02-04T12:40:12Z",
      "event_type": "malware_detected",
      "source_ip": "198.51.100.5",
      "destination_ip": "203.0.113.20",
      "user_id": "user_54321",
      "severity": "critical",
      "description": "Malware signature detected in uploaded file."
    }
  ],
  "pagination": {
    "next_page_url": "https://api.securitysystem.com/events?page=2"
  }
}

$.pagination.next_page_url

https://api.securitysystem.com/events?page=2

3

JSON Path to extract the next page URL from the API Response Body.

{
  "status": "success",
  "data": [
    {
      "event_id": "abc123xyz",
      "timestamp": "2025-02-04T12:34:56Z",
      "event_type": "unauthorized_access",
      "source_ip": "192.0.2.1",
      "destination_ip": "203.0.113.10",
      "user_id": "user_98765",
      "severity": "high",
      "description": "Multiple failed login attempts detected."
    },
    {
      "event_id": "def456uvw",
      "timestamp": "2025-02-04T12:40:12Z",
      "event_type": "malware_detected",
      "source_ip": "198.51.100.5",
      "destination_ip": "203.0.113.20",
      "user_id": "user_54321",
      "severity": "critical",
      "description": "Malware signature detected in uploaded file."
    }
  ],
  "pagination": [
    {
      "next_page_url": "https://api.securitysystem.com/events?page=2"
    }
  ]
}

$.pagination[0].next_page_url

https://api.securitysystem.com/events?page=2

In the SETTINGS tab, enter the following information for time range for data collection.

Field

Description

COLLECTION PERIOD 

Specify the maximum duration between the start time and the end time per data collection operation in days, hours, minutes, and seconds. The Collection Period determines the maximum duration between the start time and end time that the collector calculates for each data collection operation. This period defines the range of data the collector retrieves during each operation and ensures that data is collected within consistent, predefined intervals.

COLLECTION INTERVAL 

Specify the frequency at which the collector must send the new data collection API request, in minutes or hours.

For example, if the Collection Interval is set to 10 minutes, the collector sends a new API request every 10 minutes to fetch data.

COLLECTION TIMEOUT 

Enter the maximum execution time for the data collection job, in seconds.

MAX HISTORICAL FETCH DURATION 

Set the duration for the collector to retrieve data prior to the collector's setup.

The Max Historical Fetch Duration defines the maximum time range for which the collector can retrieve historical data. This duration is typically determined by the API limitations set by the service provider, as detailed in their documentation. This limit ensures that the collector operates within the allowed parameters for fetching past data. Set the maximum duration for example, 30 days or 6 months for data retrieval prior to the collector's setup. Refer to the API documentation to determine the permitted historical fetch limit and avoid exceeding request quotas or time constraints. For example, if the API allows retrieving data from the past 90 days, set the Max Historical Fetch Duration to 90 days. Based on this set duration the collector collects data created within 90 days of collector’s initialization.

The following table displays the list of features that are currently supported by the REST API cloud collector for configuring additional settings.

Feature

Supported

Not Supported

Collection Period

✓

Collection Interval

✓

Collection Timeout

✓

Maximum Historical Fetch Duration

✓