Skip to main content

CollectorsCloud Collectors Administration Guide

Table of Contents

Prerequisites to Configure the Azure Storage Analytics Cloud Collector

Before you configure the Azure Storage Analytics Cloud Collector you must complete the following prerequisites:

  • Create an Event Hub and obtain the consumer group name to be entered while configuring the Cloud Collector.

  • Obtain the connection string to be entered while configuring the Cloud Collector.

  • (Optional) Configure your Event Hub setup for enabling sufficient egress. Consider referring to the tip in this section.

  • Configure Azure Storage Account to enable the cloud collector to pull logs for blobs, queues, and tables.

Obtain the Consumer Group Name and Connection String

Before configuring the Azure Storage Analytics Cloud Collector, Ensure that you create an Event Hub if you do not have one. Use the following steps to obtain the Consumer Group Name and Connection String for your EventHub endpoint.

  1. To obtain the name of the consumer group of the Event Hub, perform the following steps.

    1. On the Azure portal, navigate to the Event Hubs namespace in which the Event Hub with the data you want to retrieve resides.

      EventHub_1_1.png
      EventHub_1_2.png

    2. Select the Event Hub where your data to be retrieved resides.

      Eventhub_5.png

      Note

      If connection errors occur after the Cloud Collector configuration, enable local authentication for a given Event Hubs namespace.

      Event_hub_5_local_authentication.png

    3. Navigate to Consumer groups to note the name of the consumer group.

      If only Exabeam is going to read data from the Event Hub, use the default consumer group. Otherwise, create a dedicated consumer group for Exabeam.

      consumer_group1.png

      Note

      Ensure that you create and use a unique consumer group. If you use the same consumer group for two cloud collector instances, it may result in event loss and ingestion issues.

  2. Create a SAS policy with listen-only permissions for Exabeam:

    1. In Event Hub, navigate to Shared access policies.

    2. Click + Add to add a new Policy on the Add SAS Policy page and enable the Listen permission.

    3. Provide a descriptive name for the token which is listen-only, then click Create.

    4. After you see the confirmation message (for example Creating SAS Policy successful), access the policy you just created and note the following.

      • SAS policy name

      • Primary key value

        Note

        The Connection string primary key acts as a connection string for the Azure Event Hub Cloud Collector.

      Event_Hub_4_2.png

      Proceed to configure the Azure Azure Storage Analytics Cloud Collector.

    Tip

    To configure your Event Hub setup for enabling sufficient egress, refer to the following tips.

    • Examine the incoming number of bytes per second, by navigating to the Event Hub page on the Azure portal.

    • Set up the Event Hub with a good throughput capacity. The egress and ingress capacity of Event Hubs is measured by throughput units (TU). Ensure that your Event Hub provides 1 Throughput Unit (TU) per 2MB/sec egress and 1MB/sec ingress capacity.

    • Ensure that your Event Hub has 32 partitions and at least 1 TU per partition.

      If you notice that your workload requirement exceeds the maximum value recommended for Event Hub Premium which is 32 partitions and 40 TU, you need to migrate to a Dedicated Event Hub Cluster.

    • Consider using the Auto-inflate throughput feature. You can enable Auto-inflate on the Event Hub namespace to automatically increase the number of TUs based on the load. For more information about throughput units, limits, and associated billing, see Event Hubs FAQs and Azure Event Hubs quotas and limits.

Configure Azure Storage Account to Forward Events to your Event Hub

Azure Storage Analytics provides metrics data for a storage account, and logs for blobs, queues, and tables. Use this data to trace requests, analyze usage trends, and diagnose issues with your storage account. For more information, see Storage Analytics in the Microsoft documentation. To enable the cloud collector to pull storage analytics logs, creating an Azure storage account that contains all of your Azure Storage data objects: blobs, files, queues, and tables with diagnostic settings is required.

If you do not have an existing storage account, to create a storage account, refer to the following steps. For detailed instructions, see Create a storage account in Microsoft Azure documentation.

Note

It is recommended to create an Event Hub specifically for storage account logs and use that Event Hub for multiple storage accounts to ensure that storage account logs are streamed to the same Event Hub. Otherwise the cloud collector may ingest any data available in the specified Event Hub.

  1. On the Azure portal, navigate to Azure Services > Storage accounts.

    Azure_Storage_account_1.png

  2. On the Storage accounts page, click Create.

    Azure_Storage_account_2.png

  3. Enter the required information in the Project details and Instance details section. For more information, see Create a storage account in Microsoft Azure documentation.

    Azure_Storage_account_3.png

  4. Click Next: Advanced. If you want to verify the details, click Review.

  5. Retain the default options for the sections: Advanced, Networking, Data protection, Encryption, and Tags. For more information on each field in these sections, see Create a storage account in Microsoft Azure documentation.

  6. Click Create.

    Azure_Storage_account_4-1.png

    The storage account is created.

    Azure_Storage_account_5_1.png

  7. Navigate to Storage accounts to view the new storage account that you created, then click the name of the storage account that you created.

    Azure_Storage_account_6.png

  8. On your storage account page, navigate to Monitor > Diagnostic Settings.

    Azure_Storage_account_7.png

  9. Configure diagnostic settings for each resource: blob, queue, table, and file. Click the resource and click Add diagnostic setting. For more information, see Diagnostic settings in Azure Monitor in Microsoft Azure documentation.

    Azure_Storage_account_8.png

  10. In the Diagnostic setting section, specify a name for the diagnostic setting, then under Logs, select allLogs.

    Additionally, to configure diagnostic setting for each of the resources, refer to the following example for blob diagnostic settings. Based on your requirement, you can select metrics logs that you want the collector to pull.

    Azure_Storage_account_9.png

  11. Select Stream to an event hub, then select the subscription, the Event Hub namespace, the Event Hub name, and the Event Hub policy name, and click Save.

    Similarly, you can configure the diagnostic settings for the resources: queue, table, and file based on your requirement.

    After the diagnostic settings are saved, the blob, queue, table, or file related logs are streamed to your Event Hub to enable the cloud collector to collect logs for storage account.