Data LakeExabeam Data Lake Collector Guide

Table of Contents

Debug an Exabeam Data Lake Log Collector Agent

In order to minimize manual debugging on agent collectors, a debug tool can be run that will gather the necessary information for Exabeam's Customer Success team.

The script will produce a compressed file which contains the collector running status and the installation folder. This file can be sent to Exabeam Customer Success.

The command for running the tool on Windows:

cd PATH_TO_EXACOLLECTOR_INSTALLER
cd tools
.\cmdbeat.exe diagnosis -installationPath "C:\Program Files" -outputPath C:\Exabeam

The command for running the tool on Linux:

Exabeam_Collector_Manager/tools/cmdbeat diagnosis -installationPath [installation_dir] -outputPath [output_dir]

Help Menu

Usage of diagnosis:
-installationPath string
    installation path for exabeam collectors
-outputPath string
    path for output the diagnosis result

In addition, if it is not apparent the agent collect is the root cause, review the following:

  • Confirm that the Database collector server is running

    systemctl status exabeam-lms-dblog

  • Check Database collector logs for error events

    journalctl -eu exabeam-lms-dblog

    • If the following event appears, then Logstash has stopped processing:

      [WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
      
      java.io.IOException: Too many open files

      Logstash has a default limit of 4096 open files. You can manually change this upper limit to resume ingesting.

      1. Confirm the current limit.

        curl -XGET 'localhost:9600/_node/stats/process?pretty'

        The maximum number of open files is listed in the max_file_descriptors of the process block.

          "process" : {
            "open_file_descriptors" : 1314,
            "peak_open_file_descriptors" : 1327,
            "max_file_descriptors" : 4096,
            "mem" : {
              "total_virtual_in_bytes" : 11518406656
            },
      2. Increase the limit by adding lines into /etc/security/limits.conf.

        sudo vim /etc/security/limits.conf

        Add the following lines:

        root  hard  nofile 65530
        root  soft  nofile 65530
        
      3. Add the following line to /etc/systemd/system/logstash.service.

        sudo vim /etc/systemd/system/logstash.service

        Add the following line:

        LimitNOFILE=65536
      4. Reload and restart services.

        sudo systemctl daemon-reload
        sudo systemctl restart logstash
      5. Verify that the limit has loaded.

        curl -XGET 'localhost:9600/_node/stats/process?pretty'

        A new file limit should appear in the max_file_open of the process block.

          "process" : {
            "open_file_descriptors" : 2235,
            "peak_open_file_descriptors" : 2235,
            "max_file_descriptors" : 65536,
            "mem" : {
              "total_virtual_in_bytes" : 21708582912
            },
  • Check for error events in Kafka

    docker exec -it kafka-host1 bash

    cd /opt/kafka/bin

    ./kafka-console-consumer.sh --zookeeper zookeeper-host1:2181 --topic lms.kafka.topic