Exabeam Site Collector for SaaSExabeam Site Collector

Table of Contents

Troubleshoot the Exabeam Site Collector

Below are troubleshooting steps for common problem scenarios in the Site Collector (SC). If the scenarios do not apply to your issue, please contact Exabeam Customer Success by starting a trouble ticket at Exabeam Community.

The Exabeam Customer Success team may require additional data to further assist in troubleshooting. Exabeam has provided a script to support capturing diagnostic information about your SC. Please run the command based on your situation:

  • To collect data just after a SC installation:

    sudo /tmp/Exabeam_Site_Collector/bin/support-package.sh
  • To collect data from a running SC:

    • For SC versions before 2.1.11

      sudo /opt/exabeam/bin/support-package.sh 
    • For SC version 2.1.11 and later

      sudo /opt/exabeam/tools/support-package.sh 
  • If using a proxy, ensure that the proxy server is configured properly and running, then check that its configurations on the SC server is correct.

    cat /etc/systemd/system/exabeam-rsc-forwarder.service | grep http_proxy
  • If using OpenVPN, check the OpenVPN client logs for error messages, then check if the OpenVPN client is running on SC server.

    # check status of openvpn client
    sudo systemctl status openvpn@<instanceID>
    
    # check logs of openvpn client
    sudo journalctl -fu openvpn@<instanceID>
  • Verify if the Forwarder and Kafka-to-GCS managers are running.

    # check status of exabeam kafka gcs1 log manager
    sudo systemctl status exabeam-kafka-gcs1-log-manager
    
    # check status of exabeam kafka gcs2 log manager
    sudo systemctl status exabeam-kafka-gcs2-log-manager
    
    # check status of exabeam regional site collector forwarder
    sudo systemctl status exabeam-rsc-forwarder
  • Check the logs of these services for error messages.

    # check logs of exabeam kafka gcs1 log manager
    sudo journalctl -fu exabeam-kafka-gcs1-log-manager
    # OR
    sudo cat /opt/exabeam/beats/gcs1/manager/logs/exabeat
    
    # check logs of exabeam kafka gcs2 collector with either method below:
    sudo journalctl -fu exabeam-kafka-gcs2-collector
    #  OR
    sudo cat /opt/exabeam/beats/gcs2/worker/logs/kafkabeat
    
    # check logs of exabeam regional site collector forwarder with either method below:
    sudo journalctl -fu exabeam-rsc-forwarder
    #  OR
    sudo cat /opt/exabeam/rscforwarder/logs/rscforwarder.log
  • Ensure that Kafka-to-GCS collectors are running in Data Lake UI:

  • On the SC server, check if services on Kafka-to-GCS collectors are running without issues.

    # check status of exabeam kafka gcs1 collector
    sudo systemctl status exabeam-kafka-gcs1-collector 
    
    #  check status of exabeam kafka gcs2 collector
    sudo systemctl status exabeam-kafka-gcs2-collector
  • Check the logs of the Kafka-to-GCS collectors services for errors.

    # check logs of exabeam kafka gcs1 collector with either method below:
    sudo journalctl -fu exabeam-kafka-gcs1-collector
    #  OR
    sudo cat /opt/exabeam/beats/gcs1/worker/logs/kafkabeat 
    
    # check logs of exabeam kafka gcs2 collector with either method below:
    sudo journalctl -fu exabeam-kafka-gcs2-collector
    #  OR
    sudo cat /opt/exabeam/beats/gcs2/worker/logs/kafkabeat
  • Verify collector configurations.

    # check exabeam kafka gcs1 collector configuration
    cat /opt/exabeam/beats/gcs1/worker/kafkabeat.yml 
    
    # check exabeam kafka gcs2 collector configuration
    cat /opt/exabeam/beats/gcs2/worker/kafkabeat.yml
  • If the logs come to SC Syslog endpoint, check if Syslog service is up and check Syslog logs for any errors, verify on the SC server.

    # check status of syslog
    sudo systemctl status logstash
    
    # check logs of syslog
    sudo journalctl -fu logstash
  • If the logs reach the Kafka endpoint (beats collectors), confirm that Kafka and Zookeeper services are up and check logs for any errors on the SC server.

    # check status of kafka
    sudo systemctl status kafka 
    
    # check status of zookeeper
    sudo systemctl status zookeeper
    
    # check logs of kafka
    sudo journalctl -fu kafka 
    
    # check logs of zookeeper
    sudo journalctl -fu zookeeper
  • Verify that the firewalld service is running and not disabled.

    firewall-cmd --state

Firewall rules may not be applied properly (for example, those in Selinux) for your environment. Run the following command:

firewall-cmd --zone=public --add-forward-port=port=389:proto=tcp:toport=389:toaddr=$<dns_server> --permanent
firewall-cmd —reload

If the above command still fails to resolve the issue, run the follow alternative command:

firewall-cmd --add-forward-port=port=389:proto=tcp:toport=389:toaddr=$<dns_server> --permanent
firewall-cmd —reload

Note

Port 389, given in this example, is used for LDAP. It may be different for your organization. Please confirm with your organization's network configuration.

During initial installation, you may encounter the message:

Starting Firewalld - [!] error (1): Failed to execute operation: Cannot send after transport endpoint shutdown

Apply the following command to try to enable throughput:

systemctl umask firewalld

During startup, you may encounter the following messages:

./tools/config.parser.sh: line 131: [: too many arguments 
./tools/config.parser.sh: line 137: [: too many arguments 
[Configuration package has been extracted to /etc/ssl/certs/exabeam striped wrapper folders] [Parse from config file at: /etc/ssl/certs/exabeam/info.yml]

To resolve this, replace the startup file with a fresh copy from the installation package.

  1. Unpack the site collector authentication package locally.

  2. Pack the files back into the archive with the same name.

  3. Upload them to the SC host and then run installation command again.

Clues to Site collector performance issues may be found when inspecting the status of your collectors as well as Logstash logs. To gather information to help Exabeam Customer Success resolve your issue, run the SC support package with the following command to collect local SC logs to attach to your troubleshooting case:

sudo ./support-package.sh

Restart or View Metrics for a Site Collector

Note

A restart will work only for SaaS or on-premises Data Lake . It is not available for Advanced Analytics .

To restart or view the condition of your collector(s):

  1. In the Exbeam UI, navigate to Settings > Collector Management > Collectors.

    DL-Settings-CollectorManagement-Collectors.png
  2. Click the More Options menu for the collector record, and then select the state or view collector details from the ACTIONS menu.

    collector management more options
    1. Restart the collector by selecting Stop and then Start.

    2. To view performance metrics for the collector, select Details to open the Collector Details panel.

      site collector details panel

      Click CANCEL to close the panel.

Inspect Logs

If restarting collector services does not resolve the issues, additional information may be found in Logstash logs. The example below shows a typical Logstash log. When troubleshooting the site collector, review your logs and look for potential error messages:

Upload to GCS [2019-08-24T08:39:00,257][INFO ][logstash.outputs.googlecloudstorage] Uploading file to dl-format-topic-ops-dist-cybo5/format1_sfo-cybo3-lp1.exabeam.corp_2019-08-24T08-38-00.part000.1e2c933b-aa08-4520-a85b-a6b756f0bcc3.log.gz rotation (by default every 30 seconds or 5 mb compressed, whichever occurs first [2019-08-24T08:39:00,270][INFO ][logstash.outputs.googlecloudstorage] Rotated out file: /tmp/logstash-gcs-format2/format2_sfo-cybo3-lp1.exabeam.corp_2019-08-24T08-38-00.part000.63dd0d1a-2bcf-458e-935f-2de1cbd7d900.log.gz

Here are some of the possible ways to discover issues from the Logstash logs and how to resolved them:

  • Monitor logs for errors:

    sudo journalctl -u logstash
  • If OpenVPN is not connected, restart the service:

    sudo systemctl start openvpn@InstanceID
  • If using GCS (default), use the following commands be used to view the number of events pending upload to GCS:

    sudo /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group group-gcs-gcs1
    sudo /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group group-gcs-gcs1 | awk ‘{s+=$5}END{print s}'
    
    sudo /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group group-gcs-gcs2
    sudo /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group group-gcs-gcs2 | awk ‘{s+=$5}END{print s}'
  • If unable to accept incoming syslog or if AD context, Splunk logs, or IR integrations are not working, verify the firewalld service is running and not disabled by running the command below:

    firewall-cmd --state
  • Confirm messages sent to the Site Collector using syslog protocol arrive to the server (add tcpdump filters for source IP as needed or grep to look for a specific message):

    tcpdump -nnvvAs0 -i any dst port 514
  • Confirm messages sent to the Site Collector using beats arrive to the server by running the command below (add tcpdump filters for a specific source IP, note that this traffic is encrypted):

    tcpdump -nn -i any dst port 9093
  • Confirm messages are in Kafka:

    • For syslog and windows log beats (winlogbeat), use the following command to view new messages arriving in Kafka

      sudo /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic lms.kafka.topic
    • For file beats (filebeat), use this command shows new messages arriving in Kafka:

      sudo /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic lms.kafka.format.topic
  • Confirm total upload lag for syslog and winlogbeat by running the command below:

    sudo /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group group-gcs-gcs1 | awk '/lms.kafka.topic/{s+=$5}END{printf ("%'\''d\n",s)}'
  • Confirm total upload lag for filebeat:

    sudo /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group group-gcs-gcs2 | awk '/lms.kafka.format.topic/{s+=$5}END{printf ("%'\''d\n",s)}'

If you want to inspect raw logs of SC components, see files located in the following locations:

Component

Location

Logstash

/opt/logstash/logs

External Kafka

/opt/exabeam/beats/<kafka_alias>/worker/logs

Forwarder

/opt/exabeam/rscforwarder/logs/rscforwarder

Kafka

/opt/kafka/logs

GCS destination

/opt/exabeam/beats/gcs1/manager/logs

For site collectors deployed before April 2020, the following instructions will guide you through redirecting feeds and agents in a step-wise manner to mitigate data loss. Please have a operating site collector host before attempting a migration.

Warning

Exabeam recommends that upgrading your SC happen at a different host than the legacy SC. This ensures there is no data loss in the transition. Otherwise, expect loss of data during the time between the shutdown of the legacy SC and when the new SC starts receiving data.

Exabeam recommends installing the new SC on a host that is not running the existing SC, using the following procedure:

  1. Deploy a new site collector onto the dedicated host.

  2. If there is a syslog source, we highly recommend (optionally) that you install a load balancer to work with the new site collector. This is useful for HA as well.

  3. Switch syslog traffic to the new site collector or load balancer. Verify that messages pending in Kafka were successfully sent. Please ensure you have removed port forwarding rules for ports 5514 and 5515. Use ports 514 and 515 for Syslog ingestion.

  4. Turn off Logstash services on the old site.

  5. Direct feeds from agent collectors, if any, to the new site collector and restart them. Verify that messages pending in Kafka were successfully sent.

  6. Let the old site collector continue to run so all remaining messages process to completion. Verify that messages pending in Kafka were successfully sent. The lag queue in Kafka must be 0.

  7. Turn off the old site collector.

If you do not have an available host to install that is not currently running the existing SC, please follow How to Uninstall Exabeam Site Collector > "How to Uninstall a Legacy Site Collector" to first remove the existing SC and then install the new SC.