Data LakeExabeam Data Lake Collector Guide

Follow the steps below to install individual log collectors on Linux and Ubuntu. This will install Exabeat (the Collector Manager) and File Collector.

Untar the collectors.tar.gz package. One option you can use is tar -xvf collectors.tar.gz.

Install all or individual log collectors using the commands below. The "-t" parameter controls which beats are being installed.

To install all beats, run the following command for SaaS and on-premises deployments.

To install file collectors only, run the following command:

To install Gzip collectors only, run the following command:

Optionally, you can set only IP address or DNS hostname for Data Lake and Kafka if they are on the same server machine with default ports.

The collector(s) are now installed and the Exabeam Log Collector services are running automatically.

sudo systemctl start exabeam-file-log-manager
sudo systemctl stop exabeam-file-log-manager
sudo systemctl restart exabeam-file-log-manager

sudo systemctl start exabeam-file-collector
sudo systemctl stop exabeam-file-collector
sudo systemctl restart exabeam-file-collector

sudo systemctl start exabeam-gzip-log-manager
sudo systemctl stop exabeam-gzip-log-manager
sudo systemctl restart exabeam-gzip-log-manager

sudo systemctl start exabeam-gzip-collector
sudo systemctl stop exabeam-gzip-collector
sudo systemctl restart exabeam-gzip-collector

sudo service exabeam-file-log-manager start
sudo service exabeam-file-log-manager stop
sudo service exabeam-file-log-manager restart

sudo service exabeam-file-collector start
sudo service exabeam-file-collector stop
sudo service exabeam-file-collector restart

sudo service exabeam-gzip-log-manager start
sudo service exabeam-gzip-log-manager stop
sudo service exabeam-gzip-log-manager restart

sudo service exabeam-gzip-collector start
sudo service exabeam-gzip-collector stop
sudo service exabeam-gzip-collector restart

The Collector Details panel can be accessed by clicking on the collector's title, the 'Template not Assigned' text when applicable, and the vertical ellipses to the right of each collector. Users can edit agent configuration information from here and view a summary of the collector health.

Figure 1. Log Collector Details panel

Ingestor – This is the IP:PORT of the Data Lake Site Collector node where this collector is gathering data. If your deployment contains multiple Data Lake Site Collector nodes, add the IP address of each node to this field as a comma-separated list.

Index – The name of the index where the data collected by this collector is stored. This cannot be changed.

Template – This shows the template that is applied to the collector. The template list will be pre-filtered to show only the template types that can be applied to the selected collector.

In order for any configuration changes to take effect, you must click SAVE & RESTART.

Limit Collector CPU Usage

Administrators can configure the CPU usage of the Data Lake collectors. In cases where the collectors are consuming CPU resources that are needed for other services, the collector resources can be throttled.

The configuration is supported on RHEL 7 and Centos 7 on systems where the systemd service manager version is 213 and above. If the system is incompatible, an error will be shown.

Note that limiting the collector's CPU usage might impact the ability of the collector to send high events per second. In extreme cases it is possible for logs to be lost if the collector is processing them at a rate that is less than the log retention rate.

To set a CPU LIMIT, navigate to Settings > Collector Management > Collectors. Select the vertical ellipsis drop-down menu of the collector you'd like to edit and click Collector Details.

Figure 2. Log Collect Details panel

There are two methods for starting collectors:

Individually – Selecting the vertical ellipsis to the right gives you the options to Start or Stop the collector as well as open a Details panel.

Grouped – Select the check box to the left of each collector that you would like to start. This will activate an Actions menu at the top right of the page:

Figure 3. Log Collector Management menu

While collectors can be configured individually it is likely that in large environments the same configuration will be used for a set of collectors. For example, an administrator would set the same configuration for collecting logs from all Windows Domain Controllers or all Unix Servers. Assigning a configuration to a set of multiple collectors is defined through templates.

Create an Exabeam Data Lake Log Collector Template

To create a template, navigate to Settings > Collector Management > Collectors. You will arrive at the Collector Management page. Select the Templates tab to see a list of your existing templates. If you have not yet created any templates, this page will be empty.

Select Create Template at the top right. This will open the template creation menu:

Figure 4. Create Configuration Template page

Name your collector and choose a service type.

File /Gzip Collectors: Installed on Windows or Linux hosts

1. As in the image below, you will be expected to list files or directories to monitor (e.g., /var/log/messages, /var/log/*.log).

   Note

   For Gzip collectors, you must enter a list of Gzipped files (e.g., /var/log/*.gz) or directories that contain only Gzipped files. This is because Gzipbeat processes each file if finds as Gzipped file and will try to unzip them.

2. Click Next.

3. Optionally, apply the template to servers. The left panel lists all of the available collectors; clicking a server will add it to the right panel of selected servers.

4. Click Save & Restart to save the template. If you have placed any collectors in the Selected Servers panel then all of those collectors will have the template applied to them and they will also be restarted. The restarting process is necessary when a new template has been applied to a collector.

   Here is a Windows DHCP configuration example:

   DHCP service can have several *.log files in \\Windows\System32\dhcp folder which DHCP service needs exclusive access to these files:

   DhcpSrvLog-Mon.log
   DhcpV6SrvLog-Mon.log
   j50.log
   j50tmp.log

   If a filebeat collector is started with the template *.log, it will lead to file access conflicts and generate DHCP service errors. The correct configuration template is \\Windows\System32\dhcp\<dhcp_service>*.log.

Windows Log Collector: Installed on Windows hosts

1. For collectors installed on Windows hosts, select Windows Event Collector as the Template Type. In the Windows Log Category section, select log and event codes categories to be collected.

   

   You can select all of the logs or alternatively the value is a comma-separated list. The accepted values are individual event IDs to include (e.g. 4624), or a range of event IDs to include (e.g. 4700-4800).

   Note

   If you specify more than 22 individual event IDs to include, Windows will prevent Data Lake from reading the event log because it limits the number of conditions that can be used in an event log query. Therefore use ranges wherever possible.

2. Click Next.

3. Optionally, apply the template to servers. The left panel lists all of the available collectors; clicking a server will add it to the right panel of selected servers.

4. Click Save & Restart to save the template. If you have placed any collectors in the Selected Servers panel then all of those collectors will have the template applied to them and they will also be restarted. The restarting process is necessary when a new template has been applied to a collector.

Run the following command to uninstall the Exabeam Collector:

cd PATH_TO_EXACOLLECTOR_INSTALLER
.\uninstall-exabeam-collector.ps1

# uninstall a particular collector
.\uninstall-exabeam-collector.ps1 -agentType "AGENT_TYPE"

# uninstall collector regardless deregistration failure
.\uninstall-exabeam-collector.ps1 -force

# get help menu of the uninstallation script
Get-Help .\uninstall-exabeam-collector.ps1 -full

Follow the steps below to uninstall all log collectors using MSI package for Windows. You can also uninstall collectors using on Windows via PowerShell.

Run the following command to uninstall the Exabeam Collector. The “-t” parameter controls which log collectors are being uninstalled.

CentOS/RedHat 7.x+ and Ubuntu 14+

cd PATH_TO_EXACOLLECTOR_INSTALLER
sudo ./uninstall-exabeam-collector.sh
sudo ./uninstall-exabeam-collector.sh -t AGENT_TYPE
sudo ./uninstall-exabeam-collector.sh -f # force to uninstall regardless of deregistration failure