Filtering Outbound Logs in Exabeam Site Collector

Exabeam Site Collector (SC) supports log filtering before uploading the logs to configured destinations. SC can drop entire events if filtering conditions are matched. SC uses Kafkabeat for the outbound message processing. Configurations are made in /opt/exabeam/beats/<TARGET>/worker/kafkabeat.yml, where <TARGET> is the folder that contains the SC destination setup.

The values in <TARGET> are the following options based on your deployment:

For SaaS: gcs1 and gcs2
For DL on-premises: lms1 and lms2

The configuration resembles the following configuration block:

kafkabeat:
  inputs:
  - type: kafka
    ...
processors:
- drop_event:
    when:
      - <condition>

Kafkabeat does not parse the event. Rather, it sends the event as-is to the destination. Filtering is based solely on the message field of an event. Listed in the following table are the five conditions in the message field that are supported:

Condition	Description	Condition Example
`contains`	Checks if a value is part of a field. The field can be a string or an array of strings. The condition accepts only a string value.	Check if an error is part of the event message contains: message: "Specific error"
`regexp`	Checks the field against a regular expression. The condition accepts only strings.	Check if the event `message` has the src_ip that fits the IP range 192.168.0.1/16 (192.168.0.0 - 192.168.255.255) regexp: message: "src_ip=192\\.168\\.\\d{1,3}\\.\\d{1,3}"
`or`	An operator that receives a list of criteria to match at least one of.	Determines match to either `message` criteria in `message ~= [DEBUG]` or `message ~= [TRACE]` or: - contains: message: "[DEBUG]" - contains: message: "[TRACE]"
`and`	An operator that receives a list of criteria to match all of.	Determines match where both `http.response.code = 200` and `status = OK` and: - equals: http.response.code: 200 - equals: status: OK
`not`	An operator that receives the condition to negate.	Drops event if `message` does not match `message ~= [ERROR]` not contains: message: "[ERROR]"

Here is an example:

We need to filter out logs from a particular Filebeat that sends logs from a given IP address. The filter will match logs that contain the text [OBSOLETE] in content or that comes from the Filebeat with ID 573d5253-4e4e-4fff-92a5-8f2f227b3af1 and IP address src_ip=195.164.*.*.

A sample log resembles:

[OBSOLETE] - Mar 24 15:00:34 2020 rt=1585062034 device=110.90.230.153 name=Stephanie Kim

The filter is written to apply all three possible match criteria:

processors:
- drop_event:
    when:
      or:
      - and:
        - contains:
            message: "\"id\":\"573d5253-4e4e-4fff-92a5-8f2f227b3af1\""
        - regexp:
            message: "src_ip=195\\.163\\.\\d{1,3}\\.\\d{1,3}"
      - contains:
          message: "[OBSOLETE]"

Exabeam Site Collector for SaaSExabeam Site Collector

Table of ContentsTable of Contents

Filtering Outbound Logs in Exabeam Site Collector