The Security Content Pipeline
Both Advanced Analytics and Data Lake collect, ingest, and process log data for use by downstream functionality. In Advanced Analytics, the processed data is analyzed using statistical modeling that profiles user and asset behavior. Machine learning is applied to detect anomalous activity. In Data Lake. processed data is indexed into a structure that optimizes searching, visualization, reporting, and other forms of presentation.
While each product has its own processors and engines for handling data ingestion tasks, the use of security content follows a similar pipeline in both products. The sections below outline the path of security content through each product. This pipeline, as shown in the image below, will serve as a framework for discussing security content functionality throughout this documentation.
Logs – Both Advanced Analytics and Data Lake collect and ingest data in the form of log files from a variety of sources. Logs can be ingested directly from servers, applications, databases, and other devices across an infrastructure, whether the source is local, remote, or cloud-based. Logs can be fetched from a SIEM or other third-party repository. Log information can also be collected from sources that provide contextual information, such as threat intelligence services and identity providers. For more information. see Understanding the Log.
Parsers – Both Advanced Analytics and Data Lake use parsing to extract meaningful values from verbose log data. Log files are associated with specific parsers based on each parser's set of conditions. When a log meets parser conditions, a set of regular expressions (regexes) are used to extract values of interest from the log and map them to Exabeam fields. Not all log data lends itself to parsing and there are some differences between what should be parsed for Advanced Analytics and for Data Lake. For more information, see Do You Need a Parser? or Exabeam Parsers.
Events – In Advanced Analytics, each parser is matched to an event builder definition. When a parsed message is output from a parser, it runs through the event building engine to be categorized as a specific event. The event becomes the key to further analysis. It determines the minimum required fields necessary for models and rules to process different types of events. For more information, see Exabeam Event Building.
Enrichers – In both Advanced Analytics and Data Lake, enrichers provide contextual information for an event. While most log information describes what users and entities are doing, context information describes who the users and entities are. Enrichers can add new values, modify values, or create new fields based on existing fields or on context tables. Enrichers can provide system-based or user-based context. For more information, and several enrichment use cases, see Exabeam Enrichment.
Sessions – In Advanced Analytics, once events have been built and enriched, they must be added to a session in order for any analysis to take place. A session is the visual representation of all the events attributed to a user in the course of a workday. Before an event can be used to train a model, visualize a timeline, or trigger a rule, it must be part of a session.
Models – In Advanced Analytics, machine learning functionality models behavior at the user, peer group, entity, and organization levels. Once an event is part of a session, it's checked against the available user-based and asset-based models to determine if the event should be used for training. When training begins, historical values are tracked so that anomalous behavior can be detected. For more information about the different types of models available, see Exabeam Models.
Rules – In Advanced Analytics, rules contain the logical expressions that define malicious or unwanted behavior. When an event matches the conditions defined in the rule, the rule is triggered and assigns points to a user's session. Two types of rules are available in Advanced Analytics. Model-based rules use historical values stored in a model and usually trigger when an event is evaluated as anomalous. Fact-based rules use field values from an event and work more like simple correlation rules when determining whether or not to trigger. For more information and some sample rules, see Exabeam Rules.