Advanced AnalyticsAdvanced Analytics Administration Guide

Models are disabled once they reach their maximum bin limit. You can either set a global configuration for the maximum number of bins or an individual configuration for each model name.

This is done by setting a max number of bins, MaxNumberOfBins in exabeam_default.conf or the model definition, for categorical models. The new limit is 10 million bins, although some models, such as ones where the feature is "Country" have a lower limit. We have put many guardrails in place to make sure models do not consume excessive memory and impact overall system health and performance. These include setting a maximum limit on bins, enabling aging for models, and verifying data which goes into models to make sure it is valid. If a model is still consuming excessive amounts of memory then we will proceed to disable that model.

To globally configure the maximum bin limit, navigate to the Modeling section located in the exabeam_default.conf file at /opt/exabeam/config/default/exabeam_default.conf.

All changes should be made to /opt/exabeam/config/custom/custom_exabeam_config.conf:

Modeling {
      ...
      # To save space we limit the number of bins saved for a histogram. This defaults to 100 if not present
      # This parameter is only for internal research
      MaxSizeOfReducedModel = 100
      MaxPointsToCluster = 250
      ReclusteringPeriod = 24 hours
      MaxNumberOfBins = 1000000

Additionally, it is possible to define a specific bin limit on a per model basis via the model definition section in the models.conf file at /opt/exabeam/config/custom/models.conf. Here is an example model, where MaxNumberOfBins is set to 5,000,000:

WTC-OH {
      ModelTemplate = "Hosts on which scheduled tasks are created"
      Description = "Models the hosts on which scheduled tasks are created"
      Category = "Asset Activity Monitoring"
      IconName = ""
      ScopeType = "ORG"
      Scope = "org"
      Feature = "dest_host"
      FeatureName = "host"
      FeatureType = "asset"
      TrainIf = """count(dest_host,'task-created')=1"""
      ModelType = "CATEGORICAL"
      BinWidth = "5"
      MaxNumberOfBins = "5000000"
      AgingWindow = ""
      CutOff = "10"
      Alpha = "2"
      ConvergenceFilter = "confidence_factor>=0.8"
      HistogramEventTypes = [
      "task-created"
      ]
      Disabled = "FALSE"
}

To identify a slow parser, your system places the parser in a cache. Every configured period, OutputParsingTimePeriodInMinutes (five minutes by default), it calculates how long it takes, on average, for the parser to parse a log. It compares this average to a configurable threshold, ParserDisableThresholdInMills, in lime.conf.

To calculate a percentage of how much the parser makes up of the total parsing time, your system divides the previously-calculated average by the total time all parsers took to parse an event in the same five minute period. It compares the percentage to another configurable threshold, ParserDisableTimePercentage, in lime.conf.

Your system conducts a second round of checks if the parser meets certain conditions:

During another five-minute period, your system checks the parser for a second time. If the parser meets the same conditions again, your system pauses the parser.

If you have a virtual or hardware deployment, you can configure OutputParsingTimePerioidInMinutes, ParserDisableThresholdInMills, and ParserDisableTimePercentage to better suit your needs.Configure Conditions for Pausing Parsers

To identify a stuck parser, your system measures how long it takes for a parser to parse a log. If the time exceeds a threshold, StuckParserWaitTimeoutMillis (100 milliseconds by default), the parser fails with a timeout exception. Your system logs the error at a DEBUG security level and notes the parser in internal error statistics.

In each configured period, ParserMaxErrorTimeWindowForChecksMillis (9000 miliseconds, or 15 minutes, by default), your system checks the internal error statistics for any parsers that have accumulated a certain number of errors. If the errors exceed a threshold, ParserMaxErrorNumberThreshold (100 errors by default), the parser is paused and removed from the internal error statistics.

If you have a virtual or hardware deployment, you can configure StuckParserWaitTimeoutMillis, ParserMaxErrorTimeWindowForChecksMillis, and ParserMaxErrorNumberThreshold.

When an entity's session has 10 million or more events in a session and an event type contributes to 70% or more of the events in that session, then that event type is disabled. If no single event type accounts for over 70% of the total event count in that session, then that entity is disabled. These thresholds, EventCountThreshold and EventDisablePercentage are configurable.

To configure the thresholds, navigate to the Container section located in the exabeam_default.conf file at /opt/exabeam/config/default/exabeam_default.conf.

All changes should be made to /opt/exabeam/config/custom/custom_exabeam_config.conf:

Container {
    ...
    EventCountThreshold = 10000000 // Martini will record the user as top user once a session or a sequence has more than
    // 10 million events
    TopNumberOfEntities = 10 // reporting top 10 users and assets
    EventDisablePercentage = 0.7 // a single event type that accounts for 70% of all event types for a disabled user
    EventCountCheckPeriod = 60 minutes
}

Acceptable values for EventCountThreshold are 10,000,000 to 20,000,000 for a typical environment. EventDisablePercentage can be a percentage decimal value between 0.1 to 0.9.