Security ContentExabeam How Content Works Guide

Exabeam Rules

Once a log has passed the event building and enrichment phase it is now ready to be processed against the "risk engine", where it will get evaluated against a set of rules that automatically ship with Exabeam.

Note

The Advanced Analytics pipeline is as follows:

parsing > event building > enrichment > session building > modeling > rule triggering

Rules contain the logical expressions that define unwanted and malicious behavior (or behavior you want to be alerted on). They provide scoring to the timeline and all the values in the UI.

Every single definition that recognizes a specific malicious behavior that you would like to add points to a timeline are defined in a Rule.

Rules are mainly defined by a logical expression, set in the RuleExpression field, and when this expression evaluates to true, the rule "triggers" and points are added to the relevant session or sequence. For example,

"""InList(process_name, 'evil.exe', 'ransomware.exe')"""

Types of Rules

There are two types of rules:

These rules use only the field values from the current event in order to determine whether to trigger or not. This is opposed to relying on historical data in models. Think of these as your simple correlation rules. If x is seen in field y, trigger rule.

These rules use historical information stored in models. The rules typically trigger when the event being evaluated is considered 'anomalous' within the context of the model. Exabeam, by default, ships a large number of rules that are separated into different rules_*.conf files based on the type of malicious behavior they trigger on. For example, rules relating to web activity will be in the rules_webactivity.conf file.  We have a file dedicated to rules related to web activity. This is done for organization, and rules can be placed in any referenced rule file.

Create Rules

Rules are meant to define some logical activity (for examples, a vpn login) you want to be alerted on. Any event that successfully goes into a timeline can trigger a rule and add points to a timeline and elevate a user to notable status. So, rules indicate the flagging of bad, suspicious, and benign behavior.

Key Rule Ingredients:

  • RuleID - The string that is the 'name' of the HOCON block in the rule configuration file. If another rule seen later (below) in the rules.conf file has the same key, the first rule configuration will be thrown out.

  • ClassifyIf - Supports the analytics engine expressions that are usable in the RuleExpression, meaning this is an additional place to place rule expression logic, though generally the logic put in this attribute usually identifies restrictions and scoping of when the rule should trigger. Many rules use this field to place a session/sequence "count" analytics engine expression that limits the rule from triggering more than once for a specific value.

    Note

    For fact based rules, set ClassifyIf to "TRUE".

  • RuleEventTypes - Events of the event type expressed in this field will be evaluated against the rule. A rule with event type web-activity-denied will never be evaluated against a network-connection-denied event.

  • RuleExpression - Rule logic. When should this rule 'fire'. For example, RuleExpression =""" process_name="malicious.exe" """.

  • RuleType - session, asset, endpoint, web, file, database, external or account-lockout. All events in Advanced Analytics fall under one of these seven event buckets. For example, process-created events are only put into the 'endpoint' sequence. So, even if the RuleEventType contains process-created, if RuleType = 'session', this rule will never fire for process-created events.

When a rule is found to not trigger when it should, it is normally one of these five fields that are edited to fix the rule.

When testing new rules, take the events you want to trigger against, and ensure that the event type is included in RuleEvent types, and that the event type is included in the ruleType (session, asset, endpoint sequence, etc.).

Rule Dependency and Chaining

Rules can have relationships with other rules through the use of the DependencyExpression attribute and WasRuleFired.

Complex sets of rules can be created using the 'Dependency Expression'. This expression checks for other triggered rules that triggered for the same event. That is, if an event triggers Rule_A, and Rule_B has "dependencyExpression='Rule_A' " then Rule_B would trigger only if rule A has triggered (in addition to its own expression). Dependency expressions can use "and", "or", and "not" to define complex dependencies.

The WasRuleFired expression is used to determine if a specific rule has previously triggered in the session or sequence and optionally on which values. If rule_X has WasRuleFired('Rule_Z') in its RuleExpression, it will trigger only if 'Rule_Z' was previously triggered. The expression WasRuleFired('Rule_Z', dest_host) indicates that the current rule should trigger only if Rule_Z has triggered previously in the session/sequence and the value of the dest_host in the event it triggered on is the same as that value in the current event. WasRuleFired can be used also to negate: !WasRuleFired('Rule_X') which means trigger the current rule only if the specified rule has not triggered. This expression is often used to ensure a certain rule triggers only once per session/sequence.

Internal Rules

Internal rules are rules with a score of 0. They are not displayed in the UI and are used by other rules as dependencies. They are used to identify events that have no security value (and therefore no score), so they identify a situation that could be significant for another rule. For example, the internal NEW_USER rule is used to identify a new user in the environment. It is used as a dependency for different rules that identify access to privileged machines, executives' machines, etc. Together they identify a situation in which a new user is accessing a privileged machine. If we want to change how we identify a new user, it has to be done only in the NEW_USER rule and will propagate to all other rules.

Another use of internal rules is to condition on information in more than one model. Since a single rule can condition on data in only one model, multiple internal rules can condition on data in different models and a single score giving rule would use these as dependencies to provide the score if all conditions materialize.

Additional Rule Guidelines

Listed below is a list of additional guidelines and features.

  • Triggered rule info is searchable in the 'triggered_rule_db' in Mongo.

  • RuleExpressions can incorporate any parsed field into the logic. For asset based rules, if you want to use a parsed field in a 'countby' expression, that parsed field must be persisted.

    • When a Model-Based-Asset-Rule uses CountBy(field_1, field_2, event_types), both field_1 and field_2 must be persisted for that event type in the PersistedEventFields definition in the enricher content_default.conf file.

  • User based rules use Count, SequenceCount, and DistinctCount for gathering session/sequence data.

  • Asset based rules use CountBy for all purposes of gathering sequence data. All asset events are 'sequence' events, and thus CountBy can be used for gathering sequence data for any event type.

Fact Based Rules

On the left is a 'User' rule, on the right an 'Asset' rule. The analytics engine needs this distinction RuleType = "asset" to score only against assets as well as an aggregation expression.

UBA (User Based) Rule

EA (Asset Based) Rule

FA-Outlook-pst {   RuleName = "A file ends with either pst or ost"   RuleDescription = "A file copied ends with either pst or ost"   ReasonTemplate = "PST/OST file copied"   AggregateReasonTemplate = "PST/OST file copied"   RuleType = "file"   RuleCategory = "File Activity"   ClassifyIf = """TRUE"""   RuleEventTypes = [     "file-write"   ]   Disabled = "FALSE"   Model = "FACT"   FactFeatureName = "src_file_name"   Score = "20.0"   RuleLabels {   mitre = ["T1114"]   }   PercentileThreshold = "0.1"   RuleExpression = """sequenceCount(src_file_name,'file-write')=1 && (endsWith(toLower(src_file_name), '.pst') || endsWith(toLower(src_file_name), '.ost'))"""   DependencyExpression = "FA-Outlook" }
A-ALERT-DISTINCT-NAMES {
  RuleName = "Various security alerts on asset"
  RuleDescription = "At least three distinct security alerts were reported for the asset. This raises the probability that the asset is compromised."
  ReasonTemplate = "Third distinct security alert on asset"
  AggregateReasonTemplate = ""
  RuleType = "asset"
  RuleCategory = "Security Alert"
  ClassifyIf = """TRUE"""
  RuleEventTypes = [
    "security-alert"
  ]
  Disabled = "FALSE"
  Model = "FACT"
  FactFeatureName = """src_host"""
  Score = "25.0"
  RuleLabels {
    mitre = ["T1066"]
  }
  PercentileThreshold = "0.1"
  RuleExpression = """DistinctCountBy(alert_name, asset, 'security-alert')=3 && !WasRuleFired('A-ALERT-DISTINCT-NAMES')"""
  DependencyExpression = "NA"
  Aggregation {
    DataExpr = """DistinctCountBy(alert_name, asset, 'security-alert')=3 && !WasRuleFired('A-ALERT-DISTINCT-NAMES')"""
    EventExpr = "TRUE"
  }
}
  • FA-Outlook-pst rule is used to detect suspicious files (like .pst and .ost files) being copied. Since, we are focusing specifically on .pst and .ost files only, this is considered fact based instead of model based, which is indicated by Model = "FACT".

  • RuleType = "file" and RuleCategory = "File Activity" indicate that the rule deals with files.

  • ClassifyIf = """TRUE""" is mandatory since it is a fact based rule.

  • The rule investigates "file-write"events to look whether .pst/.ost file is being copied. This is explained by specifying "file-write"event type in RuleEventTypes parameter.

  • FactFeatureName = "src_file_name" describes that the feature value is src_file_name which is a parsed field. Also, this value will be shown while using featureValue in the ReasonTemplate and AggregateReasonTemplate parameters.

  • Based on criticality of the rule, we can assign appropriate score for the rule. Here Score = "20.0".

  • mitre = ["T1114"] in RuleLabels indicate that this rule can be tagged with the MITRE technique T1114.

  • Percentile Threshold = "0.1" means that for the purposes of this rule we only consider events that appear below the 10th percentile to be abnormal.

  • sequenceCount(src_file_name,'file-write')=1 makes sure that the rule triggers only for different values of src_file_name in file-write events. (endsWith(toLower(src_file_name), '.pst') || endsWith(toLower(src_file_name), '.ost')) makes sure that the file extension of src_file_name is either .pst or .ost. The && between the expressions in RuleExpression indicates both conditions should be satisfied for the rule to trigger.

  • DependencyExpression = "FA-Outlook" indicates that the rule FA-Outlook-pst triggers only if FA-Outlook rule has already been triggered.

  • Rule IDA-ALERT-DISTINCT-NAMES (which starts with A-), presence of Aggregation parameter and RuleType = "asset" in Rule config specify that this rule is asset (EA) based.

  • A-ALERT-DISTINCT-NAMES rule is used to detect multiple security alerts (3 different alerts in this case) on a src_host which is a parsed field meaning source host (asset). Since, we are focusing specifically on 3 different alerts only, this is considered fact based instead of model based, which is indicated by Model = "FACT".

  • RuleCategory = "Security Alert" indicates the the rule deals with security alerts.

  • ClassifyIf = """TRUE""" is mandatory since it is a fact based rule.

  • The rule investigates "security-alert" events to detect multiple (3 different) security alerts on an asset. This is explained by specifying "security-alert" event type in RuleEventTypes parameter.

  • FactFeatureName = "src_host" describes that the feature value is src_host which is a parsed field. Also, this value will be shown while using feature Value in the ReasonTemplate and AggregateReasonTemplate parameters.

  • Based on criticality of the rule, we can assign appropriate score for the rule. Here Score = "25.0".

  • mitre = ["T1066"] in RuleLabels indicate that this rule can be tagged with the MITRE technique T1066.

  • PercentileThreshold = "0.1" means that for the purposes of this rule we only consider events that appear below the 10th percentile to be abnormal.

  • DistinctCountBy(alert_name, asset, 'security-alert')=3 makes sure that the rule triggers only if 3 distinct values of security alerts are observed on asset. !WasRuleFired('A-ALERT-DISTINCT- NAMES') makes sure that this rule does not trigger if it has already been triggered. The && between the expressions in RuleExpression indicates both conditions should be satisfied for the rule to trigger.

  • DependencyExpression = "NA" indicates that the rule is independent of any other rule.

  • Aggregation parameter is mandatory for EA rules. It consists of three different parameters. DataExpr which is used to specify expressions used in triggering the rule (like DistinctCountBy/DistinctCountByIf/sumBy/sumByIf etc.) along with expressions which may involve specific conditions (for example custom conditions like user!='System', bytes='100' etc.), EventExpr which is generally TRUE and ModelExpr which is used to specify expressions used in Models like num_observations=0, ConfidenceFactorAboveOrEqual() etc. Generally ModelExpr = """TRUE""" for FACT based rules.

The rule on the left describes a rule most commonly seen in traditional SIEMs. For the parsed field 'x' in an event, if 'x' = 'some_string', trigger the rule. With the context available to Advanced Analytics, 'Fact' based rules have a little more power than this, as shown by the rule on the right. This rule uses the 'DistinctCount' expression to check, within a session, the number of all the different values seen for a field. A 'DistinctCount' on the field 'dest_host' will return the number of dest hosts seen in logs by a user, which is how many different dest hosts were reached by a user in a session. Using an enriched field 'special_field_count_num' that is set to the number 1, a 'Fact' based rule can use the 'Sum' expression to get a count of how many times a specific value in a parsed field was seen.

For example:

Enricher (defined in a different file, see Enrichment):

EventTypes = '
Condition = "!InList(tld_domain, "com", "org", "edu", "uk", "co")
Map = [ field = {"abnormal_tld_count", value = "1"} ]RuleExpression = "Sum(abnormal_tld_count, 'web-activity-denied')=20"

will trigger when a user has been denied 20 times trying to access a website that is not a .com, .org., .edu, .uk. , .co site.

Model Based Rules

For every event that gets processed by the analytics engine, there exists a classification phase. In the classification phase, the event is evaluated against all existing models. In this phase the event is 'triaged' with corresponding models, and corresponding model data is stored with that event, which is later used by the RuleExpressions. In rules that are model based, model calculations the RuleExpression uses are created when the event is classified. Thus, when you see model logic in rule expression, such as percentile_threshold_value, this value is already predetermined.

UBA (User Based)

Model

Rule

SA-UA {
  ModelTemplate = "Security alert names for user"
  Description = "Models security alert names for the user"
  Category = "Other"
  IconName = ""
  ScopeType = "USER"
  Scope = """user"""
  Feature = """alert_name"""
  FeatureName = "alert_name"
  FeatureType = "alert_name"
  TrainIf = """count(alert_name,'security-alert')=1"""
  ModelType = "CATEGORICAL"
  AgingWindow = "32"
  CutOff = "5"
  Alpha = "0.8"
  MaxNumberOfBins = "1000000"
  ConvergenceFilter = "confidence_factor>=0.8"
  HistogramEventTypes = [
    "security-alert"
  ]
  Disabled = "FALSE"
}
SA-UA-F {
  RuleName = "First security alert name for user"
  RuleDescription = "This is the first occurrence of this security alert name for the user"
  ReasonTemplate = "First security alert with name {default|featureValue|histogram} for user"
  AggregateReasonTemplate = "First security alert name for user: {default|featureValue|histogram}"
  RuleType = "session"
  RuleCategory = "Security Alert"
  ClassifyIf = """count(alert_name,'security-alert')=1"""
  RuleEventTypes = [    "security-alert"
  ]
  Disabled = "FALSE"
  Model = "SA-UA"
  FactFeatureName = "alert_name"
  Score = "10.0"
  RuleLabels {
    mitre = ["T1078"]
  }
  PercentileThreshold = "0.1"
  RuleExpression = """num_observations=0"""
  DependencyExpression = "NA"
}
  • SA-UA is used to model Security alert names for user. Refer to Modelsfor details on various model attributes.

  • SA-UA-F (which ends with -F) is used to detect First Security alert name for the user.

  • Model = "SA-UA" demonstrates that the rule is model based and it depends on the data trained by model SA-UA.

  • RuleType = "session" indicates that it is a session rule.

  • RuleCategory = "Security Alert" indicates that the rule deals with Security alerts.

  • count(alert_name,'security-alert')=1 expression in ClassifyIf talks about the frequency of rule trigger. Here, this rule triggers once per alert_name(parsed field) insecurity-alert events.

  • The rule investigates "security-alert" events to detect first security alert name for the user. This is explained by specifying "security-alert" event type in RuleEventTypes parameter.

  • FactFeatureName = "alert_name" describes that the feature value is alert_name which is a parsed field. Also, this value will be shown while using featureValue in the ReasonTemplate and AggregateReasonTemplate parameters.

  • Based on criticality of the rule, we can assign appropriate score for the rule. Here Score = "10.0".

  • mitre = ["T1078"] in RuleLabels indicate that the this rule can be tagged with the MITRE technique T1078.

  • PercentileThreshold = "0.1" means that for the purposes of this rule we only consider events that appear below the 10th percentile to be abnormal.

  • num_observations in RuleExpression indicates how many times the current value (feature) was observed. 0 means never observed before (first time).

  • DependencyExpression = "NA" indicates that the rule does not depend or is independent of any other rule.

EA (Asset Based) Rule

All asset rules use 'countBy' instead of 'count' for all event types.

Model

Rule

A-FLSh-Count {
ModelTemplate = "Count of failed logons from host"
Description = "Models the number of failed logons from this asset"
Category = "Assets"
IconName = ""
ScopeType = "DEVICE"
Scope = """src_host"""
Feature = """DistinctCountBy(event_id,src_host,'failed-logon')"""
FeatureName = "activity"
FeatureType = "quantity"
TrainIf = """TRUE"""
ModelType = "NUMERICAL_CLUSTERED"
AgingWindow = ""
CutOff = "5"
Alpha = "1"
MaxNumberOfBins = "1000000"
ConvergenceFilter = "confidence_factor>=0.8"
HistogramEventTypes = ["sequence-end"]
SequenceTypes = [asset]
Disabled = "FALSE"
}
A-FLSh-Count-Ac {
RuleName = "Abnormal number of failed logons from asset (L)"
RuleDescription = "Extremely abnormal number of failed logons from asset"
ReasonTemplate = "({quantity|featureValue}) failed logons from asset, expected around {quantity|percentileThresholdValue|histogram}"
AggregateReasonTemplate = ""
RuleType = "asset"
RuleCategory = "Failed Logon and Account Lockout"
ClassifyIf = """TRUE"""
RuleEventTypes = ["failed-logon"]
Disabled = "FALSE"
Model = "A-FLSh-Count"
FactFeatureName = "NA"
Score = "40.0"
ScoreTarget = src_host
RuleLabels {
mitre = ["T1110","T1078"]
}
PercentileThreshold = "0.1"
RuleExpression = """num_observations<percentile_threshold_count && ConfidenceFactorAboveOrEqual() && percentile_count_distance>5 && !WasRuleFired('A-FLSh-Count-Ac')"""
DependencyExpression = "NA"
Aggregation {
DataExpr = """!WasRuleFired('A-FLSh-Count-Ac')"""
EventExpr = "TRUE"
ModelExpr = """num_observations<percentile_threshold_count && ConfidenceFactorAboveOrEqual() && percentile_count_distance>5"""
}
}

The above model tracks how many times an asset 'gets' a failed logon event in a day.

This rule triggers when:

A failed-logon event has occurred, and the number of failed-logons in the sequence so far is now considered abnormal compared to what exists in the model. The percentile_count_distance can be used to vary the amount of abnormality you would like to trigger in. Comparing against a bigger value triggers on a 'more abnormal' event(s).