Data LakeData Lake User Guide

The Date Histogram aggregation requires a field of type date and an interval. It will then put all the events into one bucket, whose value of the specified data field lies within the same interval. Besides the easily recognized interval values (minutes, seconds, etc.) there is the value auto. When you select auto the actual time interval will be determined by Data Lake depending on how large you want the histogram to be, so that a reasonable number of buckets will be created.

Example: You can construct a Date Histogram on the event_time or @timestamp field of all messages with the interval minute. This will create a bucket for each minute and each bucket will hold all events that were generated in that minute.

A Histogram is similar to a Date Histogram except that it can be used on any numerical field. As with the Date Histogram, you specify a number field and an interval. The aggregation then builds a bucket for each interval and includes all events whose value falls inside this interval.

Example: You can construct a Histogram on the event_code field with the interval of hour. This will create a bucket for each hour and each bucket will hold all events that were generated in that hour.

The Range aggregation is like a manual histogram aggregation. You also need to specify a field of type number, but you have to specify each interval manually. This is useful if you either want differently sized intervals or intervals that overlap.

Example: Filtering on the riskscore field, you can plot the count of critical users (those with a risk score greater than 90), high risk users (those with a risk score between 80 and 90), and non-risky users (those below 80).

A Terms aggregation creates buckets based on the values of a field. You specify a field (which can be any type) and it will create a bucket for each of the disparate values that exist in that field.

Example: You can run a Terms aggregation on the field src_country which holds the country name of the user. Afterward you would have a bucket for each country and in each bucket the events of all users in the country.

A Filter aggregation is extremely flexible. You can specify a filter for each bucket and all events that match the filter will be in that bucket. Each filter is a query, as we discussed in the Search chapter of this document. The filter that you specify for each bucket is entirely at your discretion.

Example: You can create a filter aggregation with one query being src_country: canada to see all the events that originated in Canada.

Significant Terms are "uncommonly common" terms in a set of events. In other words, this aggregation returns unusual or interesting terms in a given subset of events. Given a subset of events, this aggregation finds all of the terms which appear in this subset more often that is expected from term occurrences in the entire events set. It then builds a bucket for each of the significant terms - each bucket contains all the events of the subset in which this term appears. The size parameter controls how many buckets are constructed, i.e. how many significant terms are calculated.

The subset on which to operate the significant terms aggregation can be constructed by a filter or you can use another bucket aggregation first on all events and then choose Significant Terms as a sub-aggregation which is computed for the events in each bucket.

Example: Using the search field at the top to filter events for event_data.FailureReason and setting the size parameter to 5 will return the top 5 most unusual failure reasons for a failed logon.

Date Range aggregation is similar to Range aggregation, except that from and to values can be expressed. It is an aggregation dedicated to date values.

The date range expression itself starts with an anchor date, which can either be now, or a date string ending with ||. This anchor date can optionally be followed by one or more expressions.

Examples of date expressions:

This is not really an aggregation. It just returns the number of events that are in each bucket as a value for that bucket.

Example: If you want to know how many events are from which country, you can use a term aggregation on the field src_country (which will create one bucket per source country) and afterward run a count metric aggregation. Every country bucket will have the number of events as a result.

For the average and sum aggregations you need to specify a numeric field. The result for each bucket will be the sum of all values in that field or the average of all values in that field respectively.

Example: You can have the same country buckets as above again and use an average aggregation on the user.logons field to get a result of how many logons a user in that country has on average.

Like the average and sum aggregation, this aggregation needs a numeric field to run on. It will return the minimum value or maximum value that can be found in any event in the bucket for that field.

Example: If we use the country buckets again and run a maximum aggregation on the logon_count we would get for each country the highest number of logons a user had in the selected time period.

The unique count will require a field and count how many different/unique values exist in events for that bucket.

Example: This time we will use range buckets on the user.logons field, meaning we will have buckets for users with 1-100, 100-1000 and 1000- events. If we now run a unique count aggregation on the src_country field, we will get for each user's range the number of how many different countries users with x number of events would come. That could show us that there are users from 18 different countries with 1 to 100 events, from 30 different countries for 100 to 1000 events and from 4 different countries for 1000 events and above.

A percentiles aggregation is a bit different, since it won’t result in one value for each bucket, but in multiple values per bucket. These can be shown as different colored lines in a line graph, for example.

When specifying a percentile aggregation, you have to specify a number value field and multiple percentage values. The result of the aggregation will be the value for which the specified percentage of events will be inside (lower) as this value.

Example: You specify a percentiles aggregation on the field src.country_count and specify the percentile values 1, 50 and 99. This will result in three aggregated values for each bucket. Let’s assume we have just one bucket with events in it. The 1 percentile result (and e.g. the line in a line graph) will have the value 7. This means that 1% of all the events in this bucket have a count of 7 or below. The 50-percentile result is 276, meaning that 50% of all the events in this bucket have a count of 276 or below. The 99-percentile have a value of 17000, meaning that 99% of the events in the bucket have a count of 17000 or below.