Dynamic Field Extraction

To address immediate search needs on data that has not been parsed, you can define a parsed field from the Search query. This will parse fields that might have been missed at the parsing stage, without having to first define a new parser.

To perform a dynamic field extraction, use Exabeam advanced query language with an RGX_EXTRACT function inside of a SELECT clause, like the following:

SELECT RGX_EXTRACT( [fieldName,] [regexp_pattern] ) [alias_expression]

where:

fieldName – Source field name value from which the field is extracted. If omitted, the extraction uses the raw log message.
regexp_pattern – Regex expression to extract field value
alias_expression – Target field name for extracted values

Note

Limitations

RGX_EXTRACT only works for string data types.
The reserved keywords (SELECT, RGX_EXTRACT, AS, WHERE) can be either all lowercase or all uppercase.
You cannot use an existing field as a dynamic parsed field.
You can enter up to three RGX_EXTRACT expressions at a time.
RGX_EXTRACT does not currently support quick menu options for adding fields to a query (AND, AND NOT, OR) or for visualizing fields.
RGX_EXTRACT does not support the Aggregations feature available in the search results toolbar. However, when creating your search, you can leverage advanced query language syntax to produce an aggregation table. See the example below in RGX_EXTRACT Query with Aggregation.

The following functionality is not supported for dynamic parsed fields:

These fields will not appear in the Field Summary panel.
The Visualize Search buttons are disabled for all fields when dynamic parsed fields are used.
Context table lookup on a dynamic parsed field is not supported.
You cannot add or remove a dynamic parsed field in the query string from the menu that appears when clicking a field in the search results list, or on the Event Details panel.
The Convert to Rule functionality is disabled when using dynamic field extraction.
Export for dynamic parsed fields is not supported.

Sample RGX_EXTRACT Query

The following is a sample query that uses the RGX_EXTRACT function in a SELECT clause:

SELECT *, RGX_EXTRACT("Status:\s*(0x[0-9a-fA-F]+)") AS extracted_login_status WHERE vendor ="Microsoft" AND outcome="fail" AND activity_type="login"

This sample query searches the raw logs for failed login events where the vendor is Microsoft and a specific value is present in an unparsed status field. It extracts the value of the status field and parses it as a field labeled extracted_login_status.

Note

The sample query above will return results for login events that were ingested in a native JSON format.

When a value is successfully parsed using dynamic field extraction, the dynamic parsed fields will be displayed in the search results, and in the Event Details panel, as shown in the images below. Dynamic parsed fields will appear in List view and Table view, regardless of what field template is selected.

RGX_EXTRACT Query with Aggregation

Dynamically parsed fields do not support the Aggregations feature available in the search results toolbar. However, when you create your search, you can use the GROUP-BY advanced query language operator, in combination with aggregation query functions, to produce an aggregated table. In the following example, the COUNT(*) function has been added to the start of the query and the GROUP-BY clause has been added to the end:

SELECT count(*), RGX_EXTRACT("Status:\s*(0x[0-9a-fA-F]+)") AS extracted_login_status WHERE vendor ="Microsoft" AND outcome="fail" AND activity_type="login" GROUP-BY extracted_login_status

This query produces search results in a table view that shows counts for the extracted field values, as shown in the image below. For more information about using these advanced query language features, see GROUP-BY and Query Using Aggregation Functions

Exabeam SearchExabeam Search Guide

Table of ContentsTable of Contents