- Log Stream Overview
- Parser Manager
- Parsers Overview
- View Parser Details
- Create a Custom Parser
- Import Sample Logs
- Define a Subset of the Sample Logs
- Add Conditions
- Add Basic Parser Information
- Extract Event Fields
- Extract Mapped JSON Fields
- Select JSON Fields from a List of Key/Value Pairs
- Select Tokenized JSON Fields from the Values in the Sample Log
- Manually Enter JSON Path Expressions
- Sample JSON Path Expressions for Manual Entry
- Reorder Mapped JSON Fields
- Review the Matching JSON Fields and Values
- Add Logic to JSON Field Extraction
- Expressions for JSON Parser Field Extractions and Enrichment Mapping
- Extract Fields Using Regular Expressions
- Reserved Fields
- Extract Mapped JSON Fields
- Add Event Builder Rules
- Review and Save Parser
- Manage Existing Parsers
- Tokenize Non-Standard Log Files
- Duplicate a Parser
- Enable or Disable Parsers
- Parser Updates
- Supported Vendors and Products
- Live Tail
- Enrichments
- Event Filtering
- Log Stream APIs
Event Filtering
Note
License Requirement
Currently, the Event Filtering tab is available only if you have the New-Scale Analytics license.
If you have read-only access to Log Stream, any options that would permit creating or modifying event filters are unavailable. For information about access levels, see Access Log Stream
The Event Filtering tab in the Log Stream application provides access to filter which events you want to route to the New-Scale Analytics detection engine or to the Attack Surface Insights application. Events are consumed by various downstream services and applications based on a set of defined detection rules. But in some cases, it can be beneficial to filter which events or types of data are leveraged for detection or for Attack Surface Insights. This type of filtering can reduce the cost of processing events that have no security value, reduce the number of false positives, and reduce the generation of unnecessary alerts.
Event filtering is based on the enrichment rule framework. Multiple default enrichment rules are provided in Log Stream to route ingested events to either the detection engine or to Attack Surface Insights. When these default enrichment rules are enabled, they tag events with a field called m_tags. Among the values that an m_tags field can contain are a set of Discard values that specify how events should be routed for processing. These tags are applied to events based on how you configure either event filtering or enrichment rules.
The following Discard values for m_tags are available:
Discard BEAM– Thism_tagsvalue is applied to all ingested events by default. It instructs the New-Scale analytics detection engine to discard those events without using them to trigger any alarms or detections. However, you can configure the event filter to control which events should have this tag removed so they can be routed for processing by the detection engine.To configure the event filter, follow the steps below, in Defining Event Filters, to unselect vendors and products in the Detection Engine section of the Event Filtering tab. By unselecting vendors and products, you instruct the analytics engine to process events that contain that information. To find this enrichement rule in Log Stream, navigate to the Enrichments tab, click the filter icon (
) in the Category column header, and select Event Selection.Discard EM– Thism_tagsvalue is not applied to any ingested events by default and, therefore, all incoming events are processed in the Attack Service Insights application to create entities. However, you can configure the event filter to control which events you want this tag removed from so they are not routed for processing by Attack Surface Insights. For more information about how the Discard EM tag is used, see Restrict Entity Creation in the Attack Surface Insights Guide.To configure the event filter, follow the steps below, in Defining Event Filters, to select vendors and products in the Attack Surface Insights section of the Event Filtering tab. By selecting vendors and products, you instruct the Attack Surface Insights application not to process events that contain that information. To find this enrichement rule in Log Stream, navigate to the Enrichments tab, click the filter icon (
) in the Category column header, and select Event Selection.Discard_EM_USER– Thism_tagsvalue can be used to indicate that events should not be used by Attack Surface Insights to create user entities. This value is not applied to ingested events by default and can only be enabled or disabled in the Enrichments tab of the Log Stream application. When enabled, user entities are not created, but device entities will continue to be created. To find these types of enrichement rules in Log Stream, navigate to the Enrichments tab, click the filter icon () in the Enriched Fields column header, and select m_tags.Discard_EM_HOST– Thism_tagsvalue can be used to indicate that events should not be used by Attack Surface Insights to create device entities. This value is not applied to ingested events by default and can only be enabled or disabled in the Enrichments tab of the Log Stream application. When enabled, device entities are not created, but user entities will continue to be created. To find these types of enrichement rules in Log Stream, navigate to the Enrichments tab, click the filter icon () in the Enriched Fields column header, and select m_tags.
Note
When you define a Discard enrichment rule in the Enrichemtns tab, define the mapping condition using an append expression to define m_tag field as shown in the example below:
Example:append(m_tags, 'Discard_EM_USER')
Navigating the Event Filtering Tab
The filtering and routing capabilities for each engine are separated on the page. Filtering and routing for the Detection Engine is in the top section. Filtering and routing for Attack Surface Insights is on the bottom.
Each section includes the panels of information shown in the image below and described in the corresponding numbered points:
Event Filter Notification – This note indicates whether any event filtering has been implemented. If a filter has been applied, events related to specific vendors or products are being routed to the detection engine or to Attack Surface Insights.
View Vendor and Products Added – Click this link to open a panel that lists which vendors and products are selected for routing to the detection engine or Attack Surface Insights. For more information, see Defining Event Filters.
24 Hour Volume – This thumbnail graph shows how many events, in the last 24 hour period, have been routed to the detection engine or Attack Surface Insights, based on the vendor/product filtering applied.
Add Event Filter – Click this button to open a dialog box to add or edit event filtering. For more information, see Defining Event Filters.
Most Active Vendors – This panel shows which vendors and products, over a specified period of time, have routed events to the detection engine or Attack Surface Insights. The list is organized in descending order from the most active (based on the volume of parsed logs) to the least active. You can scroll through the list and click on items to view the full product and vendor names. By default, the panel shows data for the last 24 hours but you can click the calendar icon (
) to change the time period to show data for the last 7 or 30 days.Events Routed Over Time – This graph shows the number of events, over a specified period of time, that have been routed to the detection engine or Attack Surface Insights. When you hover your cursor over the graph, the time points become visible on the graph itself. Hover over a specific point to see the number of events routed at that point in time. By default, the panel shows data for the last 24 hours but you can click the calendar icon (
) to change the time period to show data for the last 7 or 30 days.
Defining Event Filters
To define event filtering, use the procedure below to select which vendors and products you want to include in the filter. When events are ingested that include the selected vendors and products, those events are routed to the detection engine or to the Attack Surface Insights. application.
Open Log Stream and click the Event Filtering tab at the top.
On the Event Filtering tab, do one of the following in either the Detection Engine or the Attack Service Insights panel to open the Add Event Filters dialog box:
Click the Add Event Filter button under the 24 Hour Volume graph on the left.
Click the View Vendor and Products Added link on the right. When the list panel opens, click the Edit Filter link at the top.
The Add Event Filters dialog box displays a list of vendors and their products that can be selected to filter events for routing to the detection engine or Attack Surface Insights.
Select the vendors and products you want to include in the filter. You can interact with the list to make selections as follows:
Use the search bar to find a specific vendor or product quickly in the long list.
Click the check box next to a vendor to add the vendor and all of its listed products to the filter.
Click the arrow to the right of a vendor to expand it and select a specific product without selecting the vendor itself or other listed products.
Click Select all to add all of the listed vendors and products to the filter.
Click Clear all to remove all vendor and product selections.
When you have selected all the desired vendors and products, click Add Event Filters. Your event filter is updated.
You can verify the updates to the event filter in the following ways:
On the Event Filtering tab, click the View Vendor and Products Added link. A panel opens and displays a list of the updated vendors and products included in the filter. When events are ingested that include data related to the selected vendors and products, those events are routed to the detection engine or Attack Surface Insights for processing.
Navigate to the Enrichments tab in Log Stream. In the table of enrichment rules, click the filter icon (
) in the Type column header and select Event Selection. For the enrichment rule that corresponds to the event filter you are updating, click the options menu (
) on the right and select View Details. A details pane opens where you can examine the Condition parameter to ensure that all of the vendors and products you selected for event filtering are listed. When events are ingested that do not include the selected product or vendor data, they are tagged with the m_tagsfield and are not routed to the detection engine or Attack Surface Insights.