Skip to main content

Log StreamLog Stream Guide

Select JSON Fields from a List of Key/Value Pairs

To select fields from a predefined list of key/value pairs:

  1. In the JSON Path Expression table at the top of the screen, start in the Key in Raw Log column by clicking Select in the Key field of row 1. The Select a Field dialog box opens and lists all of the key/value pairs found in the sample log.

    select-key.png

  2. Scroll through the listed key/value pairs to find the first field you want to map. Click to select it and then click Select key. The selected key appears in the Key field of row 1.

    select-key-db.png

  3. In the Maps to Field column in row 1, select a CIM field from the drop down menu that you want to map the key to. The options are listed according to whether they are core, detection, or informational fields for the specified activity type. You can also click + New field at the bottom of the list and create a custom field.

    select-field.png

    After you select a field, a JSON expression corresponding to the mapped field is displayed in the JSON Path Expression column.

  4. Optionally, if you want to apply conditions to the way a field value from the log is extracted for mapping to the selected CIM field, insert a logical condition in the Extraction Condition column.

    For more information about implementing this type of logic, see Add Logic to JSON Field Extraction.

  5. Optionally, you can also use the Substring Regex column to define a condition that extracts a specific value from within a string and maps only that value to a CIM field.

    For more information about implementing this type of logic, see Add Logic to JSON Field Extraction.

  6. To save the first JSON path expression, click the check at the far right side of row 1. The field mapping is saved. You can check the Sample Log Lines section at the bottom of the page, to see if the mapped field extracts any value from the sample log. Mapped values appear highlighted in various colors.

  7. To map another JSON field, click + New JSON Path Expression. A new row appears in the table.

  8. Repeat the process to select a key field, map it to a CIM field, and click the check to save the field mapping.

  9. Continue to add JSON path expression rows until you have mapped all of the core fields and any of the detection, informational, or custom fields you think will be useful.

  10. Optionally, if your sample JSON log is in a hybrid format and includes a specific type of non-JSON prefix, you can extract value from this prefix by using a regular expression. The supported hybrid log format is a JSON message that is prefixed with text metadata from the Cloud Collector source, as in the following example:

    destinationServiceName=Azure <JSON message>

    You can generate a regular expression to extract and map this information as follows:

    • At the bottom of the JSON Path Expression table, click the Regex radio button. The add expression button to the left changes to + New Regular Expression.

      new-regex.png

    • Click the + New Regular Expression button. A new row is added to the mapping table.

    • Use one of the methods outlined in Extract Fields Using Regular Expressions to define the field extraction.

    • Click the check at the far right side of the new expression row. Regex extracted values also appear highlighted in the Sample Log Lines section at the bottom of the page.

  11. After you have selected all of the necessary fields for extraction, ensure the fields are in the correct order, review the matching event type fields and log values, then continue building your custom parser by configuring event builders.