Data LakeParser Troubleshooting Guide

Table of Contents

Troubleshoot Advanced Analytics Parser

Advanced Analytics automatically identifies poor parser performance and disables such parsers in order to preserve the system health.

Note

You are shown an indicator when Advanced Analytics determines that a parser is problematic and disables it.

Find Disabled Parsers

Navigate to System Health > System Optimization, and then click on the Data Disabling tab.

AA-System-Health-Data_Disabling-Disabled-Parsers.png

You will see the list of disabled parsers. The table includes columns with the following categories:

  • Parser Name – The name of the disabled parser.

  • Average Log Line Parse Time – Average time taken by the parser to parse each event.

  • Disabled Time – Date and time when the parser was disabled.

Alternatively, query MongoDB for the contents of Disable_parser_db.current_collection, if the Advanced Analytics UI is not available:

  • For each monitoring interval, lime.log will contain "Disabled parsers for this period are: "

  • When disabling one parser , lime.log will note "Disabling parser:..."

Note

It will also output average parsing time periodically as configured through OutputParsingTimePeroidInMinutes.

To find out the historical disabled parsers, query MongoDB for the contents of Disable_parser_db.historical_collection.

Fix a Disabled Parser

Once a parser is disabled it will show with in the UI as stated in the previous section. You can also find the corresponding log message associated with the disabled parser in /opt/exabeam/data/logs/exabeam.log.

Determine which parser has been disabled:

  1. Obtain the entry in the exabeam.log for the disabled parser.

  2. Create a ticket with the Exabeam Content Team to troubleshoot the parser. Include logs for the parser from the hour before it was disabled.

  3. Once the Exabeam Content Team delivers the fixed parser, you can apply it and then restart the parser service:

    lime-stop; lime-start

Adjust the threshold:

  • If you want to see only the parsing statistics in the logs, for example, what is the average parsing time for each parser, then you can overwrite LogParser.OutputParsingTime to true.

  • If you want to turn off parser disabling, you can set LogParser.AllowDisableParser = true. There is no need to turn on the OutputParsingTime flag, because once the AllowDisableParser is enabled, it will automatically output the statistics.

Restart the parsing engine after threshold adjustments are made.

lime-stop; lime-start

Threshold Tuning

You may expect a parser to take a long time to run due to the nature of the logs or the complexity of the parser. In such cases, re-enable the parser and adjust the performance thresholds to prevent false-positive parser disabling.

Tune up the following two parameters to prevent false-positive disabling of the parser:

  • LogParser.ParserDisableThresholdInMills

  • LogParser.ParserDisableTimePercentage

To configure the thresholds, navigate to the LogParser section located in the lime_default.conf file at /opt/exabeam/config/default/.

All changes should be made to /opt/exabeam/config/custom/custom_lime_config.conf:

LogParser{
          #Output parsing performance in debug mode, be cautious this might affect performance in parsing
          OutputParsingTime = true
          OutputParsingTimePeriodInMinutes = 5
          AllowDisableParser = true // If this is enabled, output parsing time will be enabled by default.
          ParserDisableThresholdInMills = 30 //If average parsing time pass this threshold, we will disable that parser
          ParserDisableTimePercentage = 0.5
}

Acceptable values for ParserDisableThresholdInMills includes any integer value. ParserDisablePercentage can be a percentage decimal value between 0.1 to 0.9.

Note

Setting a higher parsing time percentage identifies less severe parsers.