Types of Log Parsers
Overview
Parsers are crucial for processing of logs. They facilitate extraction of selected information which can be further used for diagnostics and optimizations of a network or a system. Motadata AIOps allows you to parse logs using four distinct methods which are explained as follows:
Regex
Creating a Regex for log is very simple. You can either upload a log file using the Upload option or enter a sample log in the Log field. Then, select a part of the log and Motadata AIOps will automatically create a regex of the selected portion for you. You will only need to enter the Field name and provide a Field Type for the values. You can also edit the created Regex using the edit option next to the created regex.
The Timestamp is usually selected automatically when you select the sample log. In a rare case Motadata AIOps does not recognize it automatically, just double-click on the Timestamp field to create a regex for the field.
JSON
JSON type log files consists information in Key:Value pairs. The JSON log parser in Motadata AIOps extracts the information and segrates the Key and Value. Every Key:Value pair from the log is extracted and segregated by the parser into JSON Key and JSON Value fields.
Once you upload the log file using the Upload option or enter a sample log in Log field. AIOps will parse the logs and display the segregated key and value pairs. You can then select these Key:Value pairs and add them as Field Name and Field Value respectively by selecting them using individual checkbox. You would then need to define the Field Type for each of the added values.
Do remember, should you want to parse a different combination of fields or key:value pairs, you can always create a separate parser. For instance, you need complete log information (error ID, error message, timestamp, and other applicable fields.) but also want to parse two fields (error ID and error message) separately from the same set of logs, create two parsers. One that parses the complete log and the other one that only parses the specific fields.
You can also concat (join) two defined fields and create a custom field.To create a custom fied, click on Add Operation, then select a Field Value from a dropdown, choose the CONCAT operator using the Operator dropdown and finally choose the second Field Value. Click on Add Field to add the custom field. Do remember you will need to define the Field name and Field value for the concated field.
Delimiter
Delimiter parser uses a special character (comma, colon, or semicolon) to separate different portions of a log and extract information. You need to upload a log file using the Upload option or enter a sample log in the Log field. Also, when parsing the logs using Delimiter, you will need to mention the special character in the Delimiter field.
Ensure that you enter only a single character in the Delimiter field. Entering multiple characters at the same time may produce unexpected results.
Once the sample log is processed and fields are generated with the help of your mentioned delimiter; select each one that you wish to parse by clicking on the individual checkboxes preceding each field of the log. Once selected, you will need to define the Field name and Field Type for the added values. You can also concat (join) two defined fields and create a custom field.
To create a custom fied, click on Add Operation, then select a Field Value from a dropdown, choose the CONCAT operator using the Operator dropdown and finally choose the second Field Value. Click on Add Field to add the custom field. Do remember you will need to define the Field name and Field value for the concated field.
Custom Plugin
Custom Plugin parser comes into picture when you have highly-specific parsing needs. You can create a script-based parser that can analyze and parse the data tailored to your requirements. Write the script for your parser using the Log Parser Plugin screen. After creating a plugin, you can choose it using the Select Plugin dropdown.
