Log Parsing
Overview
Once the logs are ingested into AIOps, the next step is filtering and parsing these logs to make them meaningful to the users. Log parsing is the process of breaking down the log data into its constituent parts and assigning a meaning to it. Once the logs are parsed, users can get meaningful insights from these logs when viewed using the AIOps Log Explorer.
AIOps uses log parsers to parse the incoming log data. A log parser is a software component that can extract data fields from a log message, normalize and format them, and categorize them into various fields.
Let us understand how the Log parsing and Filtering take place in detail.
Once the logs are sent to AIOps, they could either be from a known source (Monitor) or an unknown source (a device that is not a Monitor in AIOps).
When the logs come from a known source (Monitor), a parser is assigned to the source based on the Type of the source. After that, the logs are parsed in the system and moved into the appropriate log category. After that, the logs are available in the system in a suitable format for analysis.
When the logs come from an unknown source, any one of the following two options can happen:
In case you have already configured the source in the log inventory and assigned a parser to the source, the logs will be parsed with the selected parser and the logs will move into the right category based on the Type of the device configured and the parser assigned to the device in the log inventory .
In case the source is not configured in the log inventory, the logs arriving from that source will be placed in the 'Other' category. A record corresponding to this unknown log source is created in the Log Inventory when the logs arrive from this unknown source. Once we assign type to this unknown log source and assign a parser to this log source in the log inventory, the new logs that arrive from the same source from that moment onwards will be parsed with the selected parser and the logs will move into the right category based on the Type of the device configured in the log inventory.
You can create a record corresponding to the unknown log source in advance in the Log Inventory to ensure that the logs that arrive from the source are parsed right away and are not moved into the 'Other' category.
Log Parser Library
Motadata AIOps provides an inbuilt library of parsers that are used to parse logs from a range of sources, across multiple vendors. There are 20 inbuilt parsers available in the system out of which 15 are java based parsers and the remaining 5 are regex based parsers.
These parsers are used to turn the raw logs into meaningful data by educating the system about the type of logs, the important fields present in the logs, where exactly these fields are placed in the logs.
Motadata AIOps has inbuilt parsers for several commonly used applications such as Apache, IIS, NGINX, MySQL, Oracle, and many others. These parsers can also parse logs from various network devices such as firewalls, routers, switches, load balancers, and more.
In case one of the multiple inbuilt parsers is not able to parse some specific logs from a new vendor in the market not supported by AIOps, you can also create a parser of your own to parse these logs. This can be done easily by writing custom rules for the parser. Once the custom parser is created, it can be added to the system and used for parsing the logs.
By using the Log Parser Library in Motadata AIOps, users can parse and filter logs easily, enabling them to identify and troubleshoot issues in their environment with ease.
Navigation
Go to the Main Menu, Select Settings 
. After that, go to Log 
. Select Log Parser Library. The list of all the log parsers in the system is now displayed.
Log Parser Library Screen
The following fields are available on the log parser library screen:
| Field | Description |
|---|---|
| Log Parser Name | The name of the parser. |
| Used Counts | The number of devices to which the parser is assigned. |
| Log | The sample log message used to create the parser. |
| Log Parser Type | The type of log parser. |
| Actions | The actions available for a parser. We will discuss this in detail below. |
Now, let us look into how to create a parser of your own.
Create Log Parser
In case one of the multiple inbuilt parsers is not able to parse some specific logs from a new vendor in the market not supported by AIOps, you can also create a parser of your own to parse these logs.
Navigation
Go to the Main Menu, Select Settings . After that, go to Log . Select Log Parser Library. The list of all the log parsers in the system is now displayed.
Select 
to create a log parser. The screen to create a new log parser is now displayed.
Create Log Parser Parameters
Enter the details of all the parameters on the Create Log Parser screen as per the following details:
| Field | Description |
|---|---|
| Log Parser Name | Enter the name of the parser you want to create. |
| Log Parser Type | Select the parser type from the dropdown as per the type of parser you wish to create. |
| Type | Select the type of the device generating the logs you want to parse. |
| Vendor | Select the vendor of the device generating the logs you want to parse. |
| Log Parsing Condition | This field works with the next field Log Parsing Filters for filtering logs for parsing based on keywords. The logs filtered through these fields will be parsed using the parser and the logs that do not get filtered will be moved to the 'Other' category. Select one of the following two options - All : The log event will be filtered for parsing only if ALL the keywords specified in the next field Log Parsing Filters are present in the log event - Any: The log event will be filtered for parsing if ANY of the keywords specified in the next field Log Parsing Filters are present in the log event. |
| Log Parsing Filters, If Log Contains | Specify the keywords to filter the log events. This field works with the previous field Log Parsing Condition to filter the log events based on the keywords mentioned in this field. |
| File Upload | You can use this field to upload the log file that you wish to parse. You can use the logs from this file to create the parser. This process is explained further below. |
| Log | Enter a sample log event from the log file that you wish to parse. |
| Regex | This is the auto-generated regex which is created when you click on the fields in the sample log event entered in the above Log field. The process to create an auto-generated regex is explained further below. |
| Fields | This is list of all the fields parsed from the sample log file. The list of the fields is dynamic and is updated based on the changes you make to the parser. |
| Add Operation | Click on this button to add a custom field to the parser using a concatenation operation between any two fields that you have seperated using the parser. |
| Select Plugin | This field ony shows up if you select Custom Plugin in the Log Parser Type field. Select one of the inbuilt plugins or a custom plugin that you want to use for parsing while using the Custom Plugin parser type. |
| Delimiter | This field ony shows up if you select Delimiter in the Log Parser Type field. Specify the delimiter that you want to use to seperate the fields in the logs |
Parser Creation Example
Let us take an example. Suppose you want to create a parser for Linux Syslog that parses the timestamp, process, and the process ID from all the messages in the Linux Syslog. We can use a regex parser
Enter all the details such as the parser name and start creating the parser.
Filter Conditions
Suppose you want the parser you are creating to filter and only parse the log events received in the month of February OR March, you can use the filter conditions in the parser to do the same.
Enter the keywords Feb and Mar in the Log Parsing Filters field to filter the log events that have the timestamp for either February OR March.
Select Any in the Log Parsing Condition to specify log events containing either Feb OR Mar will be filtered.
All other log events that do not contain the either of the keywords Feb OR Mar will be moved to the 'Other' category.
Suppose you want the parser you are creating to filter and only parse the logs received in March for a particular process ID, 21992 , you can use the filter conditions in the parser to do the same.
Enter the keywords Mar and 21992 in the Log Parsing Filters field to filter the log events that have the timestamp for March and have the .
Select ALL in the Log Parsing Condition to specify that the log events containing BOTH the keywords Mar and 21992 will be filtered .
All the other log events that do not have BOTH the keywords Mar and 21992 will be moved to the 'Other' category.
Generating the Regex
Enter a sample log event from the log file you want to parse in the Log field.
Select the part/field of the log event that you want to identify and seperate using the parser. A regex corresponding to the selected part of the log event will be auto generated in the field Regex.
For example, you can select the specific part of the sample log event that contains process name to auto-generate corresponding regex to identify process names.
The time stamp from the log event is parsed by default and does not require to be added to the regex.
As you select more fields/parts from the sample log event, the regex get modified to include the fields you have selected. For example, after generating the regex for process name, you might also want to include the part of the log event that includes the process ID.
Now, the regex to identify and parse the Process ID, Process Name, and the Timestamp is generated.
Provide Field Names and Create the Parser
After parsing the field names using the regex, we provide the names to the parsed fields. The names that you provide to the field will be used to identify them in the Log Search to view the log details.
For example, in the diagram below, the fields have been named Process ID, Process Name, and the timestamp.
Select Create Parser to create the parser and add it to the Log Parser Library.
Select Create Parser & Upload Logs to create the parser and add it to the Log Parser Library and upload the attached logs to the system. The logs uploaded will be parsed and available to view in the Log Search
Select Reset to erase all the current field values, if required.
All the logs parsed with the parser you created will be available in the Log Search in the Linux category with the parsed values of Process ID, Process Name, and Timestamp.