Skip to main content

Log Ingestion

Overview​

In order to explore your logs, you need to configure a logging source to send the logs to Motadata ObserveOps. The following type of logs can be ingested in Motadata from your infrastructre, some of them directly and some of them through an agent.

  • Application Logs
  • Server Logs
  • Network Logs

Application Logs​

A variety of applications are used in the current scenario across infrastructure around the world and these applications generate a large volume of log data that can be used for analysis to figure out why a particular issue occured in an application.

Example of Sending Application Logs to ObserveOps​

Let us take an example to understand how to send Apache logs to Motadata ObserveOps.

  1. Install agent on the Apache server.

  2. Go to Settings. Select Monitor Settings. After that, select Agent Monitor Settings.

  3. Select View Details against the agent for which you want to ingest the logs. Click on the Log tab to start the log configuration.

Enter the log configuration details as follows:

FieldDescription
Log Agent StatusToggle this button ON/OFF to start/stop the log ingestion for this agent.
Log DirectoryEnter the exact path where the log file is located on the server. In this case for apache logs on the linux server we can see that the path is entered as '/var/log/apache2/'. This is the path where the Apache logs are located on the server.
Log IncludeMention the file name or the extension of the file in this field to make sure that only the logs from that particular file present at the path mentioned in Log Directory above are ingested in the system. For example, if you want to ingest logs from a file 'access.log' you can do that by mentioning the file name in this field as 'access.log'. You can ingest logs from all the log files with the extension '.log' by specifiying '*.log' in this field.
Multiline LogUse this toggle button to specify that the log you are ingesting are multiline logs.
File PatternThis field is available only if you switch the Multiline Log toggle button ON. Specify the file from which you want to ingest the log data.
Log PatternThis field is available only if you switch the Multiline Log toggle button ON.Specify the log pattern of the multiline logs that you want to ingest from the file that you have specified in the previous field. Enter the regex that could be used to identify the pattern that is used to differentiate two lines of logs in multiline logs.

We just looked into how you can ingest Apache logs into Motadata ObserveOps by just pointing ObserveOps towards the log file by providing the directory and log file details in the log configuration in the agent. Similarly, you can also do the same for other application logs such as IIS, NGINX, and many more.

Example of Ingesting Multiline Logs in Motadata ObserveOps​

Ingesting multiline logs in Motadata ObserveOps requires identifying the start of each new log entry within the multiline logs. This is achieved using a specific log pattern that can be defined using regular expressions (regex). Let's take an example of multiline IBM MQ logs to illustrate this process.

IBM MQ Multiline Logs Example​

Consider IBM MQ logs where each new log entry starts with a line similar to:

04/09/24 09:46:51 - Process(9241016.5111) User(mqm) Program(amqrmppa)

In these logs, each new entry begins with a timestamp followed by the word "Process". This pattern can be utilized to indicate the beginning of a new log entry.

Defining the Log Pattern​

To help ObserveOps correctly ingest and parse these multiline logs, we need to define a regex pattern that matches the start of each new log entry. For our example, the regex would look like this:

\d+\/\d+\/\d+\s+\d+:\d+:\d+\s+-\s+Process

This regex pattern matches:

  • A date in the format MM/DD/YY (e.g., 04/09/24)
  • A time in the format HH:MM:SS (e.g., 09:46:51)
  • The literal string - Process

Configuring the Log Pattern in Motadata ObserveOps​

In the Log Pattern field, enter the regex pattern: \d+\/\d+\/\d+\s+\d+:\d+:\d+\s+-\s+Process.

By configuring the log pattern with this regex, Motadata ObserveOps will be able to recognize the start of each new log entry in the multiline IBM MQ logs. This allows ObserveOps to accurately ingest and parse the logs, ensuring that the log entries are correctly identified and processed.

This approach can be applied to other types of multiline logs by defining an appropriate regex pattern that matches the unique start of each log entry for those logs.