Skip to main content

Log Ingestion

Overview​

In order to explore your logs, you need to configure a logging source to send the logs to Motadata AIOps. The following type of logs can be ingested in Motadata from your infrastructre, some of them directly and some of them through an agent.

  • Application Logs
  • Server Logs
  • Network Logs

Application Logs​

A variety of applications are used in the current scenario across infrastructure around the world and these applications generate a large volume of log data that can be used for analysis to figure out why a particular issue occured in an application.

Example of Sending Application Logs to AIOps​

Let us take an example to understand how to send Apache logs to Motadata AIOps.

  1. Install agent on the Apache server.

  2. Go to Settings. Select Monitoring Settings. After that, select Agent Monitor Settings.

  3. Select View Details against the agent for which you want to ingest the logs. Click on the Log tab to start the log configuration.

Enter the log configuration details as follows:

FieldDescription
Log Agent StatusToggle this button ON/OFF to start/stop the log ingestion for this agent.
Log DirectoryEnter the exact path where the log file is located on the server. In this case for apache logs on the linux server we can see that the path is entered as '/var/log/apache2/'. This is the path where the Apache logs are located on the server.
Log IncludeMention the file name or the extension of the file in this field to make sure that only the logs from that particular file present at the path mentioned in Log Directory above are ingested in the system. For example, if you want to ingest logs from a file 'access.log' you can do that by mentioning the file name in this field as 'access.log'. You can ingest logs from all the log files with the extension '.log' by specifiying '*.log' in this field.
Multiline Log

We just looked into how you can ingest Apache logs into Motadata AIOps by just pointing AIOps towards the log file by providing the directory and log file details in the log configuration in the agent. Similarly, you can also do the same for other application logs such as IIS, NGINX, and many more.

Server Logs​

There are multiple servers in infrastructure setups used by IT teams to provide users access to a variety of services and applications. These servers in turn are accessed by a range of users who use it for a variety of purposes

For example, a web server might contain a log of page requests that users might have made. Apart from that there might be multiple requests including access logs and error logs. You might even want to analyse the Syslog from linux servers.

All of these server logs can be ingested by AIOps. We will now take examples of Linux Syslog and Windows Event Logs to see how we can send the server logs to AIOps.

Example of Sending Linux Server logs to AIOps​

Let us take an example to understand how to send Syslog from Linux Server to Motadata AIOps.

  1. Log in to the Linux Server for which you want to send the Syslog to AIOps.

  2. Open the rsyslog.conf file which is typically located in '/etc'.

  3. Look for the following text in the file:

    $IncludeConfig /etc/rsyslog.d/*.conf

    This text is typically located at the end of the file.

  4. Now we need to provide the AIOps server details to send the Syslog to the AIOps server. In order to do that we need to enter the Motadata server IP below the text located in the step above so now the text looks as follows:


    $IncludeConfig /etc/rsyslog.d/*.conf
    *.* @ServerIP:PortNumber

    where 'ServerIP' is the IP address of the Motadata AIOps server and 'PortNumber' is the log forwarding port number on the linux server.

    note

    Write @ServerIP to send UDP logs.

    note

    Write @@ServerIP to send TCP logs.

  5. Now, restart the rsyslog service to start sending the Syslog to Motadata AIOps.

Example of Sending Windows Server logs to AIOps​

Now, let us see how we can send the Windows event logs to AIOps.

  1. Install agent on the Windows server.

  2. Go to Settings. Select Monitoring Settings. After that, select Agent Monitor Settings.

  3. Select View Details against the agent for which you want to ingest the logs. Click on the Log tab to start the log configuration.

Enter the log configuration details for Windows event logs as follows:

FieldDescription
NameSpecify the type of Windows event log that you want to send to AIOps i.e., Application, Security, or System.
LevelsSpecify the event level of the Windows event log from the dropdown that you want to send to AIOps i.e., Trace, Critical, Error, Warning, Informational, or Verbose.
EventsSpecify the Event Id of the Windows event log that you want to send to AIOps.
note

In case you do not specify any specific event level, then logs of ALL the event levels from the type of log selected will be sent to AIOps. Similarly, if you do not specify any Event ID, then ALL the logs from the selected log type and the selected event level will be sent to AIOps. For Example, if you specify the Name as Application and Levels as Critical, then ALL the critical application logs will be sent to AIOps.

We just looked into how you can ingest Windows event logs into Motadata AIOps. Now, let us look into how you can send network logs to AIOps.

Network Logs​

Just like other types of logs, network logs are a valuable source of information for maintaining the performance and security of your IT infrastructure. In Motadata AIOps, ingesting network logs follows a familiar process as sending syslog data.

By utilizing the same method employed for sending syslog, you can seamlessly incorporate network logs into Motadata AIOps for comprehensive monitoring and analysis. This approach ensures consistency and ease of use when dealing with different types of log sources.

Let's explore how to ingest network logs into Motadata AIOps using the established method.