Motadata Log Pipeline in ObserveOps
The Motadata Log Pipeline is the processing backbone of log management in ObserveOps (formerly known as AIOps). Every log that enters the system passes through this pipeline before it reaches storage. The pipeline cleans raw log text, extracts structured fields, enriches context, and routes logs to the right retention bucket — automatically.
Why a Pipeline Matters
Raw logs arrive in dozens of formats. A Linux syslog looks nothing like an IIS access log or a Cisco network event. Without a pipeline, you store raw text and search through noise. With the pipeline, every log exits as structured data. You search fields, not strings. You find answers in seconds.
The pipeline also runs before storage. That means you can drop logs you don't need, generate metrics from log data, and control exactly what gets stored and for how long — all without modifying the original sources.
Pipeline Architecture
The Motadata Log Pipeline operates as a sequential set of stages. Every log follows the same path.
Log Source
↓
Ingestion (Agent / Syslog / Cloud / TCP-UDP / Events)
↓
Pipeline Matching (which pipeline applies to this log?)
↓
Parsing (extract structured fields from raw text)
↓
Processing (enrich, transform, remap, or remove fields)
↓
Exclusion Check (drop matching logs before storage)
↓
Index Matching (which retention bucket?)
↓
Storage (structured log stored for the retention period)
Stage Details
Ingestion
Logs enter ObserveOps from any source: MotaAgent installed on servers, syslog over the network, cloud provider APIs, TCP/UDP listeners, or event-based integrations. Each source connects through a log ingestion profile.
See Log Ingestion for how to connect sources.
Pipeline Matching
ObserveOps checks the incoming log against every active pipeline in order. A pipeline has two filters: a log type filter and optional prefilters. When a log matches a pipeline's filters, it enters that pipeline for processing.
Multiple pipelines can match the same log. Logs check all pipelines from top to bottom. Once a parser in any pipeline successfully parses the log, processing stops.
Parsing
Inside a pipeline, parsers run in order. Each parser tries to match the raw log text. The first parser that matches extracts structured fields from the text. Parsing turns a raw string like "May 01 10:23:14 server01 sshd: Failed password for root" into fields like timestamp, host, process, and message.
ObserveOps includes four parser types:
| Parser Type | How It Works |
|---|---|
| Regex Parser | Extracts fields using named capture groups in a regular expression |
| Delimiter Parser | Splits log text into fields using a character delimiter |
| Remapper | Renames or copies fields from one name to another |
| Custom Plugin | Applies a pre-built plugin for a known log format |
Processing
After parsing, each processor in the pipeline runs in sequence. Processors can enrich fields, transform values, remove noise, or build new fields from existing ones.
| Processor Type | What It Does |
|---|---|
| Remapper | Renames or copies a field |
| URL Enricher | Parses a URL field into protocol, host, path, and query components |
| User-Agent Enricher | Parses a user agent string into browser, OS, and device fields |
| Geo IP Enricher | Looks up an IP address and adds country, city, and coordinates |
| Tag Remapper | Moves a field value into the log tags |
| Field Remover | Deletes a field from the log before storage |
| Arithmetic Processor | Calculates a new numeric field from existing numeric fields |
| String Builder | Builds a new string field by combining existing field values |
Exclusion Check
After processing, ObserveOps checks the log against all active exclusion rules. If the log matches any rule, it is dropped before storage. The log still counts in metrics. This lets you reduce storage costs without losing visibility into log volumes.
See Exclude Unwanted Logs for how to create exclusion rules.
Index Matching
Every log that passes the exclusion check is assigned to an index. ObserveOps checks active indexes in order from top to bottom. The log lands in the first index whose inclusion criteria match. The index retention period controls how long the log is stored.
ObserveOps ships with four pre-configured indexes: 5-day, 7-day, 15-day, and 30-day retention. The 30-day index captures everything that no other index claims.
See Configure Indexes for how to set up custom retention rules.
Where the Pipeline Lives
All pipeline configuration is in one place:
Settings > Observability Pipeline > Log Pipeline
From there you configure the full pipeline: Live Trail, active pipelines, parsers, processors, Generate Metrics rules, Exclude Unwanted Logs rules, Configure Indexes, and Forward Logs destinations. Each is an expandable section within the same screen.
What the Pipeline Gives You
| Outcome | How the Pipeline Produces It |
|---|---|
| Structured, searchable logs | Parsers extract named fields from raw text |
| Enriched context | Processors add GeoIP, URL, and user agent details |
| Reduced storage costs | Exclusion rules drop logs you don't need |
| Metrics from log data | Generate Metrics turns log fields into queryable time-series data |
| Controlled retention | Indexes route logs to the right retention bucket |
| External forwarding | Forward Logs sends a copy to external SIEM or syslog destinations |