Motadata Log Pipeline in ObserveOps
The Motadata Log Pipeline is the processing backbone of log management in ObserveOps . Every log that enters the system passes through this pipeline before it reaches storage. The pipeline cleans raw log text, extracts structured fields, enriches context, and routes logs to the right retention bucket, automatically.
Why a Pipeline Matters
Raw logs arrive in dozens of formats. A Linux syslog looks nothing like an IIS access log or a Cisco network event. Without a pipeline, you store raw text and search through noise. With the pipeline, every log exits as structured data. You search fields, not strings. You find answers in seconds.
The pipeline also runs before storage. That means you can drop logs you don't need, generate metrics from log data, and control exactly what gets stored and for how long, all without modifying the original sources.
Pipeline Architecture
The Motadata Log Pipeline operates as a sequential set of stages. Every log follows the same path.
Log Source
↓
Ingestion (Agent / Syslog / Cloud / TCP-UDP / Events)
↓
Pipeline Matching (from which pipeline will the log flow?)
↓
Parsing (extract structured fields from raw text)
↓
Processing (enrich, transform, remap, or remove fields)
↓
Exclusion Check (drop matching logs before storage)
↓
Index Matching & Storage(which retention bucket?)
↓
Log Forwarding (forwards logs to configured custom destinations)
Stage Details
Ingestion
Logs enter ObserveOps from any source: MotaAgent installed on servers, syslog over the network, cloud provider APIs, TCP/UDP listeners, or event-based integrations. Each source connects through a log ingestion profile.
See Log Ingestion for how to connect sources.
Pipeline Matching
ObserveOps checks the incoming log against every active pipeline in order. A pipeline has two filters: a log type filter and optional pre-filters. When a log matches a pipeline's filters, it enters that pipeline for processing.
Multiple pipelines can match the same log. Logs check all pipelines from top to bottom. Once a parser in any pipeline successfully parses the log, log stops flowing through other pipeline.
Parsing
Inside a pipeline, parsers run in order. Each parser tries to match the raw log text. The first parser that matches extracts structured fields from the text.
ObserveOps includes four parser types:
| Parser Type | How It Works |
|---|---|
| Regex Parser | Extracts fields by auto constructing regexes for selected fields. |
| Delimiter Parser | Splits log text into fields using a character delimiter. |
| JSON Parser | Parses selected fields from JSON logs. |
| Custom Plugin | Applies a pre-built custom plugin for a known log format. |
Processing
After parsing, each processor in the pipeline runs in sequence. Processors can enrich fields, transform values, remove noise, or build new fields from existing ones.
| Processor Type | What It Does |
|---|---|
| Remapper | Renames or copies a field/ tag to a new field/ tag. |
| URL Enricher | Parses a URL field. |
| User-Agent Enricher | Parses a user agent string. |
| Geo IP Enricher | Looks up an IP address and adds geo-locations. |
| Tag Remapper | Categorizes logs using filters. |
| Field Remover | Deletes a field from the log. |
| Arithmetic Processor | Calculates a new numeric field from existing numeric fields. |
| String Builder | Builds a new string field by combining existing field values. |
Exclusion Check
After processing, ObserveOps checks the log against all active exclusion rules. If the log matches any rule, it is dropped before storage. The log still counts in metrics. This lets you reduce storage costs without losing visibility into trends derived from logs.
See Exclude Unwanted Logs for how to create exclusion rules.
Index Matching
Every log that passes the exclusion check is assigned to an index. ObserveOps checks active indexes in order from top to bottom. The log lands in the first index whose inclusion criteria match. The index retention period controls how long the log is stored. If the log does not match any inclusion criteria in the top three indexes then the log enters in the default fourth index by default.
The fourth index cannot be disabled. You can modify the retention days for any index, but apply changes carefully, as retention updates are applied retrospectively to logs that were already stored in the index before the change.
See Configure Indexes for how to set up custom retention rules.
Where the Pipeline Lives
All pipeline configuration is in one place:
Settings > Observability Pipeline > Log Pipeline
From there you configure the full pipeline: Live Trail, active pipelines, parsers, processors, Generate Metrics rules, Exclude Unwanted Logs rules, Configure Indexes, and Forward Logs destinations. Each is an expandable section within the same screen.
What the Pipeline Gives You
| Outcome | How the Pipeline Produces It |
|---|---|
| Structured, searchable logs | Parsers extract named fields from raw text. |
| Enriched context | Processors remap, transform and add GeoIP, URL, and user agent details. |
| Reduced storage costs | Exclusion rules drop logs you don't need. |
| Metrics from log data | Generate Metrics turns log fields into queryable time-series data. |
| Controlled retention | Indexes route logs to the right retention bucket. |
| External forwarding | Forward Logs sends a copy to external SIEM or syslog destinations. |