Observability Pipeline Overview in ObserveOps
The Observability Pipeline in ObserveOps is the central processing engine for all incoming logs. Every log that enters the system passes through the pipeline before it reaches storage. The pipeline lets you control what gets parsed, what gets enriched, what gets dropped, and how long each log is kept.
What the Observability Pipeline Does
The pipeline sits between log ingestion and log storage. It processes every log automatically. You configure it once, and it handles every log that matches your rules from that point forward.
The pipeline has six stages, each with its own configuration section:
| Stage | What It Does |
|---|---|
| Ingest Logs | Receives logs from all connected sources like servers, applications, databases, network devices, cloud services, and security tools See Log Ingestion for setup. |
| Live Trail | Streams every incoming log in real time so you can verify sources. |
| Configure Log Pipelines | Defines which logs get processed and how as through parsers and processors you configure. |
| Generate Metrics | Turns log fields into queryable time-series metrics. |
| Exclude Unwanted Logs | Drops noisy logs before storage while still counting them in metrics. |
| Configure Indexes | Routes logs to retention buckets with configurable retention periods. |
| Log Forwarding | Forward Logs is also available from the pipeline to send a copy of ingested logs to external destinations over TCP or UDP. |
How to Access the Observability Pipeline
Go to Settings > Observability Pipeline > Log Pipeline.
The screen shows all pipeline sections as collapsible panels. Expand any section to view or configure it.
How a Log Flows Through the Pipeline
- A log arrives from any event source.
- ObserveOps checks the log against every active pipeline in order. The log enters each pipeline whose filters match.
- Inside the pipeline, the first matching parser extracts structured fields from the raw log text.
- As soon as a specific parser parses a log, the corresponding pipeline is selected for processing that log. The log does not continue further to check any other pipeline.
- Each processor runs in order: enriching, transforming, or removing fields.
- ObserveOps checks the processed log against exclusion rules. If it matches, the log is dropped before storage. It still counts in metrics.
- The log lands in the first index whose inclusion criteria match and is kept for that index's retention period.