Observability Pipeline Overview in ObserveOps
The Observability Pipeline in ObserveOps is the central processing engine for all incoming logs. Every log that enters the system passes through the pipeline before it reaches storage. The pipeline lets you control what gets parsed, what gets enriched, what gets dropped, and how long each log is kept.
What the Observability Pipeline Does
The pipeline sits between log ingestion and log storage. It processes every log automatically. You configure it once, and it handles every log that matches your rules from that point forward.
The pipeline has six stages, each with its own configuration section:
| Stage | What It Does |
|---|---|
| Ingest Logs | Receives logs from all connected sources — agents, syslog, cloud APIs, TCP/UDP, and events. See Log Ingestion for setup. |
| Live Trail | Streams every incoming log in real time so you can verify sources and test parsers |
| Configure Log Pipelines | Defines which logs get processed and how — through parsers and processors you configure |
| Generate Metrics | Turns log fields into queryable time-series metrics without additional agents |
| Exclude Unwanted Logs | Drops noisy logs before storage while still counting them in volume metrics |
| Configure Indexes | Routes logs to retention buckets with configurable retention periods |
Forward Logs is also available from the pipeline to send a copy of ingested logs to external destinations over Syslog-TCP or Syslog-UDP.
How to Access the Observability Pipeline
Go to Settings > Observability Pipeline > Log Pipeline.
The screen shows all pipeline sections as collapsible panels. Expand any section to view or configure it.
How a Log Flows Through the Pipeline
- A log arrives from any connected source.
- ObserveOps checks the log against every active pipeline in order. The log enters each pipeline whose filters match.
- Inside the pipeline, the first matching parser extracts structured fields from the raw log text.
- Each processor runs in order: enriching, transforming, or removing fields.
- ObserveOps checks the processed log against exclusion rules. If it matches, the log is dropped before storage. It still counts in metrics.
- The log lands in the first index whose inclusion criteria match and is kept for that index's retention period.
Where Each Feature Lives
All six stages live in one place: Settings > Observability Pipeline > Log Pipeline. Each is an expandable section on the same screen.
| Section | What You Configure There |
|---|---|
| Live Trail | Real-time log stream for verification |
| Configure Log Pipelines | Pipelines, parsers, and processors |
| Generate Metrics | Metric rules derived from log data |
| Exclude Unwanted Logs | Drop rules for noisy log sources |
| Configure Indexes | Retention indexes and inclusion criteria |
| Forward Logs | External syslog forwarding destinations |