Skip to main content

Create a Log Parser in ObserveOps

A parser extracts structured fields from raw log text. You add parsers inside a pipeline. When a log enters the pipeline, ObserveOps tries each parser in order. The first parser that matches the log extracts its fields. Unmatched logs move to the next parser in the pipeline.

Prerequisites​

Add a Parser to a Pipeline​

Open a pipeline from Settings > Observability Pipeline > Log Pipeline. Click + Add Parser inside the pipeline.

The Create Parser screen opens with three tabs.

Tab 1 — Indexed Logs​

This tab loads the logs that match this pipeline's filter conditions and have arrived in the last 15 minutes. Columns shown: TIMESTAMP, MESSAGE, SEVERITY, LOG TYPE, SOURCE.

Create Parser screen showing the Indexed Logs tab with qualifying logs from the last 15 minutes

Show Unparsed Logs: Click this button to filter the stream to only logs that have not been parsed by any parser yet. Use this to focus on logs your current parser needs to handle.

Click any log to open a popup showing its parsed fields, the same detail view as Log Explorer.

Click any log also fills it in as the sample log in the test panel below, in the exact raw format the log arrived. No changes are made to the raw text.

Tab 2 — Live Trail​

This tab streams only the logs that qualify this pipeline's filter conditions in real time.

Create Parser screen showing the Live Trail tab with a real-time log stream for qualifying logs

Clicking any log from this tab also fills it as the sample log below, in its exact raw format.

Tab 3 — Upload Log File​

Upload a sample log file to test your parser against. ObserveOps runs the parser against the file and shows the results.

Create Parser screen showing the Upload Log File tab with test output and field distribution

Clicking any log from the upload output also fills it as the sample log below in its exact raw format.

Parser Types​

ObserveOps supports four parser types. Select the type when you create a parser.

Log parser types showing Regex Parser, Delimiter Parser, Remapper, and Custom Plugin options

Regex Parser​

The Regex parser extracts fields using a regular expression. You write the pattern and name each capture group. ObserveOps extracts each named group as a field.

How to configure:

  1. Select Regex as the parser type.
  2. Write the regex pattern in the Pattern field.
  3. Name each capture group in your regex. The group name becomes the field name in the parsed log.
  4. Paste or select a sample log using the tabs above, then click Test.

Test results:

When you run a test on a sample log or uploaded file, ObserveOps shows:

  • The extracted fields and their values
  • The field distribution alongside each field, showing how often that field appears across the sample set

This field distribution view works the same way it did in the previous version of the product.

Delimiter Parser​

The Delimiter parser splits log lines at a specific character or string. You pick which positions to extract and name them.

How to configure:

  1. Select Delimiter as the parser type.
  2. Enter the delimiter character or string (for example, , for CSV or | for pipe-separated logs).
  3. After setting the delimiter, ObserveOps shows a checkbox next to each detected field position.
  4. Select the checkboxes for the fields you want to extract. Only checked positions are parsed.
  5. Enter a name for each selected field.
  6. Click Test to verify the output.
note

You must check the fields you want first. You cannot name a field until you select its checkbox. Only checked fields appear in the parsed output.

Remapper Parser​

The Remapper parser copies the value of a source field into a target field. Use it when your logs arrive with field names that differ from the naming convention you want in storage.

How to configure:

  1. Select Remapper as the parser type.
  2. Set the Source field whose value you want to copy.
  3. Set the Target field to write the value into.
  4. Choose whether to keep the source field or remove it after copying.
  5. Paste or select a sample log and click Test to verify the output.

Custom Plugin Parser​

The Custom Plugin parser lets you select a pre-built plugin from the ObserveOps plugin library to handle log formats with specialized parsing requirements.

How to configure:

  1. Select Custom Plugin as the parser type.
  2. Choose the plugin from the available plugin list.
  3. Configure the plugin parameters as required.
  4. Click Test to verify the output against your sample log.

The Custom Plugin parser works the same way it did in the previous version.

Parser Ordering​

Inside a pipeline, parsers run in order from top to bottom. ObserveOps tries each parser against the incoming log until one matches. Once a parser matches, parsing is complete. The log does not move to any further parsers.

Parsers always appear above processors in the pipeline view. You can drag parsers to reorder them among themselves, but you cannot move a parser below a processor.

Troubleshooting​

No logs are appearing in the Indexed Logs tab

Cause: No logs matching this pipeline's filters arrived in the last 15 minutes.

Fix: Confirm logs are flowing in using Live Trail. Check that the pipeline filter is correct and logs are being ingested.

The parser test runs but no fields are extracted

Cause: The pattern or configuration does not match the sample log format.

Fix: For Regex, check the pattern against the raw log text. For Delimiter, verify the delimiter character is correct and present in the log. For JSON, confirm the log is valid JSON.

Clicking a log is not filling in the sample log

Cause: The log was cut off or the UI session needs a refresh.

Fix: Reload the page and open the parser again. If the problem persists, use the Upload Sample File tab to provide the log manually.