Runbooks
A runbook is a ready-to-run automation that reads data from a monitor or changes state on it, without the operator logging in to the server. ObserveOps ships with inbuilt runbooks for the common operational tasks, and lets you build custom runbooks of your own.
Runbooks run on demand from the monitor's Runbook tab, on a schedule, or automatically as the action of an alert policy.
How Runbooks Work
A runbook is bound to one or more monitors. When it runs, ObserveOps resolves the monitor's credential profile, opens a session over the configured protocol (SSH, Telnet, PowerShell over WinRM, SNMP, Trace Route, Database, HTTP, or a custom script), executes the script, parses the output, and shows the result on the runbook result screen.
Runbooks fall into two broad classes:
- Check runbooks read state and return it. They do not change the server. A check that returns a "no" answer, for example "service is not running", still shows Succeeded because the runbook ran.
- Fix runbooks change state on the server (stop a service, kill a process, reboot the host, clear a cache). These include the inbuilt runbooks in the System Management, Troubleshooting, and Network Config categories.
Confirm the correct monitor is targeted before clicking Run. The runbook fires immediately against every assigned monitor.
Where You Can Use a Runbook
| Trigger | How it runs |
|---|---|
| On demand | Open the monitor, go to the Runbook tab, click Run on the assigned runbook. |
| Scheduled | Attach a schedule to the runbook from the row menu. The runbook fires at the next scheduled time against every monitor it is assigned to. |
| Alert policy | Pick a runbook as the policy's action. The runbook fires when the policy's severity threshold is breached. |
Runbook Types
ObserveOps supports eight runbook types. The type decides which protocol the runbook uses and which fields appear on the create form.
| Type | Use for |
|---|---|
| SSH Script | Shell commands on Linux, Solaris, AIX, or any SSH-reachable host. |
| Telnet | Legacy devices reachable only over Telnet. |
| Powershell | Windows hosts over WinRM. |
| SNMP | SNMP scalar or tabular walks against network devices. |
| Trace Route | ICMP or UDP hop-by-hop reachability. |
| Database | SQL against supported database engines through the credential profile. |
| HTTP | REST or web endpoint checks with headers, parameters, and body. |
| Custom | GO or shell scripts the ObserveOps agent runs on the monitor. |
Example
A policy raises a Critical alert when CPU utilization on a Windows monitor stays above 90% for 5 minutes. The policy's action runs the inbuilt List Top 5 CPU-Consuming Processes runbook on the affected monitor. The result screen shows the top 5 processes, and the NOC engineer sees the cause of the spike in seconds, without opening an RDP session.
Troubleshooting
| Issue | Cause | Fix |
|---|---|---|
| Runbook does not appear in the monitor's Runbook tab | The runbook is not assigned to this monitor. | Open the row menu on the runbook, click Assign Monitor, pick the monitor, and save. |
| Fix runbook fired on the wrong monitor | The runbook was assigned to multiple monitors and Run targets all of them. | Review the assigned monitors from the row menu's Remove Assigned Monitor before running, or trigger the runbook from the monitor's own Runbook tab. |
| Runbook result stays on "In Progress" | The target monitor is unreachable, or the credential profile is wrong. | Open the monitor, check status, test the credential, and retry. |