Application Performance Monitoring (APM) Policy

Overview

Once you have configured an APM, it can be used to create policies that leverage application performance indicators to generate actionable alerts and notify users through default channels. Creating an APM policy ensures a proactive monitoring mechanism, enabling efficient application performance monitoring and rapid troubleshooting, which further minimizes potential service disruptions.

Navigation

Go to Menu and select Settings . After that, Go to Policy Settings . Then, select APM Policy.

APM Policy Screen

List of created APM policies will be displayed on this screen. Below is a gist of options visible on the screen:

Field	Description
Policy Name	Name of the configured policy.
Used Count	Display the number of Monitor(s) and Trace Service(s) associated with the policy.
APM Policy Type	Displays the type of configured policy.
Tag	Displays the configured tag details for the policy.
Severity	Displays the current severity level of the APM policy.
Threshold Value	Display the configured policy threshold values.
Metric	Displays the current metric name associated with the policy.
Status	Use the toggle button to enable or disable the specific policy.
Actions	Click on the Ellipsis icon to access the below actions for APM policy: - Edit: Allows you to edit parameters of the policy. - Delete: You can delete the policy using the delete option.

Create APM Policy

You can create an APM policy by clicking on the Create Policy button. Below-mentioned parameters will be displayed on the next screen to configure the policy.

Field	Description
Policy Name	Provide a descriptive name for policy.
Tag	Provide a Tag for the policy.

Since configurations for different APM Policy Types differ, let's look at them separately:

Trace Metrics

If you select Trace Metrics as APM Policy type, it can be further divided as:

|Field|Description| |Threshold Alert|Triggers when the selected metric crosses a fixed value you define.| |Baseline Alert|Alerts when current behavior deviates from its usual historical pattern.| |Anomaly|Detects unexpected behavior automatically without requiring strict thresholds.|Detects unexpected behavior automatically without requiring strict thresholds.| |Forecast|Predicts future breaches based on trends and alerts before limits are hit.|

Threshold Alert

Use-Case

Suppose you are running a payment processing microservice in your application environment and you want to ensure its response time stays within acceptable limits. You need Motadata ObserveOps to raise an alert whenever the average response time for that service exceeds a fixed threshold for example, 2 seconds — indicating potential performance degradation. You can create a Trace Metrics Threshold Alert APM policy to do the same.

Set Conditions

Field	Description
APM Policy Type	Select Trace Metrics as APM Policy Type and further select the Threshold Alert card.
Counter	Use the dropdown to select a counter.
Source Filter	Use the dropdown to either select a service or a Monitor.
Source	List of services or monitors will be available based on the Source Filter selection.
Critical/Major/Warning	Provide a numerical threshold value for each severity level respectively. You can also change the operator suitable to your needs using the dropdown.
Notify if threshold value breach within	Use the dropdown to select the duration for repeat notification.
Abnormality occurrence	Select the number of occurrences you wish to be notified for when threshold value has been breached.
Auto clear	Select the time duration to auto clear the abnormality alert. This does not clear the base alert.

Assumption Based Scenarios

To understand how Abnormality occurrence and Notify if threshold value breach within work together, consider the following assumptions:

• The polling period is 1 minute.
• Notify if threshold value breach within is set to 5 minutes.
• Abnormality occurrence is set to 3. ObserveOps checks for 3 consecutive threshold breaches within the 5-minute window before firing an alert. The window resets every time the metric returns within the threshold.

The threshold values set for this example are:

• Warning: Response time exceeds 2000ms
• Major: Response time exceeds 3500ms
• Critical: Response time exceeds 5000ms

Scenario 1 — No Alert Triggered

Response time breaches the threshold at 10:00 and 10:01, but returns within the normal range at 10:02. No alert fires because 3 consecutive breaches did not occur within the 5-minute window.

Scenario 2 — Alert Triggered

Response time exceeds the threshold at 10:00, 10:01, and 10:02 — three consecutive breaches within 5 minutes. ObserveOps fires an alert at 10:02. The severity depends on which threshold was crossed:

• If response time crossed 2000ms each time: Warning alert fires.
• If response time crossed 3500ms each time: Major alert fires.
• If response time crossed 5000ms each time: Critical alert fires.

Scenario 3 — Alert Triggered After Window Reset

Response time exceeds the threshold at 10:00 and 10:01, then returns to normal at 10:02. The 5-minute evaluation window resets. Response time then breaches the threshold again at 10:03, 10:04, and 10:05 — three consecutive times. ObserveOps fires an alert at 10:05.

note

If the metric crosses multiple severity thresholds at the same time, ObserveOps raises the alert at the highest applicable severity. For example, a response time of 6000ms qualifies for both Major and Critical. ObserveOps raises the alert as Critical because it is the highest qualified severity.

Set Alert Message

Field	Description
Subject	Modify the subject-line for the alert message.
Message	Modify the alert message from here.

info

For a list of default supported Macros to modify the default set message, Please visit: Alert Macros

Notification

Field	Description
Notify	There are multiple ways you can populate this field: Username of registered user in MotadataObserveOps (formerly known as AIOps) (The alert notification will be sent over email and SMS to the mapped email address and phone number respectively). - Any email address (In case the recipient whom you wish to notify is not a registered user, you can enter an email address). Mention Teams Handle, if you have integrated Microsoft Teams with Motadata ObserveOps you can simple mention the Teams Handle provided at the time of the configuring the integration. It will send a notification to the teams channel when the policy is breached. - Use @ to mention a registered username, email address, or Microsoft Teams handle. - Use # to select a user profile for alert notification. - Use ! to select a Syslog Forwarder or SNMP Trap Forwarder integration profile and forward the alert to the configured external destination.
If severity is	Select the severity level using individual checkboxes in the dropdown.You can select multiple, all, or a single option as per your requirement. You can also have different recipients notified at different severity levels. For instance, you can notify johndoe@motadata.com when severity level hits Critical and send an alert notification to janedoe@motadata.com when severity level is Major.
Play Sound	Activate this toggle to enable sound notifications when an alert is triggered.
If Severity is	Choose the severity level at which the sound notification should be triggered. This option becomes visible only when the Play Sound toggle is switched ON.
Renotification	Turning on the toggle will resend the alert at a specific interval defined by you. If turned off, Motadata ObserveOps will not renotify about the alert.
Renotify	Similar to Notification field, enter the username or email address of the recipient. Also choose a preset duration for renotification along with the severity level at which they system will renotify you if the alert severity is not changed.
Do not renotify if acknowledged	If the toggle is turned on, Motadata ObserveOps will not send a renotification to the recipient if they mark the alert as acknowledged.

Declare Incident

Field	Description
Select Integration Profile to Trigger	Choose an Integration profile to be executed from the dropdown when the chosen alert severity is triggered.
When Alert Severity is	Select an Alert severity from the dropdown.
Create Integration Profile	Select this button to start creating a new Integration profile that you might want to assign to the policy you are creating.

You can click on to define a new Alert severity and Integration Profile combination. You can have separate Integration profile triggered at different levels of Alert severity.

Baseline Alert

The Baseline Alert for Trace Metrics is a powerful feature within Motadata ObserveOps that enables proactive monitoring of application performance by comparing real-time trace metric values to a dynamically generated baseline. By analyzing historical data and establishing a baseline range, this alerting mechanism helps identify deviations in service behavior, allowing organizations to detect potential performance issues and take preventive action.

With the Baseline Alert, organizations gain deeper insights into the normal behavior of their trace metrics by leveraging historical data from the past 15 days. By dynamically adjusting the baseline range using advanced statistical techniques, the alerting mechanism adapts to changes in service behavior, ensuring accurate detection of deviations in real-time.

When a trace metric value breaches the dynamic baseline threshold during a policy evaluation, the Baseline Alert triggers a policy violation. Organizations can customize the actions taken when a violation occurs, such as sending notifications to stakeholders, to ensure corrective action.

Use-Case

The Baseline Alert is particularly useful in scenarios where it is crucial to maintain optimal application performance and prevent service degradation. Suppose you are monitoring an order management service and its average response time normally stays around 300ms based on historical patterns. Rather than setting a rigid threshold, you want Motadata ObserveOps to alert you whenever the response time deviates significantly from this established baseline — for example, if it suddenly climbs 50% above the expected range. By continuously evaluating trace metric values against the established baseline, this approach provides early warning signs of performance bottlenecks, abnormal latency spikes, or unexpected variations in key service metrics. This empowers your teams to address potential issues before they escalate, minimizing downtime and enhancing overall operational efficiency.

Baseline APM Policy Mechanism

• Policy Evaluation

The Baseline policy evaluations start as soon as the policy is created. During each evaluation, the system considers the trace metric data from the last 15 days to create a baseline range.

• Baseline Calculation

To generate the baseline range, the system utilizes advanced statistical methods. These techniques analyze the historical data points within the last 15 days and produce a dynamically adjusted baseline.

• Baseline Threshold Violation

A baseline threshold violation occurs when a trace metric's data point deviates from the expected behavior, surpassing the predefined acceptable range known as the baseline threshold. This violation triggers an alert based on the configured parameters within the baseline policy.

In the baseline policy configuration, users set the criteria for triggering an alert by specifying the number of times a data point can deviate from the baseline threshold within a defined time window, as configured in the Abnormality Occurrence and Notify if the Threshold value breach within fields, respectively.

• Actions on Baseline Policy Triggering

Once the baseline alert is triggered, you can define specific actions to be taken, such as sending notifications to relevant stakeholders.

Set Conditions

Field	Description
APM Policy Type	Select Trace Metrics as APM Policy Type and further select the Baseline Alert card.
Counter	Use the dropdown to select a counter.
Source Filter	Use the dropdown to either select a service or a Monitor.
Source	List of services or monitors will be available based on the Source Filter selection.
Absolute/Relative	- Select Absolute to define specific numerical values. When selecting this option, users specify the exact values that metrics should deviate from the baseline to trigger an alert. - Select Relative to set thresholds based on percentages. This option allows users to define deviations from the baseline as a percentage, triggering alerts when metrics deviate by the specified percentage.
Critical/Major/Warning	Provide a numerical threshold value for each severity level respectively. You can also change the operator suitable to your needs using the dropdown.
Notify if threshold value breach within	Use the dropdown to select the duration for repeat notification.
Abnormality occurrence	Select the number of occurrences you wish to be notified for when threshold value has been breached.
Auto clear	Select the time duration to auto clear the abnormality alert. This does not clear the base alert.

Set Alert Message

Field	Description
Subject	Modify the subject-line for the alert message.
Message	Modify the alert message from here.

info

For a list of default supported Macros to modify the default set message, Please visit: Alert Macros

Notification

Field	Description
Notify	There are multiple ways you can populate this field: - Username of registered user in MotadataObserveOps (formerly known as AIOps) (The alert notification will be sent over email and SMS to the mapped email address and phone number respectively). - Any email address (In case the recipient whom you wish to notify is not a registered user, you can enter an email address). Mention Teams Handle, if you have integrated Microsoft Teams with Motadata ObserveOps you can simple mention the Teams Handle provided at the time of the configuring the integration. It will send a notification to the teams channel when the policy is breached. - Use @ to mention a registered username, email address, or Microsoft Teams handle. - Use # to select a user profile for alert notification. - Use ! to select a Syslog Forwarder or SNMP Trap Forwarder integration profile and forward the alert to the configured external destination.
If severity is	Select the severity level using individual checkboxes in the dropdown.You can select multiple, all, or a single option as per your requirement. You can also have different recipients notified at different severity levels. For instance, you can notify johndoe@motadata.com when severity level hits Critical and send an alert notification to janedoe@motadata.com when severity level is Major.
Play Sound	Activate this toggle to enable sound notifications when an alert is triggered.
If Severity is	Choose the severity level at which the sound notification should be triggered. This option becomes visible only when the Play Sound toggle is switched ON.
Renotification	Turning on the toggle will resend the alert at a specific interval defined by you. If turned off, Motadata ObserveOps will not renotify about the alert.
Renotify	Similar to Notification field, enter the username or email address of the recipient. Also choose a preset duration for renotification along with the severity level at which they system will renotify you if the alert severity is not changed.
Do not renotify if acknowledged	If the toggle is turned on, Motadata ObserveOps will not send a renotification to the recipient if they mark the alert as acknowledged.

Declare Incident

Field	Description
Select Integration Profile to Trigger	Choose an Integration profile to be executed from the dropdown when the chosen alert severity is triggered.
When Alert Severity is	Select an Alert severity from the dropdown.
Create Integration Profile	Select this button to start creating a new Integration profile that you might want to assign to the policy you are creating.

You can click on to define a new Alert severity and Integration Profile combination. You can have separate Integration profile triggered at different levels of Alert severity.

Trace Analytics

Set Conditions

Field	Description
APM Policy Type	Select Trace Analytics as APM Policy Type.
Counter	Use the dropdown to select a counter.
Aggregation	Use the dropdown to apply an aggregation type for the selected counter.
Operator	Use the dropdown menu to select an operator best suitable for your requirements.
Value	Provide a numerical threshold value for the severity level.
Source Filter	Use the dropdown to either select a Service or Source Host.
Source	List of Source Host(s) or Service(s) will be available based on the Source Filter selection.
Result By	Define what constitutes an alert.
Filters	You can configure pre-filters to further filter the condition.
Severity	Select a severity level to associate with the configured threshold value. You can only select single severity level.
Notify if Threshold value breach within	Use the dropdown to select the duration for repeat notification if the threshold value is breached again within stipulated time.

Set Alert Message

Field	Description
Subject	Modify the subject-line for the alert message.
Message	Modify the alert message from here.

info

For a list of default supported Macros to modify the default set message, Please visit: Alert Macros

Notification

Field	Description
Notify	There are multiple ways you can populate this field: - Username of registered user in MotadataObserveOps (The alert notification will be sent over email and SMS to the mapped email address and phone number respectively). - Any email address (In case the recipient whom you wish to notify is not a registered user, you can enter an email address). - Mention Teams Handle, if you have integrated Microsoft Teams with Motadata ObserveOps you can simple mention the Teams Handle provided at the time of the configuring the integration. It will send a notification to the teams channel when the policy is breached. - Use @ to mention a registered username, email address, or Microsoft Teams handle. - Use # to select a user profile for alert notification. - Use ! to select a Syslog Forwarder or SNMP Trap Forwarder integration profile and forward the alert to the configured external destination.
Play Sound	Activate this toggle to enable sound notifications when an alert is triggered.

Select the Create Policy button to create the policy based on the details entered.

Select the Reset button to erase all the current field values, if required.