Amazon SageMaker Endpoint Monitoring in ObserveOps

Overview

The Amazon SageMaker Endpoint integration with ObserveOps (formerly known as AIOps) collects operational telemetry from Amazon SageMaker inference endpoints. It monitors endpoint state, production variant configuration, per-variant resource utilization and latency, invocation rates, error counts, and endpoint-level performance including overhead latency and model cache hit rates.

These metrics help ML platform teams detect inference errors, track model latency, optimize instance scaling, and ensure endpoint availability for real-time prediction workloads.

Prerequisites

• Amazon SageMaker is enabled in the target AWS account and region with at least one inference endpoint deployed.
• The IAM role or user used for integration has read access to SageMaker resources (sagemaker:DescribeEndpoint, sagemaker:ListEndpoints, cloudwatch:GetMetricData).
• Required SageMaker and CloudWatch API endpoints are reachable from ObserveOps.
• The AWS account is added in discovery with correct credentials and region configuration.

List of Supported KPIs

Endpoint Configuration

Metric	Description	Type
aws.sagemaker.endpoint.state	Current state of the SageMaker endpoint.	String
aws.sagemaker.endpoint.failure.reason	Reason for the most recent endpoint creation or update failure.	String
aws.sagemaker.variant	Name identifier of a production variant on the endpoint.	String
aws.sagemaker.variant.instances	Number of instances running the production variant.	Count
aws.sagemaker.variant.serverless.concurrency	Number of concurrent invocations for a serverless inference variant.	Count

Variant Performance

Metric	Description	Type
aws.sagemaker.variant.cpu.percent	CPU utilization of the variant instances as a percentage.	Percent
aws.sagemaker.variant.gpu.percent	GPU utilization of the variant instances as a percentage.	Percent
aws.sagemaker.variant.memory.percent	Memory utilization of the variant instances as a percentage.	Percent
aws.sagemaker.variant.gpu.memory.percent	GPU memory utilization of the variant instances as a percentage.	Percent
aws.sagemaker.variant.disk.percent	Disk utilization of the variant instances as a percentage.	Percent
aws.sagemaker.variant.model.latency.ms	Average model inference latency for the variant in milliseconds.	Milliseconds
aws.sagemaker.variant.invocations.per.sec	Number of invocation requests received by the variant per second.	Count
aws.sagemaker.variant.invocation.4xx.errors	Number of variant invocations that returned 4xx client errors.	Count
aws.sagemaker.variant.invocation.5xx.errors	Number of variant invocations that returned 5xx server errors.	Count
aws.sagemaker.variant.invocation.model.errors	Number of variant invocations that returned model errors.	Count

Endpoint Performance

Metric	Description	Type
aws.sagemaker.endpoint.invocations.per.sec	Total invocation requests received by the endpoint per second.	Count
aws.sagemaker.endpoint.model.latency.ms	Average model inference latency across all variants in milliseconds.	Milliseconds
aws.sagemaker.endpoint.overhead.latency.ms	Overhead latency added by SageMaker infrastructure in milliseconds.	Milliseconds
aws.sagemaker.endpoint.invocation.4xx.errors	Number of endpoint invocations that returned 4xx client errors.	Count
aws.sagemaker.endpoint.invocation.5xx.errors	Number of endpoint invocations that returned 5xx server errors.	Count
aws.sagemaker.endpoint.invocation.model.errors	Number of endpoint invocations that returned model errors.	Count
aws.sagemaker.endpoint.invocations.per.instance	Average number of invocations per instance across all variants.	Count
aws.sagemaker.endpoint.model.cache.hit.percent	Percentage of invocations served from the model cache.	Percent
aws.sagemaker.endpoint.model.loading.ms	Average time to load a model from storage in milliseconds.	Milliseconds