Amazon Data Firehose Monitoring in ObserveOps
Overview
The Amazon Data Firehose integration with ObserveOps (formerly known as AIOps) collects delivery stream configuration and throughput telemetry from Amazon Data Firehose, AWS's fully managed data delivery service. It monitors stream state, encryption settings, data ingestion rates, delivery success percentages, data freshness for each destination type, and stream rate limits.
These metrics help data engineering teams detect delivery failures, track data freshness, identify throttling, and verify that records reach their intended destinations at expected rates.
Prerequisites
- Amazon Data Firehose is enabled in the target AWS account and region with at least one delivery stream configured.
- The IAM role or user used for integration has read access to Firehose resources (
firehose:ListDeliveryStreams, firehose:DescribeDeliveryStream, cloudwatch:GetMetricData). - Required Firehose and CloudWatch API endpoints are reachable from ObserveOps.
- The AWS account is added in discovery with correct credentials and region configuration.
List of Supported KPIs
Stream Configuration
| Metric | Description | Type |
|---|
| aws.firehose.state | Current state of the Firehose delivery stream. | String |
| aws.firehose.type | Type of Firehose delivery stream. | String |
| aws.firehose.destination.type | Type of destination for the delivery stream. | String |
| aws.firehose.encryption.state | Indicates whether server-side encryption is enabled on the stream. | String |
| aws.firehose.failure.type | Type of the most recent delivery failure. | String |
| aws.firehose.create.time | Timestamp when the delivery stream was created. | Timestamp |
| aws.firehose.last.update.time | Timestamp of the most recent stream configuration update. | Timestamp |
Data Ingestion
| Metric | Description | Type |
|---|
| aws.firehose.incoming.bytes | Total bytes ingested into the delivery stream. | Bytes |
| aws.firehose.incoming.bytes.per.sec | Inbound data rate to the delivery stream in bytes per second. | Bytes/s |
| aws.firehose.incoming.records | Total number of records ingested into the delivery stream. | Count |
| aws.firehose.incoming.put.requests | Total number of PutRecord and PutRecordBatch API calls. | Count |
| aws.firehose.throttled.records | Number of records throttled due to stream limits. | Count |
Delivery Success
| Metric | Description | Type |
|---|
| aws.firehose.delivery.to.s3.success.percent | Percentage of records successfully delivered to S3. | Percent |
| aws.firehose.delivery.to.redshift.success.percent | Percentage of records successfully delivered to Redshift. | Percent |
| aws.firehose.delivery.to.opensearch.success.percent | Percentage of records successfully delivered to OpenSearch. | Percent |
| aws.firehose.delivery.to.splunk.success.percent | Percentage of records successfully delivered to Splunk. | Percent |
| aws.firehose.delivery.to.http.success.percent | Percentage of records successfully delivered to HTTP endpoint. | Percent |
Delivery Freshness
| Metric | Description | Type |
|---|
| aws.firehose.delivery.to.s3.data.freshness.seconds | Age of the oldest undelivered record to S3 in seconds. | Seconds |
| aws.firehose.delivery.to.splunk.data.freshness.seconds | Age of the oldest undelivered record to Splunk in seconds. | Seconds |
| aws.firehose.delivery.to.http.data.freshness.seconds | Age of the oldest undelivered record to HTTP endpoint in seconds. | Seconds |
| aws.firehose.delivery.to.opensearch.data.freshness.seconds | Age of the oldest undelivered record to OpenSearch in seconds. | Seconds |
| aws.firehose.delivery.to.splunk.ack.latency.seconds | Latency for Splunk delivery acknowledgments in seconds. | Seconds |
Stream Limits
| Metric | Description | Type |
|---|
| aws.firehose.bytes.per.sec.limit | Maximum allowed inbound data rate in bytes per second. | Bytes/s |
| aws.firehose.records.per.sec.limit | Maximum allowed record ingestion rate per second. | Count |
| aws.firehose.kinesis.lag.ms | Lag between Kinesis Data Streams source and Firehose delivery in milliseconds. | Milliseconds |