[bug]: Duplicated metric data values

[peterpakos]

What did you do

We are scraping custom CloudWatch metrics coming from Artillery.

Configuration snippet:

region: eu-west-2
role_arn: arn:aws:iam::xxx:role/grafana
delay_seconds: 0
use_get_metric_data: true
metrics:
  - aws_namespace: PerformanceTesting
    aws_metric_name: "http.requests"
    aws_dimensions: [Name, Test]
    aws_statistics: [Sum]

Metric data shown in CloudWatch UI:

Metric data returned from CloudWatch API:

{
    "MetricDataResults": [
        {
            "Id": "m1",
            "Label": "http.requests",
            "Timestamps": [
                "2024-09-24T12:31:00+00:00",
                "2024-09-24T12:30:00+00:00"
            ],
            "Values": [
                16.0,
                44.0
            ],
            "StatusCode": "Complete"
        }
    ],
    "Messages": []
}

What did you expect to see?

Prometheus metrics in Grafana matching metrics in CloudWatch.

What did you see instead? Under which circumstances?

First 2 data points matching CloudWatch then the last value repeated for 5-10 minutes.

Metric data in Grafana:

Are you currently working around this issue?

No

Environment

# docker images | grep cloudwatch-exporter
prom/cloudwatch-exporter             latest    70a238bc9c70   5 weeks ago     297MB

[matthiasr]

I think this is a consequence of the range_seconds setting. On every scrape, the exporter will look that far back in CloudWatch – by default, 10 minutes. This is one of the many ugly things we need to do to bridge the gap between Prometheus' and Cloudwatch's metric-and-time models 😔

[matthiasr]

You may want to enable set_timestamp, that way Prometheus should know that it's seeing the same sample over and over.

[peterpakos]

@matthiasr AFAIK set_timestamp is set to true by default.

[matthiasr]

Ah, true, then I'm not 100% sure what is going on. sum and range queries obscure the timing of the underlying samples in Prometheus. Could you run an instant query (table view in Prometheus UI / Grafana Explorer) with the time set to 2024-09-24 13:45 for performancetesting_http_requests_sum{…}[20m](with enough labels instead of … to limit it to one raw time series)?

[matthiasr]

this will return a single array-shaped result of (timestamp, value) pairs, those are what is really stored in Prometheus.

[peterpakos]

@matthiasr this is just one raw time series:

[matthiasr]

Hmm, then I really don't know where these are coming from. I would expect at least the timestamps of the samples in Grafana Cloud to match the CloudWatch timestamps, but they do not. I'm not familiar with the GC ingestion pipeline; looking at the agent and alloy documentation they do default to honor_timestamps: true.

What I would try to do is track down where they are getting lost. The first thing to check would be the timestamps in the exporter's /metrics endpoint directly. Do these match the timestamps in CloudWatch, or those in Grafana Cloud, for the same time series and value? For a time series that has stopped in CW, do they still keep updating? This is all that we control on the exporter side – if anything along the ingestion chain does not honor timestamps, the behavior you are observing is what I expect to happen.