feat: support success metric on ratio indicator

[cbeneke]

Implements #1538

This PR adds support for a success metric on the Ratio indicator. Any 2 out of 3 of total, error and success can be confiugred, and the required total and error will be derived following the total = error + success logic.

[metalmatze]

For simplicity's sake I've always preferred having only one "correct" version to specify the SLO in the configuration.
Is there any immediate use case for this, other than having the option?

[cbeneke]

Thanks for the feedback already!

Unfortunately there are still enough systems which do not provide metrics for the „total / error“ combination. While this is generally the accepted standard, especially in message transfer / processing I am seeing the „error / success“ combination often enough or systems which only expose total and successful requests.

While this can be worked around using recording rules that is quite unflexible and requires backfilling the data on every change, which is cumbersome. Having direct support for these metric combinations for SLO measurement would be great :)

[Angelin01]

Adding my two cents here. We are currently using Sloth instead of Pyrra due to the exact lack of this feature.

I'd say about half of our systems have error ratios defined as (total - success) / total

In Sloth, we can simply say something like:

      error: |
        sum(rate(istio_request_duration_milliseconds_bucket{
          destination_service="service-foo",
          le="+Inf"
        }[{{ .window }}]))
        -
        sum(rate(istio_request_duration_milliseconds_bucket{
          destination_service="service-foo",
          le="250.0"
        }[{{ .window }}]))
      total: |
        sum(rate(istio_requests_total{
          destination_service="service-foo"
        }[{{ .window }}]))

But due to Pyrra's design, this doesn't seem possible. Am I wrong?

[metalmatze]

That's right @Angelin01,

the entire idea is that users configure VectorSelectors to match time series and the rest is taken care of. It fundamentally wouldn't work if we'd allow users to input arbitrary queries in Pyrra.

This won't change, even if we end up merging success, error, and total support.

[cbeneke]

While „arbitrary queries“ would still not be supported, the example brought forward by @Angelin01 could be covered with a total and success metric (needs clarification which of the two mentioned total queries would take precedence though).

@metalmatze do you see any issue with going forward with this format of implementation? I generally understand the direction to limit options, but the combination of requiring a „single metric“ and „which kind of metric“ is a limiting factor for usability of Pyrra currently.

[metalmatze]

Got it.
If we can make it work I'll merge it.
We should add more tests and make sure the pass.