NETOBSERV-2657: Allow external/contributed recording rules in Network Health by leandroberetta · Pull Request #2500 · netobserv/netobserv-operator

leandroberetta · 2026-02-26T21:17:51Z

Description

PrometheusRule CRD won't allow to have annotations in rules of type "record", for this reason, today is not possible to for third party users to bring new recording rules, just alerts. This PR changes that, allowing users to bring their own recording rules using the following convention:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: my-recording-rules
  namespace: netobserv
  labels:
    netobserv: "true"
  annotations:
    netobserv.io/network-health: |
      {
        "my_metric_per_namespace": {
          "summary": "Custom metric is {{ $value }} in the namespace {{ $labels.namespace }}",
          "description": "Custom metric is {{ $value }} in the namespace {{ $labels.namespace }}",
          "netobserv_io_network_health": "{\"unit\":\"%\",\"upperBound\":\"100\",\"namespaceLabels\":[\"namespace\"],\"recordingThresholds\":{\"info\":\"10\",\"warning\":\"25\",\"critical\":\"50\"}}"
        }
      }
spec:
  groups:
    - name: MyRecordingRules
      interval: 30s
      rules:
        - record: my_metric_per_namespace
          expr: (count by (namespace) (kube_pod_info) * 0 + 20)
          labels:
            netobserv: "true"

Dependencies

netobserv/netobserv-web-console#1311

Checklist

If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.

codecov · 2026-02-27T00:26:11Z

Codecov Report

❌ Patch coverage is 40.00000% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.07%. Comparing base (3baee62) to head (05b1033).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...ntroller/consoleplugin/consoleplugin_reconciler.go	23.07%	17 Missing and 3 partials ⚠️
.../controller/consoleplugin/consoleplugin_objects.go	60.00%	1 Missing and 1 partial ⚠️
internal/controller/flowcollector_controller.go	77.77%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2500      +/-   ##
==========================================
- Coverage   72.39%   72.07%   -0.33%     
==========================================
  Files         105      105              
  Lines       10851    10889      +38     
==========================================
- Hits         7856     7848       -8     
- Misses       2518     2557      +39     
- Partials      477      484       +7

Flag	Coverage Δ
unittests	`72.07% <40.00%> (-0.33%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
.../controller/consoleplugin/consoleplugin_objects.go	`89.95% <60.00%> (-0.34%)`	⬇️
internal/controller/flowcollector_controller.go	`80.37% <77.77%> (-0.24%)`	⬇️
...ntroller/consoleplugin/consoleplugin_reconciler.go	`68.00% <23.07%> (-8.00%)`	⬇️

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

internal/controller/consoleplugin/consoleplugin_objects.go

internal/controller/flowcollector_controller.go

jotak

thanks!
/lgtm

openshift-ci-robot · 2026-03-09T20:09:19Z

memodi

Hi @leandroberetta - few questions/comments:

could you add documentation for this feature describing the example and the fields that are necessary ?
It looks like users can add recording for any metric they would like and it would show up on Network Health page - my concerns is - netobserv NS is deleted when flowcollector is deleted and users would lose their rules, I wonder how do we make users aware of this?
do we provide hint in UI to differentiate user defined rules?

memodi · 2026-03-11T20:34:43Z

internal/controller/consoleplugin/consoleplugin_reconciler.go

+	list := &monitoringv1.PrometheusRuleList{}
+	if err := cl.List(ctx, list, client.MatchingLabels{"netobserv": "true"}); err != nil {
+		log.FromContext(ctx).Error(err, "Failed to list PrometheusRules for recording annotations")
+		return out


when there are transient issues (API request failing etc.) retrieving the rules, it would overwrite existing rules with empty recording rules?

Good catch, I'll fix it.

memodi · 2026-03-12T01:20:26Z

internal/controller/consoleplugin/consoleplugin_reconciler.go

+						raw, hasAnnot := pr.Annotations[recordingAnnotationsAnnotation]
+						if hasAnnot && raw != "" {
+							var perRule map[string]map[string]string
+							if err := json.Unmarshal([]byte(raw), &perRule); err != nil {


is simply validating JSON is sufficient? do we want to validate specific fields?

I think is ok, but we can improve this on a follow up PR is your ok.

openshift-ci · 2026-03-17T12:03:29Z

New changes are detected. LGTM label has been removed.

openshift-ci · 2026-03-17T12:03:32Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from jotak. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

DOWNSTREAM_OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

leandroberetta · 2026-03-17T12:05:08Z

Hi @leandroberetta - few questions/comments:

could you add documentation for this feature describing the example and the fields that are necessary ?

It looks like users can add recording for any metric they would like and it would show up on Network Health page - my concerns is - netobserv NS is deleted when flowcollector is deleted and users would lose their rules, I wonder how do we make users aware of this?

do we provide hint in UI to differentiate user defined rules?

Hi @memodi, sorry for the delay and thanks for the review.

could you add documentation for this feature describing the example and the fields that are necessary ?

I added the documentation to the HealthRules.md

It looks like users can add recording for any metric they would like and it would show up on Network Health > page - my concerns is - netobserv NS is deleted when flowcollector is deleted and users would lose their rules, I wonder how do we make users aware of this?

PrometheusRules can be on any namespace. I added a mention of this in the docs.

do we provide hint in UI to differentiate user defined rules?

We aren't today. I will create an issue for that if you're ok.

leandroberetta · 2026-03-17T12:16:25Z

@memodi in the description of the issue you have a simple HealthRule which you can update to play with the thresholds, the recording rules will appear in the Namespaces tab.

memodi · 2026-03-18T17:52:02Z

Hi @leandroberetta - thanks for addressing feedback. could you resolve merge conflicts and add /ok-to-test label after that?

memodi · 2026-03-18T19:35:46Z

/ok-to-test

github-actions · 2026-03-18T19:49:41Z

New images:

quay.io/netobserv/network-observability-operator:ff365a2
quay.io/netobserv/network-observability-operator-bundle:v0.0.0-sha-ff365a2
quay.io/netobserv/network-observability-operator-catalog:v0.0.0-sha-ff365a2

They will expire in two weeks.

To deploy this build:

# Direct deployment, from operator repo
IMAGE=quay.io/netobserv/network-observability-operator:ff365a2 make deploy

# Or using operator-sdk
operator-sdk run bundle quay.io/netobserv/network-observability-operator-bundle:v0.0.0-sha-ff365a2

Or as a Catalog Source:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: netobserv-dev
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/netobserv/network-observability-operator-catalog:v0.0.0-sha-ff365a2
  displayName: NetObserv development catalog
  publisher: Me
  updateStrategy:
    registryPoll:
      interval: 1m

memodi · 2026-03-18T20:13:12Z

/ok-to-test

memodi · 2026-03-19T14:10:12Z

/ok-to-test

memodi · 2026-03-19T22:17:38Z

docs/HealthRules.md

+
+### Lifecycle and namespace
+
+Custom PrometheusRules are **not** owned by the FlowCollector. If you put them in the **NetObserv namespace** (e.g. `netobserv`), that namespace may be removed when the FlowCollector is uninstalled (depending on install mode), and your rules would be deleted. To avoid that, create your PrometheusRules in **another namespace** (e.g. `monitoring` or a dedicated `netobserv-rules`); they will still be picked up cluster-wide. If you do keep rules in the NetObserv namespace, back them up (e.g. in Git) and re-apply after reinstalling.


I think we should add that those ns will need:
openshift.io/cluster-monitoring: "true" label

That annotation is used for letting know Prometheus that that namespace should be scraped. That is not required, but I think the example in the description was outdated and for that reason, you didn't see any rule in the page.

I've updated the docs to improve and add more clarity to the requirements.

A label netobserv true was missing and also the annotaion was renamed some commits ago (now is network-health). Sorry @memodi !

openshift-ci-robot · 2026-03-20T14:41:37Z

openshift-ci-robot · 2026-03-20T14:54:15Z

memodi · 2026-03-24T20:41:02Z

/ok-to-test

github-actions · 2026-03-24T20:49:12Z

New images:

quay.io/netobserv/network-observability-operator:305b85f
quay.io/netobserv/network-observability-operator-bundle:v0.0.0-sha-305b85f
quay.io/netobserv/network-observability-operator-catalog:v0.0.0-sha-305b85f

They will expire in two weeks.

To deploy this build:

# Direct deployment, from operator repo
IMAGE=quay.io/netobserv/network-observability-operator:305b85f make deploy

# Or using operator-sdk
operator-sdk run bundle quay.io/netobserv/network-observability-operator-bundle:v0.0.0-sha-305b85f

Or as a Catalog Source:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: netobserv-dev
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/netobserv/network-observability-operator-catalog:v0.0.0-sha-305b85f
  displayName: NetObserv development catalog
  publisher: Me
  updateStrategy:
    registryPoll:
      interval: 1m

memodi · 2026-03-24T21:15:02Z

/label qe-approved

thanks @leandroberetta

openshift-ci-robot · 2026-03-24T21:15:09Z

leandroberetta mentioned this pull request Feb 26, 2026

NETOBSERV-2657: Allow external/contributed recording rules in Network Health netobserv/netobserv-web-console#1311

Open

10 tasks

leandroberetta requested a review from jotak February 26, 2026 21:31

leandroberetta self-assigned this Feb 26, 2026

leandroberetta changed the title ~~Allow non-builtin (external/contributed) recording rules in Network Health~~ Allow external/contributed recording rules in Network Health Feb 26, 2026