Swap usage alarm enhancement

[DavidePrincipi]

The current node-monitor alarm logic is based on the SwapFree/SwapTotal ratio. However after an high memory usage peak the ratio remains high and the alarm is not reset.

Improve the monitoring tool's ability to detect high Linux swap usage by adding a secondary factor to the existing SwapFree/SwapTotal ratio check. The enhancement should provide better insight into memory pressure during and after swap peaks.

Requirements:

1. Current Metric:

   • Continue using the /proc/meminfo SwapFree/SwapTotal ratio to monitor overall swap utilization.

2. New Metric:

   • Introduce monitoring of /proc/vmstat for:
     • pswpin: Number of pages swapped into memory.
     • pswpout: Number of pages swapped out of memory.
   • Implement tracking of the rate of change in these counters over a configurable interval (e.g., every 10 seconds).

3. Thresholds:

   • Add configurable thresholds for both pswpin and pswpout rates to trigger alerts when they exceed a certain limit (indicating sustained memory pressure).

4. Alerting Logic:

   • Trigger an alert if:
     • The SwapFree/SwapTotal ratio raises above the defined threshold and
     • A high rate of pswpin or pswpout events is detected over a defined period (indicating active swap use despite acceptable swap levels).

Discussion @nrauso https://mattermost.nethesis.it/nethesis/pl/5kgxh85pep8atgjd1ppce64ikr

[gsanchietti]

The new monitor system uses node_exporter metrics to generate the alerts.
Before proceeding with QA of this issue, please make sure to have installed the latest version from #7162

Test case 1: no swap

• Create a node without a SWAP
• Verify an alert is fired
• Create a swap:

  dd if=/dev/zero of=/swap bs=1M count=10
  chmod 0600 /swap
  mkswap /swap
  swapon /swap
  

• Verify the alert is resolved after few minutes

Example of alert inside journal:

ALERT CRITICAL swap:notpresent:1

Test case 2: full swap

• After test case 1
• Create a very small SWAP file (10 MB, as in test case 1) and increase swappiness:

  echo 200 > /proc/sys/vm/swappiness
  

• Install and configure many modules on the node
• Verify an alert is raised then the SWAP became full

To fill the memory very quickly, save this code to hog.py:

import ctypes

def memleak():
    # Create C api callable
    inc_ref = ctypes.pythonapi.Py_IncRef
    inc_ref.argtypes = [ctypes.py_object]
    inc_ref.restype = None

    # Allocate a large object
    obj = list(range(10_000_000))

    # Increment its ref count
    inc_ref(obj)

    # obj will have a dangling reference after this function exits

while True:
    memleak()  # leaks memory

Then execute with:

python3 hog.py

[DavidePrincipi]

In testing core 3.5.0-dev.5

[DavidePrincipi]

VERIFIED

Test case 1

{
  "status": "success",
  "data": {
    "alerts": [
      {
        "labels": {
          "alertname": "swap_notpresent",
          "instance": "10.5.4.2:9100",
          "job": "providers",
          "node": "2",
          "severity": "critical"
        },
        "annotations": {
          "description": "Swap is not configured on this host.\n ",
          "summary": "Swap not configured (instance 10.5.4.2:9100)"
        },
        "state": "firing",
        "activeAt": "2025-03-05T11:01:23.728781774Z",
        "value": "0e+00"
      }
    ]
  }
}

Test case 2

{
  "status": "success",
  "data": {
    "alerts": [
      {
        "labels": {
          "alertname": "swap_full",
          "instance": "10.5.4.2:9100",
          "job": "providers",
          "node": "2",
          "severity": "warning"
        },
        "annotations": {
          "description": "Swap is filling up (>80%)\n  VALUE = 99.88276670574443\n  LABELS = map[instance:10.5.4.2:9100 job:providers node:2]",
          "summary": "Host swap is filling up (instance 10.5.4.2:9100)"
        },
        "state": "pending",
        "activeAt": "2025-03-05T11:09:23.728781774Z",
        "value": "9.988276670574443e+01"
      }
    ]
  }
}