docs: add steps for clearing out the WAL

[sed-i]

Via @marino-mrc

otelcol gave:

Exporting failed. Dropping data.
...
Permanent error: Permanent error: context deadline exceeded
...
Permanent error: Permanent error: Permanent error: remote write returned HTTP status 429 Too Many Requests; the request has been rejected because the tenant exceeded the ingestion rate limit

On mimir-write side,

no space left on device

One option is to kill the pvc,

kubectl scale statefulset mimir-write -n cos --replicas=0
kubectl delete pvc   mimir-write-data-db5430bc-mimir-write-0   mimir-write-data-db5430bc-mimir-write-1   mimir-write-data-db5430bc-mimir-write-2   mimir-write-recovery-data-db5430bc-mimir-write-0   mimir-write-recovery-data-db5430bc-mimir-write-1   mimir-write-recovery-data-db5430bc-mimir-write-2   -n cos
kubectl scale statefulset mimir-write -n cos --replicas=3

A quicker option is to juju ssh and delete the WAL files.

We should:

1. Add a doc showing how to clear out the WAL. Maybe even add charm action purge-wal.
2. Include the above error messages verbatim in our docs, so they are searchable.

---

Reported from: https://documentation.ubuntu.com/observability/latest/how-to/deploy-and-manage/

[sinapah]

A small question from my side is: can't we mitigate the WAL from growing too big in the first place? AFAICT, one of the main ways the WAL can grow to the point where the issue above happens is if the ingestion rate suddenly grows too much or if a config value such as max_chunk_age is set too high, resulting in flushing happening too infrequently. Shouldn't we prioritize those approaches? Clearing out the WAL will lead to data loss. Instead, we can prevent the WAL getting too big in the first place by flushing more frequently. It would be more CPU-intensive, but in most env's that shouldn't be an issue (I think).