Add storage.wal.failover.write_and_sync.latency
#20566
Merged
+20
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for cockroachdb-api-docs canceled.
|
✅ Deploy Preview for cockroachdb-interactivetutorials-docs canceled.
|
Files changed:
|
rmloveland
added a commit
to rmloveland/cockroach
that referenced
this pull request
Oct 14, 2025
This change marks the `storage.wal.failover.write_and_sync.latency` metric as "Essential" so it gets automatically pulled into the 'Essential Metrics' documentation at e.g., https://www.cockroachlabs.com/docs/stable/essential-metrics-self-hosted.html#storage This is necessary since we are adding some words about this metric to the docs via cockroachdb/docs#20566 We would like to then backport this change to all supported versions of CockroachDB which have WAL failover (i.e., v24.1 and later).
✅ Netlify Preview
To edit notification comments on pull requests, go to your Netlify project configuration. |
sumeerbhola
requested changes
Oct 15, 2025
There was a problem hiding this comment.
@sumeerbhola reviewed 1 of 1 files at r1, all commit messages.
Reviewable status:complete! 0 of 0 LGTMs obtained (waiting on @rmloveland)
src/current/_includes/v25.4/wal-failover-metrics.md line 7 at r1 (raw file):
- `storage.wal.failover.switch.count`: Count of the number of times WAL writing has switched from primary to secondary store, and vice versa. - `storage.wal.fsync.latency` monitors the latencies of WAL files. If you have WAL failover enabled and are failing over, `storage.wal.fsync.latency` will include the latency of the stalled primary. - `storage.wal.failover.write_and_sync.latency` metric is up one level from `storage.wal.fsync.latency`, and during the failover will report the latency actually observed by higher levels (which should be ~equivalent to the latency of the secondary).
It is not just during the failover. We should say something like.
When WAL failover is configured in a cluster, the operator should monitor this metric which shows the effective latency observed by the higher layer writing to the WAL. This metric is expected to stay low in a healthy system, regardless of whether WAL files are being written to the primary or secondary.
|
This change is |
|
thanks @sumeerbhola, i've updated in 998c744 - PTAL once we're happy with this change and it's had docs review I would like to backport it to the WAL failover docs for all versions where this metric is supported which previous versions have this metric available? is it everything v24.1+ or only a subset? |
craig bot
pushed a commit
to cockroachdb/cockroach
that referenced
this pull request
Oct 15, 2025
155395: storage: mark add'l WAL latency metric essential r=rmloveland a=rmloveland This change marks the `storage.wal.failover.write_and_sync.latency` metric as "Essential" so it gets automatically pulled into the 'Essential Metrics' documentation at e.g., https://www.cockroachlabs.com/docs/stable/essential-metrics-self-hosted.html#storage This is necessary since we are adding some words about this metric to the docs via cockroachdb/docs#20566 We would like to then backport this change to all supported versions of CockroachDB which have WAL failover (i.e., v24.1 and later). Addresses part of DOC-13184 Co-authored-by: Rich Loveland <rich@cockroachlabs.com>
rmloveland
added a commit
to rmloveland/cockroach
that referenced
this pull request
Oct 16, 2025
This change marks the `storage.wal.failover.write_and_sync.latency` metric as "Essential" so it gets automatically pulled into the 'Essential Metrics' documentation at e.g., https://www.cockroachlabs.com/docs/stable/essential-metrics-self-hosted.html#storage This is necessary since we are adding some words about this metric to the docs via cockroachdb/docs#20566 We would like to then backport this change to all supported versions of CockroachDB which have WAL failover (i.e., v24.1 and later).
rmloveland
added a commit
to rmloveland/cockroach
that referenced
this pull request
Oct 16, 2025
This change marks the `storage.wal.failover.write_and_sync.latency` metric as "Essential" so it gets automatically pulled into the 'Essential Metrics' documentation at e.g., https://www.cockroachlabs.com/docs/stable/essential-metrics-self-hosted.html#storage This is necessary since we are adding some words about this metric to the docs via cockroachdb/docs#20566 We would like to then backport this change to all supported versions of CockroachDB which have WAL failover (i.e., v24.1 and later).
craig bot
pushed a commit
to cockroachdb/cockroach
that referenced
this pull request
Oct 20, 2025
155395: storage: mark add'l WAL latency metric essential r=rmloveland a=rmloveland This change marks the `storage.wal.failover.write_and_sync.latency` metric as "Essential" so it gets automatically pulled into the 'Essential Metrics' documentation at e.g., https://www.cockroachlabs.com/docs/stable/essential-metrics-self-hosted.html#storage This is necessary since we are adding some words about this metric to the docs via cockroachdb/docs#20566 We would like to then backport this change to all supported versions of CockroachDB which have WAL failover (i.e., v24.1 and later). Addresses part of DOC-13184 Co-authored-by: Rich Loveland <rich@cockroachlabs.com>
sumeerbhola
approved these changes
Oct 23, 2025
There was a problem hiding this comment.
which previous versions have this metric available? is it everything v24.1+ or only a subset?
Since v24.1 cockroachdb/cockroach#123232
@sumeerbhola reviewed 1 of 1 files at r2, all commit messages.
Reviewable status:
|
TFTRs! Backporting to all supported versions v24.1+ before merge |
Fixes DOC-13184 Summary of changes: - Add a mention of the `storage.wal.failover.write_and_sync.latency` metric to the `wal-failover-metrics.md` include file, which will pull it into the 'WAL failover' and 'cockroach start' pages. - We're also doing a cockroachdb/cockroach PR to mark this metric as 'essential', so it shows up in the list of Storage essential metrics at e.g. https://www.cockroachlabs.com/docs/v25.3/essential-metrics-self-hosted.html#storage
rmloveland
force-pushed
the
20251014-DOC-13184-update-wal-failover-with-latency-metric
branch
from
998c744 to
d0d710f
Compare
rmloveland
deleted the
20251014-DOC-13184-update-wal-failover-with-latency-metric
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes DOC-13184
Summary of changes:
Add a mention of the
storage.wal.failover.write_and_sync.latencymetric to thewal-failover-metrics.mdinclude file, which will pull it into the 'WAL failover' and 'cockroach start' pages.We're also doing storage: mark add'l WAL latency metric essential cockroach#155395 to mark this metric as 'essential', so it shows up in the list of Storage essential metrics at e.g. https://www.cockroachlabs.com/docs/v25.3/essential-metrics-self-hosted.html#storage