From c16c04936ac58b3e13af7d739d32e3e64deb67d7 Mon Sep 17 00:00:00 2001 From: Subhash Khileri Date: Mon, 20 Apr 2026 16:01:33 +0530 Subject: [PATCH 1/3] docs: document skip tag support for overlay E2E tests Add documentation for the new tag-based test skipping feature that auto-derives --grep-invert from JOB_NAME in run-e2e.sh. Updated run-e2e reference, CI pipeline tutorial, and spec files guide. Co-Authored-By: Claude Opus 4.6 --- docs/overlay/reference/run-e2e.md | 49 ++++++++++++++++++++++- docs/overlay/test-structure/spec-files.md | 27 +++++++++++++ docs/overlay/tutorials/ci-pipeline.md | 31 ++++++++++++++ 3 files changed, 106 insertions(+), 1 deletion(-) diff --git a/docs/overlay/reference/run-e2e.md b/docs/overlay/reference/run-e2e.md index 2f3b33c..31421ed 100644 --- a/docs/overlay/reference/run-e2e.md +++ b/docs/overlay/reference/run-e2e.md @@ -82,7 +82,54 @@ A workspace is discovered when it has a `workspaces//e2e-tests/` directory |----------|-------------|---------| | `E2E_NIGHTLY_MODE` | When `true`, uses released OCI images from metadata; defaults `E2E_TEST_UTILS_VERSION` to `latest` | `false` | | `GIT_PR_NUMBER` | PR number for OCI URL generation (uses PR-built images) | - | -| `JOB_NAME` | CI job name; if contains `periodic-`, disables metadata injection | - | +| `JOB_NAME` | CI job name; if contains `periodic-`, disables metadata injection. Also used to [auto-derive skip tags](#skip-tags). | - | + +## Skip Tags + +When `JOB_NAME` is set (by OpenShift CI), the script auto-derives a Playwright tag and injects `--grep-invert` to exclude tests tagged with it. This lets test authors skip specific tests in specific CI jobs using standard Playwright tags. + +### How It Works + +The job suffix is extracted from `JOB_NAME` by stripping everything up to and including `-e2e-`: + +| JOB_NAME (suffix shown) | Derived tag | `--grep-invert` | +|--------------------------|-------------|-----------------| +| `...-e2e-ocp-helm` | `@skip-ocp-helm` | `@skip-ocp-helm` | +| `...-e2e-ocp-helm-nightly` | `@skip-ocp-helm-nightly` | `@skip-ocp-helm-nightly` | +| `...-e2e-ocp-operator` | `@skip-ocp-operator` | `@skip-ocp-operator` | +| `...-e2e-ocp-operator-nightly` | `@skip-ocp-operator-nightly` | `@skip-ocp-operator-nightly` | + +If `JOB_NAME` doesn't contain `-e2e-`, no tag is derived and no filtering is applied. + +### Tagging Tests + +Add Playwright tags to `test.describe` or individual `test` calls: + +```typescript +// Skip in ocp-helm-nightly job +test.describe("My Plugin", { tag: "@skip-ocp-helm-nightly" }, () => { ... }); + +// Skip in all ocp-helm jobs (PR + nightly, since regex matches as substring) +test("expensive test", { tag: "@skip-ocp-helm" }, async () => { ... }); + +// Multiple tags — skip in both helm and operator nightly +test.describe("Suite", { + tag: ["@skip-ocp-helm-nightly", "@skip-ocp-operator-nightly"], +}, () => { ... }); +``` + +### From a Workspace Directory + +Skip tags work the same way when running tests from a workspace — pass `--grep-invert` manually: + +```bash +cd workspaces/tech-radar/e2e-tests +yarn test -- --grep-invert "@skip-ocp-helm" +``` + +::: info Precedence +The auto-derived `--grep-invert` is prepended to the Playwright arguments. If you also pass `--grep-invert` on the command line, Playwright uses the last value (last wins), so your manual flag takes precedence. +::: ## How It Works diff --git a/docs/overlay/test-structure/spec-files.md b/docs/overlay/test-structure/spec-files.md index 51e4aad..4398e06 100644 --- a/docs/overlay/test-structure/spec-files.md +++ b/docs/overlay/test-structure/spec-files.md @@ -52,6 +52,33 @@ test.describe("Test ", () => { In CI, namespaces are automatically deleted after all tests complete via the built-in teardown reporter. No manual cleanup code is needed. See [Namespace Cleanup](/guide/core-concepts/playwright-fixtures#namespace-cleanup-teardown) for details. ::: +## Tagging Tests + +Use Playwright tags to control which tests run in which CI jobs. Tags are added via the second argument to `test.describe` or `test`: + +```typescript +// Skip this suite in the ocp-helm-nightly CI job +test.describe("My Plugin", { tag: "@skip-ocp-helm-nightly" }, () => { + test("test case", async () => { ... }); +}); + +// Skip a single test in ocp-helm jobs (both PR and nightly) +test("expensive test", { tag: "@skip-ocp-helm" }, async () => { ... }); + +// Multiple tags +test.describe("Suite", { + tag: ["@skip-ocp-helm-nightly", "@skip-ocp-operator-nightly"], +}, () => { ... }); +``` + +When running in CI, `run-e2e.sh` auto-derives the skip tag from `JOB_NAME` and passes `--grep-invert` to Playwright. From a workspace directory, pass it manually: + +```bash +yarn test -- --grep-invert "@skip-ocp-helm" +``` + +See [Skip Tags](/overlay/reference/run-e2e#skip-tags) for the full list of tags and derivation logic. + ## Imports Import test utilities from `@red-hat-developer-hub/e2e-test-utils`: diff --git a/docs/overlay/tutorials/ci-pipeline.md b/docs/overlay/tutorials/ci-pipeline.md index 29fd474..8dbfb26 100644 --- a/docs/overlay/tutorials/ci-pipeline.md +++ b/docs/overlay/tutorials/ci-pipeline.md @@ -93,6 +93,37 @@ To test nightly job changes via the release repo: 2. Comment `/test e2e-ocp-helm-nightly` on the PR 3. The rehearse job runs with `JOB_MODE=nightly`, testing all workspaces +## Skipping Tests by CI Job + +Tests can opt out of specific CI jobs using Playwright tags. The `run-e2e.sh` script auto-derives a skip tag from `JOB_NAME` and passes `--grep-invert` to Playwright. + +For example, to skip a test in the nightly helm job: + +```typescript +test.describe("My Plugin", { tag: "@skip-ocp-helm-nightly" }, () => { + // This entire suite is excluded when JOB_NAME contains "e2e-ocp-helm-nightly" +}); +``` + +### Available Tags + +| Tag | Skipped in | +|-----|-----------| +| `@skip-ocp-helm` | `e2e-ocp-helm` (PR check) | +| `@skip-ocp-helm-nightly` | `e2e-ocp-helm-nightly` | +| `@skip-ocp-operator` | `e2e-ocp-operator` (PR check) | +| `@skip-ocp-operator-nightly` | `e2e-ocp-operator-nightly` | + +Multiple tags can be combined on a single test or describe block: + +```typescript +test.describe("Suite", { + tag: ["@skip-ocp-helm-nightly", "@skip-ocp-operator-nightly"], +}, () => { ... }); +``` + +See [Skip Tags](/overlay/reference/run-e2e#skip-tags) for the full derivation logic and how to use tags from a workspace directory. + ## PR OCI Image Builds When testing a PR, the plugins need to be built into OCI images that RHDH can use. This is handled through the `/publish` command. From b7c6c17f1183ff51b39af73e4dd31494326083da Mon Sep 17 00:00:00 2001 From: Subhash Khileri Date: Mon, 20 Apr 2026 16:08:20 +0530 Subject: [PATCH 2/3] fix: update skip tag docs to reflect exact matching with lookahead MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tags now match exactly — @skip-ocp-helm won't filter @skip-ocp-helm-nightly. Updated tables and examples to clarify per-job isolation. Co-Authored-By: Claude Opus 4.6 --- docs/overlay/reference/run-e2e.md | 18 +++++++++--------- docs/overlay/tutorials/ci-pipeline.md | 14 ++++++++------ 2 files changed, 17 insertions(+), 15 deletions(-) diff --git a/docs/overlay/reference/run-e2e.md b/docs/overlay/reference/run-e2e.md index 31421ed..d07c753 100644 --- a/docs/overlay/reference/run-e2e.md +++ b/docs/overlay/reference/run-e2e.md @@ -90,14 +90,14 @@ When `JOB_NAME` is set (by OpenShift CI), the script auto-derives a Playwright t ### How It Works -The job suffix is extracted from `JOB_NAME` by stripping everything up to and including `-e2e-`: +The job suffix is extracted from `JOB_NAME` by stripping everything up to and including `-e2e-`. A negative lookahead `(?!-)` is appended so each tag matches exactly — `@skip-ocp-helm` won't accidentally filter `@skip-ocp-helm-nightly`: -| JOB_NAME (suffix shown) | Derived tag | `--grep-invert` | -|--------------------------|-------------|-----------------| -| `...-e2e-ocp-helm` | `@skip-ocp-helm` | `@skip-ocp-helm` | -| `...-e2e-ocp-helm-nightly` | `@skip-ocp-helm-nightly` | `@skip-ocp-helm-nightly` | -| `...-e2e-ocp-operator` | `@skip-ocp-operator` | `@skip-ocp-operator` | -| `...-e2e-ocp-operator-nightly` | `@skip-ocp-operator-nightly` | `@skip-ocp-operator-nightly` | +| JOB_NAME (suffix shown) | `--grep-invert` pattern | +|--------------------------|-------------------------| +| `...-e2e-ocp-helm` | `@skip-ocp-helm(?!-)` | +| `...-e2e-ocp-helm-nightly` | `@skip-ocp-helm-nightly(?!-)` | +| `...-e2e-ocp-operator` | `@skip-ocp-operator(?!-)` | +| `...-e2e-ocp-operator-nightly` | `@skip-ocp-operator-nightly(?!-)` | If `JOB_NAME` doesn't contain `-e2e-`, no tag is derived and no filtering is applied. @@ -106,10 +106,10 @@ If `JOB_NAME` doesn't contain `-e2e-`, no tag is derived and no filtering is app Add Playwright tags to `test.describe` or individual `test` calls: ```typescript -// Skip in ocp-helm-nightly job +// Skip only in ocp-helm-nightly job test.describe("My Plugin", { tag: "@skip-ocp-helm-nightly" }, () => { ... }); -// Skip in all ocp-helm jobs (PR + nightly, since regex matches as substring) +// Skip only in ocp-helm PR check job (won't affect nightly) test("expensive test", { tag: "@skip-ocp-helm" }, async () => { ... }); // Multiple tags — skip in both helm and operator nightly diff --git a/docs/overlay/tutorials/ci-pipeline.md b/docs/overlay/tutorials/ci-pipeline.md index 8dbfb26..cde78e0 100644 --- a/docs/overlay/tutorials/ci-pipeline.md +++ b/docs/overlay/tutorials/ci-pipeline.md @@ -107,12 +107,14 @@ test.describe("My Plugin", { tag: "@skip-ocp-helm-nightly" }, () => { ### Available Tags -| Tag | Skipped in | -|-----|-----------| -| `@skip-ocp-helm` | `e2e-ocp-helm` (PR check) | -| `@skip-ocp-helm-nightly` | `e2e-ocp-helm-nightly` | -| `@skip-ocp-operator` | `e2e-ocp-operator` (PR check) | -| `@skip-ocp-operator-nightly` | `e2e-ocp-operator-nightly` | +| Tag | Skipped in | Not skipped in | +|-----|-----------|----------------| +| `@skip-ocp-helm` | `e2e-ocp-helm` (PR check) | `e2e-ocp-helm-nightly` | +| `@skip-ocp-helm-nightly` | `e2e-ocp-helm-nightly` | `e2e-ocp-helm` (PR check) | +| `@skip-ocp-operator` | `e2e-ocp-operator` (PR check) | `e2e-ocp-operator-nightly` | +| `@skip-ocp-operator-nightly` | `e2e-ocp-operator-nightly` | `e2e-ocp-operator` (PR check) | + +Each tag matches exactly — `@skip-ocp-helm` won't accidentally skip tests in the nightly job. Multiple tags can be combined on a single test or describe block: From 6e5540537a2a47f79be6ebec584a8c5bd8a556f0 Mon Sep 17 00:00:00 2001 From: Subhash Khileri Date: Mon, 20 Apr 2026 16:50:03 +0530 Subject: [PATCH 3/3] docs: add skipping and disabling tests guidance --- docs/overlay/test-structure/spec-files.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/overlay/test-structure/spec-files.md b/docs/overlay/test-structure/spec-files.md index 4398e06..47df08d 100644 --- a/docs/overlay/test-structure/spec-files.md +++ b/docs/overlay/test-structure/spec-files.md @@ -52,7 +52,15 @@ test.describe("Test ", () => { In CI, namespaces are automatically deleted after all tests complete via the built-in teardown reporter. No manual cleanup code is needed. See [Namespace Cleanup](/guide/core-concepts/playwright-fixtures#namespace-cleanup-teardown) for details. ::: -## Tagging Tests +## Skipping and Disabling Tests + +| Mechanism | When to use | Report | +|-----------|-------------|--------| +| `{ tag: "@skip-ocp-helm" }` | Test works, but skip in a specific CI job | Hidden from that job and report | +| `test.fixme()` | Test is broken or flaky, needs fixing | Shows as "skipped" everywhere | +| `test.skip(condition, reason)` | Skip based on runtime (e.g., missing secret) | Shows as "skipped" with reason | + +### Tagging Tests Use Playwright tags to control which tests run in which CI jobs. Tags are added via the second argument to `test.describe` or `test`: