diff --git a/docs/overlay/reference/run-e2e.md b/docs/overlay/reference/run-e2e.md index 2f3b33c..d07c753 100644 --- a/docs/overlay/reference/run-e2e.md +++ b/docs/overlay/reference/run-e2e.md @@ -82,7 +82,54 @@ A workspace is discovered when it has a `workspaces//e2e-tests/` directory |----------|-------------|---------| | `E2E_NIGHTLY_MODE` | When `true`, uses released OCI images from metadata; defaults `E2E_TEST_UTILS_VERSION` to `latest` | `false` | | `GIT_PR_NUMBER` | PR number for OCI URL generation (uses PR-built images) | - | -| `JOB_NAME` | CI job name; if contains `periodic-`, disables metadata injection | - | +| `JOB_NAME` | CI job name; if contains `periodic-`, disables metadata injection. Also used to [auto-derive skip tags](#skip-tags). | - | + +## Skip Tags + +When `JOB_NAME` is set (by OpenShift CI), the script auto-derives a Playwright tag and injects `--grep-invert` to exclude tests tagged with it. This lets test authors skip specific tests in specific CI jobs using standard Playwright tags. + +### How It Works + +The job suffix is extracted from `JOB_NAME` by stripping everything up to and including `-e2e-`. A negative lookahead `(?!-)` is appended so each tag matches exactly — `@skip-ocp-helm` won't accidentally filter `@skip-ocp-helm-nightly`: + +| JOB_NAME (suffix shown) | `--grep-invert` pattern | +|--------------------------|-------------------------| +| `...-e2e-ocp-helm` | `@skip-ocp-helm(?!-)` | +| `...-e2e-ocp-helm-nightly` | `@skip-ocp-helm-nightly(?!-)` | +| `...-e2e-ocp-operator` | `@skip-ocp-operator(?!-)` | +| `...-e2e-ocp-operator-nightly` | `@skip-ocp-operator-nightly(?!-)` | + +If `JOB_NAME` doesn't contain `-e2e-`, no tag is derived and no filtering is applied. + +### Tagging Tests + +Add Playwright tags to `test.describe` or individual `test` calls: + +```typescript +// Skip only in ocp-helm-nightly job +test.describe("My Plugin", { tag: "@skip-ocp-helm-nightly" }, () => { ... }); + +// Skip only in ocp-helm PR check job (won't affect nightly) +test("expensive test", { tag: "@skip-ocp-helm" }, async () => { ... }); + +// Multiple tags — skip in both helm and operator nightly +test.describe("Suite", { + tag: ["@skip-ocp-helm-nightly", "@skip-ocp-operator-nightly"], +}, () => { ... }); +``` + +### From a Workspace Directory + +Skip tags work the same way when running tests from a workspace — pass `--grep-invert` manually: + +```bash +cd workspaces/tech-radar/e2e-tests +yarn test -- --grep-invert "@skip-ocp-helm" +``` + +::: info Precedence +The auto-derived `--grep-invert` is prepended to the Playwright arguments. If you also pass `--grep-invert` on the command line, Playwright uses the last value (last wins), so your manual flag takes precedence. +::: ## How It Works diff --git a/docs/overlay/test-structure/spec-files.md b/docs/overlay/test-structure/spec-files.md index 51e4aad..47df08d 100644 --- a/docs/overlay/test-structure/spec-files.md +++ b/docs/overlay/test-structure/spec-files.md @@ -52,6 +52,41 @@ test.describe("Test ", () => { In CI, namespaces are automatically deleted after all tests complete via the built-in teardown reporter. No manual cleanup code is needed. See [Namespace Cleanup](/guide/core-concepts/playwright-fixtures#namespace-cleanup-teardown) for details. ::: +## Skipping and Disabling Tests + +| Mechanism | When to use | Report | +|-----------|-------------|--------| +| `{ tag: "@skip-ocp-helm" }` | Test works, but skip in a specific CI job | Hidden from that job and report | +| `test.fixme()` | Test is broken or flaky, needs fixing | Shows as "skipped" everywhere | +| `test.skip(condition, reason)` | Skip based on runtime (e.g., missing secret) | Shows as "skipped" with reason | + +### Tagging Tests + +Use Playwright tags to control which tests run in which CI jobs. Tags are added via the second argument to `test.describe` or `test`: + +```typescript +// Skip this suite in the ocp-helm-nightly CI job +test.describe("My Plugin", { tag: "@skip-ocp-helm-nightly" }, () => { + test("test case", async () => { ... }); +}); + +// Skip a single test in ocp-helm jobs (both PR and nightly) +test("expensive test", { tag: "@skip-ocp-helm" }, async () => { ... }); + +// Multiple tags +test.describe("Suite", { + tag: ["@skip-ocp-helm-nightly", "@skip-ocp-operator-nightly"], +}, () => { ... }); +``` + +When running in CI, `run-e2e.sh` auto-derives the skip tag from `JOB_NAME` and passes `--grep-invert` to Playwright. From a workspace directory, pass it manually: + +```bash +yarn test -- --grep-invert "@skip-ocp-helm" +``` + +See [Skip Tags](/overlay/reference/run-e2e#skip-tags) for the full list of tags and derivation logic. + ## Imports Import test utilities from `@red-hat-developer-hub/e2e-test-utils`: diff --git a/docs/overlay/tutorials/ci-pipeline.md b/docs/overlay/tutorials/ci-pipeline.md index 29fd474..cde78e0 100644 --- a/docs/overlay/tutorials/ci-pipeline.md +++ b/docs/overlay/tutorials/ci-pipeline.md @@ -93,6 +93,39 @@ To test nightly job changes via the release repo: 2. Comment `/test e2e-ocp-helm-nightly` on the PR 3. The rehearse job runs with `JOB_MODE=nightly`, testing all workspaces +## Skipping Tests by CI Job + +Tests can opt out of specific CI jobs using Playwright tags. The `run-e2e.sh` script auto-derives a skip tag from `JOB_NAME` and passes `--grep-invert` to Playwright. + +For example, to skip a test in the nightly helm job: + +```typescript +test.describe("My Plugin", { tag: "@skip-ocp-helm-nightly" }, () => { + // This entire suite is excluded when JOB_NAME contains "e2e-ocp-helm-nightly" +}); +``` + +### Available Tags + +| Tag | Skipped in | Not skipped in | +|-----|-----------|----------------| +| `@skip-ocp-helm` | `e2e-ocp-helm` (PR check) | `e2e-ocp-helm-nightly` | +| `@skip-ocp-helm-nightly` | `e2e-ocp-helm-nightly` | `e2e-ocp-helm` (PR check) | +| `@skip-ocp-operator` | `e2e-ocp-operator` (PR check) | `e2e-ocp-operator-nightly` | +| `@skip-ocp-operator-nightly` | `e2e-ocp-operator-nightly` | `e2e-ocp-operator` (PR check) | + +Each tag matches exactly — `@skip-ocp-helm` won't accidentally skip tests in the nightly job. + +Multiple tags can be combined on a single test or describe block: + +```typescript +test.describe("Suite", { + tag: ["@skip-ocp-helm-nightly", "@skip-ocp-operator-nightly"], +}, () => { ... }); +``` + +See [Skip Tags](/overlay/reference/run-e2e#skip-tags) for the full derivation logic and how to use tags from a workspace directory. + ## PR OCI Image Builds When testing a PR, the plugins need to be built into OCI images that RHDH can use. This is handled through the `/publish` command.