Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 48 additions & 1 deletion docs/overlay/reference/run-e2e.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,54 @@ A workspace is discovered when it has a `workspaces/<name>/e2e-tests/` directory
|----------|-------------|---------|
| `E2E_NIGHTLY_MODE` | When `true`, uses released OCI images from metadata; defaults `E2E_TEST_UTILS_VERSION` to `latest` | `false` |
| `GIT_PR_NUMBER` | PR number for OCI URL generation (uses PR-built images) | - |
| `JOB_NAME` | CI job name; if contains `periodic-`, disables metadata injection | - |
| `JOB_NAME` | CI job name; if contains `periodic-`, disables metadata injection. Also used to [auto-derive skip tags](#skip-tags). | - |

## Skip Tags

When `JOB_NAME` is set (by OpenShift CI), the script auto-derives a Playwright tag and injects `--grep-invert` to exclude tests tagged with it. This lets test authors skip specific tests in specific CI jobs using standard Playwright tags.

### How It Works

The job suffix is extracted from `JOB_NAME` by stripping everything up to and including `-e2e-`. A negative lookahead `(?!-)` is appended so each tag matches exactly — `@skip-ocp-helm` won't accidentally filter `@skip-ocp-helm-nightly`:

| JOB_NAME (suffix shown) | `--grep-invert` pattern |
|--------------------------|-------------------------|
| `...-e2e-ocp-helm` | `@skip-ocp-helm(?!-)` |
| `...-e2e-ocp-helm-nightly` | `@skip-ocp-helm-nightly(?!-)` |
| `...-e2e-ocp-operator` | `@skip-ocp-operator(?!-)` |
| `...-e2e-ocp-operator-nightly` | `@skip-ocp-operator-nightly(?!-)` |

If `JOB_NAME` doesn't contain `-e2e-`, no tag is derived and no filtering is applied.

### Tagging Tests

Add Playwright tags to `test.describe` or individual `test` calls:

```typescript
// Skip only in ocp-helm-nightly job
test.describe("My Plugin", { tag: "@skip-ocp-helm-nightly" }, () => { ... });

// Skip only in ocp-helm PR check job (won't affect nightly)
test("expensive test", { tag: "@skip-ocp-helm" }, async () => { ... });

// Multiple tags — skip in both helm and operator nightly
test.describe("Suite", {
tag: ["@skip-ocp-helm-nightly", "@skip-ocp-operator-nightly"],
}, () => { ... });
```

### From a Workspace Directory

Skip tags work the same way when running tests from a workspace — pass `--grep-invert` manually:

```bash
cd workspaces/tech-radar/e2e-tests
yarn test -- --grep-invert "@skip-ocp-helm"
```

::: info Precedence
The auto-derived `--grep-invert` is prepended to the Playwright arguments. If you also pass `--grep-invert` on the command line, Playwright uses the last value (last wins), so your manual flag takes precedence.
:::

## How It Works

Expand Down
35 changes: 35 additions & 0 deletions docs/overlay/test-structure/spec-files.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,41 @@ test.describe("Test <plugin>", () => {
In CI, namespaces are automatically deleted after all tests complete via the built-in teardown reporter. No manual cleanup code is needed. See [Namespace Cleanup](/guide/core-concepts/playwright-fixtures#namespace-cleanup-teardown) for details.
:::

## Skipping and Disabling Tests

| Mechanism | When to use | Report |
|-----------|-------------|--------|
| `{ tag: "@skip-ocp-helm" }` | Test works, but skip in a specific CI job | Hidden from that job and report |
| `test.fixme()` | Test is broken or flaky, needs fixing | Shows as "skipped" everywhere |
| `test.skip(condition, reason)` | Skip based on runtime (e.g., missing secret) | Shows as "skipped" with reason |

### Tagging Tests

Use Playwright tags to control which tests run in which CI jobs. Tags are added via the second argument to `test.describe` or `test`:

```typescript
// Skip this suite in the ocp-helm-nightly CI job
test.describe("My Plugin", { tag: "@skip-ocp-helm-nightly" }, () => {
test("test case", async () => { ... });
});

// Skip a single test in ocp-helm jobs (both PR and nightly)
test("expensive test", { tag: "@skip-ocp-helm" }, async () => { ... });

// Multiple tags
test.describe("Suite", {
tag: ["@skip-ocp-helm-nightly", "@skip-ocp-operator-nightly"],
}, () => { ... });
```

When running in CI, `run-e2e.sh` auto-derives the skip tag from `JOB_NAME` and passes `--grep-invert` to Playwright. From a workspace directory, pass it manually:

```bash
yarn test -- --grep-invert "@skip-ocp-helm"
```

See [Skip Tags](/overlay/reference/run-e2e#skip-tags) for the full list of tags and derivation logic.

## Imports

Import test utilities from `@red-hat-developer-hub/e2e-test-utils`:
Expand Down
33 changes: 33 additions & 0 deletions docs/overlay/tutorials/ci-pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,39 @@ To test nightly job changes via the release repo:
2. Comment `/test e2e-ocp-helm-nightly` on the PR
3. The rehearse job runs with `JOB_MODE=nightly`, testing all workspaces

## Skipping Tests by CI Job

Tests can opt out of specific CI jobs using Playwright tags. The `run-e2e.sh` script auto-derives a skip tag from `JOB_NAME` and passes `--grep-invert` to Playwright.

For example, to skip a test in the nightly helm job:

```typescript
test.describe("My Plugin", { tag: "@skip-ocp-helm-nightly" }, () => {
// This entire suite is excluded when JOB_NAME contains "e2e-ocp-helm-nightly"
});
```

### Available Tags

| Tag | Skipped in | Not skipped in |
|-----|-----------|----------------|
| `@skip-ocp-helm` | `e2e-ocp-helm` (PR check) | `e2e-ocp-helm-nightly` |
| `@skip-ocp-helm-nightly` | `e2e-ocp-helm-nightly` | `e2e-ocp-helm` (PR check) |
| `@skip-ocp-operator` | `e2e-ocp-operator` (PR check) | `e2e-ocp-operator-nightly` |
| `@skip-ocp-operator-nightly` | `e2e-ocp-operator-nightly` | `e2e-ocp-operator` (PR check) |

Each tag matches exactly — `@skip-ocp-helm` won't accidentally skip tests in the nightly job.

Multiple tags can be combined on a single test or describe block:

```typescript
test.describe("Suite", {
tag: ["@skip-ocp-helm-nightly", "@skip-ocp-operator-nightly"],
}, () => { ... });
```

See [Skip Tags](/overlay/reference/run-e2e#skip-tags) for the full derivation logic and how to use tags from a workspace directory.

## PR OCI Image Builds

When testing a PR, the plugins need to be built into OCI images that RHDH can use. This is handled through the `/publish` command.
Expand Down
Loading