-
Notifications
You must be signed in to change notification settings - Fork 17
HYPERFLEET-872 - docs: add Konflux release pipeline design and ADR #128
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
ciaranRoche
wants to merge
3
commits into
main
Choose a base branch
from
HYPERFLEET-872-konflux-pipeline-docs
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
cc5d86c
HYPERFLEET-872 - docs: add Konflux release pipeline design, ADR, and …
ciaranRoche 8db7c2d
HYPERFLEET-872 - chore: reorganise release docs into docs/release/
ciaranRoche 7b00206
HYPERFLEET-872 - docs: normalise release timelines to working days
ciaranRoche File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| --- | ||
| Status: Active | ||
| Owner: Ciaran Roche | ||
| Last Updated: 2026-04-21 | ||
| --- | ||
|
|
||
| # 0013 — Konflux for Container Image Build and Release | ||
|
|
||
| ## Context | ||
|
|
||
| HyperFleet builds container images for three components (API, Sentinel, Adapter) using Prow's ci-operator, publishing to `registry.ci.openshift.org`. This setup lacks supply chain security: no signed provenance records, no SBOM generation, no image signing. | ||
|
|
||
| Red Hat requires all services to use Konflux for image builds to achieve SLSA Level 3 compliance. Konflux provides provenance via Tekton Chains, automatic SBOM generation, image signing with cosign, and Enterprise Contract policy enforcement — capabilities Prow does not offer. | ||
|
|
||
| Meanwhile, HyperFleet's E2E testing infrastructure runs on Prow with GKE clusters, GCP credentials, and an established test framework that would be costly and complex to replicate in Konflux. | ||
|
|
||
| ## Decision | ||
|
|
||
| Adopt a hybrid model: **Konflux handles image build and release, Prow retains E2E testing.** | ||
|
|
||
| The architecture has three layers: | ||
|
|
||
| - **PaC (Pipelines as Code)** — trigger layer. Watches GitHub webhooks and matches git events (push to main, version tags) to `.tekton/` pipeline files in each component repo. | ||
| - **Konflux** — orchestration layer. Builds images, generates SBOM, signs provenance (Tekton Chains), runs Enterprise Contract validation, and releases to `quay.io/redhat-services-prod/hyperfleet/`. | ||
| - **Prow** — test layer. Runs nightly and RC E2E tests against Konflux-built images from Quay. Retains PR presubmit checks (lint, unit tests). | ||
|
|
||
| Key design choices: | ||
|
|
||
| - One Application (`hyperfleet`) with three Components, releasing through a single ReleasePlanAdmission with auto-release (`block-releases: false`) | ||
| - Tag-based triggers using CEL expressions (not globs) to distinguish RC tags from release tags | ||
| - `app-interface-standard` Enterprise Contract policy for service-type releases | ||
| - RC E2E triggered via `hyperfleet-release` repo tag -> GitHub Action -> Gangway -> Prow | ||
| - `rh-push-to-external-registry` release pipeline with Pyxis registration and Slack notifications | ||
|
|
||
| See [Konflux Release Pipeline Design](../docs/release/konflux-release-pipeline-design.md) for the full design document. | ||
|
|
||
| ## Consequences | ||
|
|
||
| **Gains:** | ||
|
|
||
| - SLSA Level 3 supply chain security (provenance, SBOM, image signing) with zero extra configuration | ||
| - Simplified RC testing — one reusable Prow job with parameterized image coordinates replaces per-release-branch pipeline definitions | ||
| - Auto-release on merge to main eliminates manual image publishing | ||
| - Public Quay images remove pull secret complexity for Prow and partner environments | ||
| - Alignment with Red Hat release engineering standards | ||
|
|
||
| **Trade-offs:** | ||
|
|
||
| - Two build systems during transition (Konflux for images, Prow for PR validation and E2E) | ||
| - RC E2E triggering depends on Gangway API availability | ||
| - Team must learn Konflux primitives (RPA, ITS, PaC, Snapshots) | ||
|
|
||
| ## Alternatives Considered | ||
|
|
||
| | Alternative | Why Rejected | | ||
| |-------------|--------------| | ||
| | Full Prow (status quo) | No path to SLSA compliance. No provenance, SBOM, or signing. Does not meet Red Hat release engineering requirements. | | ||
| | Full Konflux (including E2E) | Would require replicating GKE cluster provisioning, GCP credentials, and the entire hyperfleet-e2e framework inside Konflux ITS pipelines. High cost, low benefit — Prow already works. | | ||
| | Hybrid with ITS wrapper for E2E | An IntegrationTestScenario that triggers Prow via Gangway and polls for results. Adds complexity (polling, timeouts, error handling) as a Tekton pipeline when a simpler GitHub Action achieves the same result with better DX (retrigger via `gh workflow run`). Deferred to fast-follow if automated gating is needed. | | ||
| | Separate RPAs per build context | One RPA for nightly (auto-release) + one for releases (gated via `block-releases`). Rejected because `block-releases` is per-RPA not per-Snapshot — every build from the Application would create blocked releases on the gated RPA, generating noise without meaningful safety. | | ||
| | Glob patterns for tag matching | PaC glob patterns (e.g., `v*`) cannot distinguish `v1.0.0-rc1` from `v1.0.0`. CEL expressions with regex anchoring are required. Verified against PaC source code and hypershift production configuration. | | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| # Release Documentation | ||
|
|
||
| ## Start Here | ||
|
|
||
| | Document | Purpose | | ||
| |----------|---------| | ||
| | [Konflux Release Pipeline Design](./konflux-release-pipeline-design.md) | How the build and release pipeline works — architecture, flows, and design decisions | | ||
| | [HyperFleet Release Process](./hyperfleet-release-process.md) | The release process — cadence, checklists, branching, bug handling, hotfixes | | ||
| | [Glossary](./glossary.md) | Definitions of terms used across the release docs | | ||
| | [ADR 0013](../../adrs/0013-konflux-build-and-release.md) | Decision record for adopting Konflux | | ||
|
|
||
| ## Prow Test and Release | ||
|
|
||
| The `test-release/` subdirectory contains Prow-specific docs for CI job setup and E2E testing infrastructure. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| --- | ||
| Status: Active | ||
| Owner: HyperFleet Team | ||
| Last Updated: 2026-04-21 | ||
| --- | ||
|
|
||
| # Release Glossary | ||
|
|
||
| > **Audience:** HyperFleet developers, release owners, and anyone reading the release documentation. | ||
|
|
||
| Key terms used across the release docs. Written for someone encountering these systems for the first time. | ||
|
|
||
| --- | ||
|
|
||
| ## Konflux and Build Pipeline | ||
|
|
||
| | Term | Definition | | ||
| |------|------------| | ||
| | **Konflux** | Red Hat's build and release platform. Orchestrates container image builds, signing, policy validation, and registry publishing. | | ||
| | **PaC** (Pipelines as Code) | Trigger layer. Watches GitHub webhooks and matches events to `.tekton/` pipeline files in component repos via annotations. | | ||
| | **Tekton** | Kubernetes-native pipeline framework used by Konflux. Defines Pipelines, Tasks, and PipelineRuns as custom resources. | | ||
| | **Tekton Chains** | Controller that automatically signs provenance records (in-toto format) with cosign after every build. Zero configuration needed. | | ||
| | **Snapshot** | Immutable record of all component images at specific commits. Created after each build. Releases operate on Snapshots, not individual images. | | ||
| | **RPA** (ReleasePlanAdmission) | Defines how releases are processed: component-to-registry mapping, tags, EC policy, release pipeline. Lives in `konflux-release-data`. | | ||
| | **ITS** (IntegrationTestScenario) | A test pipeline that runs against a Snapshot before release. Mandatory ITS blocks release on failure. | | ||
| | **EC** (Enterprise Contract) | Policy engine validating container images meet compliance requirements. Checks provenance, acceptable bundles, base images. | | ||
| | **CEL** (Common Expression Language) | Used in PaC annotations for event matching. RE2 regex via `matches()`. Required for distinguishing RC from release tags. | | ||
| | **Provenance** | Signed record proving how and where a container image was built. Enables verification that an image came from a specific commit via Konflux's pipeline. | | ||
| | **SBOM** (Software Bill of Materials) | Machine-readable inventory of all packages in a container image. Generated automatically by Konflux builds. | | ||
| | **SLSA** (Supply-chain Levels for Software Artifacts) | Framework for build integrity. Konflux achieves Level 3 by default — builds on hardened infrastructure with signed provenance. | | ||
|
|
||
| ## Prow and Testing | ||
|
|
||
| | Term | Definition | | ||
| |------|------------| | ||
| | **Prow** | Kubernetes-based CI system used by OpenShift. Runs PR validation, E2E tests, and periodic jobs. | | ||
| | **ci-operator** | OpenShift CI tool that orchestrates container image builds and test execution. Configured via files in the `openshift/release` repo. | | ||
| | **presubmit** | Prow job triggered on PR creation. Runs lint, unit tests, and build checks. Must pass before merge. | | ||
| | **postsubmit** | Prow job triggered after code is merged to main. Runs post-merge validations. | | ||
| | **periodic** | Prow job triggered on a cron schedule. Used for nightly E2E tests and regression testing. | | ||
| | **Gangway** | Prow API that allows external services to trigger Prow jobs programmatically. Used for RC E2E triggering via GitHub Actions. | | ||
| | **E2E** (End-to-End) | Tests that validate complete user workflows across all system components deployed to a real environment. | | ||
|
|
||
| ## Release Process | ||
|
|
||
| | Term | Definition | | ||
| |------|------------| | ||
| | **RC** (Release Candidate) | Pre-release version tagged for validation before GA. Example: `v1.5.0-rc1`. Requires full E2E testing before promotion. | | ||
| | **GA** (General Availability) | Final production release version. Example: `v1.5.0`. Signals production-ready status to partner teams. | | ||
| | **Feature Freeze** | Point in the release cycle when the release branch is created. No new features after this — only bug fixes. | | ||
| | **Code Freeze** | Point after stabilization when only Blocker/Critical fixes are permitted. Strict change control with Release Owner approval. | | ||
| | **Release Owner** | Person responsible for coordinating the release: approving cherry-picks, tracking bugs, ensuring release criteria are met. | | ||
| | **RELEASE_MANIFEST.yaml** | File in `hyperfleet-release` repo tracking which component versions constitute a validated release. Updated before each RC and GA tag. | | ||
| | **Cherry-Pick** | Applying a specific commit from one branch to another. HyperFleet fixes main first, then cherry-picks to the release branch. | | ||
| | **Hotfix** | Urgent fix applied to a released version outside the normal release cycle. Patches skip RC — straight to a patch tag. | | ||
| | **N-1 Compatibility** | Supporting one version back (e.g., v1.5 compatible with v1.4). | | ||
|
|
||
| ## Infrastructure | ||
|
|
||
| | Term | Definition | | ||
| |------|------------| | ||
| | **stone-prd-rh01** | Konflux production cluster hosted by Red Hat. Runs the PaC controller, Tekton Chains, and release pipelines for HyperFleet. | | ||
| | **Quay** | Red Hat's container registry. HyperFleet images are published to `quay.io/redhat-services-prod/hyperfleet/`. | | ||
| | **Pyxis** | Red Hat's container metadata catalog. Images registered here are automatically scanned for CVEs. | | ||
| | **ArgoCD** | GitOps-based deployment tool. Partner teams use ArgoCD to consume HyperFleet images from Quay. | | ||
| | **konflux-release-data** | GitLab repo containing RPA, constraint, and tenant configuration for Konflux releases. Source of truth for release pipeline config. | |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.