CLOUDP-295785 - staging support for image building #336

MaciejKaras · 2025-08-11T19:00:03Z

Summary

PR integrates staging build scenario with atomic_pipeline.py and the e2e tests. staging repositories are created in AWS ECR under /staging dir. List of new staging/ repositories:

Example AWS page with details:

Previously all of the evergreen patch builds were targeting 268558157000.dkr.ecr.us-east-1.amazonaws.com/dev repository. Now, depending on the BUILD_SCENARIO environment variable images will be pushed to and pulled from
different registry:

PATCH_REGISTRY="268558157000.dkr.ecr.us-east-1.amazonaws.com/dev"
STAGING_REGISTRY="268558157000.dkr.ecr.us-east-1.amazonaws.com/staging"
RELEASE_REGISTRY="quay.io/mongodb"

Additional changes:

re-added missing latest tag for staging builds. This is used for local testing
moved all registry variables declarations to root-context. Previously they were duplicated in both scripts/dev/contexts/evg-private-context and scripts/dev/contexts/local-defaults-context
fixes multi arch smoke tests configuration. We can now run them also for released images (tbd)
started to use OPERATOR_VERSION instead of VERSION_ID. VERSION_ID was only related to patch builds, which did not apply for staging and release builds
BASE_REPO_URL_SHARED is now 268558157000.dkr.ecr.us-east-1.amazonaws.com/staging because all latest tags are pushed on master builds
build_scenario now needs to be passed explicitly to pipeline.py as well as version (apart from agent image)

Proof of Work

Building all images is passing for staging scenario -> https://spruce.mongodb.com/version/68c7ce53672666000716401a/tasks?page=0&sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC

Building all images is passing for release scenario (apart from kubectl which requires tag) -> https://spruce.mongodb.com/version/68c911ef8d38f100074fa2db/tasks?sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC

Example image stored together with signature -> https://us-east-1.console.aws.amazon.com/ecr/repositories/private/268558157000/staging/mongodb-kubernetes-init-database?region=us-east-1

Remaining work

Add guide on how to run development scenario for building the images using evergreen and directly pipeline.py
Fix dummy version in the release_info.json - this is already planned in here

Checklist

Have you linked a jira ticket and/or is the ticket in the title?
Have you checked whether your jira ticket required DOCSP changes?
Have you added changelog file?
- use skip-changelog label if not needed
- refer to Changelog files and Release Notes section in CONTRIBUTING.md for more details

github-actions · 2025-08-11T19:00:46Z

⚠️ (this preview might not be accurate if the PR is not rebased on current master branch)

MCK 1.5.0 Release Notes

New Features

Improve automation agent certificate rotation: the agent now restarts automatically when its certificate is renewed, ensuring smooth operation without manual intervention and allowing seamless certificate updates without requiring manual Pod restarts.

Bug Fixes

MongoDBMultiCluster: fix resource stuck in Pending state if any clusterSpecList item has 0 members. After the fix, a value of 0 members is handled correctly, similarly to how it's done in the MongoDB resource.
MultiClusterSharded: Blocked removing non-zero member cluster from MongoDB resource. This prevents from scaling down member cluster without current configuration available, which could lead to unexpected issues.

scripts/release/build/build_scenario.py

scripts/dev/contexts/evg-private-context

scripts/dev/contexts/variables/om60

scripts/dev/contexts/local-defaults-context

build_info.json

scripts/dev/contexts/local-defaults-context

scripts/release/build/build_scenario.py

build_info.json

scripts/release/atomic_pipeline.py

scripts/release/build/build_scenario.py

scripts/release/release_info_test.py

scripts/release/build/build_scenario.py

scripts/dev/contexts/evg-private-context

build_info.json

…344) # Summary ⚠️ **Important notice** This PR contains some changes from #336, but they are not used yet and don't impact the PRs or patches. They are included because previously this PR was stacked on the staging PR and it is much easier to include them. The changes that are included: - `latest_tag` support - this is needed for staging builds, but like mentioned earlier, staging builds are not yet used - replace `268558157000.dkr.ecr.us-east-1.amazonaws.com/dev` with `BASE_REPO_URL`. This will be used to distinguish different repo urls: dev, staging and release. Currently hardcoded to `268558157000.dkr.ecr.us-east-1.amazonaws.com/dev` --- **This change is made to unblock the release of MCK 1.3.0. It is not final state of the release mechanism and most of it will be replaced by image promotion process.** Created new `.evergreen-release.yml` file that contains all release tasks including integration with `kubectl-mongodb` plugin release task. All of the variants are triggered only when `github_tag` is added. Additional changes: - each released image will be also released with additional `olm_tag` that has dynamic timestamp part. It will prevent accidental overriding the tags used by OLM. The tag syntax is `{version}-olm-{timestamp_suffix}` where timestamp suffix is in `%Y%m%d%H%M%S` format - created separate `release_operator_pipeline` evergreen function that uses `release` build scenario and version provided by `git_tag` - fixed and bumped preflight script ## Proof of Work List of tasks that are triggered when doing manual patch: <img width="2036" height="1017" alt="Screenshot 2025-09-03 at 11 00 16" src="https://github.com/user-attachments/assets/b3e7e707-3929-4f88-bc4f-2f998a16482a" /> ⚠️ This PR was tested by running evergreen command locally: ``` sudo evergreen patch -p mongodb-kubernetes -a release -d "Release test" -f -y -u --browse --path .evergreen.yml --param RELEASE_OPERATOR_VERSION=1.3.0-rc ``` Link to evg job -> https://spruce.mongodb.com/version/68b81b45285a950007bc8398 ## Checklist - [x] Have you linked a jira ticket and/or is the ticket in the title? - [x] Have you checked whether your jira ticket required DOCSP changes? - [x] Have you added changelog file? - use `skip-changelog` label if not needed - refer to [Changelog files and Release Notes](https://github.com/mongodb/mongodb-kubernetes/blob/master/CONTRIBUTING.md#changelog-files-and-release-notes) section in CONTRIBUTING.md for more details

…ongodb#344) # Summary ⚠️ **Important notice** This PR contains some changes from mongodb#336, but they are not used yet and don't impact the PRs or patches. They are included because previously this PR was stacked on the staging PR and it is much easier to include them. The changes that are included: - `latest_tag` support - this is needed for staging builds, but like mentioned earlier, staging builds are not yet used - replace `268558157000.dkr.ecr.us-east-1.amazonaws.com/dev` with `BASE_REPO_URL`. This will be used to distinguish different repo urls: dev, staging and release. Currently hardcoded to `268558157000.dkr.ecr.us-east-1.amazonaws.com/dev` --- **This change is made to unblock the release of MCK 1.3.0. It is not final state of the release mechanism and most of it will be replaced by image promotion process.** Created new `.evergreen-release.yml` file that contains all release tasks including integration with `kubectl-mongodb` plugin release task. All of the variants are triggered only when `github_tag` is added. Additional changes: - each released image will be also released with additional `olm_tag` that has dynamic timestamp part. It will prevent accidental overriding the tags used by OLM. The tag syntax is `{version}-olm-{timestamp_suffix}` where timestamp suffix is in `%Y%m%d%H%M%S` format - created separate `release_operator_pipeline` evergreen function that uses `release` build scenario and version provided by `git_tag` - fixed and bumped preflight script ## Proof of Work List of tasks that are triggered when doing manual patch: <img width="2036" height="1017" alt="Screenshot 2025-09-03 at 11 00 16" src="https://github.com/user-attachments/assets/b3e7e707-3929-4f88-bc4f-2f998a16482a" /> ⚠️ This PR was tested by running evergreen command locally: ``` sudo evergreen patch -p mongodb-kubernetes -a release -d "Release test" -f -y -u --browse --path .evergreen.yml --param RELEASE_OPERATOR_VERSION=1.3.0-rc ``` Link to evg job -> https://spruce.mongodb.com/version/68b81b45285a950007bc8398 ## Checklist - [x] Have you linked a jira ticket and/or is the ticket in the title? - [x] Have you checked whether your jira ticket required DOCSP changes? - [x] Have you added changelog file? - use `skip-changelog` label if not needed - refer to [Changelog files and Release Notes](https://github.com/mongodb/mongodb-kubernetes/blob/master/CONTRIBUTING.md#changelog-files-and-release-notes) section in CONTRIBUTING.md for more details

nammn

awesome work! Just some questions.
I wonder whether we can make things more straighforward than the following:

1. evg-private-context -> bash script and context which decides which env vars to use
1. .evergreen.yml -> calls pipeline.sh
1. pipeline.sh -> calls pipeline.py

i wonder whether we can simplify/unify things a bit more. Maybe joining 2 and 3 or extract the decision logic from evg-private-context into 2.

WDYT?

scripts/dev/contexts/variables/e2e_smoke_base

scripts/release/pipeline_ops_manager.sh

scripts/release/release_info.py

scripts/release/retag_images_to_staging.sh

nammn · 2025-09-17T14:54:08Z

EVERGREEN.md

@@ -0,0 +1,43 @@
+# Evergreen CI/CD Configuration Guide


I believe we will have all the documantion on our ci/cd in a google doc (?) if yes - we should link to that one instead. Otherwise - we will scatter things around a lot of places and things will go even more stale

I thought about having all CI/CD documentation in the codebase tbh. That is why I have started this doc.

problem is that ci/cd (evg) is only viewable and available internally. Not much external contributors can rely on + we will have release related things in google docs (right?) and this here - I think this will make things more difficult. I rather have that md file really short and just link to the google doc you are writing/have prepared

The release guide already exists, but is has less to do with evergreen CI/CD documentation. This EVERGREEN.md files describes the structure, aliases and usage of evergreen in our project. I really think it is beneficial to keep it close to code, so important changes in the Evergreen configuration/usage can be easily documented.

I think there was a discussion with @viveksinghggits on where to put things. The content is great, but I want us to be consistent with what everyone decided

I think that we agreed to link the EVERGREEN.md in the WiKi ToC and in related documents.

scripts/release/build/build_scenario.py

.evergreen.yml

scripts/dev/contexts/e2e_smoke_ibm_power

scripts/dev/contexts/root-context

scripts/release/atomic_pipeline.py

scripts/release/release_info.py

mircea-cosbuc · 2025-09-19T11:32:07Z

scripts/release/retag_images_to_staging.sh

+
+    if [[ -n "${digest_arm64}" ]]; then
+      docker pull "${source_image}@${digest_arm64}"
+      docker tag "${source_image}@${digest_arm64}" "${target_image}-arm64"


Why not skopeo copy ?

Good question. I didn't, but Łukasz did and he experienced issues trying skopeo and crane as well.

I've tested now that using skopeo copy --preserve-digests --all works. I will try to use it in the promotion pipeline later. After that this script will not be needed anymore

Got it, this should be mandatory for the promotion process (or an equivalent that preserves digests).

Do we still need this script as an intermediary step to the promotion pipeline? I assume yes, just making sure.

This script is not used in promoting the images, just for initial supply of the OM images in staging repo. I've added a comment on the top of the script:

# Utility used to retag and push container images from one repo to another # Useful for migrating images between different repositories (e.g. dev -> staging)

viveksinghggits

LGTM, just left a small comment.

viveksinghggits · 2025-09-23T11:20:43Z

EVERGREEN.md

Hi @MaciejKaras ,
Are you doing to add the section that we talked about, about how to load build info for a particular scenario? For example if I want to do something similar to

build_scenario = BuildScenario.infer_scenario_from_environment() kubectl_plugin_build_info = load_build_info(build_scenario).binaries[KUBECTL_PLUGIN_BINARY_NAME]

How would I do that now?

You should expose the build_scenario from your script and user should provide it. In evg-private-context there is BUILD_SCENARIO env variable set that you can use when calling the script in evg task. For local runs you can make it a default Development value

Ok, makes sense. So we will have to two evg variants one will be run in PR patches and one will be run in staging. If I am using the same script in these variants, I can make my script accept the build_scenario. Which can even be hardcoded in the variants. Please correct me if I am wrong.
Also, are we going to document this? Or you documenting it doesn't make much sense.

we could consider to use a script that just returns the env vars for build scenarios. evg-private-context can just and any other user the same way. It follow my suggestion in the call to extract that logic out to be re-usable

So we will have to two evg variants one will be run in PR patches and one will be run in staging. If I am using the same script in these variants, I can make my script accept the build_scenario. Which can even be hardcoded in the variants. Please correct me if I am wrong.

@viveksinghggits You don't need separate variants for PR patches and staging builds. Just call the switch_context evg function and it will create .generated context files with the BUILD_SCENARIO environment variable available for you. This is what we do in all other tasks, including e2e tests or image building tasks. I believe clone task already sets up context files so you just need to read it in your script.

we could consider to use a script that just returns the env vars for build scenarios. evg-private-context can just and any other user the same way. It follow my suggestion in the call to extract that logic out to be re-usable

@nammn We could extract that logic, I agree. But still we need BUILD_SCENARIO to be provided and it's currently calculated in evg-private-context and it only makes sense to calculate it there. The evg variables that are used to calculate BUILD_SCENARIO are not present locally.

EVERGREEN.md

# Conflicts: # scripts/dev/contexts/public_kind_code_snippets

# Conflicts: # scripts/dev/contexts/e2e_mdb_kind_ubi_cloudqa # scripts/dev/contexts/e2e_smoke_ibm_power # scripts/dev/contexts/e2e_smoke_ibm_z # scripts/dev/contexts/e2e_static_mdb_kind_ubi_cloudqa # scripts/dev/contexts/e2e_static_smoke_arm # scripts/dev/contexts/e2e_static_smoke_ibm_power # scripts/dev/contexts/e2e_static_smoke_ibm_z # scripts/dev/contexts/variables/om60_image # scripts/dev/contexts/variables/om70_image # scripts/dev/contexts/variables/om80_image

viveksinghggits

LGTM

scripts/dev/contexts/evg-private-context

# Conflicts: # scripts/dev/contexts/migrate_all_agents # scripts/dev/contexts/preflight_release_images # scripts/dev/contexts/preflight_release_images_check_only # scripts/dev/contexts/release_agent

…71-1` used in tests

…ging # Conflicts: # .evergreen-functions.yml

# Conflicts: # .evergreen-functions.yml

MaciejKaras changed the base branch from master to maciejk/use-ar-related-versions August 11, 2025 19:00

MaciejKaras commented Aug 11, 2025

View reviewed changes

scripts/release/build/build_scenario.py Outdated Show resolved Hide resolved

MaciejKaras commented Aug 11, 2025

View reviewed changes

scripts/dev/contexts/evg-private-context Outdated Show resolved Hide resolved

MaciejKaras added the skip-changelog Use this label in Pull Request to not require new changelog entry file label Aug 11, 2025

Base automatically changed from maciejk/use-ar-related-versions to master August 12, 2025 10:58

MaciejKaras commented Aug 13, 2025

View reviewed changes

scripts/dev/contexts/variables/om60 Outdated Show resolved Hide resolved

MaciejKaras commented Aug 13, 2025

View reviewed changes

scripts/dev/contexts/local-defaults-context Show resolved Hide resolved

MaciejKaras marked this pull request as ready for review August 13, 2025 12:55

MaciejKaras requested a review from a team as a code owner August 13, 2025 12:55

MaciejKaras requested review from m1kola, lucian-tosa, mircea-cosbuc, nammn, Julien-Ben and viveksinghggits and removed request for m1kola and lucian-tosa August 13, 2025 12:55

mircea-cosbuc reviewed Aug 13, 2025

View reviewed changes

build_info.json Show resolved Hide resolved

scripts/dev/contexts/local-defaults-context Show resolved Hide resolved

scripts/release/build/build_scenario.py Outdated Show resolved Hide resolved

build_info.json Outdated Show resolved Hide resolved

Julien-Ben reviewed Aug 13, 2025

View reviewed changes

scripts/release/atomic_pipeline.py Outdated Show resolved Hide resolved

MaciejKaras commented Aug 13, 2025

View reviewed changes

scripts/release/build/build_scenario.py Show resolved Hide resolved

nammn reviewed Aug 13, 2025

View reviewed changes

scripts/release/release_info_test.py Outdated Show resolved Hide resolved

scripts/release/build/build_scenario.py Outdated Show resolved Hide resolved

scripts/dev/contexts/evg-private-context Outdated Show resolved Hide resolved

build_info.json Show resolved Hide resolved

This was referenced Aug 29, 2025

Enable staging build scenario that was commented out by default #362

Closed

CLOUDP-295785 - release tasks integration with atomic_pipeline.py #344

Merged

MaciejKaras marked this pull request as draft September 3, 2025 17:25

MaciejKaras force-pushed the maciejk/ar-image-staging branch from 1801c66 to dcd030d Compare September 3, 2025 17:26

MaciejKaras force-pushed the maciejk/ar-image-staging branch from 439da21 to a68d85e Compare September 9, 2025 08:58

MaciejKaras changed the base branch from master to maciejk/ar-evg-aliases September 9, 2025 08:58

MaciejKaras added 2 commits September 17, 2025 11:09

Merge branch 'master' into maciejk/ar-image-staging

dd3345b

Fix missing overrides for repositories

f61dba9

MaciejKaras force-pushed the maciejk/ar-image-staging branch from ea0e23a to f61dba9 Compare September 17, 2025 11:02

Merge branch 'master' into maciejk/ar-image-staging

39840fc

MaciejKaras force-pushed the maciejk/ar-image-staging branch from f593df1 to 39840fc Compare September 17, 2025 12:17

nammn reviewed Sep 17, 2025

View reviewed changes

Julien-Ben reviewed Sep 18, 2025

View reviewed changes

scripts/release/build/build_scenario.py Show resolved Hide resolved

mircea-cosbuc reviewed Sep 19, 2025

View reviewed changes

MaciejKaras added 2 commits September 22, 2025 10:36

Fixes for review comments

1f99905

Add comment on dummy version

1d66ef8

Julien-Ben approved these changes Sep 23, 2025

View reviewed changes

viveksinghggits reviewed Sep 23, 2025

View reviewed changes

EVERGREEN.md Show resolved Hide resolved

MaciejKaras requested review from mircea-cosbuc, viveksinghggits and nammn September 30, 2025 10:48

MaciejKaras added 4 commits September 30, 2025 14:39

Move meko-tests.release to quay.io/mongodb/mongodb-kubernetes-tests

a5bbb9f

Merge branch 'master' into maciejk/ar-image-staging

3d4c986

# Conflicts: # scripts/dev/contexts/public_kind_code_snippets

Don't build agents on each PR

5d6a716

viveksinghggits reviewed Oct 7, 2025

View reviewed changes

scripts/dev/contexts/evg-private-context Show resolved Hide resolved

scripts/dev/contexts/evg-private-context Show resolved Hide resolved

viveksinghggits approved these changes Oct 8, 2025

View reviewed changes

MaciejKaras added 6 commits October 8, 2025 15:59

Fix switch-context issues + context cleanup

9b40c45

Merge branch 'master' into maciejk/ar-image-staging

5634ce5

# Conflicts: # scripts/dev/contexts/migrate_all_agents # scripts/dev/contexts/preflight_release_images # scripts/dev/contexts/preflight_release_images_check_only # scripts/dev/contexts/release_agent

Remove obsolete 108.0.0.8694-1 agent version + bumping `107.0.17.87…

9a7f17d

…71-1` used in tests

Fix switch-context issues + context cleanup

655cc54

Merge branch 'maciejk/refac-context-switch' into maciejk/ar-image-sta…

1934fae

…ging # Conflicts: # .evergreen-functions.yml

Merge branch 'master' into maciejk/ar-image-staging

b406574

# Conflicts: # .evergreen-functions.yml

MaciejKaras merged commit 749c1a3 into master Oct 9, 2025
29 of 37 checks passed

MaciejKaras deleted the maciejk/ar-image-staging branch October 9, 2025 16:40

CLOUDP-295785 - staging support for image building #336

CLOUDP-295785 - staging support for image building #336

Uh oh!

Conversation

MaciejKaras commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Proof of Work

Remaining work

Checklist

Uh oh!

github-actions bot commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

MCK 1.5.0 Release Notes

New Features

Bug Fixes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nammn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaciejKaras Sep 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viveksinghggits left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaciejKaras commented Aug 11, 2025 •

edited

Loading

github-actions bot commented Aug 11, 2025 •

edited

Loading

MaciejKaras Sep 22, 2025 •

edited

Loading

MaciejKaras Sep 30, 2025 •

edited

Loading