Migrate to GitHub Actions #8835
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
scholtzan
force-pushed
the
migrate-to-github-actions-with-tests
branch
from
78627cb to
5985aba
Compare
Integration report for "Migrate to GitHub actions with tests"
|
Integration report for "Migrate to GitHub actions with tests"
|
Integration report for "Migrate to GitHub actions with tests"
|
scholtzan
changed the title
BenWu
reviewed
Feb 12, 2026
| pip install --no-deps -r requirements.txt | ||
| pip-sync --pip-args=--no-deps | ||
| - name: Save cached virtualenv | ||
| if: steps.cache-venv.outputs.cache-hit != 'true' |
Contributor
There was a problem hiding this comment.
I don't see a cache-venv step. Is that missing? This build job outputs a Cache save failed. warning which I'm guessing is because the key is already used by a previous build. Did you mean to attempt to pull from the cache before this?
| PATH=".venv/bin:$PATH" pip-compile --allow-unsafe --generate-hashes --quiet | ||
| git diff --exit-code -G '^ *[^# ]' -- requirements.txt | ||
|
|
||
| verify-format-yaml: |
Contributor
There was a problem hiding this comment.
The run page shows a bunch of yaml warnings in the annotations section which is useful but those should be fixed as part of this.
.github/workflows/build.yml
Outdated
| needs: [generate-dags, main-generate-sql-and-dags, decide-runs] | ||
| if: | | ||
| (github.ref != 'refs/heads/main') && | ||
| (needs.decide-runs.outputs.validate-sql == 'true' || needs.decide-runs.outputs.validate-routines == 'true' || needs.decide-runs.outputs.deploy == 'true') |
Contributor
There was a problem hiding this comment.
outputs.deploy = "${{ github.ref }}" == "refs/heads/main" so it would always be false
Suggested change
| (needs.decide-runs.outputs.validate-sql == 'true' || needs.decide-runs.outputs.validate-routines == 'true' || needs.decide-runs.outputs.deploy == 'true') | |
| (needs.decide-runs.outputs.validate-sql == 'true' || needs.decide-runs.outputs.validate-routines == 'true') |
|
|
||
| At this point, the table exists in Bigquery so you are able to: | ||
| - [Find and re-run the CI](https://app.circleci.com/pipelines/github/mozilla/bigquery-etl?) of your PR and make sure that all tests pass | ||
| - [Find and re-run the CI](https://github.com/mozilla/bigquery-etl/actions) of your PR and make sure that all tests pass |
Contributor
There was a problem hiding this comment.
The top-level actions page has a lot of other stuff in it which would make it harder to find a specific run
Suggested change
| - [Find and re-run the CI](https://github.com/mozilla/bigquery-etl/actions) of your PR and make sure that all tests pass | |
| - [Find and re-run the CI](https://github.com/mozilla/bigquery-etl/actions/workflows/build.yml) of your PR and make sure that all tests pass |
| 1. Open a new PR with these changes. | ||
| 1. PR reviewed and approved. | ||
| 1. Find and run again the [CI pipeline](https://app.circleci.com/pipelines/github/mozilla/bigquery-etl?) for the PR. | ||
| 1. Find and run again the [CI pipeline](https://github.com/mozilla/bigquery-etl/actions) for the PR. |
Contributor
There was a problem hiding this comment.
Suggested change
| 1. Find and run again the [CI pipeline](https://github.com/mozilla/bigquery-etl/actions) for the PR. | |
| 1. Find and run again the [CI pipeline](https://github.com/mozilla/bigquery-etl/actions/workflows/build.yml) for the PR. |
tests/test_entrypoint.py
Outdated
Comment on lines
36
to
37
| # running bq in CircleCI will fail since it's not installed | ||
| # in GitHub Actions, bq is installed but may fail for other reasons |
Contributor
There was a problem hiding this comment.
This reads like it's one (confusing) sentence
Suggested change
| # running bq in CircleCI will fail since it's not installed | |
| # in GitHub Actions, bq is installed but may fail for other reasons | |
| # Running bq in CircleCI will fail since it's not installed. | |
| # In GitHub Actions, bq is installed but may fail for other reasons |
tests/test_entrypoint.py
Outdated
Comment on lines
85
to
86
| # running bq in CircleCI will fail since it's not installed | ||
| # in GitHub Actions, bq is installed but may fail for other reasons |
Contributor
There was a problem hiding this comment.
Suggested change
| # running bq in CircleCI will fail since it's not installed | |
| # in GitHub Actions, bq is installed but may fail for other reasons | |
| # Running bq in CircleCI will fail since it's not installed. | |
| # In GitHub Actions, bq is installed but may fail for other reasons |
tests/test_entrypoint.py
Outdated
Comment on lines
141
to
142
| # running bq in CircleCI will fail since it's not installed | ||
| # in GitHub Actions, bq is installed but may fail for other reasons |
Contributor
There was a problem hiding this comment.
Suggested change
| # running bq in CircleCI will fail since it's not installed | |
| # in GitHub Actions, bq is installed but may fail for other reasons | |
| # Running bq in CircleCI will fail since it's not installed. | |
| # In GitHub Actions, bq is installed but may fail for other reasons |
Integration report for "Migrate to GitHub Actions"
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR migrates bigquery-etl to Github Actions. mozilla/telemetry-airflow#2339 needs to be merged alongside it as this will also change where docker images will get published to (GAR).
CircleCI will need to be disabled once this is merged.
I did test most of the CI tasks (also those that will just run on
main) in github.com/mozilla/bigquery-etl/actions/runs/21886512482 (except for syncing the DAGs submodules) and made some SQL changes there to trigger stage deploys. Some CI tests run a few seconds faster, some a few seconds slower compared to CircleCI (https://app.circleci.com/pipelines/github/mozilla/bigquery-etl/57677/workflows/61769702-d64a-49c9-a571-de1de5c9999a). Overall, the runtime seems to be about the same.An annoying thing with Github Actions is that it creates a deployment when accessing environment variables in the
devenvironment. The stage deploy credentials need to come from that environment (see https://github.com/mozilla/dataservices-infra/pull/1512#discussion_r2775354202). There is an open roadmap item to decouple deployments from using the environment variables: https://github.com/orgs/community/discussions/36919 but until this is implemented every time a CI task runs that accesses an environment variable it adds a message to the PR.Related Tickets & Documents
Reviewer, please follow this checklist