Skip to content

Commit 735277b

Browse files
sl-at-ibmsl-at-ibm
committed
Introduce codecoverage combined calculation in CI
most (local) test coverage/reporting in place draft for codecov orchestrator TEMP reduce it scope for testing codecov relative path of workflows, fix workflows called with full syntax nope, revert to abs path and nothing more standardize yaml naming standardize yaml naming 2 secret inheritance chain overhaul add env to unit test job add env block to all testing flows trying to debug uploader: trying to debug uploader 2 trying to debug uploader 3 trying to debug uploader 4 trying to debug uploader 5 remove debug what is in the downloaded things anyway? coverage calculation and posting to PR, attempt 1 coverage calculation and posting to PR, attempt 2 coverage calculation and posting to PR, attempt 3 coverage calculation and posting to PR, attempt 4 coverage calculation and posting to PR, attempt 5 coverage calculation and posting to PR, attempt 6 coverage calculation and posting to PR, attempt 7 coverage calculation and posting to PR, attempt 8 publish html codecov report publish html codecov report 2 publish html codecov report 3 publish html codecov report 4 remove and postpone the 'gh pages html publish' thing minor coverage message rephrase html report retention workaround 1 html report retention workaround 2 reinstate full test; coverage json uploaded for push-to-main suspend farr-related tests on HCD until dev rer.serv. back in action revert the exclusion of farr in hcd tests, i.e. include them again
1 parent 1d81b09 commit 735277b

File tree

12 files changed

+660
-487
lines changed

12 files changed

+660
-487
lines changed

‎.github/workflows/codecov_aggregator.yml‎

Lines changed: 182 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,182 @@
1+
name: CI codecov aggregator
2+
on:
3+
push:
4+
branches: [main]
5+
pull_request:
6+
types: [opened, synchronize, reopened, closed]
7+
branches: [main]
8+
9+
permissions:
10+
# required to post on the PR
11+
pull-requests: write
12+
# required for the local IT (which pulls HCD from ECR)
13+
id-token: write
14+
contents: write
15+
# required to publish GH pages
16+
pages: write
17+
18+
concurrency:
19+
group: preview-pages-${{ github.ref }}
20+
cancel-in-progress: true
21+
22+
env:
23+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
24+
25+
jobs:
26+
main_it:
27+
uses: ./.github/workflows/main.yml
28+
if: ${{ github.event.action != 'closed' }}
29+
with:
30+
coverage_prefix: main_it
31+
sha: ${{ github.sha }}
32+
secrets:
33+
ASTRA_DB_APPLICATION_TOKEN: ${{ secrets.ASTRA_DB_APPLICATION_TOKEN }}
34+
ASTRA_DB_API_ENDPOINT: ${{ secrets.ASTRA_DB_API_ENDPOINT }}
35+
HEADER_EMBEDDING_API_KEY_OPENAI: ${{ secrets.HEADER_EMBEDDING_API_KEY_OPENAI }}
36+
LEGACY_INSERTMANY_BEHAVIOUR_PRE2193: "yes"
37+
38+
local_it:
39+
uses: ./.github/workflows/local.yml
40+
if: ${{ github.event.action != 'closed' }}
41+
with:
42+
coverage_prefix: local_it
43+
sha: ${{ github.sha }}
44+
secrets:
45+
AWS_ECR_REGION: ${{ secrets.AWS_ECR_REGION }}
46+
AWS_ECR_ACCOUNT_ID: ${{ secrets.AWS_ECR_ACCOUNT_ID }}
47+
AWS_ECR_ROLE_NAME: ${{ secrets.AWS_ECR_ROLE_NAME }}
48+
AWS_ECR_REPOSITORY: ${{ secrets.AWS_ECR_REPOSITORY }}
49+
AWS_ECR_REGISTRY: ${{ secrets.AWS_ECR_REGISTRY }}
50+
HEADER_EMBEDDING_API_KEY_OPENAI: ${{ secrets.HEADER_EMBEDDING_API_KEY_OPENAI }}
51+
# hardcoding the target DB
52+
DOCKER_COMPOSE_LOCAL_DATA_API: "yes"
53+
# turn on header-based reranker auth
54+
HEADER_RERANKING_API_KEY_NVIDIA: ${{ secrets.HEADER_RERANKING_API_KEY_NVIDIA }}
55+
56+
ut:
57+
uses: ./.github/workflows/unit.yml
58+
if: ${{ github.event.action != 'closed' }}
59+
with:
60+
coverage_prefix: ut
61+
sha: ${{ github.sha }}
62+
coverage_python_version: "3.11"
63+
secrets:
64+
# basic secrets
65+
ASTRA_DB_APPLICATION_TOKEN: ${{ secrets.ASTRA_DB_APPLICATION_TOKEN }}
66+
ASTRA_DB_API_ENDPOINT: ${{ secrets.ASTRA_DB_API_ENDPOINT }}
67+
68+
aggregate:
69+
runs-on: ubuntu-latest
70+
needs: [main_it, local_it, ut]
71+
steps:
72+
- name: Checkout code
73+
if: ${{ github.event.action != 'closed' }}
74+
uses: actions/checkout@v2
75+
76+
- name: Set up Python
77+
if: ${{ github.event.action != 'closed' }}
78+
uses: actions/setup-python@v2
79+
with:
80+
python-version: 3.11
81+
82+
- name: Install dependencies
83+
if: ${{ github.event.action != 'closed' }}
84+
run: |
85+
python -m pip install --upgrade pip
86+
pipx install uv
87+
make venv
88+
89+
- name: Download all coverage artifacts from this run
90+
if: ${{ github.event.action != 'closed' }}
91+
uses: actions/download-artifact@v4
92+
with:
93+
path: ./all_coverage
94+
95+
- name: Combine coverage data
96+
if: ${{ github.event.action != 'closed' }}
97+
run: uv run coverage combine $(find all_coverage/ -name ".coverage" 2>/dev/null)
98+
99+
- name: produce coverage JSON report
100+
if: ${{ github.event.action != 'closed' }}
101+
run: uv run coverage json
102+
103+
- name: verify JSON report output
104+
if: ${{ github.event.action != 'closed' }}
105+
run: cut -c -200 coverage.json
106+
107+
- name: Upload coverage JSON report for mid-term storage
108+
if: ${{ github.event_name == 'push' }}
109+
uses: actions/upload-artifact@v4
110+
id: upload_json
111+
with:
112+
name: coverage-json-push-${{ github.sha }}
113+
path: coverage.json
114+
retention-days: 30
115+
116+
- name: produce coverage HTML report (WIP)
117+
if: ${{ github.event.action != 'closed' }}
118+
run: |
119+
uv run coverage html
120+
test -f htmlcov/index.html
121+
echo "commit=${GITHUB_SHA}" > htmlcov/_preview_build.txt
122+
123+
# This uploads the coverage html report to make it available for offline inspection
124+
- name: Upload coverage HTML report for reference
125+
uses: actions/upload-artifact@v4
126+
id: upload_html
127+
with:
128+
name: coverage-html-report-${{ inputs.sha }}
129+
path: htmlcov
130+
retention-days: 7
131+
include-hidden-files: true
132+
133+
# Note: the following would be to publish a PR-preview of the code coverage
134+
# HTML report. Not completely set up, something does not work with publishing
135+
# to GH Pages. Postponed.
136+
137+
# - name: Sanity check coverage output (WIP)
138+
# run: |
139+
# echo "PWD=$PWD"
140+
# ls -la
141+
# ls -la htmlcov || true
142+
# test -f htmlcov/index.html || (echo "ERROR: htmlcov/index.html not found." && exit 1)
143+
# echo "Files in htmlcov:"
144+
# find htmlcov -type f | head -100
145+
146+
# - name: Deploy PR preview to Pages (WIP)
147+
# if: ${{ github.event.action != 'closed' }}
148+
# uses: rossjrw/pr-preview-action@v1
149+
# with:
150+
# source-dir: htmlcov
151+
152+
- name: Remove PR preview
153+
if: ${{ github.event.action == 'closed' }}
154+
uses: rossjrw/pr-preview-action@v1
155+
with:
156+
action: remove
157+
158+
- name: generate code coverage short report
159+
if: ${{ github.event.action != 'closed' }}
160+
id: generate_coverage_report
161+
shell: bash
162+
run: |
163+
set -o pipefail
164+
OUTPUT=$(uv run coverage report)
165+
# Multi-line safe write to $GITHUB_OUTPUT
166+
echo "text<<EOF" >> "$GITHUB_OUTPUT"
167+
echo "$OUTPUT" >> "$GITHUB_OUTPUT"
168+
echo "EOF" >> "$GITHUB_OUTPUT"
169+
170+
- name: Comment on PR
171+
if: ${{ github.event.action != 'closed' && github.event_name == 'pull_request' }}
172+
uses: actions/github-script@v7
173+
with:
174+
github-token: ${{ secrets.GITHUB_TOKEN }}
175+
script: |
176+
const prNumber = context.payload.pull_request.number;
177+
const body = `### Coverage report\nfor commit: \`${{ github.sha }}\`.\ndownload detailed report [here](${{ steps.upload_html.outputs.artifact-url }}).\n\`\`\`\n${{ steps.generate_coverage_report.outputs.text }}\n\`\`\``;
178+
await github.rest.issues.createComment({
179+
...context.repo,
180+
issue_number: prNumber,
181+
body
182+
});

‎.github/workflows/local.yaml‎

Lines changed: 0 additions & 58 deletions
This file was deleted.

‎.github/workflows/local.yml‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
name: Run base integration tests on a local Data API
2+
3+
on:
4+
workflow_call:
5+
inputs:
6+
coverage_prefix:
7+
description: "Code coverage file base name"
8+
type: string
9+
required: true
10+
sha:
11+
description: "Commit SHA of the code being tested"
12+
type: string
13+
required: true
14+
secrets:
15+
AWS_ECR_REGION:
16+
required: true
17+
AWS_ECR_ACCOUNT_ID:
18+
required: true
19+
AWS_ECR_ROLE_NAME:
20+
required: true
21+
AWS_ECR_REPOSITORY:
22+
required: true
23+
AWS_ECR_REGISTRY:
24+
required: true
25+
HEADER_EMBEDDING_API_KEY_OPENAI:
26+
required: true
27+
DOCKER_COMPOSE_LOCAL_DATA_API:
28+
required: true
29+
HEADER_RERANKING_API_KEY_NVIDIA:
30+
required: true
31+
32+
permissions:
33+
id-token: write # required for OIDC
34+
contents: read # required for actions/checkout if you use it
35+
36+
jobs:
37+
test:
38+
env:
39+
AWS_ECR_REGION: ${{ secrets.AWS_ECR_REGION }}
40+
AWS_ECR_ACCOUNT_ID: ${{ secrets.AWS_ECR_ACCOUNT_ID }}
41+
AWS_ECR_ROLE_NAME: ${{ secrets.AWS_ECR_ROLE_NAME }}
42+
AWS_ECR_REPOSITORY: ${{ secrets.AWS_ECR_REPOSITORY }}
43+
AWS_ECR_REGISTRY: ${{ secrets.AWS_ECR_REGISTRY }}
44+
HEADER_EMBEDDING_API_KEY_OPENAI: ${{ secrets.HEADER_EMBEDDING_API_KEY_OPENAI }}
45+
DOCKER_COMPOSE_LOCAL_DATA_API: ${{ secrets.DOCKER_COMPOSE_LOCAL_DATA_API }}
46+
HEADER_RERANKING_API_KEY_NVIDIA: ${{ secrets.HEADER_RERANKING_API_KEY_NVIDIA }}
47+
runs-on: ubuntu-latest
48+
49+
steps:
50+
- name: Checkout code
51+
uses: actions/checkout@v2
52+
53+
- name: Set up Python
54+
uses: actions/setup-python@v2
55+
with:
56+
python-version: 3.12
57+
58+
- name: Install dependencies
59+
run: |
60+
python -m pip install --upgrade pip
61+
pipx install uv
62+
make venv
63+
64+
- name: Configure AWS credentials from OIDC
65+
uses: aws-actions/configure-aws-credentials@v4
66+
with:
67+
role-to-assume: arn:aws:iam::${{ env.AWS_ECR_ACCOUNT_ID }}:role/${{ env.AWS_ECR_ROLE_NAME }}
68+
aws-region: ${{ env.AWS_ECR_REGION }}
69+
70+
- name: Login to Amazon ECR
71+
uses: aws-actions/amazon-ecr-login@v2
72+
73+
- name: Run pytest
74+
run: |
75+
uv run pytest --cov=astrapy/ tests/base/integration -vv
76+
77+
- name: Upload coverage data (unique per commit)
78+
uses: actions/upload-artifact@v4
79+
with:
80+
name: ${{ inputs.coverage_prefix }}-${{ inputs.sha }}
81+
path: .coverage
82+
retention-days: 1
83+
include-hidden-files: true

‎.github/workflows/main.yml‎

Lines changed: 29 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,33 @@
11
name: Run base integration tests on Astra DB
22

33
on:
4-
push:
5-
branches:
6-
- main
7-
pull_request:
8-
branches:
9-
- main
4+
workflow_call:
5+
inputs:
6+
coverage_prefix:
7+
description: "Code coverage file base name"
8+
type: string
9+
required: true
10+
sha:
11+
description: "Commit SHA of the code being tested"
12+
type: string
13+
required: true
14+
secrets:
15+
ASTRA_DB_APPLICATION_TOKEN:
16+
required: true
17+
ASTRA_DB_API_ENDPOINT:
18+
required: true
19+
HEADER_EMBEDDING_API_KEY_OPENAI:
20+
required: true
21+
LEGACY_INSERTMANY_BEHAVIOUR_PRE2193:
22+
required: true
1023

1124
jobs:
1225
test:
1326
env:
14-
# basic secrets
1527
ASTRA_DB_APPLICATION_TOKEN: ${{ secrets.ASTRA_DB_APPLICATION_TOKEN }}
1628
ASTRA_DB_API_ENDPOINT: ${{ secrets.ASTRA_DB_API_ENDPOINT }}
1729
HEADER_EMBEDDING_API_KEY_OPENAI: ${{ secrets.HEADER_EMBEDDING_API_KEY_OPENAI }}
18-
LEGACY_INSERTMANY_BEHAVIOUR_PRE2193: "yes"
30+
LEGACY_INSERTMANY_BEHAVIOUR_PRE2193: ${{ secrets.LEGACY_INSERTMANY_BEHAVIOUR_PRE2193 }}
1931
runs-on: ubuntu-latest
2032

2133
steps:
@@ -35,4 +47,12 @@ jobs:
3547
3648
- name: Run pytest
3749
run: |
38-
uv run pytest tests/base/integration
50+
uv run pytest --cov=astrapy/ tests/base/integration -vv
51+
52+
- name: Upload coverage data (unique per commit)
53+
uses: actions/upload-artifact@v4
54+
with:
55+
name: ${{ inputs.coverage_prefix }}-${{ inputs.sha }}
56+
path: .coverage
57+
retention-days: 1
58+
include-hidden-files: true

0 commit comments

Comments
 (0)