From 9cf7a7ad2c6931e618fa5aa6e45241252d9bab7f Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Mon, 19 Jan 2026 22:35:56 +0000 Subject: [PATCH 01/50] Migrating from azure pipelines to GHA with minor updates for Ruff and Docker --- .github/workflows/README.md | 301 +++++++++++++++++++++++++ .github/workflows/build.yaml | 76 +++++++ .github/workflows/clang_format.yaml | 27 ++- .github/workflows/format-and-lint.yaml | 26 ++- .github/workflows/fprettify.yaml | 52 +++++ .github/workflows/isort.yaml | 21 +- .github/workflows/mypy.yaml | 2 +- .github/workflows/pylint.yaml | 24 +- .github/workflows/pypi.yaml | 2 +- .github/workflows/tapenade.yaml | 41 ++++ azure/README.md | 170 -------------- azure/azure_build.yaml | 157 ------------- azure/azure_pypi.yaml | 29 --- azure/azure_style.yaml | 208 ----------------- azure/azure_tapenade.yaml | 45 ---- azure/azure_template.yaml | 139 ------------ azure/clang-format.sh | 70 ------ azure/combine-config.py | 28 --- azure/fprettify.sh | 35 --- combine-config.py | 59 +++++ ruff.toml | 13 +- 21 files changed, 614 insertions(+), 911 deletions(-) create mode 100644 .github/workflows/README.md create mode 100644 .github/workflows/build.yaml create mode 100644 .github/workflows/fprettify.yaml create mode 100644 .github/workflows/tapenade.yaml delete mode 100644 azure/README.md delete mode 100644 azure/azure_build.yaml delete mode 100644 azure/azure_pypi.yaml delete mode 100644 azure/azure_style.yaml delete mode 100644 azure/azure_tapenade.yaml delete mode 100644 azure/azure_template.yaml delete mode 100644 azure/clang-format.sh delete mode 100644 azure/combine-config.py delete mode 100644 azure/fprettify.sh create mode 100644 combine-config.py diff --git a/.github/workflows/README.md b/.github/workflows/README.md new file mode 100644 index 0000000..c86ac50 --- /dev/null +++ b/.github/workflows/README.md @@ -0,0 +1,301 @@ +# GitHub Actions Workflows + +Reusable GitHub Actions workflows for Supercritical repositories. These workflows are called from individual repositories using the `workflow_call` trigger. + +## Available Workflows + +| Workflow | Description | +| :------- | :---------- | +| `build.yaml` | Build and test code in Docker container | +| `format-and-lint.yaml` | Python formatting and linting with ruff/pre-commit | +| `pypi.yaml` | Publish Python packages to PyPI | +| `tapenade.yaml` | Tapenade automatic differentiation checks | +| `clang_format.yaml` | C/C++ formatting checks | +| `fprettify.yaml` | Fortran 90 formatting checks | +| `isort.yaml` | Python import sorting checks | +| `pylint.yaml` | Python linting with pylint | +| `mypy.yaml` | Python type checking | +| `branch-name-check.yml` | Enforce branch naming conventions | + +--- + +## Workflow Options + +### build.yaml + +Docker-based build and test workflow using the `scritical/private-dev` image. + +| Name | Type | Default | Description | +| :--- | :--- | :------ | :---------- | +| `TIMEOUT` | number | `120` | Runtime allowed for the job, in minutes | +| `GCC_CONFIG` | string | `""` | Path to GCC configuration file (from repository root) | +| `BUILD_SCRIPT` | string | `.github/build_real.sh` | Path to build script. Empty string skips this step | +| `TEST_SCRIPT` | string | `.github/test_real.sh` | Path to test script. Empty string skips this step | + +**Required Secrets:** +| Name | Description | +| :--- | :---------- | +| `DOCKER_OAT` | Docker registry Organization Access Token | + +--- + +### format-and-lint.yaml + +Python formatting and linting using pre-commit with ruff. + +| Name | Type | Default | Description | +| :--- | :--- | :------ | :---------- | +| `MCCABE` | boolean | `false` | Enable McCabe complexity check (pass/fail, max complexity = 10) | + +**Configuration Override:** Create a `ruff.toml` in your repo with: +```toml +extend = "~/.config/ruff/ruff.toml" + +# Local overrides here +[lint] +ignore = ["N802"] +``` + +--- + +### pypi.yaml + +Publish Python packages to PyPI on tagged releases. + +**Required Secrets:** +| Name | Description | +| :--- | :---------- | +| `PYPI_API_TOKEN` | PyPI API token for publishing | + +--- + +### tapenade.yaml + +Run Tapenade automatic differentiation and check for uncommitted changes. + +| Name | Type | Default | Description | +| :--- | :--- | :------ | :---------- | +| `TIMEOUT` | number | `10` | Runtime allowed for the job, in minutes | +| `TAPENADE_SCRIPT` | string | `.github/build_tapenade.sh` | Path to Tapenade build script | + +Uses Tapenade version 3.16. + +--- + +### clang_format.yaml + +C/C++ code formatting checks using clang-format. + +| Name | Type | Default | Description | +| :--- | :--- | :------ | :---------- | +| `TIMEOUT` | number | `10` | Runtime allowed for the job, in minutes | + +**Configuration Override:** Create a `.clang-format` file in your repo root. + +--- + +### fprettify.yaml + +Fortran 90 code formatting checks using fprettify. + +| Name | Type | Default | Description | +| :--- | :--- | :------ | :---------- | +| `TIMEOUT` | number | `10` | Runtime allowed for the job, in minutes | + +**Configuration Override:** Create a `.fprettify.rc` file in your repo root. + +--- + +### isort.yaml + +Python import sorting checks. + +**Configuration Override:** Create a `.isort.cfg` file in your repo root with overrides. The global and local configs are merged automatically using `combine-config.py`. + +--- + +### pylint.yaml + +Python linting with pylint. + +**Configuration Override:** Create a `.pylintrc` file in your repo root with overrides. The global and local configs are merged automatically using `combine-config.py`. + +--- + +### mypy.yaml + +Python type checking with mypy. + +No configurable inputs. Uses Python 3.11. + +--- + +## Setting Up Workflows + +### Step 1: Create Workflow File + +Create `.github/workflows/ci.yaml` in your repository: + +```yaml +name: CI + +on: + push: + branches: [main] + pull_request: + branches: [main] + +jobs: + build: + uses: scritical/.github/.github/workflows/build.yaml@main + with: + GCC_CONFIG: config/defaults/config.LINUX_GFORTRAN.mk + BUILD_SCRIPT: .github/build_real.sh + TEST_SCRIPT: .github/test_real.sh + secrets: + DOCKER_OAT: ${{ secrets.DOCKER_OAT }} + + format-and-lint: + uses: scritical/.github/.github/workflows/format-and-lint.yaml@main + + clang-format: + uses: scritical/.github/.github/workflows/clang_format.yaml@main + + fprettify: + uses: scritical/.github/.github/workflows/fprettify.yaml@main +``` + +### Step 2: Write Build Script + +Create `.github/build_real.sh` in your repository: + +```bash +#!/bin/bash +set -e +cp $CONFIG_FILE config/config.mk +make +pip install . +``` + +### Step 3: Write Test Script + +Create `.github/test_real.sh` in your repository: + +```bash +#!/bin/bash +set -e +./input_files/get-input-files.sh +testflo -v . -n 1 +``` + +### Step 4: Add Secrets + +In your repository settings, add the required secrets: +- `DOCKER_OAT` - Docker Organization Access Token for pulling private images + +--- + +## PyPI Publishing + +For repositories that publish to PyPI, extend your workflow: + +```yaml +name: CI + +on: + push: + branches: [main] + tags: + - 'v*.*.*' + pull_request: + branches: [main] + +jobs: + format-and-lint: + uses: scritical/.github/.github/workflows/format-and-lint.yaml@main + + build: + uses: scritical/.github/.github/workflows/build.yaml@main + with: + GCC_CONFIG: config/defaults/config.LINUX_GFORTRAN.mk + secrets: + DOCKER_OAT: ${{ secrets.DOCKER_OAT }} + + pypi: + needs: [format-and-lint, build] + uses: scritical/.github/.github/workflows/pypi.yaml@main + secrets: + PYPI_API_TOKEN: ${{ secrets.PYPI_API_TOKEN }} +``` + +--- + +## Complex Builds + +For repositories requiring both real and complex builds: + +```yaml +jobs: + build-real: + uses: scritical/.github/.github/workflows/build.yaml@main + with: + GCC_CONFIG: config/defaults/config.LINUX_GFORTRAN.mk + BUILD_SCRIPT: .github/build_real.sh + TEST_SCRIPT: .github/test_real.sh + secrets: + DOCKER_OAT: ${{ secrets.DOCKER_OAT }} + + build-complex: + uses: scritical/.github/.github/workflows/build.yaml@main + with: + GCC_CONFIG: config/defaults/config.LINUX_GFORTRAN.mk + BUILD_SCRIPT: .github/build_complex.sh + TEST_SCRIPT: .github/test_complex.sh + secrets: + DOCKER_OAT: ${{ secrets.DOCKER_OAT }} +``` + +--- + +## Configuration Inheritance + +Several workflows support configuration inheritance/overrides: + +| Tool | Method | Local Config File | +| :--- | :----- | :---------------- | +| Ruff | Native `extend` keyword | `ruff.toml` | +| Pylint | Merged via `combine-config.py` | `.pylintrc` | +| isort | Merged via `combine-config.py` | `.isort.cfg` | +| clang-format | Local file takes precedence | `.clang-format` | +| fprettify | Local file takes precedence | `.fprettify.rc` | + +### Example: Ruff Override + +```toml +# ruff.toml +extend = "~/.config/ruff/ruff.toml" + +[lint] +ignore = ["N802", "N803"] # Allow uppercase function/argument names +``` + +### Example: Pylint Override + +```ini +# .pylintrc (partial - only overrides) +[MESSAGES CONTROL] +disable = C0114,C0115 + +[FORMAT] +max-line-length = 150 +``` + +--- + +## Adding Status Badge + +Add a status badge to your README: + +```markdown +![CI](https://github.com/scritical/REPO_NAME/actions/workflows/ci.yaml/badge.svg) +``` diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml new file mode 100644 index 0000000..b89d70f --- /dev/null +++ b/.github/workflows/build.yaml @@ -0,0 +1,76 @@ +on: + workflow_call: + inputs: + TIMEOUT: + type: number + default: 120 + description: "Timeout for the job in minutes" + GCC_CONFIG: + type: string + default: "" + description: "Path to GCC configuration file" + BUILD_SCRIPT: + type: string + default: ".github/build_real.sh" + description: "Path to build script" + TEST_SCRIPT: + type: string + default: ".github/test_real.sh" + description: "Path to test script" + secrets: + DOCKER_OAT: + required: true + description: "Docker registry Organization Access Token" + +env: + SCRITICAL_HOMEDIR: /home/scriticaluser + DOCKER_TAG: u24-gcc-ompi-latest + +jobs: + build-and-test: + runs-on: ubuntu-24.04 + timeout-minutes: ${{ inputs.TIMEOUT }} + steps: + - name: Checkout repository + uses: actions/checkout@v5 + + - name: Get repository name + id: repo + run: echo "name=${GITHUB_REPOSITORY##*/}" >> $GITHUB_OUTPUT + + - name: Login to Docker Hub + run: echo "${{ secrets.DOCKER_OAT }}" | docker login -u scritical --password-stdin + + - name: Pull Docker image + run: docker pull scritical/private-dev:${{ env.DOCKER_TAG }} + + - name: Start Docker container + run: | + REPO_NAME="${{ steps.repo.outputs.name }}" + docker run -t -d --name app \ + --mount "type=bind,src=${{ github.workspace }},target=${{ env.SCRITICAL_HOMEDIR }}/mount/${REPO_NAME}" \ + scritical/private-dev:${{ env.DOCKER_TAG }} /bin/bash + + # Copy repo to working directory + docker exec app /bin/bash -c "cp -r ${{ env.SCRITICAL_HOMEDIR }}/mount/${REPO_NAME} ${{ env.SCRITICAL_HOMEDIR }}/repos/${REPO_NAME}" + + - name: Build + if: inputs.BUILD_SCRIPT != '' + run: | + REPO_NAME="${{ steps.repo.outputs.name }}" + docker exec \ + -e CONFIG_FILE="${{ inputs.GCC_CONFIG }}" \ + app /bin/bash -c ". ${{ env.SCRITICAL_HOMEDIR }}/.bashrc_scritical && cd ${{ env.SCRITICAL_HOMEDIR }}/repos/${REPO_NAME} && /bin/bash ${{ inputs.BUILD_SCRIPT }}" + + - name: Test + if: inputs.TEST_SCRIPT != '' + run: | + REPO_NAME="${{ steps.repo.outputs.name }}" + docker exec \ + -e AGENT_NAME="${{ runner.name }}" \ + -e BUILD_REASON="${{ github.event_name }}" \ + app /bin/bash -c ". ${{ env.SCRITICAL_HOMEDIR }}/.bashrc_scritical && cd ${{ env.SCRITICAL_HOMEDIR }}/repos/${REPO_NAME} && source ~/MPIOversubscribe.sh && /bin/bash ${{ inputs.TEST_SCRIPT }}" + + - name: Cleanup + if: always() + run: docker stop app && docker rm app || true diff --git a/.github/workflows/clang_format.yaml b/.github/workflows/clang_format.yaml index 03aeea9..e2b8bb9 100644 --- a/.github/workflows/clang_format.yaml +++ b/.github/workflows/clang_format.yaml @@ -1,13 +1,32 @@ on: workflow_call: + inputs: + TIMEOUT: + type: number + default: 10 + description: "Timeout for the job in minutes" jobs: clang_format: runs-on: ubuntu-24.04 + timeout-minutes: ${{ inputs.TIMEOUT }} steps: - - name: Install dependencies + - name: Checkout repository + uses: actions/checkout@v5 + + - name: Install clang-format + run: sudo apt-get install clang-format-10 -y + + - name: Get clang-format config run: | - sudo apt-get install clang-format-10 -y - - name: Run clang_format + LOCAL_CONFIG_FILE=".clang-format" + GLOBAL_CONFIG_URL="https://raw.githubusercontent.com/scritical/.github/main/.clang-format" + + if [[ ! -f "$LOCAL_CONFIG_FILE" ]]; then + echo "Downloading global clang-format config..." + wget "$GLOBAL_CONFIG_URL" -O "$LOCAL_CONFIG_FILE" || echo "No global config found" + fi + + - name: Run clang-format run: | - bash ./.github/clang_format.sh + find . -iname '*.h' -o -iname '*.hpp' -o -iname '*.c' -o -iname '*.cpp' | xargs clang-format -style=file --dry-run --Werror diff --git a/.github/workflows/format-and-lint.yaml b/.github/workflows/format-and-lint.yaml index 4ff965c..e426cf0 100644 --- a/.github/workflows/format-and-lint.yaml +++ b/.github/workflows/format-and-lint.yaml @@ -1,5 +1,10 @@ on: workflow_call: + inputs: + MCCABE: + type: boolean + default: false + description: "Enable McCabe complexity check (pass/fail)" jobs: format-and-lint: @@ -9,23 +14,32 @@ jobs: - name: Set up Python uses: actions/setup-python@v6 with: - python-version: 3.12 + python-version: "3.11" - name: Install dependencies run: | - wget https://raw.githubusercontent.com/mdolab/.github/main/.pre-commit-config.yaml + wget https://raw.githubusercontent.com/scritical/.github/main/.pre-commit-config.yaml pip install pre-commit - # Set the ruff config in the .github repo as the global config, then the local config should extend it by including the line `extend = "~/.config/ruff/ruff.toml"` + # Set the ruff config in the .github repo as the global config + # Repos can extend it by including `extend = "~/.config/ruff/ruff.toml"` in their local ruff.toml mkdir -p ~/.config/ruff - url=https://raw.githubusercontent.com/mdolab/.github/main/ruff.toml - wget $url -O ~/.config/ruff/ruff.toml + wget https://raw.githubusercontent.com/scritical/.github/main/ruff.toml -O ~/.config/ruff/ruff.toml - echo "Ruff config:" + echo "=== Global ruff config ===" cat ~/.config/ruff/ruff.toml if [[ -f "ruff.toml" ]]; then + echo "=== Local ruff config ===" cat ruff.toml fi - name: Run pre-commit checks run: | pre-commit run --all-files --show-diff-on-failure + + - name: McCabe complexity check + if: inputs.MCCABE + run: | + echo "=== McCabe Complexity Report ===" + echo "Checking for functions with complexity > 10..." + pip install ruff + ruff check --select=C90 --output-format=text . diff --git a/.github/workflows/fprettify.yaml b/.github/workflows/fprettify.yaml new file mode 100644 index 0000000..f22bc4e --- /dev/null +++ b/.github/workflows/fprettify.yaml @@ -0,0 +1,52 @@ +on: + workflow_call: + inputs: + TIMEOUT: + type: number + default: 10 + description: "Timeout for the job in minutes" + +jobs: + fprettify: + runs-on: ubuntu-24.04 + timeout-minutes: ${{ inputs.TIMEOUT }} + steps: + - name: Checkout repository + uses: actions/checkout@v5 + + - name: Set up Python + uses: actions/setup-python@v6 + with: + python-version: "3.11" + + - name: Install fprettify + run: pip install fprettify==0.3.7 + + - name: Get fprettify config + run: | + LOCAL_CONFIG_FILE=".fprettify.rc" + GLOBAL_CONFIG_URL="https://raw.githubusercontent.com/scritical/.github/main/.fprettify.rc" + + if [[ ! -f "$LOCAL_CONFIG_FILE" ]]; then + echo "Downloading global fprettify config..." + wget "$GLOBAL_CONFIG_URL" -O "$LOCAL_CONFIG_FILE" || echo "No global config found, using defaults" + fi + + - name: Run fprettify + run: | + CONFIG_FILE=".fprettify.rc" + CONFIG_ARG="" + if [[ -f "$CONFIG_FILE" ]]; then + CONFIG_ARG="--config-file $CONFIG_FILE" + fi + + # Run fprettify on non-differentiated Fortran 90 source files + find . -type f -iname '*.f90' ! -iname '*_b.f90' ! -iname '*_d.f90' -print0 | while read -d $'\0' fileName + do + echo "Checking $fileName" + fprettify $CONFIG_ARG "$fileName" + done + + - name: Check for changes + run: | + git diff --summary --exit-code || (echo "fprettify produced changes. Please format and commit." && exit 1) diff --git a/.github/workflows/isort.yaml b/.github/workflows/isort.yaml index 37c8e44..de5271a 100644 --- a/.github/workflows/isort.yaml +++ b/.github/workflows/isort.yaml @@ -9,11 +9,22 @@ jobs: - name: Set up Python uses: actions/setup-python@v6 with: - python-version: 3.9 - - name: Get isort config file + python-version: "3.11" + - name: Prepare isort config run: | - # copy over the isort config file - if [[ ! -f ".isort.cfg" ]]; then - wget https://raw.githubusercontent.com/mdolab/.github/main/.isort.cfg + # Download global config + wget https://raw.githubusercontent.com/scritical/.github/main/.isort.cfg -O .isort.cfg-global + + # Check for local overrides and merge if present + if [[ -f ".isort.cfg" ]]; then + echo "Found local .isort.cfg, merging with global config..." + wget https://raw.githubusercontent.com/scritical/.github/main/combine-config.py + python combine-config.py .isort.cfg-global .isort.cfg .isort.cfg-merged + mv .isort.cfg-merged .isort.cfg + rm combine-config.py + else + mv .isort.cfg-global .isort.cfg fi + rm -f .isort.cfg-global + - uses: isort/isort-action@master diff --git a/.github/workflows/mypy.yaml b/.github/workflows/mypy.yaml index 97c4221..30f6e10 100644 --- a/.github/workflows/mypy.yaml +++ b/.github/workflows/mypy.yaml @@ -6,7 +6,7 @@ jobs: runs-on: ubuntu-24.04 strategy: matrix: - python-version: [3.9, 3.11] + python-version: "3.11" steps: - uses: actions/checkout@v5 - name: Set up Python ${{ matrix.python-version }} diff --git a/.github/workflows/pylint.yaml b/.github/workflows/pylint.yaml index 01921c6..cd42e13 100644 --- a/.github/workflows/pylint.yaml +++ b/.github/workflows/pylint.yaml @@ -9,13 +9,27 @@ jobs: - name: Set up Python uses: actions/setup-python@v6 with: - python-version: 3.9 + python-version: "3.11" - name: Install dependencies run: | - pip install wheel - pip install pylint + pip install wheel pylint + - name: Prepare pylint config + run: | + # Download global config + wget https://raw.githubusercontent.com/scritical/.github/main/.pylintrc -O .pylintrc-global + + # Check for local overrides and merge if present + if [[ -f ".pylintrc" ]]; then + echo "Found local .pylintrc, merging with global config..." + wget https://raw.githubusercontent.com/scritical/.github/main/combine-config.py + python combine-config.py .pylintrc-global .pylintrc .pylintrc-merged + mv .pylintrc-merged .pylintrc + rm combine-config.py + else + mv .pylintrc-global .pylintrc + fi + rm -f .pylintrc-global + - name: Run pylint run: | - # copy over the pylint config file - wget https://raw.githubusercontent.com/mdolab/.github/main/.pylintrc find . -type f -name "*.py" -not -path "*/doc/*" | xargs pylint diff --git a/.github/workflows/pypi.yaml b/.github/workflows/pypi.yaml index 1ecfb57..a9243ae 100644 --- a/.github/workflows/pypi.yaml +++ b/.github/workflows/pypi.yaml @@ -12,7 +12,7 @@ jobs: - uses: actions/checkout@v5 - uses: actions/setup-python@v6 with: - python-version: 3.9 + python-version: "3.11" - name: Build package run: | python --version diff --git a/.github/workflows/tapenade.yaml b/.github/workflows/tapenade.yaml new file mode 100644 index 0000000..abe9d8e --- /dev/null +++ b/.github/workflows/tapenade.yaml @@ -0,0 +1,41 @@ +on: + workflow_call: + inputs: + TIMEOUT: + type: number + default: 10 + description: "Timeout for the job in minutes" + TAPENADE_SCRIPT: + type: string + default: ".github/build_tapenade.sh" + description: "Path to Tapenade build script" + +jobs: + tapenade: + runs-on: ubuntu-24.04 + timeout-minutes: ${{ inputs.TIMEOUT }} + steps: + - name: Checkout repository + uses: actions/checkout@v5 + + - name: Set up Java + uses: actions/setup-java@v4 + with: + distribution: temurin + java-version: 8 + + - name: Install and run Tapenade + run: | + # Download Tapenade 3.16 + TAPENADE_URL="https://gitlab.inria.fr/tapenade/tapenade/-/package_files/112870/download" + wget "$TAPENADE_URL" -O tapenade_3.16.tar + tar -xzf tapenade_3.16.tar + export PATH=$(pwd)/tapenade_3.16/bin:$PATH + + # Run Tapenade + touch config.mk # empty config file to make "make" happy + bash ${{ inputs.TAPENADE_SCRIPT }} + + - name: Check for changes + run: | + git diff --exit-code || (echo "Tapenade produced changes. Please regenerate and commit." && exit 1) diff --git a/azure/README.md b/azure/README.md deleted file mode 100644 index 32c5aed..0000000 --- a/azure/README.md +++ /dev/null @@ -1,170 +0,0 @@ -# Azure -The Supercritical Azure Pipelines page is found on the [Azure Website](https://dev.azure.com/scritical/). -The process outlined here is split into Public and Private projects, used accordingly for public and private repositories. -However, at the moment we only support Private projects, but the processes support public repositories. -The pipelines for each repository can be found by selecting its parent project and then the pipelines button. -Each pipeline is set up using the templates located in this `.github` repository so only configuration files are needed in each repository. - -The templates are organized into the following files: -- `azure_build.yaml` which handles the build and test jobs for the code -- `azure_pypi.yaml` which handles the PyPI deployment if applicable -- `azure_style.yaml` which handles style checks on the code -- `azure_template.yaml` which handles the job itself, calling the necessary sub-templates. - -## Template Options -| Name | Type | Default | Description. | -| :--------------------- | :------ | :-------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------ | -| `REPO_NAME` | string | | Name of repository | -| `COMPLEX` | boolean | `false` | Flag for triggering complex build and tests | -| `TIMEOUT_BUILD` | number | `120` | Runtime allowed for each build and test job, in minutes | -| `GCC_CONFIG` | string | `None` | Path to GCC configuration file (from repository root) | -| `INTEL_CONFIG` | string | `None` | Path to Intel configuration file (from repository root) | -| `BUILD_REAL` | string | `.github/build_real.sh` | Path to Bash script with commands to build real code. Using `None` will skip this step. | -| `TEST_REAL` | string | `.github/text_real.sh` | Path to Bash script to run real tests. Using `None` will skip this step. | -| `BUILD_COMPLEX` | string | `.github/build_complex.sh` | Path to Bash script with commands to build complex code. Using `None` will skip this step. | -| `TEST_COMPLEX` | string | `.github/text_complex.sh` | Path to Bash script with commands to run complex tests. Using `None` will skip this step. | -| `IMAGE` | string | `public` | Select Docker image. Can be `public`, `private`, or `auto`. `auto` uses the private image on trusted builds and the public image otherwise. | -| `COVERAGE` | boolean | `false` | Flag to report test coverage to `codecov` | -| `TIMEOUT_STYLE` | number | `10` | Runtime allowed for each style check, in minutes | -| `IGNORE_STYLE` | boolean | `false` | Flag to allow `formatting and linting` checks to fail without failing the pipeline | -| `BASE_FORMAT_AND_LINT` | boolean | `true` | Flag to trigger the `base_format_and_lint` check | -| `ISORT` | boolean | `false` | Flag to trigger the `isort` check | -| `PYLINT` | boolean | `false` | Flag to trigger the `pylint` check | -| `CLANG_FORMAT` | boolean | `false` | Flag to trigger the `clang-format` check | -| `FPRETTIFY` | boolean | `false` | Flag to trigger the `fprettify` check | -| `MYPY` | boolean | `false` | Flag to trigger the `mypy` check | -| `TAPENADE` | boolean | `false` | Flag to trigger the Tapenade check | -| `TIMEOUT_TAPENADE` | number | `10` | Runtime allowed for the Tapenade check, in minutes | -| `TAPENADE_SCRIPT` | string | `.github/build_tapenade.sh` | Path to Bash script with commands to run Tapenade | -| `TAPENADE_VERSION` | string | `3.10` | Version of Tapenade to use | - -## Setting up a pipeline -### Step 1: Setup Azure Pipelines YAML File: - -1. Create `azure_template.yaml` in the `.github/` directory of your repository -2. Add triggers (see example below) - - Only set triggers for the `main` branch and pull requests to `main` -3. Add resources (see example below) - - This resource pulls the `azure_template` from the `scritical/.github` repository -4. Add parameters (see example below and the options table above) - -``` -trigger: -- main - -pr: -- main - -resources: - repositories: - - repository: azure_template - type: github - name: scritical/.github - endpoint: scritical - -extends: - template: azure/azure_template.yaml@azure_template - parameters: - REPO_NAME: adflow - IGNORE_STYLE: true - COMPLEX: true - GCC_CONFIG: config/defaults/config.LINUX_GFORTRAN.mk - INTEL_CONFIG: config/defaults/config.LINUX_INTEL.mk -``` - -### Step 2: Write Build Bash Script: - -1. Create `build_real.sh` in the `.github/` directory of your repository -2. Add `#!/bin/bash` followed by `set -e` as the first two lines -3. Add configuration file copy if needed (see example below). Note that the environment variable `$CONFIG_FILE` is provided by the Azure template and is automatically set depending on the compiler used for each build. -4. Add make command if needed (see example below) -5. Add python install command (see example below) -6. Repeat with `build_complex.sh` if needed - -``` -#!/bin/bash -set -e -cp $CONFIG_FILE config/config.mk -make -pip install . -``` - -### Step 3: Write Test Bash Script: - -1. Create `test_real.sh` in `.github/` directory of your repository -2. Add `#!/bin/bash` followed by `set -e` as the first two lines -3. Add input file download if needed (see example below) -4. Add `testflo` command -5. Repeat with `test_complex.sh` if needed - -``` -#!/bin/bash -set -e -./input_files/get-input-files.sh -testflo -v . -n 1 -``` - -### Step 4: Push branch to `scritical/` repository and create pull request - -1. Push branch directly to `scritical` repository, not your personal fork - -### Step 5: Create new Pipeline on Microsoft Azure - -1. Go to [Azure Pipelines](https://dev.azure.com/scritical/) and select public / private as needed by current repository -2. Select "Pipelines" and "New pipeline" -3. Select "Github" and authorize the Azure Pipeline app if you have not already -4. Select correct scritical repository (again, not personal fork) -5. Select "Existing Azure Pipelines YAML file" -6. In pop-up menu, set "Branch" to your new Azure-transition branch and "Path" to `.github/azure-pipelines.yaml` -7. Click "continue" - -### Step 6: Add Badge to README - -1. Go to the pipeline on Azure -2. Click on the three dots next to "Run pipeline" -3. Click on "status badge" -4. Set "Branch" to "main" -5. Copy the text for "Sample markdown" and paste it into README - -### Step 7: Create PR - -1. Create pull request to `main` -2. Ensure the Azure jobs start and appear in the pull request - -## PyPI with Azure -If using PyPI for a repository, the yaml file does not follow this same structure, but imports components as stages instead of as an extend. -An example yaml file using PyPI is: - -``` -trigger: - branches: - include: - - main - tags: - include: - - v*.*.* - -pr: -- main - -resources: - repositories: - - repository: azure_template - type: github - name: scritical/.github - endpoint: scritical - -stages: -- template: azure/azure_template.yaml@azure_template - parameters: - REPO_NAME: REPOSITORY NAME - -- stage: - dependsOn: - - Test_Real - - Style - displayName: PyPI - condition: and(succeeded(), contains(variables['build.sourceBranch'], 'tags')) - jobs: - - template: azure/azure_pypi.yaml@azure_template -``` diff --git a/azure/azure_build.yaml b/azure/azure_build.yaml deleted file mode 100644 index effbc18..0000000 --- a/azure/azure_build.yaml +++ /dev/null @@ -1,157 +0,0 @@ -parameters: - - name: REPO_NAME - type: string - - name: TIMEOUT - type: number - - name: GCC_CONFIG - type: string - - name: INTEL_CONFIG - type: string - - name: BUILD - type: string - - name: TEST - type: string - - name: IMAGE - type: string - - name: COVERAGE - type: boolean - - name: C_COVERAGE - type: boolean - -jobs: - - job: - pool: - vmImage: "ubuntu-24.04" - timeoutInMinutes: ${{ parameters.TIMEOUT }} - strategy: - matrix: - "u24-gcc-ompi-latest": - DOCKER_TAG: u24-gcc-ompi-latest - CONFIG_FILE: ${{ parameters.GCC_CONFIG }} - # "u22-gcc-ompi-stable": - # DOCKER_TAG: u22-gcc-ompi-stable - # CONFIG_FILE: ${{ parameters.GCC_CONFIG }} - variables: - - group: Docker - - group: Codecov - - name: SCRITICAL_HOMEDIR - value: /home/scriticaluser - - name: DOCKER_WORKING_DIR - value: ${{ variables.SCRITICAL_HOMEDIR }}/repos/${{ parameters.REPO_NAME }} - - name: DOCKER_MOUNT_DIR - value: ${{ variables.SCRITICAL_HOMEDIR }}/azure/${{ parameters.REPO_NAME }} - - name: BASHRC - value: ${{ variables.SCRITICAL_HOMEDIR }}/.bashrc_scritical - steps: - - checkout: self - - checkout: azure_template - - script: | - # This is a trusted build if DOCKER_USERNAME is defined - if [[ ! -z $(DOCKER_USERNAME) ]] ; then - echo $(DOCKER_PASSWORD) | docker login -u $(DOCKER_USERNAME) --password-stdin; - fi - # "auto" pulls the private image for trusted builds - if [[ "${{ parameters.IMAGE }}" == "private" ]] || [[ "${{ parameters.IMAGE }}" == "auto" && ! -z $(DOCKER_USERNAME) ]] ; then - export DOCKER_IMAGE=private-dev; - else - export DOCKER_IMAGE=public-dev; - fi - docker pull scritical/$DOCKER_IMAGE:$(DOCKER_TAG); - docker run -t -d --name app --mount "type=bind,src=$(pwd)/${{ parameters.REPO_NAME }},target=${{ variables.DOCKER_MOUNT_DIR }}" scritical/$DOCKER_IMAGE:$(DOCKER_TAG) /bin/bash; - docker exec app /bin/bash -c "rm -rf ${{ variables.DOCKER_WORKING_DIR }} && cp -r ${{ variables.DOCKER_MOUNT_DIR }} ${{ variables.DOCKER_WORKING_DIR }}"; - displayName: Prepare Repository - - script: docker exec -e CONFIG_FILE=$(CONFIG_FILE) app /bin/bash -c ". ${{ variables.BASHRC }} && cd ${{ variables.DOCKER_WORKING_DIR }} && /bin/bash ${{ parameters.BUILD }}" - displayName: Build - condition: and(succeeded(), ne('${{ parameters.BUILD }}', 'None')) - - script: | - set -e - docker exec -e AGENT_NAME="$AGENT_NAME" -e BUILD_REASON=$(Build.Reason) app /bin/bash -c ". ${{ variables.BASHRC }} && cd ${{ variables.DOCKER_WORKING_DIR }} && source ~/MPIOversubscribe.sh && /bin/bash ${{ parameters.TEST }}" - displayName: Run Tests - condition: and(succeeded(), ne('${{ parameters.TEST }}', 'None')) - - script: | - cd ${{ parameters.REPO_NAME }} - - # validate the uploader - curl https://keybase.io/codecovsecurity/pgp_keys.asc | gpg --no-default-keyring --keyring trustedkeys.gpg --import - curl -Os https://uploader.codecov.io/latest/linux/codecov - curl -Os https://uploader.codecov.io/latest/linux/codecov.SHA256SUM - curl -Os https://uploader.codecov.io/latest/linux/codecov.SHA256SUM.sig - gpgv codecov.SHA256SUM.sig codecov.SHA256SUM - shasum -a 256 -c codecov.SHA256SUM - - # exit if checksum fails - if [[ $? -ne 0 ]] ; then - echo "checksum failed" - exit 1 - fi - - # clean validation files - rm codecov.SHA256SUM* - - # grab the token - if [[ ! -z $(${{ parameters.REPO_NAME }}_CODECOV_TOKEN) ]] ; then - echo "codecov token found!" - export CODECOV_TOKEN=$(${{ parameters.REPO_NAME }}_CODECOV_TOKEN) - export CODECOV_ARGS="-t $CODECOV_TOKEN" - else - export CODECOV_ARGS="" - fi - - # convert coverage file to xml inside Docker - sudo docker exec app /bin/bash -c ". ${{ variables.BASHRC }} && cd ${{ variables.DOCKER_WORKING_DIR }} && \ - mv \$(find . -name ".coverage" -type f) . && coverage xml" - - # copy coverage file out of Docker - docker cp app:${{ variables.DOCKER_WORKING_DIR }}/coverage.xml $(pwd) - - # run codecov - chmod +x codecov - ./codecov $CODECOV_ARGS - condition: and(succeeded(), ${{ parameters.COVERAGE }}) - displayName: Coverage - - script: | - cd ${{ parameters.REPO_NAME }} - - # validate the uploader - curl https://keybase.io/codecovsecurity/pgp_keys.asc | gpg --no-default-keyring --keyring trustedkeys.gpg --import - curl -Os https://uploader.codecov.io/latest/linux/codecov - curl -Os https://uploader.codecov.io/latest/linux/codecov.SHA256SUM - curl -Os https://uploader.codecov.io/latest/linux/codecov.SHA256SUM.sig - gpgv codecov.SHA256SUM.sig codecov.SHA256SUM - shasum -a 256 -c codecov.SHA256SUM - - # exit if checksum fails - if [[ $? -ne 0 ]] ; then - echo "checksum failed" - exit 1 - fi - - # clean validation files - rm codecov.SHA256SUM* - - # grab the token - if [[ ! -z $(${{ parameters.REPO_NAME }}_CODECOV_TOKEN) ]] ; then - echo "codecov token found!" - export CODECOV_TOKEN=$(${{ parameters.REPO_NAME }}_CODECOV_TOKEN) - export CODECOV_ARGS="-t $CODECOV_TOKEN" - else - export CODECOV_ARGS="" - fi - - # convert coverage file to xml inside Docker - sudo docker exec app /bin/bash -c ". ${{ variables.BASHRC }} && cd ${{ variables.DOCKER_WORKING_DIR }} && \ - lcov --capture --directory . --output-file coverage.info && \ - lcov --remove coverage.info '/usr/*' --output-file coverage.info && \ - lcov --remove coverage.info '*/numpy/*' --output-file coverage.info && \ - lcov --remove coverage.info '*/.pyenv/versions/*' --output-file coverage.info && \ - lcov --remove coverage.info '*mpi4py*' --output-file coverage.info" && \ - lcov --list coverage.info - - # copy coverage file out of Docker - docker cp app:${{ variables.DOCKER_WORKING_DIR }}/coverage.info $(pwd) - - # run codecov - chmod +x codecov - ./codecov $CODECOV_ARGS - condition: ${{ parameters.C_COVERAGE }} - displayName: C Coverage diff --git a/azure/azure_pypi.yaml b/azure/azure_pypi.yaml deleted file mode 100644 index 7ed4c46..0000000 --- a/azure/azure_pypi.yaml +++ /dev/null @@ -1,29 +0,0 @@ -parameters: - - name: SERVICE_CONNECTION - type: string - default: pypi - - name: TIMEOUT - type: number - default: 10 - -jobs: - - job: PyPI_Deploy - pool: - vmImage: "ubuntu-22.04" - timeoutInMinutes: ${{ parameters.TIMEOUT }} - steps: - - task: UsePythonVersion@0 - inputs: - versionSpec: "3.8" - - script: | - python3 --version - python3 -m pip install -U pip - pip3 install setuptools wheel twine - python3 setup.py bdist_wheel - displayName: "Install Packages" - - task: TwineAuthenticate@1 - displayName: "Twine Authenticate" - inputs: - pythonUploadServiceConnection: ${{ parameters.SERVICE_CONNECTION }} - - script: python3 -m twine upload --config-file $(PYPIRC_PATH) dist/*.whl - displayName: "Twine Upload" diff --git a/azure/azure_style.yaml b/azure/azure_style.yaml deleted file mode 100644 index 7bbd391..0000000 --- a/azure/azure_style.yaml +++ /dev/null @@ -1,208 +0,0 @@ -parameters: - - name: REPO_NAME - type: string - - name: IMAGE - type: string - # The following parameters have defaults because this template could be used by repos directly - - name: TIMEOUT - type: number - default: 10 - - name: IGNORE_STYLE - type: boolean - default: false - - name: BASE_FORMAT_AND_LINT - type: boolean - default: true - - name: ISORT - type: boolean - default: false - - name: PYLINT - type: boolean - default: false - - name: CLANG_FORMAT - type: boolean - default: false - - name: FPRETTIFY - type: boolean - default: false - - name: MYPY - type: boolean - default: false -jobs: - - job: base_format_and_lint - pool: - vmImage: "ubuntu-22.04" - timeoutInMinutes: ${{ parameters.TIMEOUT }} - continueOnError: ${{ parameters.IGNORE_STYLE }} - condition: ${{ parameters.BASE_FORMAT_AND_LINT }} - steps: - - checkout: self - - checkout: azure_template - - task: UsePythonVersion@0 - inputs: - versionSpec: "3.11" - - script: | - cd ${{ parameters.REPO_NAME }} - cp ../.github/.pre-commit-config.yaml . - pip install pre-commit - - # Set the ruff config in the .github repo as the global config, then the local config should extend it by including the line `extend = "~/.config/ruff/ruff.toml"` - mkdir -p ~/.config/ruff - cp ../.github/ruff.toml ~/.config/ruff/ruff.toml - - echo "Ruff config:" - cat ~/.config/ruff/ruff.toml - if [[ -f "ruff.toml" ]]; then - cat ruff.toml - fi - - # Run the pre-commit checks - pre-commit run --all-files --show-diff-on-failure - - - job: isort - pool: - vmImage: "ubuntu-22.04" - timeoutInMinutes: ${{ parameters.TIMEOUT }} - continueOnError: ${{ parameters.IGNORE_STYLE }} - condition: ${{ parameters.ISORT }} - steps: - - checkout: self - - checkout: azure_template - - task: UsePythonVersion@0 - inputs: - versionSpec: "3.11" - - script: | - cd ${{ parameters.REPO_NAME }} - - # copy over the isort config file - if [[ ! -f ".isort.cfg" ]]; then - cp ../.github/.isort.cfg . - fi - - pip install wheel - pip install isort - isort . -c - - - job: pylint - pool: - vmImage: "ubuntu-22.04" - timeoutInMinutes: ${{ parameters.TIMEOUT }} - continueOnError: ${{ parameters.IGNORE_STYLE }} - condition: ${{ parameters.PYLINT }} - steps: - - checkout: self - - checkout: azure_template - - task: UsePythonVersion@0 - inputs: - versionSpec: "3.11" - - script: | - cd ${{ parameters.REPO_NAME }} - - # copy over the pylint config file - cp ../.github/.pylintrc . - - pip install pylint - find . -type f -name "*.py" -not -path "*/doc/*" | xargs pylint - - - job: clang_format - pool: - vmImage: "ubuntu-22.04" - timeoutInMinutes: ${{ parameters.TIMEOUT }} - continueOnError: ${{ parameters.IGNORE_STYLE }} - condition: ${{ parameters.CLANG_FORMAT }} - steps: - - checkout: self - - checkout: azure_template - - script: | - # Install prerequisites - sudo apt-get install clang-format-10 -y - - cd ${{ parameters.REPO_NAME }} - - # Check if we can access script, if not exit - if [[ ! -f ../.github/azure/clang-format.sh ]]; then - echo "clang-format.sh not found. Exiting." - exit 1 - fi - - # Run the formatting - bash ../.github/azure/clang-format.sh --dry-run || exit $? - - - job: fprettify - pool: - vmImage: "ubuntu-22.04" - timeoutInMinutes: ${{ parameters.TIMEOUT }} - continueOnError: ${{ parameters.IGNORE_STYLE }} - condition: ${{ parameters.FPRETTIFY }} - steps: - - checkout: self - - checkout: azure_template - - task: UsePythonVersion@0 - inputs: - versionSpec: "3.11" - - script: | - cd ${{ parameters.REPO_NAME }} - - # Check if we can access script, if not exit - if [[ ! -f ../.github/azure/fprettify.sh ]]; then - echo "fprettify.sh not found. Exiting." - exit 1 - fi - - # Install fprettify - pip install fprettify==0.3.7 - - # Run the formatting - bash ../.github/azure/fprettify.sh || exit $? - - # Exit with an error if any of the tracked files changed - git diff --summary --exit-code - - - job: mypy - pool: - vmImage: "ubuntu-22.04" - timeoutInMinutes: ${{ parameters.TIMEOUT }} - continueOnError: ${{ parameters.IGNORE_STYLE }} - condition: ${{ parameters.MYPY }} - strategy: - matrix: - "u24-gcc-ompi-latest": - DOCKER_TAG: u24-gcc-ompi-latest - variables: - - group: Docker - - name: SCRITICAL_HOMEDIR - value: /home/scriticaluser - - name: DOCKER_WORKING_DIR - value: ${{ variables.SCRITICAL_HOMEDIR }}/repos/${{ parameters.REPO_NAME }} - - name: DOCKER_MOUNT_DIR - value: ${{ variables.SCRITICAL_HOMEDIR }}/azure/${{ parameters.REPO_NAME }} - - name: BASHRC - value: ${{ variables.SCRITICAL_HOMEDIR }}/.bashrc_scritical - steps: - - checkout: self - - checkout: azure_template - - task: UsePythonVersion@0 - inputs: - versionSpec: "3.11" - - script: | - # This is a trusted build if DOCKER_USERNAME is defined - if [[ ! -z $(DOCKER_USERNAME) ]] ; then - echo $(DOCKER_PASSWORD) | docker login -u $(DOCKER_USERNAME) --password-stdin; - fi - # "auto" pulls the private image for trusted builds - if [[ "${{ parameters.IMAGE }}" == "private" ]] || [[ "${{ parameters.IMAGE }}" == "auto" && ! -z $(DOCKER_USERNAME) ]] ; then - export DOCKER_IMAGE=private-dev; - else - export DOCKER_IMAGE=public-dev; - fi - docker pull scritical/$DOCKER_IMAGE:$(DOCKER_TAG); - docker run -t -d --name app --mount "type=bind,src=$(pwd)/${{ parameters.REPO_NAME }},target=${{ variables.DOCKER_MOUNT_DIR }}" scritical/$DOCKER_IMAGE:$(DOCKER_TAG) /bin/bash; - docker exec app /bin/bash -c "rm -rf ${{ variables.DOCKER_WORKING_DIR }} && cp -r ${{ variables.DOCKER_MOUNT_DIR }} ${{ variables.DOCKER_WORKING_DIR }}"; - displayName: Prepare Repository - - script: | - set -e - docker exec app /bin/bash -c ". ${{ variables.BASHRC }} && cd ${{ variables.DOCKER_WORKING_DIR }} && mypy" - displayName: Run mypy - condition: succeeded() - - diff --git a/azure/azure_tapenade.yaml b/azure/azure_tapenade.yaml deleted file mode 100644 index a6209e4..0000000 --- a/azure/azure_tapenade.yaml +++ /dev/null @@ -1,45 +0,0 @@ -parameters: - - name: REPO_NAME - type: string - - name: TIMEOUT - type: number - - name: TAPENADE_SCRIPT - type: string - - name: TAPENADE_VERSION - type: string - -jobs: - - job: tapenade - pool: - vmImage: "ubuntu-22.04" - timeoutInMinutes: ${{ parameters.TIMEOUT }} - steps: - - task: JavaToolInstaller@0 - inputs: - versionSpec: "8" - jdkArchitectureOption: "x64" - jdkSourceOption: "PreInstalled" - - script: | - if [[ "${{ parameters.TAPENADE_VERSION }}" == "3.10" ]]; then - export TAPENADE_URL=https://websites.umich.edu/~mdolaboratory/misc/tapenade3.10.tar - export TAPENADE_HOME=$(pwd)/tapenade${{ parameters.TAPENADE_VERSION }} - elif [[ "${{ parameters.TAPENADE_VERSION }}" == "3.16" ]]; then - # The following link corresponds to tapenade_3.16-v2-723-ge8da61555.tar - export TAPENADE_URL="https://gitlab.inria.fr/tapenade/tapenade/-/package_files/112870/download -O tapenade_3.16.tar" - export TAPENADE_HOME=$(pwd)/tapenade_${{ parameters.TAPENADE_VERSION }} - fi - - # Install Tapenade - wget $TAPENADE_URL - tar -xzf $TAPENADE_HOME.tar - export PATH=$TAPENADE_HOME/bin:$PATH - - # Run Tapenade - touch config.mk # empty config file to make "make" happy - bash ${{ parameters.TAPENADE_SCRIPT }} - if [ $? -ne 0 ]; then - echo "Tapenade runscript failed" - exit 1 - fi - - git diff --exit-code diff --git a/azure/azure_template.yaml b/azure/azure_template.yaml deleted file mode 100644 index 783c31e..0000000 --- a/azure/azure_template.yaml +++ /dev/null @@ -1,139 +0,0 @@ -parameters: - - name: REPO_NAME - type: string - # Build and test - - name: COMPLEX - type: boolean - default: false - - name: TIMEOUT_BUILD - type: number - default: 120 - - name: GCC_CONFIG - type: string - default: None - - name: INTEL_CONFIG - type: string - default: None - - name: BUILD_REAL - type: string - default: .github/build_real.sh - - name: TEST_REAL - type: string - default: .github/test_real.sh - - name: BUILD_COMPLEX - type: string - default: .github/build_complex.sh - - name: TEST_COMPLEX - type: string - default: .github/test_complex.sh - - name: IMAGE - type: string - default: private - - name: COVERAGE - type: boolean - default: false - - name: C_COVERAGE - type: boolean - default: false - # Style checks - - name: TIMEOUT_STYLE - type: number - default: 10 - - name: IGNORE_STYLE - type: boolean - default: false - - name: BASE_FORMAT_AND_LINT - type: boolean - default: true - - name: ISORT - type: boolean - default: false - - name: PYLINT - type: boolean - default: false - - name: CLANG_FORMAT - type: boolean - default: false - - name: FPRETTIFY - type: boolean - default: false - - name: MYPY - type: boolean - default: false - # Tapenade - - name: TAPENADE - type: boolean - default: false - - name: TIMEOUT_TAPENADE - type: number - default: 10 - - name: TAPENADE_SCRIPT - type: string - default: .github/build_tapenade.sh - - name: TAPENADE_VERSION - type: string - default: 3.10 - - -stages: - - stage: Test_Real - dependsOn: [] - displayName: Real Test - jobs: - - template: azure_build.yaml - parameters: - REPO_NAME: ${{ parameters.REPO_NAME }} - TIMEOUT: ${{ parameters.TIMEOUT_BUILD }} - GCC_CONFIG: ${{ parameters.GCC_CONFIG }} - INTEL_CONFIG: ${{ parameters.INTEL_CONFIG }} - BUILD: ${{ parameters.BUILD_REAL }} - TEST: ${{ parameters.TEST_REAL }} - IMAGE: ${{ parameters.IMAGE }} - COVERAGE: ${{ parameters.COVERAGE }} - C_COVERAGE: ${{ parameters.C_COVERAGE }} - - - stage: Test_Complex - dependsOn: [] - displayName: Complex Test - condition: ${{ parameters.COMPLEX }} - jobs: - - template: azure_build.yaml - parameters: - REPO_NAME: ${{ parameters.REPO_NAME }} - TIMEOUT: ${{ parameters.TIMEOUT_BUILD }} - GCC_CONFIG: ${{ parameters.GCC_CONFIG }} - INTEL_CONFIG: ${{ parameters.INTEL_CONFIG }} - BUILD: ${{ parameters.BUILD_COMPLEX }} - TEST: ${{ parameters.TEST_COMPLEX }} - IMAGE: ${{ parameters.IMAGE }} - COVERAGE: ${{ parameters.COVERAGE }} - C_COVERAGE: ${{ parameters.C_COVERAGE }} - - - stage: Style - dependsOn: [] - displayName: Style Checks - jobs: - - template: azure_style.yaml - parameters: - REPO_NAME: ${{ parameters.REPO_NAME }} - IMAGE: ${{ parameters.IMAGE }} - TIMEOUT: ${{ parameters.TIMEOUT_STYLE }} - IGNORE_STYLE: ${{ parameters.IGNORE_STYLE }} - BASE_FORMAT_AND_LINT: ${{ parameters.BASE_FORMAT_AND_LINT }} - ISORT: ${{ parameters.ISORT }} - PYLINT: ${{ parameters.PYLINT }} - CLANG_FORMAT: ${{ parameters.CLANG_FORMAT }} - FPRETTIFY: ${{ parameters.FPRETTIFY }} - MYPY: ${{ parameters.MYPY }} - - - stage: Tapenade - dependsOn: [] - displayName: Tapenade Checks - condition: ${{ parameters.TAPENADE }} - jobs: - - template: azure_tapenade.yaml - parameters: - REPO_NAME: ${{ parameters.REPO_NAME }} - TIMEOUT: ${{ parameters.TIMEOUT_TAPENADE }} - TAPENADE_SCRIPT: ${{ parameters.TAPENADE_SCRIPT }} - TAPENADE_VERSION: ${{ parameters.TAPENADE_VERSION }} diff --git a/azure/clang-format.sh b/azure/clang-format.sh deleted file mode 100644 index ac6ded6..0000000 --- a/azure/clang-format.sh +++ /dev/null @@ -1,70 +0,0 @@ -#!/bin/bash - -# clang-format expects a .clang-format file in the root (local dir) where the command -# is run. Need to check for existing files first. If none copy over the global file -# and remove once done to make sure its not added to git. - -# -------------- Command line input -------------- -USAGE=" -usage: [-d] - -Argument description: - -d|--dry-run Do not make changes, only simulate formatting changes -" -DRY_RUN="" -while [[ "$#" -gt 0 ]]; do - case $1 in - -d|--dry-run) DRY_RUN="--dry-run"; shift ;; - -h|--help) echo "$USAGE"; exit 1 ;; - *) echo "Unknown parameter passed: $1"; echo "$USAGE"; exit 1 ;; - esac - shift -done - -# Set some constants -CLANGFORMAT_CONFIG_FILE="" -LOCAL_CONFIG_FILE=".clang-format" -GLOBAL_CONFIG_FILE="../.github/.clang-format" - -# Initialize variables -global=0 - -if [[ -f "$LOCAL_CONFIG_FILE" ]]; then - # Use local config file - echo "Using local config file: $LOCAL_CONFIG_FILE" - CLANGFORMAT_CONFIG_FILE="$LOCAL_CONFIG_FILE" -elif [[ -f "$GLOBAL_CONFIG_FILE" ]]; then - # Use global config file - echo "Using .github shared config file: $GLOBAL_CONFIG_FILE" - CLANGFORMAT_CONFIG_FILE="$GLOBAL_CONFIG_FILE" - cp "$GLOBAL_CONFIG_FILE" . - global=1 -else - echo "No clang-format config was found. Exiting" - exit 2 -fi - -# Setting common values -clang_format_args="-style=file --verbose --Werror" -if [[ -z "$DRY_RUN" ]]; then - # Apply the formatting inplace - clang_format_args="$clang_format_args -i" -else - # Only do a dry run - clang_format_args="$clang_format_args --dry-run" -fi - -echo "Running clang-format with args: $clang_format_args" - -# Run the formatting -find . -iname '*.h' -o -iname '*.hpp' -o -iname '*.c' -o -iname '*.cpp' | xargs clang-format $clang_format_args - -# Store the exit code in case we are running dry-run and find -clang_exit_code=$? - -if [[ 1 == "$global" ]]; then - # Remove only of we copied over - rm "$LOCAL_CONFIG_FILE" -fi - -exit $clang_exit_code \ No newline at end of file diff --git a/azure/combine-config.py b/azure/combine-config.py deleted file mode 100644 index 9de9d6a..0000000 --- a/azure/combine-config.py +++ /dev/null @@ -1,28 +0,0 @@ -""" -This script will read in two files via command line arguments -It will then merge them, and write it out to the appropriate output file name -""" -import argparse -import configparser - -parser = argparse.ArgumentParser() -parser.add_argument("default", type=str, help="file name for the default config file") -parser.add_argument("repo", type=str, help="file name for the repo-specific config file") -parser.add_argument("output", type=str, help="output file name") -args = parser.parse_args() - -# default config -config_def = configparser.ConfigParser() -config_def.read(args.default) -# project config -config_repo = configparser.ConfigParser() -config_repo.read(args.repo) - -# merge them together, with existing keys overwritten by project config -for config in config_def.keys(): - for key in config_repo[config].keys(): - config_def[config][key] = config_repo[config][key] - -# write out the file -with open(args.output, "w") as configfile: - config_def.write(configfile) diff --git a/azure/fprettify.sh b/azure/fprettify.sh deleted file mode 100644 index 91333e8..0000000 --- a/azure/fprettify.sh +++ /dev/null @@ -1,35 +0,0 @@ -#!/bin/bash - -FPRETTIFY_CONFIG_FILE="" -LOCAL_CONFIG_FILE=".fprettify.rc" -GLOBAL_CONFIG_FILE="../.github/.fprettify.rc" - -if [[ ! -z "$1" ]]; then - # Offer the option of supplying config file through an argument - echo "Using user supplied fprettify config file: $1" - FPRETTIFY_CONFIG_FILE="$1" -elif [[ -f "$LOCAL_CONFIG_FILE" ]]; then - # Use local config file - echo "Using local fprettify config file: $LOCAL_CONFIG_FILE" - FPRETTIFY_CONFIG_FILE="$LOCAL_CONFIG_FILE" -elif [[ -f "$GLOBAL_CONFIG_FILE" ]]; then - # Use global config file - echo "Using .github shared fprettify config file: $GLOBAL_CONFIG_FILE" - FPRETTIFY_CONFIG_FILE="$GLOBAL_CONFIG_FILE" -else - echo "No fprettify config was found. Exiting" - exit 2 -fi - -# Run fprettify on non-differentiated Fortran 90 source files -find . -type f -iname '*.f90' ! -iname '*_b.f90' ! -iname '*_d.f90' -print0 | while read -d $'\0' fileName -do - echo "Checking $fileName" - fprettify --config-file "$FPRETTIFY_CONFIG_FILE" "$fileName" - - # Check if this file changed and print a message if it did - git diff --summary --exit-code $fileName - if [ $? -ne 0 ]; then - echo "$fileName was formatted" - fi -done diff --git a/combine-config.py b/combine-config.py new file mode 100644 index 0000000..ad07de0 --- /dev/null +++ b/combine-config.py @@ -0,0 +1,59 @@ +#!/usr/bin/env python3 +""" +Merge two INI-style configuration files. + +This script reads a default config and a repo-specific config, +merges them (with repo-specific values taking precedence), +and writes the result to an output file. + +Usage: + python combine-config.py default.cfg repo.cfg output.cfg + +If the repo config file doesn't exist, the default config is used as-is. +""" +import argparse +import configparser +import sys +from pathlib import Path + + +def merge_configs(default_path: str, repo_path: str, output_path: str) -> None: + """Merge default and repo configs, writing result to output.""" + # Read default config + config = configparser.ConfigParser() + config.read(default_path) + + # Read and merge repo config if it exists + repo_file = Path(repo_path) + if repo_file.exists(): + config_repo = configparser.ConfigParser() + config_repo.read(repo_path) + + # Merge: repo config overrides default + for section in config_repo.sections(): + if not config.has_section(section): + config.add_section(section) + for key, value in config_repo.items(section): + config.set(section, key, value) + + # Write merged config + with open(output_path, "w") as f: + config.write(f) + + +def main() -> int: + """Parse arguments and run merge.""" + parser = argparse.ArgumentParser( + description="Merge two INI-style configuration files" + ) + parser.add_argument("default", type=str, help="Path to default config file") + parser.add_argument("repo", type=str, help="Path to repo-specific config file") + parser.add_argument("output", type=str, help="Path for merged output file") + args = parser.parse_args() + + merge_configs(args.default, args.repo, args.output) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/ruff.toml b/ruff.toml index d59433f..96f0ce3 100644 --- a/ruff.toml +++ b/ruff.toml @@ -18,7 +18,7 @@ ignore = [ "E501", # line length, exceeded by some docstrings "W605", # TODO (to be fixed), invalid escape sequence, necessary for sphinx directives in docstrings but should switch to raw string "D301", # Same reason as W605 - "D10", # D100-107, missing docstring in public function, class etc (yes we should have docstrings for all public methods, but that is a pipe dream) + "D10", # D100-107, missing docstring in public function, class etc (yes we should have docstrings for all public methods, but that is a pipe dream) "D400", # First line of docstring should end with a period (assumes the summary fits on one line, which is not always the case) "D205", # 1 blank line required between summary line and description (Makes same assumption as D400) "D200", # One-line docstring should fit on one line, don't like this because it enforces inconsistent formatting between one-line and multi-line docstrings @@ -27,17 +27,14 @@ ignore = [ "D404", # First word of the docstring should not be "This" (Probably a good rule to follow going forward but it occurs in so many places in our code and it's not a big enough issue to warrant fixing) ] -select = [ - "B", - "D", - "E", - "F", - "W", -] +select = ["B", "D", "E", "F", "W"] [lint.pydocstyle] convention = "numpy" +[lint.mccabe] +max-complexity = 10 + [format] quote-style = "double" indent-style = "space" From 1d0a80e665caf4b98fac0a772e2cc6b7fbe45f3f Mon Sep 17 00:00:00 2001 From: "Anthony V. Ozdemir" <20494900+anthony-ozdemir@users.noreply.github.com> Date: Tue, 3 Feb 2026 15:32:57 -0500 Subject: [PATCH 02/50] Fix strategy matrix for mypy workflow --- .github/workflows/mypy.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/mypy.yaml b/.github/workflows/mypy.yaml index 30f6e10..37bd251 100644 --- a/.github/workflows/mypy.yaml +++ b/.github/workflows/mypy.yaml @@ -6,7 +6,7 @@ jobs: runs-on: ubuntu-24.04 strategy: matrix: - python-version: "3.11" + python-version: ["3.11"] steps: - uses: actions/checkout@v5 - name: Set up Python ${{ matrix.python-version }} From b3cfd71eca5b5b9ce85cad6a59ab2773abe6c5a6 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin <25615947+lamkina@users.noreply.github.com> Date: Wed, 4 Feb 2026 10:50:37 -0800 Subject: [PATCH 03/50] Update .github/workflows/fprettify.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .github/workflows/fprettify.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/fprettify.yaml b/.github/workflows/fprettify.yaml index f22bc4e..9b52cd1 100644 --- a/.github/workflows/fprettify.yaml +++ b/.github/workflows/fprettify.yaml @@ -29,7 +29,7 @@ jobs: if [[ ! -f "$LOCAL_CONFIG_FILE" ]]; then echo "Downloading global fprettify config..." - wget "$GLOBAL_CONFIG_URL" -O "$LOCAL_CONFIG_FILE" || echo "No global config found, using defaults" + wget "$GLOBAL_CONFIG_URL" -O "$LOCAL_CONFIG_FILE" || echo "No global config found; fprettify will use its built-in defaults" fi - name: Run fprettify From 8db7f40c78463bd100acfea119092897c96aa6a8 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin <25615947+lamkina@users.noreply.github.com> Date: Wed, 4 Feb 2026 10:50:54 -0800 Subject: [PATCH 04/50] Update .github/workflows/clang_format.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .github/workflows/clang_format.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/clang_format.yaml b/.github/workflows/clang_format.yaml index e2b8bb9..016b6a9 100644 --- a/.github/workflows/clang_format.yaml +++ b/.github/workflows/clang_format.yaml @@ -24,7 +24,7 @@ jobs: if [[ ! -f "$LOCAL_CONFIG_FILE" ]]; then echo "Downloading global clang-format config..." - wget "$GLOBAL_CONFIG_URL" -O "$LOCAL_CONFIG_FILE" || echo "No global config found" + wget "$GLOBAL_CONFIG_URL" -O "$LOCAL_CONFIG_FILE" || echo "No global clang-format config found; proceeding without a config file. The subsequent clang-format step may fail because -style=file requires a .clang-format configuration." fi - name: Run clang-format From 88c8ce4e6d678f77e3c35444620fefacea2f4e0a Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 4 Feb 2026 18:55:35 +0000 Subject: [PATCH 05/50] Removing reference to azure in root README --- README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/README.md b/README.md index 6b306c3..6738095 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,6 @@ This repo stores the following shared repository settings/configurations/templat - `pylint` - `codecov` - `mypy` -- Shared configurations for Azure Pipelines - Shared configurations for GitHub Actions - Issue/PR labels shared across the organization - Configuration script for the build bot From ec760ccd0c29f617bc53fc5c138514d66168ad32 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 4 Feb 2026 18:56:27 +0000 Subject: [PATCH 06/50] Removing pypi workflow --- .github/workflows/pypi.yaml | 26 -------------------------- 1 file changed, 26 deletions(-) delete mode 100644 .github/workflows/pypi.yaml diff --git a/.github/workflows/pypi.yaml b/.github/workflows/pypi.yaml deleted file mode 100644 index a9243ae..0000000 --- a/.github/workflows/pypi.yaml +++ /dev/null @@ -1,26 +0,0 @@ -on: - workflow_call: - secrets: - PYPI_API_TOKEN: - required: true - -jobs: - pypi: - if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags') - runs-on: ubuntu-24.04 - steps: - - uses: actions/checkout@v5 - - uses: actions/setup-python@v6 - with: - python-version: "3.11" - - name: Build package - run: | - python --version - python -m pip install -U pip - pip install setuptools wheel - python setup.py bdist_wheel - - name: Publish package - uses: pypa/gh-action-pypi-publish@release/v1 - with: - user: __token__ - password: ${{ secrets.PYPI_API_TOKEN }} From 88e9fd6c3d2012e2ebf820fe1b630bf15e30634d Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 4 Feb 2026 22:48:06 +0000 Subject: [PATCH 07/50] Adding specific ruff workflow and removing old format and lint workflow --- .github/workflows/format-and-lint.yaml | 45 ----------------------- .github/workflows/ruff.yml | 49 ++++++++++++++++++++++++++ ruff.toml | 2 +- 3 files changed, 50 insertions(+), 46 deletions(-) delete mode 100644 .github/workflows/format-and-lint.yaml create mode 100644 .github/workflows/ruff.yml diff --git a/.github/workflows/format-and-lint.yaml b/.github/workflows/format-and-lint.yaml deleted file mode 100644 index e426cf0..0000000 --- a/.github/workflows/format-and-lint.yaml +++ /dev/null @@ -1,45 +0,0 @@ -on: - workflow_call: - inputs: - MCCABE: - type: boolean - default: false - description: "Enable McCabe complexity check (pass/fail)" - -jobs: - format-and-lint: - runs-on: ubuntu-24.04 - steps: - - uses: actions/checkout@v5 - - name: Set up Python - uses: actions/setup-python@v6 - with: - python-version: "3.11" - - name: Install dependencies - run: | - wget https://raw.githubusercontent.com/scritical/.github/main/.pre-commit-config.yaml - pip install pre-commit - - # Set the ruff config in the .github repo as the global config - # Repos can extend it by including `extend = "~/.config/ruff/ruff.toml"` in their local ruff.toml - mkdir -p ~/.config/ruff - wget https://raw.githubusercontent.com/scritical/.github/main/ruff.toml -O ~/.config/ruff/ruff.toml - - echo "=== Global ruff config ===" - cat ~/.config/ruff/ruff.toml - if [[ -f "ruff.toml" ]]; then - echo "=== Local ruff config ===" - cat ruff.toml - fi - - - name: Run pre-commit checks - run: | - pre-commit run --all-files --show-diff-on-failure - - - name: McCabe complexity check - if: inputs.MCCABE - run: | - echo "=== McCabe Complexity Report ===" - echo "Checking for functions with complexity > 10..." - pip install ruff - ruff check --select=C90 --output-format=text . diff --git a/.github/workflows/ruff.yml b/.github/workflows/ruff.yml new file mode 100644 index 0000000..6563141 --- /dev/null +++ b/.github/workflows/ruff.yml @@ -0,0 +1,49 @@ +on: + workflow_call: + inputs: + MCCABE: + type: boolean + default: false + description: "Enable McCabe complexity check (pass/fail)" + + +jobs: + ruff: + runs-on: ubuntu-24.04 + steps: + - name: Checkout project + uses: actions/checkout@v5 + + - name: Checkout org-wide configs + uses: actions/checkout@v5 + with: + repository: scritical/.github + path: .org-dotgithub + fetch-depth: 1 + + - name: Set up Python + uses: actions/setup-python@v6 + with: + python-version: "3.11" + + - name: Install Ruff + run: | + python -m pip install --upgrade pip + pip install ruff + + - name: Ruff formatting + run: | + echo "=== Ruff Formatting Check ===" + ruff format --check --exit-non-zero-on-fix --config .org-dotgithub/ruff.toml --output-format=github . + + - name: Ruff linting + run: | + echo "=== Ruff Linting Check ===" + ruff check --exit-zero-on-fix --config .org-dotgithub/ruff.toml --output-format=github . + + - name: McCabe complexity check + if: inputs.MCCABE + run: | + echo "=== McCabe Complexity Report ===" + echo "Checking for functions with complexity > 10..." + ruff check --select=C90 --output-format=github . \ No newline at end of file diff --git a/ruff.toml b/ruff.toml index 96f0ce3..080c8b9 100644 --- a/ruff.toml +++ b/ruff.toml @@ -1,5 +1,5 @@ # Minimum Python syntax support -target-version = "py39" +target-version = "py311" line-length = 120 From ff3a1e4fc133920392369eef731ccbe7e1f54d8b Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 4 Feb 2026 22:58:21 +0000 Subject: [PATCH 08/50] updating ruff yaml extension --- .github/workflows/{ruff.yml => ruff.yaml} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename .github/workflows/{ruff.yml => ruff.yaml} (100%) diff --git a/.github/workflows/ruff.yml b/.github/workflows/ruff.yaml similarity index 100% rename from .github/workflows/ruff.yml rename to .github/workflows/ruff.yaml From 34c54313552db11271d70d0f58d04ac0c9ae0ef2 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 4 Feb 2026 23:11:55 +0000 Subject: [PATCH 09/50] Fixing indentation --- .github/workflows/ruff.yaml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/ruff.yaml b/.github/workflows/ruff.yaml index 6563141..0538bd5 100644 --- a/.github/workflows/ruff.yaml +++ b/.github/workflows/ruff.yaml @@ -1,10 +1,10 @@ on: workflow_call: inputs: - MCCABE: - type: boolean - default: false - description: "Enable McCabe complexity check (pass/fail)" + MCCABE: + type: boolean + default: false + description: "Enable McCabe complexity check (pass/fail)" jobs: From e484d0c71ded9099a4367211497e67b9b3832d5d Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Thu, 5 Feb 2026 00:41:19 +0000 Subject: [PATCH 10/50] Updating ruff workflow to fix gha error --- .github/workflows/ruff.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ruff.yaml b/.github/workflows/ruff.yaml index 0538bd5..d7f21d4 100644 --- a/.github/workflows/ruff.yaml +++ b/.github/workflows/ruff.yaml @@ -34,16 +34,16 @@ jobs: - name: Ruff formatting run: | echo "=== Ruff Formatting Check ===" - ruff format --check --exit-non-zero-on-fix --config .org-dotgithub/ruff.toml --output-format=github . + ruff format --check --exit-non-zero-on-fix --config .org-dotgithub/ruff.toml - name: Ruff linting run: | echo "=== Ruff Linting Check ===" - ruff check --exit-zero-on-fix --config .org-dotgithub/ruff.toml --output-format=github . + ruff check --exit-zero-on-fix --config .org-dotgithub/ruff.toml - name: McCabe complexity check if: inputs.MCCABE run: | echo "=== McCabe Complexity Report ===" echo "Checking for functions with complexity > 10..." - ruff check --select=C90 --output-format=github . \ No newline at end of file + ruff check --select=C90 \ No newline at end of file From f280eb305085f282f761169c72ea8dca583b8f64 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Thu, 5 Feb 2026 00:47:55 +0000 Subject: [PATCH 11/50] Testing ruff pwd --- .github/workflows/ruff.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/ruff.yaml b/.github/workflows/ruff.yaml index d7f21d4..e062c08 100644 --- a/.github/workflows/ruff.yaml +++ b/.github/workflows/ruff.yaml @@ -34,6 +34,8 @@ jobs: - name: Ruff formatting run: | echo "=== Ruff Formatting Check ===" + CURRENT_REPO_DIR=$(pwd) + echo "Current repo directory: $CURRENT_REPO_DIR" ruff format --check --exit-non-zero-on-fix --config .org-dotgithub/ruff.toml - name: Ruff linting From 042b4ebe3a6e2a6221993e9a89e4d23084d8c5ad Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Thu, 5 Feb 2026 02:25:18 +0000 Subject: [PATCH 12/50] Fixing ruff directory issues --- .github/workflows/ruff.yaml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/ruff.yaml b/.github/workflows/ruff.yaml index e062c08..4a29ad2 100644 --- a/.github/workflows/ruff.yaml +++ b/.github/workflows/ruff.yaml @@ -11,14 +11,14 @@ jobs: ruff: runs-on: ubuntu-24.04 steps: - - name: Checkout project + - name: Checkout repository uses: actions/checkout@v5 - name: Checkout org-wide configs uses: actions/checkout@v5 with: repository: scritical/.github - path: .org-dotgithub + path: ~/.org-dotgithub fetch-depth: 1 - name: Set up Python @@ -36,12 +36,12 @@ jobs: echo "=== Ruff Formatting Check ===" CURRENT_REPO_DIR=$(pwd) echo "Current repo directory: $CURRENT_REPO_DIR" - ruff format --check --exit-non-zero-on-fix --config .org-dotgithub/ruff.toml + ruff format --check --exit-non-zero-on-fix --config ~/.org-dotgithub/ruff.toml - name: Ruff linting run: | echo "=== Ruff Linting Check ===" - ruff check --exit-zero-on-fix --config .org-dotgithub/ruff.toml + ruff check --exit-zero-on-fix --config ~/.org-dotgithub/ruff.toml - name: McCabe complexity check if: inputs.MCCABE From d98f17717aaba09d426c25ac93b67b49c11bc733 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Thu, 5 Feb 2026 02:28:49 +0000 Subject: [PATCH 13/50] Debugging ruff formatting path --- .github/workflows/ruff.yaml | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ruff.yaml b/.github/workflows/ruff.yaml index 4a29ad2..ec2c944 100644 --- a/.github/workflows/ruff.yaml +++ b/.github/workflows/ruff.yaml @@ -18,7 +18,7 @@ jobs: uses: actions/checkout@v5 with: repository: scritical/.github - path: ~/.org-dotgithub + path: .org-dotgithub fetch-depth: 1 - name: Set up Python @@ -36,12 +36,16 @@ jobs: echo "=== Ruff Formatting Check ===" CURRENT_REPO_DIR=$(pwd) echo "Current repo directory: $CURRENT_REPO_DIR" - ruff format --check --exit-non-zero-on-fix --config ~/.org-dotgithub/ruff.toml + ls -la + echo "" + echo "One level up: ${CURRENT_REPO_DIR}/.." + ls -la .. + ruff format --check --exit-non-zero-on-fix --config .org-dotgithub/ruff.toml - name: Ruff linting run: | echo "=== Ruff Linting Check ===" - ruff check --exit-zero-on-fix --config ~/.org-dotgithub/ruff.toml + ruff check --exit-zero-on-fix --config .org-dotgithub/ruff.toml - name: McCabe complexity check if: inputs.MCCABE From ea9670009368454724382604b017b9e8ad862811 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Thu, 5 Feb 2026 02:35:30 +0000 Subject: [PATCH 14/50] Trying to fix path issues with gha and ruff --- .github/workflows/ruff.yaml | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/.github/workflows/ruff.yaml b/.github/workflows/ruff.yaml index ec2c944..4762a58 100644 --- a/.github/workflows/ruff.yaml +++ b/.github/workflows/ruff.yaml @@ -10,9 +10,14 @@ on: jobs: ruff: runs-on: ubuntu-24.04 + defaults: + run: + working-directory: project steps: - name: Checkout repository uses: actions/checkout@v5 + with: + path: project - name: Checkout org-wide configs uses: actions/checkout@v5 @@ -34,22 +39,16 @@ jobs: - name: Ruff formatting run: | echo "=== Ruff Formatting Check ===" - CURRENT_REPO_DIR=$(pwd) - echo "Current repo directory: $CURRENT_REPO_DIR" - ls -la - echo "" - echo "One level up: ${CURRENT_REPO_DIR}/.." - ls -la .. - ruff format --check --exit-non-zero-on-fix --config .org-dotgithub/ruff.toml + ruff format --check --exit-non-zero-on-fix --config ../.org-dotgithub/ruff.toml . - name: Ruff linting run: | echo "=== Ruff Linting Check ===" - ruff check --exit-zero-on-fix --config .org-dotgithub/ruff.toml + ruff check --exit-zero-on-fix --config ../.org-dotgithub/ruff.toml . - name: McCabe complexity check if: inputs.MCCABE run: | echo "=== McCabe Complexity Report ===" echo "Checking for functions with complexity > 10..." - ruff check --select=C90 \ No newline at end of file + ruff check --select=C90 . \ No newline at end of file From 5ac78f50e1c4e295ab94b3c82baa2f610c2cb038 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Thu, 5 Feb 2026 02:39:37 +0000 Subject: [PATCH 15/50] Fixing an argument bug --- .github/workflows/ruff.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ruff.yaml b/.github/workflows/ruff.yaml index 4762a58..f8bf8f6 100644 --- a/.github/workflows/ruff.yaml +++ b/.github/workflows/ruff.yaml @@ -44,7 +44,7 @@ jobs: - name: Ruff linting run: | echo "=== Ruff Linting Check ===" - ruff check --exit-zero-on-fix --config ../.org-dotgithub/ruff.toml . + ruff check --exit-non-zero-on-fix --config ../.org-dotgithub/ruff.toml . - name: McCabe complexity check if: inputs.MCCABE From 3f5145967a2b63bf4bba33bfb3dd1a1a1b3e4902 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Thu, 5 Feb 2026 03:34:24 +0000 Subject: [PATCH 16/50] Trying to fix build bug with paths --- .github/workflows/build.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index b89d70f..a64e2fe 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -52,7 +52,7 @@ jobs: scritical/private-dev:${{ env.DOCKER_TAG }} /bin/bash # Copy repo to working directory - docker exec app /bin/bash -c "cp -r ${{ env.SCRITICAL_HOMEDIR }}/mount/${REPO_NAME} ${{ env.SCRITICAL_HOMEDIR }}/repos/${REPO_NAME}" + docker exec app /bin/bash -c "rm -rf ${{ env.SCRITICAL_HOMEDIR }}/repos/${REPO_NAME} && mkdir -p ${{ env.SCRITICAL_HOMEDIR }}/repos/${REPO_NAME} && cp -a ${{ env.SCRITICAL_HOMEDIR }}/mount/${REPO_NAME}/. ${{ env.SCRITICAL_HOMEDIR }}/repos/${REPO_NAME}/" - name: Build if: inputs.BUILD_SCRIPT != '' From b6f50962e92a0748f03dec7aae8ff534d0c14902 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Thu, 5 Feb 2026 04:04:44 +0000 Subject: [PATCH 17/50] Making the branch name check a callable workflow --- .github/workflows/branch-name-check.yml | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/.github/workflows/branch-name-check.yml b/.github/workflows/branch-name-check.yml index f02ea69..3e60609 100644 --- a/.github/workflows/branch-name-check.yml +++ b/.github/workflows/branch-name-check.yml @@ -1,15 +1,14 @@ name: Branch Name Check on: - pull_request: - types: [opened, synchronize, reopened, edited] + workflow_call: branches: - main - 'client-*' jobs: check-branch-name: - runs-on: ubuntu-latest + runs-on: ubuntu-24.04 steps: - name: Check branch name for main if: github.base_ref == 'main' From 342d9dcd77144db126b78c2f4bca068178230dab Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Thu, 5 Feb 2026 04:06:48 +0000 Subject: [PATCH 18/50] Fixing bug with branch workflow --- .github/workflows/branch-name-check.yml | 5 ----- 1 file changed, 5 deletions(-) diff --git a/.github/workflows/branch-name-check.yml b/.github/workflows/branch-name-check.yml index 3e60609..cd70185 100644 --- a/.github/workflows/branch-name-check.yml +++ b/.github/workflows/branch-name-check.yml @@ -1,10 +1,5 @@ -name: Branch Name Check - on: workflow_call: - branches: - - main - - 'client-*' jobs: check-branch-name: From c08da3d4746f596284027245d7401ddf3e818f5c Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Thu, 5 Feb 2026 04:23:53 +0000 Subject: [PATCH 19/50] Removing pylint --- .github/workflows/pylint.yaml | 35 ----------------------------------- 1 file changed, 35 deletions(-) delete mode 100644 .github/workflows/pylint.yaml diff --git a/.github/workflows/pylint.yaml b/.github/workflows/pylint.yaml deleted file mode 100644 index cd42e13..0000000 --- a/.github/workflows/pylint.yaml +++ /dev/null @@ -1,35 +0,0 @@ -on: - workflow_call: - -jobs: - pylint: - runs-on: ubuntu-24.04 - steps: - - uses: actions/checkout@v5 - - name: Set up Python - uses: actions/setup-python@v6 - with: - python-version: "3.11" - - name: Install dependencies - run: | - pip install wheel pylint - - name: Prepare pylint config - run: | - # Download global config - wget https://raw.githubusercontent.com/scritical/.github/main/.pylintrc -O .pylintrc-global - - # Check for local overrides and merge if present - if [[ -f ".pylintrc" ]]; then - echo "Found local .pylintrc, merging with global config..." - wget https://raw.githubusercontent.com/scritical/.github/main/combine-config.py - python combine-config.py .pylintrc-global .pylintrc .pylintrc-merged - mv .pylintrc-merged .pylintrc - rm combine-config.py - else - mv .pylintrc-global .pylintrc - fi - rm -f .pylintrc-global - - - name: Run pylint - run: | - find . -type f -name "*.py" -not -path "*/doc/*" | xargs pylint From 25b908fffc4d8f222695231a556a4566cbcdea26 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Thu, 5 Feb 2026 04:27:12 +0000 Subject: [PATCH 20/50] Improving docstrings --- combine-config.py | 54 ++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 44 insertions(+), 10 deletions(-) diff --git a/combine-config.py b/combine-config.py index ad07de0..9b8c282 100644 --- a/combine-config.py +++ b/combine-config.py @@ -1,15 +1,25 @@ #!/usr/bin/env python3 -""" -Merge two INI-style configuration files. +"""Merge two INI-style configuration files. + +This script merges a *default* INI config with an optional *repo-specific* INI +config, writing the merged result to an output path. If the repo-specific +config file does not exist, the default config is written as-is. + +Notes +----- +The merge behavior is: + +- Sections present only in the default config are preserved. +- Sections present in the repo config are added if missing. +- For keys present in both configs, the repo config value overrides the + default. -This script reads a default config and a repo-specific config, -merges them (with repo-specific values taking precedence), -and writes the result to an output file. +Examples +-------- +From the command line:: -Usage: - python combine-config.py default.cfg repo.cfg output.cfg + python combine-config.py default.cfg repo.cfg merged.cfg -If the repo config file doesn't exist, the default config is used as-is. """ import argparse import configparser @@ -18,7 +28,23 @@ def merge_configs(default_path: str, repo_path: str, output_path: str) -> None: - """Merge default and repo configs, writing result to output.""" + """Merge a default config with an optional repo config. + + Parameters + ---------- + default_path : str + Path to the default INI-style configuration file. + repo_path : str + Path to the repo-specific INI-style configuration file. If this file + does not exist, the default config is used without modification. + output_path : str + Path to write the merged INI configuration. + + Returns + ------- + None + This function writes the merged configuration to ``output_path``. + """ # Read default config config = configparser.ConfigParser() config.read(default_path) @@ -42,7 +68,15 @@ def merge_configs(default_path: str, repo_path: str, output_path: str) -> None: def main() -> int: - """Parse arguments and run merge.""" + """CLI entry point. + + Parses command-line arguments and writes a merged configuration file. + + Returns + ------- + int + Process exit code. Returns ``0`` on successful completion. + """ parser = argparse.ArgumentParser( description="Merge two INI-style configuration files" ) From 5baa5892c80dafc52f47b66fa721def7eeda03bb Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Tue, 10 Feb 2026 21:52:38 +0000 Subject: [PATCH 21/50] Changing all .yml to .yaml for consistency --- .codecov.yml => .codecov.yaml | 0 .github/{dependabot.yml => dependabot.yaml} | 0 .github/{stale.yml => stale.yaml} | 0 .../workflows/{branch-name-check.yml => branch-name-check.yaml} | 0 4 files changed, 0 insertions(+), 0 deletions(-) rename .codecov.yml => .codecov.yaml (100%) rename .github/{dependabot.yml => dependabot.yaml} (100%) rename .github/{stale.yml => stale.yaml} (100%) rename .github/workflows/{branch-name-check.yml => branch-name-check.yaml} (100%) diff --git a/.codecov.yml b/.codecov.yaml similarity index 100% rename from .codecov.yml rename to .codecov.yaml diff --git a/.github/dependabot.yml b/.github/dependabot.yaml similarity index 100% rename from .github/dependabot.yml rename to .github/dependabot.yaml diff --git a/.github/stale.yml b/.github/stale.yaml similarity index 100% rename from .github/stale.yml rename to .github/stale.yaml diff --git a/.github/workflows/branch-name-check.yml b/.github/workflows/branch-name-check.yaml similarity index 100% rename from .github/workflows/branch-name-check.yml rename to .github/workflows/branch-name-check.yaml From e7fb61b0712e516300b62bd01b47a72870bb8ef6 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Tue, 10 Feb 2026 21:55:20 +0000 Subject: [PATCH 22/50] Adding docker login action instead of command --- .github/workflows/build.yaml | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index a64e2fe..b8c7259 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -9,6 +9,10 @@ on: type: string default: "" description: "Path to GCC configuration file" + INTEL_CONFIG: + type: string + default: "" + description: "Path to Intel configuration file" BUILD_SCRIPT: type: string default: ".github/build_real.sh" @@ -39,8 +43,10 @@ jobs: run: echo "name=${GITHUB_REPOSITORY##*/}" >> $GITHUB_OUTPUT - name: Login to Docker Hub - run: echo "${{ secrets.DOCKER_OAT }}" | docker login -u scritical --password-stdin - + uses: docker/login-action@v3 + with: + username: ${{ secrets.DOCKER_USER }} + password: ${{ secrets.DOCKER_OAT }} - name: Pull Docker image run: docker pull scritical/private-dev:${{ env.DOCKER_TAG }} From 94e32930b072e0b90b503b819e0e296276ec02fe Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 11 Feb 2026 00:21:54 +0000 Subject: [PATCH 23/50] Removing isort workflow and config in favor of ruff --- .github/workflows/isort.yaml | 30 ------------------------------ .isort.cfg | 9 --------- 2 files changed, 39 deletions(-) delete mode 100644 .github/workflows/isort.yaml delete mode 100644 .isort.cfg diff --git a/.github/workflows/isort.yaml b/.github/workflows/isort.yaml deleted file mode 100644 index de5271a..0000000 --- a/.github/workflows/isort.yaml +++ /dev/null @@ -1,30 +0,0 @@ -on: - workflow_call: - -jobs: - isort: - runs-on: ubuntu-24.04 - steps: - - uses: actions/checkout@v5 - - name: Set up Python - uses: actions/setup-python@v6 - with: - python-version: "3.11" - - name: Prepare isort config - run: | - # Download global config - wget https://raw.githubusercontent.com/scritical/.github/main/.isort.cfg -O .isort.cfg-global - - # Check for local overrides and merge if present - if [[ -f ".isort.cfg" ]]; then - echo "Found local .isort.cfg, merging with global config..." - wget https://raw.githubusercontent.com/scritical/.github/main/combine-config.py - python combine-config.py .isort.cfg-global .isort.cfg .isort.cfg-merged - mv .isort.cfg-merged .isort.cfg - rm combine-config.py - else - mv .isort.cfg-global .isort.cfg - fi - rm -f .isort.cfg-global - - - uses: isort/isort-action@master diff --git a/.isort.cfg b/.isort.cfg deleted file mode 100644 index d6b8b9d..0000000 --- a/.isort.cfg +++ /dev/null @@ -1,9 +0,0 @@ -[isort] -profile=black -line_length=120 -force_sort_within_sections=true -import_heading_stdlib=Standard Python modules -import_heading_thirdparty=External modules -import_heading_firstparty=First party modules -import_heading_localfolder=Local modules -skip_glob=**__init__.py,**setup.py From f8a702ec5d68b03b1d9e21f12f060668381d573b Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 11 Feb 2026 00:23:51 +0000 Subject: [PATCH 24/50] Adding import sorting to the select --- ruff.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ruff.toml b/ruff.toml index 080c8b9..3d0d9e3 100644 --- a/ruff.toml +++ b/ruff.toml @@ -27,7 +27,7 @@ ignore = [ "D404", # First word of the docstring should not be "This" (Probably a good rule to follow going forward but it occurs in so many places in our code and it's not a big enough issue to warrant fixing) ] -select = ["B", "D", "E", "F", "W"] +select = ["B", "D", "E", "F", "W", "I"] [lint.pydocstyle] convention = "numpy" From da811a1b84b9be3a515c9e27c3df5506f443edae Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 11 Feb 2026 00:50:07 +0000 Subject: [PATCH 25/50] Refactoring build script and adding intel, removing standalone mypy workflow, and updating settings for ruffimport sorting --- .github/workflows/build.yaml | 72 ++++++++++++++++++++++++++---------- .github/workflows/mypy.yaml | 23 ------------ .github/workflows/ruff.yaml | 12 +++--- ruff.toml | 3 ++ 4 files changed, 62 insertions(+), 48 deletions(-) delete mode 100644 .github/workflows/mypy.yaml diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index b8c7259..c262e14 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -21,62 +21,94 @@ on: type: string default: ".github/test_real.sh" description: "Path to test script" + BUILD: + type: boolean + default: true + description: "Whether to run the build step" + TEST: + type: boolean + default: true + description: "Whether to run the test step" + MYPY: + type: boolean + default: false + description: "Whether to run mypy type checking" secrets: + DOCKER_USER: + required: true + description: "Docker registry username" DOCKER_OAT: required: true description: "Docker registry Organization Access Token" env: SCRITICAL_HOMEDIR: /home/scriticaluser - DOCKER_TAG: u24-gcc-ompi-latest + BASHRC: ${{ env.SCRITICAL_HOMEDIR }}/.bashrc_scritical jobs: build-and-test: runs-on: ubuntu-24.04 timeout-minutes: ${{ inputs.TIMEOUT }} + strategy: + matrix: + include: + - DOCKER_TAG: u24-gcc-ompi-latest + CONFIG_FILE: ${{ inputs.GCC_CONFIG }} + - DOCKER_TAG: u24-intel-ompi-latest + CONFIG_FILE: ${{ inputs.INTEL_CONFIG }} steps: - name: Checkout repository - uses: actions/checkout@v5 - - - name: Get repository name - id: repo - run: echo "name=${GITHUB_REPOSITORY##*/}" >> $GITHUB_OUTPUT + uses: actions/checkout@v6 + - name: Set repo environment variables + run: | + repo_name="${{ github.event.repository.name }}" + echo "REPO_NAME=$repo_name" >> "$GITHUB_ENV" + echo "DOCKER_WORKING_DIR=${{ env.SCRITICAL_HOMEDIR }}/repos/$repo_name" >> "$GITHUB_ENV" + echo "DOCKER_MOUNT_DIR=${{ env.SCRITICAL_HOMEDIR }}/mount/$repo_name" >> "$GITHUB_ENV" + - name: Login to Docker Hub uses: docker/login-action@v3 with: username: ${{ secrets.DOCKER_USER }} password: ${{ secrets.DOCKER_OAT }} - - name: Pull Docker image - run: docker pull scritical/private-dev:${{ env.DOCKER_TAG }} - name: Start Docker container run: | - REPO_NAME="${{ steps.repo.outputs.name }}" docker run -t -d --name app \ - --mount "type=bind,src=${{ github.workspace }},target=${{ env.SCRITICAL_HOMEDIR }}/mount/${REPO_NAME}" \ - scritical/private-dev:${{ env.DOCKER_TAG }} /bin/bash + --mount "type=bind,src=${{ github.workspace }},target=${{ env.DOCKER_MOUNT_DIR }}" \ + scritical/private-dev:${{ matrix.DOCKER_TAG }} /bin/bash # Copy repo to working directory - docker exec app /bin/bash -c "rm -rf ${{ env.SCRITICAL_HOMEDIR }}/repos/${REPO_NAME} && mkdir -p ${{ env.SCRITICAL_HOMEDIR }}/repos/${REPO_NAME} && cp -a ${{ env.SCRITICAL_HOMEDIR }}/mount/${REPO_NAME}/. ${{ env.SCRITICAL_HOMEDIR }}/repos/${REPO_NAME}/" + docker exec app /bin/bash -c "rm -rf $DOCKER_WORKING_DIR && mkdir -p $DOCKER_WORKING_DIR && cp -a $DOCKER_MOUNT_DIR/. $DOCKER_WORKING_DIR/" - name: Build - if: inputs.BUILD_SCRIPT != '' + if: inputs.BUILD_SCRIPT != '' && inputs.BUILD run: | - REPO_NAME="${{ steps.repo.outputs.name }}" docker exec \ - -e CONFIG_FILE="${{ inputs.GCC_CONFIG }}" \ - app /bin/bash -c ". ${{ env.SCRITICAL_HOMEDIR }}/.bashrc_scritical && cd ${{ env.SCRITICAL_HOMEDIR }}/repos/${REPO_NAME} && /bin/bash ${{ inputs.BUILD_SCRIPT }}" + -e CONFIG_FILE="${{ matrix.CONFIG_FILE }}" \ + app /bin/bash -c ". $BASHRC && cd $DOCKER_WORKING_DIR && /bin/bash ${{ inputs.BUILD_SCRIPT }}" - name: Test - if: inputs.TEST_SCRIPT != '' + if: inputs.TEST_SCRIPT != '' && inputs.TEST + run: | + docker exec \ + -e AGENT_NAME="${{ runner.name }}" \ + -e BUILD_REASON="${{ github.event_name }}" \ + app /bin/bash -c ". $BASHRC && cd $DOCKER_WORKING_DIR && source ~/MPIOversubscribe.sh && /bin/bash ${{ inputs.TEST_SCRIPT }}" + + - name: MyPy type checking + if: inputs.MYPY run: | - REPO_NAME="${{ steps.repo.outputs.name }}" docker exec \ -e AGENT_NAME="${{ runner.name }}" \ -e BUILD_REASON="${{ github.event_name }}" \ - app /bin/bash -c ". ${{ env.SCRITICAL_HOMEDIR }}/.bashrc_scritical && cd ${{ env.SCRITICAL_HOMEDIR }}/repos/${REPO_NAME} && source ~/MPIOversubscribe.sh && /bin/bash ${{ inputs.TEST_SCRIPT }}" + app /bin/bash -c ". $BASHRC && cd $DOCKER_WORKING_DIR && mypy" - name: Cleanup if: always() - run: docker stop app && docker rm app || true + run: | + # Stop and remove the container only if it exists to avoid noisy errors. + if docker ps -a --format '{{.Names}}' | grep -qx app; then + docker stop app && docker rm app || true + fi diff --git a/.github/workflows/mypy.yaml b/.github/workflows/mypy.yaml deleted file mode 100644 index 37bd251..0000000 --- a/.github/workflows/mypy.yaml +++ /dev/null @@ -1,23 +0,0 @@ -on: - workflow_call: - -jobs: - mypy: - runs-on: ubuntu-24.04 - strategy: - matrix: - python-version: ["3.11"] - steps: - - uses: actions/checkout@v5 - - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v6 - with: - python-version: ${{ matrix.python-version }} - - name: Install dependencies - run: | - pip install wheel - pip install mypy - - name: Mypy check - run: | - mypy --any-exprs-report mypy-output . - cat mypy-output/any-exprs.txt diff --git a/.github/workflows/ruff.yaml b/.github/workflows/ruff.yaml index f8bf8f6..f768d05 100644 --- a/.github/workflows/ruff.yaml +++ b/.github/workflows/ruff.yaml @@ -13,14 +13,16 @@ jobs: defaults: run: working-directory: project + env: + RUFF_CONFIG: ../.org-dotgithub/ruff.toml steps: - name: Checkout repository - uses: actions/checkout@v5 + uses: actions/checkout@v6 with: path: project - name: Checkout org-wide configs - uses: actions/checkout@v5 + uses: actions/checkout@v6 with: repository: scritical/.github path: .org-dotgithub @@ -39,12 +41,12 @@ jobs: - name: Ruff formatting run: | echo "=== Ruff Formatting Check ===" - ruff format --check --exit-non-zero-on-fix --config ../.org-dotgithub/ruff.toml . - + ruff format --check --exit-non-zero-on-fix --config $RUFF_CONFIG . + - name: Ruff linting run: | echo "=== Ruff Linting Check ===" - ruff check --exit-non-zero-on-fix --config ../.org-dotgithub/ruff.toml . + ruff check --exit-non-zero-on-fix --config $RUFF_CONFIG . - name: McCabe complexity check if: inputs.MCCABE diff --git a/ruff.toml b/ruff.toml index 3d0d9e3..4ddf9eb 100644 --- a/ruff.toml +++ b/ruff.toml @@ -35,6 +35,9 @@ convention = "numpy" [lint.mccabe] max-complexity = 10 +[lint.isort] +force-sort-within-sections = true + [format] quote-style = "double" indent-style = "space" From b20cc7150c3a4490cb21ae39e300834b8d52cf13 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 11 Feb 2026 00:53:34 +0000 Subject: [PATCH 26/50] Adding explicit tapenade version --- .github/workflows/tapenade.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/tapenade.yaml b/.github/workflows/tapenade.yaml index abe9d8e..3bc90f7 100644 --- a/.github/workflows/tapenade.yaml +++ b/.github/workflows/tapenade.yaml @@ -16,7 +16,7 @@ jobs: timeout-minutes: ${{ inputs.TIMEOUT }} steps: - name: Checkout repository - uses: actions/checkout@v5 + uses: actions/checkout@v6 - name: Set up Java uses: actions/setup-java@v4 @@ -26,7 +26,7 @@ jobs: - name: Install and run Tapenade run: | - # Download Tapenade 3.16 + # Download Tapenade 3.16 (tapenade_3.16-v2-723-ge8da61555.tar) TAPENADE_URL="https://gitlab.inria.fr/tapenade/tapenade/-/package_files/112870/download" wget "$TAPENADE_URL" -O tapenade_3.16.tar tar -xzf tapenade_3.16.tar From 80d874b15a5efaa06d23744e8f4a3318d2cdf453 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 11 Feb 2026 00:53:51 +0000 Subject: [PATCH 27/50] Updating readme for new workflow setup --- .github/workflows/README.md | 112 ++++++------------------------------ 1 file changed, 18 insertions(+), 94 deletions(-) diff --git a/.github/workflows/README.md b/.github/workflows/README.md index c86ac50..e262296 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -7,15 +7,11 @@ Reusable GitHub Actions workflows for Supercritical repositories. These workflow | Workflow | Description | | :------- | :---------- | | `build.yaml` | Build and test code in Docker container | -| `format-and-lint.yaml` | Python formatting and linting with ruff/pre-commit | -| `pypi.yaml` | Publish Python packages to PyPI | | `tapenade.yaml` | Tapenade automatic differentiation checks | | `clang_format.yaml` | C/C++ formatting checks | | `fprettify.yaml` | Fortran 90 formatting checks | -| `isort.yaml` | Python import sorting checks | -| `pylint.yaml` | Python linting with pylint | -| `mypy.yaml` | Python type checking | -| `branch-name-check.yml` | Enforce branch naming conventions | +| `ruff.yaml` | Python formatting and linting with Ruff | +| `branch-name-check.yaml` | Enforce branch naming conventions | --- @@ -29,19 +25,24 @@ Docker-based build and test workflow using the `scritical/private-dev` image. | :--- | :--- | :------ | :---------- | | `TIMEOUT` | number | `120` | Runtime allowed for the job, in minutes | | `GCC_CONFIG` | string | `""` | Path to GCC configuration file (from repository root) | +| `INTEL_CONFIG` | string | `""` | Path to Intel configuration file (from repository root) | | `BUILD_SCRIPT` | string | `.github/build_real.sh` | Path to build script. Empty string skips this step | | `TEST_SCRIPT` | string | `.github/test_real.sh` | Path to test script. Empty string skips this step | +| `BUILD` | boolean | `true` | Whether to run the build step | +| `TEST` | boolean | `true` | Whether to run the test step | +| `MYPY` | boolean | `false` | Whether to run mypy type checking | **Required Secrets:** | Name | Description | | :--- | :---------- | +| `DOCKER_USER` | Docker registry username | | `DOCKER_OAT` | Docker registry Organization Access Token | --- -### format-and-lint.yaml +### ruff.yaml -Python formatting and linting using pre-commit with ruff. +Python formatting and linting using Ruff. | Name | Type | Default | Description | | :--- | :--- | :------ | :---------- | @@ -58,17 +59,6 @@ ignore = ["N802"] --- -### pypi.yaml - -Publish Python packages to PyPI on tagged releases. - -**Required Secrets:** -| Name | Description | -| :--- | :---------- | -| `PYPI_API_TOKEN` | PyPI API token for publishing | - ---- - ### tapenade.yaml Run Tapenade automatic differentiation and check for uncommitted changes. @@ -78,7 +68,9 @@ Run Tapenade automatic differentiation and check for uncommitted changes. | `TIMEOUT` | number | `10` | Runtime allowed for the job, in minutes | | `TAPENADE_SCRIPT` | string | `.github/build_tapenade.sh` | Path to Tapenade build script | -Uses Tapenade version 3.16. +Uses Tapenade version 3.16 from tapenade_3.16-v2-723-ge8da61555.tar. Using a different version will create a diff. + +Here is the link to our tapenade version: https://gitlab.inria.fr/tapenade/tapenade/-/package_files/112870/download --- @@ -106,30 +98,6 @@ Fortran 90 code formatting checks using fprettify. --- -### isort.yaml - -Python import sorting checks. - -**Configuration Override:** Create a `.isort.cfg` file in your repo root with overrides. The global and local configs are merged automatically using `combine-config.py`. - ---- - -### pylint.yaml - -Python linting with pylint. - -**Configuration Override:** Create a `.pylintrc` file in your repo root with overrides. The global and local configs are merged automatically using `combine-config.py`. - ---- - -### mypy.yaml - -Python type checking with mypy. - -No configurable inputs. Uses Python 3.11. - ---- - ## Setting Up Workflows ### Step 1: Create Workflow File @@ -153,10 +121,11 @@ jobs: BUILD_SCRIPT: .github/build_real.sh TEST_SCRIPT: .github/test_real.sh secrets: + DOCKER_USER: ${{ secrets.DOCKER_USER }} DOCKER_OAT: ${{ secrets.DOCKER_OAT }} - format-and-lint: - uses: scritical/.github/.github/workflows/format-and-lint.yaml@main + ruff: + uses: scritical/.github/.github/workflows/ruff.yaml@main clang-format: uses: scritical/.github/.github/workflows/clang_format.yaml@main @@ -191,45 +160,11 @@ testflo -v . -n 1 ### Step 4: Add Secrets In your repository settings, add the required secrets: +- `DOCKER_USER` - Docker organization name - `DOCKER_OAT` - Docker Organization Access Token for pulling private images --- -## PyPI Publishing - -For repositories that publish to PyPI, extend your workflow: - -```yaml -name: CI - -on: - push: - branches: [main] - tags: - - 'v*.*.*' - pull_request: - branches: [main] - -jobs: - format-and-lint: - uses: scritical/.github/.github/workflows/format-and-lint.yaml@main - - build: - uses: scritical/.github/.github/workflows/build.yaml@main - with: - GCC_CONFIG: config/defaults/config.LINUX_GFORTRAN.mk - secrets: - DOCKER_OAT: ${{ secrets.DOCKER_OAT }} - - pypi: - needs: [format-and-lint, build] - uses: scritical/.github/.github/workflows/pypi.yaml@main - secrets: - PYPI_API_TOKEN: ${{ secrets.PYPI_API_TOKEN }} -``` - ---- - ## Complex Builds For repositories requiring both real and complex builds: @@ -243,6 +178,7 @@ jobs: BUILD_SCRIPT: .github/build_real.sh TEST_SCRIPT: .github/test_real.sh secrets: + DOCKER_USER: ${{ secrets.DOCKER_USER }} DOCKER_OAT: ${{ secrets.DOCKER_OAT }} build-complex: @@ -252,6 +188,7 @@ jobs: BUILD_SCRIPT: .github/build_complex.sh TEST_SCRIPT: .github/test_complex.sh secrets: + DOCKER_USER: ${{ secrets.DOCKER_USER }} DOCKER_OAT: ${{ secrets.DOCKER_OAT }} ``` @@ -264,8 +201,6 @@ Several workflows support configuration inheritance/overrides: | Tool | Method | Local Config File | | :--- | :----- | :---------------- | | Ruff | Native `extend` keyword | `ruff.toml` | -| Pylint | Merged via `combine-config.py` | `.pylintrc` | -| isort | Merged via `combine-config.py` | `.isort.cfg` | | clang-format | Local file takes precedence | `.clang-format` | | fprettify | Local file takes precedence | `.fprettify.rc` | @@ -279,17 +214,6 @@ extend = "~/.config/ruff/ruff.toml" ignore = ["N802", "N803"] # Allow uppercase function/argument names ``` -### Example: Pylint Override - -```ini -# .pylintrc (partial - only overrides) -[MESSAGES CONTROL] -disable = C0114,C0115 - -[FORMAT] -max-line-length = 150 -``` - --- ## Adding Status Badge From 4da3a7875d2e582615421b22cdea3c71137fc32c Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 11 Feb 2026 00:54:03 +0000 Subject: [PATCH 28/50] Bumping clang format to version 20 --- .github/workflows/clang_format.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/clang_format.yaml b/.github/workflows/clang_format.yaml index 016b6a9..b76cdd8 100644 --- a/.github/workflows/clang_format.yaml +++ b/.github/workflows/clang_format.yaml @@ -12,10 +12,10 @@ jobs: timeout-minutes: ${{ inputs.TIMEOUT }} steps: - name: Checkout repository - uses: actions/checkout@v5 + uses: actions/checkout@v6 - name: Install clang-format - run: sudo apt-get install clang-format-10 -y + run: sudo apt-get install clang-format-20 -y - name: Get clang-format config run: | From 053c38487c45ba26e6c67ee03be3ce62f8bf61f5 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 11 Feb 2026 00:54:37 +0000 Subject: [PATCH 29/50] Bumping actions checkout to v6 --- .github/workflows/fprettify.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/fprettify.yaml b/.github/workflows/fprettify.yaml index 9b52cd1..9f65560 100644 --- a/.github/workflows/fprettify.yaml +++ b/.github/workflows/fprettify.yaml @@ -12,7 +12,7 @@ jobs: timeout-minutes: ${{ inputs.TIMEOUT }} steps: - name: Checkout repository - uses: actions/checkout@v5 + uses: actions/checkout@v6 - name: Set up Python uses: actions/setup-python@v6 From 3f327d952897c223c2240596873297bb0fcbf39c Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 11 Feb 2026 18:34:11 +0000 Subject: [PATCH 30/50] Removing MDO Lab codeowners --- .github/CODEOWNERS | 1 - 1 file changed, 1 deletion(-) delete mode 100644 .github/CODEOWNERS diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS deleted file mode 100644 index 5155aa3..0000000 --- a/.github/CODEOWNERS +++ /dev/null @@ -1 +0,0 @@ -* @mdolab/infrastructure-maintainers From 2ac4151e223ce802cc82440b981286890c77c5ff Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 11 Feb 2026 18:46:25 +0000 Subject: [PATCH 31/50] Removing import sorting by default from Ruff config and adding sorting step to ruff workflow --- .github/workflows/ruff.yaml | 11 ++++++++++- ruff.toml | 2 +- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ruff.yaml b/.github/workflows/ruff.yaml index f768d05..c84dc17 100644 --- a/.github/workflows/ruff.yaml +++ b/.github/workflows/ruff.yaml @@ -5,7 +5,10 @@ on: type: boolean default: false description: "Enable McCabe complexity check (pass/fail)" - + ISORT: + type: boolean + default: false + description: "Enable isort import sorting check (pass/fail)" jobs: ruff: @@ -47,6 +50,12 @@ jobs: run: | echo "=== Ruff Linting Check ===" ruff check --exit-non-zero-on-fix --config $RUFF_CONFIG . + + - name: isort check + if: inputs.ISORT + run: | + echo "=== isort Import Sorting Check ===" + ruff check --exit-non-zero-on-fix --select=I --config $RUFF_CONFIG . - name: McCabe complexity check if: inputs.MCCABE diff --git a/ruff.toml b/ruff.toml index 4ddf9eb..c2d4df6 100644 --- a/ruff.toml +++ b/ruff.toml @@ -27,7 +27,7 @@ ignore = [ "D404", # First word of the docstring should not be "This" (Probably a good rule to follow going forward but it occurs in so many places in our code and it's not a big enough issue to warrant fixing) ] -select = ["B", "D", "E", "F", "W", "I"] +select = ["B", "D", "E", "F", "W"] [lint.pydocstyle] convention = "numpy" From 05cfcee188c6e587c40ae0ef1d59c0cbae23113e Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 11 Feb 2026 18:46:52 +0000 Subject: [PATCH 32/50] Updating readme, refactoring build workflow, and adding mypy to build workflow to run in docker images --- .github/workflows/README.md | 10 ++++++++++ .github/workflows/build.yaml | 1 + 2 files changed, 11 insertions(+) diff --git a/.github/workflows/README.md b/.github/workflows/README.md index e262296..c37a341 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -20,6 +20,7 @@ Reusable GitHub Actions workflows for Supercritical repositories. These workflow ### build.yaml Docker-based build and test workflow using the `scritical/private-dev` image. +Runs separate GCC and Intel jobs when the matching config input is provided. | Name | Type | Default | Description | | :--- | :--- | :------ | :---------- | @@ -43,10 +44,12 @@ Docker-based build and test workflow using the `scritical/private-dev` image. ### ruff.yaml Python formatting and linting using Ruff. +The workflow checks out the org-wide Ruff configuration from `scritical/.github` and uses `ruff.toml` from that repo as the base config. | Name | Type | Default | Description | | :--- | :--- | :------ | :---------- | | `MCCABE` | boolean | `false` | Enable McCabe complexity check (pass/fail, max complexity = 10) | +| `ISORT` | boolean | `false` | Enable import sorting check (pass/fail) | **Configuration Override:** Create a `ruff.toml` in your repo with: ```toml @@ -98,6 +101,13 @@ Fortran 90 code formatting checks using fprettify. --- +### branch-name-check.yaml + +Enforces branch naming conventions for pull requests: + +- For PRs targeting `main`, source branches must start with `feature-`, `bugfix-`, or `hotfix-`. +- For PRs targeting `client-*`, source branches must start with `feature-`, `bugfix-`, or `hotfix-`. + ## Setting Up Workflows ### Step 1: Create Workflow File diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index c262e14..3ae46ad 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -47,6 +47,7 @@ env: jobs: build-and-test: + if: matrix.CONFIG_FILE != '' runs-on: ubuntu-24.04 timeout-minutes: ${{ inputs.TIMEOUT }} strategy: From 582951cf90a426d7747e3c69e7c5a2fcaa2e4e82 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 11 Feb 2026 22:05:53 +0000 Subject: [PATCH 33/50] Minor update to the build workflow for turning intel/gcc on or off --- .github/workflows/build.yaml | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 3ae46ad..6e4c6ed 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -13,6 +13,14 @@ on: type: string default: "" description: "Path to Intel configuration file" + INTEL: + type: boolean + default: false + description: "Whether to run Intel-specific build and test steps (if false, runs GCC steps)" + GCC: + type: boolean + default: true + description: "Whether to run GCC-specific build and test steps (if false, runs Intel steps)" BUILD_SCRIPT: type: string default: ".github/build_real.sh" @@ -47,15 +55,19 @@ env: jobs: build-and-test: - if: matrix.CONFIG_FILE != '' + if: | + (matrix.TOOLCHAIN == 'gcc' && inputs.GCC) || + (matrix.TOOLCHAIN == 'intel' && inputs.INTEL) runs-on: ubuntu-24.04 timeout-minutes: ${{ inputs.TIMEOUT }} strategy: matrix: include: - - DOCKER_TAG: u24-gcc-ompi-latest + - TOOLCHAIN: gcc + DOCKER_TAG: u24-gcc-ompi-latest CONFIG_FILE: ${{ inputs.GCC_CONFIG }} - - DOCKER_TAG: u24-intel-ompi-latest + - TOOLCHAIN: intel + DOCKER_TAG: u24-intel-ompi-latest CONFIG_FILE: ${{ inputs.INTEL_CONFIG }} steps: - name: Checkout repository From 7d7ef200bd4255fde42ceaa0ffca30294eb98910 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 11 Feb 2026 22:13:16 +0000 Subject: [PATCH 34/50] Fixing env bug --- .github/workflows/build.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 6e4c6ed..ac33dee 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -51,7 +51,7 @@ on: env: SCRITICAL_HOMEDIR: /home/scriticaluser - BASHRC: ${{ env.SCRITICAL_HOMEDIR }}/.bashrc_scritical + BASHRC: /home/scriticaluser/.bashrc_scritical jobs: build-and-test: From 30a1c97442ec611fc9edb0be6e8a026a90338d09 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 11 Feb 2026 22:15:35 +0000 Subject: [PATCH 35/50] Improving the logic checks for the build workflow for intel/gcc --- .github/workflows/build.yaml | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index ac33dee..93f7526 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -55,9 +55,6 @@ env: jobs: build-and-test: - if: | - (matrix.TOOLCHAIN == 'gcc' && inputs.GCC) || - (matrix.TOOLCHAIN == 'intel' && inputs.INTEL) runs-on: ubuntu-24.04 timeout-minutes: ${{ inputs.TIMEOUT }} strategy: @@ -73,7 +70,19 @@ jobs: - name: Checkout repository uses: actions/checkout@v6 + - name: Check toolchain enabled + id: enable + run: | + enabled=false + if [[ "${{ matrix.TOOLCHAIN }}" == "gcc" && "${{ inputs.GCC }}" == "true" ]]; then + enabled=true + elif [[ "${{ matrix.TOOLCHAIN }}" == "intel" && "${{ inputs.INTEL }}" == "true" ]]; then + enabled=true + fi + echo "enabled=$enabled" >> "$GITHUB_OUTPUT" + - name: Set repo environment variables + if: steps.enable.outputs.enabled == 'true' run: | repo_name="${{ github.event.repository.name }}" echo "REPO_NAME=$repo_name" >> "$GITHUB_ENV" @@ -81,12 +90,14 @@ jobs: echo "DOCKER_MOUNT_DIR=${{ env.SCRITICAL_HOMEDIR }}/mount/$repo_name" >> "$GITHUB_ENV" - name: Login to Docker Hub + if: steps.enable.outputs.enabled == 'true' uses: docker/login-action@v3 with: username: ${{ secrets.DOCKER_USER }} password: ${{ secrets.DOCKER_OAT }} - name: Start Docker container + if: steps.enable.outputs.enabled == 'true' run: | docker run -t -d --name app \ --mount "type=bind,src=${{ github.workspace }},target=${{ env.DOCKER_MOUNT_DIR }}" \ @@ -96,14 +107,14 @@ jobs: docker exec app /bin/bash -c "rm -rf $DOCKER_WORKING_DIR && mkdir -p $DOCKER_WORKING_DIR && cp -a $DOCKER_MOUNT_DIR/. $DOCKER_WORKING_DIR/" - name: Build - if: inputs.BUILD_SCRIPT != '' && inputs.BUILD + if: steps.enable.outputs.enabled == 'true' && inputs.BUILD_SCRIPT != '' && inputs.BUILD run: | docker exec \ -e CONFIG_FILE="${{ matrix.CONFIG_FILE }}" \ app /bin/bash -c ". $BASHRC && cd $DOCKER_WORKING_DIR && /bin/bash ${{ inputs.BUILD_SCRIPT }}" - name: Test - if: inputs.TEST_SCRIPT != '' && inputs.TEST + if: steps.enable.outputs.enabled == 'true' && inputs.TEST_SCRIPT != '' && inputs.TEST run: | docker exec \ -e AGENT_NAME="${{ runner.name }}" \ @@ -111,7 +122,7 @@ jobs: app /bin/bash -c ". $BASHRC && cd $DOCKER_WORKING_DIR && source ~/MPIOversubscribe.sh && /bin/bash ${{ inputs.TEST_SCRIPT }}" - name: MyPy type checking - if: inputs.MYPY + if: steps.enable.outputs.enabled == 'true' && inputs.MYPY run: | docker exec \ -e AGENT_NAME="${{ runner.name }}" \ From a2db2c1b73b88681f603ec1ac517f21f884c0fdd Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 11 Feb 2026 22:25:00 +0000 Subject: [PATCH 36/50] Fixing mypy --- .github/workflows/build.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 93f7526..a5b931e 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -127,7 +127,7 @@ jobs: docker exec \ -e AGENT_NAME="${{ runner.name }}" \ -e BUILD_REASON="${{ github.event_name }}" \ - app /bin/bash -c ". $BASHRC && cd $DOCKER_WORKING_DIR && mypy" + app /bin/bash -c ". $BASHRC && cd $DOCKER_WORKING_DIR && mypy ." - name: Cleanup if: always() From 08d7ff21066e0c5512b12adfe4dc6d7a32574d4a Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 11 Feb 2026 22:50:08 +0000 Subject: [PATCH 37/50] Creating composite actions for build workflow to split jobs --- .github/actions/build-and-test/action.yml | 100 ++++++++++++++++++++ .github/workflows/build.yaml | 108 +++++++--------------- 2 files changed, 131 insertions(+), 77 deletions(-) create mode 100644 .github/actions/build-and-test/action.yml diff --git a/.github/actions/build-and-test/action.yml b/.github/actions/build-and-test/action.yml new file mode 100644 index 0000000..526f847 --- /dev/null +++ b/.github/actions/build-and-test/action.yml @@ -0,0 +1,100 @@ +name: Build and test + +description: Run dockerized build/test/mypy steps for a specific toolchain. + +inputs: + docker_tag: + description: Docker image tag to use. + required: true + config_file: + description: Path to compiler config file (from repository root). + required: true + build_script: + description: Path to build script. + required: true + test_script: + description: Path to test script. + required: true + build: + description: Whether to run the build step. + required: false + default: "true" + test: + description: Whether to run the test step. + required: false + default: "true" + mypy: + description: Whether to run mypy type checking. + required: false + default: "false" + docker_user: + description: Docker registry username. + required: true + docker_oat: + description: Docker registry Organization Access Token. + required: true + +runs: + using: composite + steps: + - name: Checkout repository + uses: actions/checkout@v6 + + - name: Set repo environment variables + shell: bash + run: | + repo_name="${{ github.event.repository.name }}" + echo "REPO_NAME=$repo_name" >> "$GITHUB_ENV" + echo "DOCKER_WORKING_DIR=${{ env.SCRITICAL_HOMEDIR }}/repos/$repo_name" >> "$GITHUB_ENV" + echo "DOCKER_MOUNT_DIR=${{ env.SCRITICAL_HOMEDIR }}/mount/$repo_name" >> "$GITHUB_ENV" + + - name: Login to Docker Hub + uses: docker/login-action@v3 + with: + username: ${{ inputs.docker_user }} + password: ${{ inputs.docker_oat }} + + - name: Start Docker container + shell: bash + run: | + docker run -t -d --name app \ + --mount "type=bind,src=${{ github.workspace }},target=${{ env.DOCKER_MOUNT_DIR }}" \ + scritical/private-dev:${{ inputs.docker_tag }} /bin/bash + + # Copy repo to working directory + docker exec app /bin/bash -c "rm -rf $DOCKER_WORKING_DIR && mkdir -p $DOCKER_WORKING_DIR && cp -a $DOCKER_MOUNT_DIR/. $DOCKER_WORKING_DIR/" + + - name: Build + if: ${{ inputs.build_script != '' && inputs.build == 'true' }} + shell: bash + run: | + docker exec \ + -e CONFIG_FILE="${{ inputs.config_file }}" \ + app /bin/bash -c ". $BASHRC && cd $DOCKER_WORKING_DIR && /bin/bash ${{ inputs.build_script }}" + + - name: Test + if: ${{ inputs.test_script != '' && inputs.test == 'true' }} + shell: bash + run: | + docker exec \ + -e AGENT_NAME="${{ runner.name }}" \ + -e BUILD_REASON="${{ github.event_name }}" \ + app /bin/bash -c ". $BASHRC && cd $DOCKER_WORKING_DIR && source ~/MPIOversubscribe.sh && /bin/bash ${{ inputs.test_script }}" + + - name: MyPy type checking + if: ${{ inputs.mypy == 'true' }} + shell: bash + run: | + docker exec \ + -e AGENT_NAME="${{ runner.name }}" \ + -e BUILD_REASON="${{ github.event_name }}" \ + app /bin/bash -c ". $BASHRC && cd $DOCKER_WORKING_DIR && mypy ." + + - name: Cleanup + if: ${{ always() }} + shell: bash + run: | + # Stop and remove the container only if it exists to avoid noisy errors. + if docker ps -a --format '{{.Names}}' | grep -qx app; then + docker stop app && docker rm app || true + fi diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index a5b931e..4088074 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -54,85 +54,39 @@ env: BASHRC: /home/scriticaluser/.bashrc_scritical jobs: - build-and-test: + build-gcc: + if: inputs.GCC runs-on: ubuntu-24.04 timeout-minutes: ${{ inputs.TIMEOUT }} - strategy: - matrix: - include: - - TOOLCHAIN: gcc - DOCKER_TAG: u24-gcc-ompi-latest - CONFIG_FILE: ${{ inputs.GCC_CONFIG }} - - TOOLCHAIN: intel - DOCKER_TAG: u24-intel-ompi-latest - CONFIG_FILE: ${{ inputs.INTEL_CONFIG }} steps: - - name: Checkout repository - uses: actions/checkout@v6 - - - name: Check toolchain enabled - id: enable - run: | - enabled=false - if [[ "${{ matrix.TOOLCHAIN }}" == "gcc" && "${{ inputs.GCC }}" == "true" ]]; then - enabled=true - elif [[ "${{ matrix.TOOLCHAIN }}" == "intel" && "${{ inputs.INTEL }}" == "true" ]]; then - enabled=true - fi - echo "enabled=$enabled" >> "$GITHUB_OUTPUT" - - - name: Set repo environment variables - if: steps.enable.outputs.enabled == 'true' - run: | - repo_name="${{ github.event.repository.name }}" - echo "REPO_NAME=$repo_name" >> "$GITHUB_ENV" - echo "DOCKER_WORKING_DIR=${{ env.SCRITICAL_HOMEDIR }}/repos/$repo_name" >> "$GITHUB_ENV" - echo "DOCKER_MOUNT_DIR=${{ env.SCRITICAL_HOMEDIR }}/mount/$repo_name" >> "$GITHUB_ENV" - - - name: Login to Docker Hub - if: steps.enable.outputs.enabled == 'true' - uses: docker/login-action@v3 + - name: Build and test (GCC) + uses: ./.github/actions/build-and-test with: - username: ${{ secrets.DOCKER_USER }} - password: ${{ secrets.DOCKER_OAT }} - - - name: Start Docker container - if: steps.enable.outputs.enabled == 'true' - run: | - docker run -t -d --name app \ - --mount "type=bind,src=${{ github.workspace }},target=${{ env.DOCKER_MOUNT_DIR }}" \ - scritical/private-dev:${{ matrix.DOCKER_TAG }} /bin/bash - - # Copy repo to working directory - docker exec app /bin/bash -c "rm -rf $DOCKER_WORKING_DIR && mkdir -p $DOCKER_WORKING_DIR && cp -a $DOCKER_MOUNT_DIR/. $DOCKER_WORKING_DIR/" - - - name: Build - if: steps.enable.outputs.enabled == 'true' && inputs.BUILD_SCRIPT != '' && inputs.BUILD - run: | - docker exec \ - -e CONFIG_FILE="${{ matrix.CONFIG_FILE }}" \ - app /bin/bash -c ". $BASHRC && cd $DOCKER_WORKING_DIR && /bin/bash ${{ inputs.BUILD_SCRIPT }}" + docker_tag: u24-gcc-ompi-latest + config_file: ${{ inputs.GCC_CONFIG }} + build_script: ${{ inputs.BUILD_SCRIPT }} + test_script: ${{ inputs.TEST_SCRIPT }} + build: ${{ inputs.BUILD }} + test: ${{ inputs.TEST }} + mypy: ${{ inputs.MYPY }} + docker_user: ${{ secrets.DOCKER_USER }} + docker_oat: ${{ secrets.DOCKER_OAT }} - - name: Test - if: steps.enable.outputs.enabled == 'true' && inputs.TEST_SCRIPT != '' && inputs.TEST - run: | - docker exec \ - -e AGENT_NAME="${{ runner.name }}" \ - -e BUILD_REASON="${{ github.event_name }}" \ - app /bin/bash -c ". $BASHRC && cd $DOCKER_WORKING_DIR && source ~/MPIOversubscribe.sh && /bin/bash ${{ inputs.TEST_SCRIPT }}" - - - name: MyPy type checking - if: steps.enable.outputs.enabled == 'true' && inputs.MYPY - run: | - docker exec \ - -e AGENT_NAME="${{ runner.name }}" \ - -e BUILD_REASON="${{ github.event_name }}" \ - app /bin/bash -c ". $BASHRC && cd $DOCKER_WORKING_DIR && mypy ." - - - name: Cleanup - if: always() - run: | - # Stop and remove the container only if it exists to avoid noisy errors. - if docker ps -a --format '{{.Names}}' | grep -qx app; then - docker stop app && docker rm app || true - fi + build-intel: + if: inputs.INTEL + runs-on: ubuntu-24.04 + timeout-minutes: ${{ inputs.TIMEOUT }} + steps: + - name: Build and test (Intel) + uses: ./.github/actions/build-and-test + with: + docker_tag: u24-intel-ompi-latest + config_file: ${{ inputs.INTEL_CONFIG }} + build_script: ${{ inputs.BUILD_SCRIPT }} + test_script: ${{ inputs.TEST_SCRIPT }} + build: ${{ inputs.BUILD }} + test: ${{ inputs.TEST }} + mypy: ${{ inputs.MYPY }} + docker_user: ${{ secrets.DOCKER_USER }} + docker_oat: ${{ secrets.DOCKER_OAT }} + From 930109c072be9f93ae0dadb4c6803ad569cbf6ed Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 11 Feb 2026 22:51:38 +0000 Subject: [PATCH 38/50] Fixing .yml to .yaml --- .github/actions/build-and-test/{action.yml => action.yaml} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename .github/actions/build-and-test/{action.yml => action.yaml} (100%) diff --git a/.github/actions/build-and-test/action.yml b/.github/actions/build-and-test/action.yaml similarity index 100% rename from .github/actions/build-and-test/action.yml rename to .github/actions/build-and-test/action.yaml From 502dadf60d25759ff575b85c74d0d5ae934a5530 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 11 Feb 2026 22:55:11 +0000 Subject: [PATCH 39/50] Adding a checkout step for composite actions workflow --- .github/workflows/build.yaml | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 4088074..7867b80 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -59,8 +59,15 @@ jobs: runs-on: ubuntu-24.04 timeout-minutes: ${{ inputs.TIMEOUT }} steps: + - name: Checkout org workflows + uses: actions/checkout@v6 + with: + repository: scritical/.github + ref: feature-migrate_azure2gha + path: .org-dotgithub + - name: Build and test (GCC) - uses: ./.github/actions/build-and-test + uses: ./.org-dotgithub/.github/actions/build-and-test with: docker_tag: u24-gcc-ompi-latest config_file: ${{ inputs.GCC_CONFIG }} @@ -77,8 +84,15 @@ jobs: runs-on: ubuntu-24.04 timeout-minutes: ${{ inputs.TIMEOUT }} steps: + - name: Checkout org workflows + uses: actions/checkout@v6 + with: + repository: scritical/.github + ref: feature-migrate_azure2gha + path: .org-dotgithub + - name: Build and test (Intel) - uses: ./.github/actions/build-and-test + uses: ./.org-dotgithub/.github/actions/build-and-test with: docker_tag: u24-intel-ompi-latest config_file: ${{ inputs.INTEL_CONFIG }} From 8d0b709311ddc3c40710caef11cb57bd945dd41f Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 11 Feb 2026 23:47:33 +0000 Subject: [PATCH 40/50] Removing composite actions and trying yaml anchor approach --- .github/actions/build-and-test/action.yaml | 100 --------------------- .github/workflows/build.yaml | 95 ++++++++++++-------- 2 files changed, 60 insertions(+), 135 deletions(-) delete mode 100644 .github/actions/build-and-test/action.yaml diff --git a/.github/actions/build-and-test/action.yaml b/.github/actions/build-and-test/action.yaml deleted file mode 100644 index 526f847..0000000 --- a/.github/actions/build-and-test/action.yaml +++ /dev/null @@ -1,100 +0,0 @@ -name: Build and test - -description: Run dockerized build/test/mypy steps for a specific toolchain. - -inputs: - docker_tag: - description: Docker image tag to use. - required: true - config_file: - description: Path to compiler config file (from repository root). - required: true - build_script: - description: Path to build script. - required: true - test_script: - description: Path to test script. - required: true - build: - description: Whether to run the build step. - required: false - default: "true" - test: - description: Whether to run the test step. - required: false - default: "true" - mypy: - description: Whether to run mypy type checking. - required: false - default: "false" - docker_user: - description: Docker registry username. - required: true - docker_oat: - description: Docker registry Organization Access Token. - required: true - -runs: - using: composite - steps: - - name: Checkout repository - uses: actions/checkout@v6 - - - name: Set repo environment variables - shell: bash - run: | - repo_name="${{ github.event.repository.name }}" - echo "REPO_NAME=$repo_name" >> "$GITHUB_ENV" - echo "DOCKER_WORKING_DIR=${{ env.SCRITICAL_HOMEDIR }}/repos/$repo_name" >> "$GITHUB_ENV" - echo "DOCKER_MOUNT_DIR=${{ env.SCRITICAL_HOMEDIR }}/mount/$repo_name" >> "$GITHUB_ENV" - - - name: Login to Docker Hub - uses: docker/login-action@v3 - with: - username: ${{ inputs.docker_user }} - password: ${{ inputs.docker_oat }} - - - name: Start Docker container - shell: bash - run: | - docker run -t -d --name app \ - --mount "type=bind,src=${{ github.workspace }},target=${{ env.DOCKER_MOUNT_DIR }}" \ - scritical/private-dev:${{ inputs.docker_tag }} /bin/bash - - # Copy repo to working directory - docker exec app /bin/bash -c "rm -rf $DOCKER_WORKING_DIR && mkdir -p $DOCKER_WORKING_DIR && cp -a $DOCKER_MOUNT_DIR/. $DOCKER_WORKING_DIR/" - - - name: Build - if: ${{ inputs.build_script != '' && inputs.build == 'true' }} - shell: bash - run: | - docker exec \ - -e CONFIG_FILE="${{ inputs.config_file }}" \ - app /bin/bash -c ". $BASHRC && cd $DOCKER_WORKING_DIR && /bin/bash ${{ inputs.build_script }}" - - - name: Test - if: ${{ inputs.test_script != '' && inputs.test == 'true' }} - shell: bash - run: | - docker exec \ - -e AGENT_NAME="${{ runner.name }}" \ - -e BUILD_REASON="${{ github.event_name }}" \ - app /bin/bash -c ". $BASHRC && cd $DOCKER_WORKING_DIR && source ~/MPIOversubscribe.sh && /bin/bash ${{ inputs.test_script }}" - - - name: MyPy type checking - if: ${{ inputs.mypy == 'true' }} - shell: bash - run: | - docker exec \ - -e AGENT_NAME="${{ runner.name }}" \ - -e BUILD_REASON="${{ github.event_name }}" \ - app /bin/bash -c ". $BASHRC && cd $DOCKER_WORKING_DIR && mypy ." - - - name: Cleanup - if: ${{ always() }} - shell: bash - run: | - # Stop and remove the container only if it exists to avoid noisy errors. - if docker ps -a --format '{{.Names}}' | grep -qx app; then - docker stop app && docker rm app || true - fi diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 7867b80..24bbd85 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -58,49 +58,74 @@ jobs: if: inputs.GCC runs-on: ubuntu-24.04 timeout-minutes: ${{ inputs.TIMEOUT }} + env: + DOCKER_TAG: u24-gcc-ompi-latest + CONFIG_FILE: ${{ inputs.GCC_CONFIG }} steps: - - name: Checkout org workflows + - &build_steps + name: Checkout repository uses: actions/checkout@v6 - with: - repository: scritical/.github - ref: feature-migrate_azure2gha - path: .org-dotgithub - - name: Build and test (GCC) - uses: ./.org-dotgithub/.github/actions/build-and-test + - name: Set repo environment variables + run: | + repo_name="${{ github.event.repository.name }}" + echo "REPO_NAME=$repo_name" >> "$GITHUB_ENV" + echo "DOCKER_WORKING_DIR=${{ env.SCRITICAL_HOMEDIR }}/repos/$repo_name" >> "$GITHUB_ENV" + echo "DOCKER_MOUNT_DIR=${{ env.SCRITICAL_HOMEDIR }}/mount/$repo_name" >> "$GITHUB_ENV" + + - name: Login to Docker Hub + uses: docker/login-action@v3 with: - docker_tag: u24-gcc-ompi-latest - config_file: ${{ inputs.GCC_CONFIG }} - build_script: ${{ inputs.BUILD_SCRIPT }} - test_script: ${{ inputs.TEST_SCRIPT }} - build: ${{ inputs.BUILD }} - test: ${{ inputs.TEST }} - mypy: ${{ inputs.MYPY }} - docker_user: ${{ secrets.DOCKER_USER }} - docker_oat: ${{ secrets.DOCKER_OAT }} + username: ${{ secrets.DOCKER_USER }} + password: ${{ secrets.DOCKER_OAT }} + + - name: Start Docker container + run: | + docker run -t -d --name app \ + --mount "type=bind,src=${{ github.workspace }},target=${{ env.DOCKER_MOUNT_DIR }}" \ + scritical/private-dev:${{ env.DOCKER_TAG }} /bin/bash + + # Copy repo to working directory + docker exec app /bin/bash -c "rm -rf $DOCKER_WORKING_DIR && mkdir -p $DOCKER_WORKING_DIR && cp -a $DOCKER_MOUNT_DIR/. $DOCKER_WORKING_DIR/" + + - name: Build + if: inputs.BUILD_SCRIPT != '' && inputs.BUILD + run: | + docker exec \ + -e CONFIG_FILE="${{ env.CONFIG_FILE }}" \ + app /bin/bash -c ". $BASHRC && cd $DOCKER_WORKING_DIR && /bin/bash ${{ inputs.BUILD_SCRIPT }}" + + - name: Test + if: inputs.TEST_SCRIPT != '' && inputs.TEST + run: | + docker exec \ + -e AGENT_NAME="${{ runner.name }}" \ + -e BUILD_REASON="${{ github.event_name }}" \ + app /bin/bash -c ". $BASHRC && cd $DOCKER_WORKING_DIR && source ~/MPIOversubscribe.sh && /bin/bash ${{ inputs.TEST_SCRIPT }}" + + - name: MyPy type checking + if: inputs.MYPY + run: | + docker exec \ + -e AGENT_NAME="${{ runner.name }}" \ + -e BUILD_REASON="${{ github.event_name }}" \ + app /bin/bash -c ". $BASHRC && cd $DOCKER_WORKING_DIR && mypy ." + + - name: Cleanup + if: always() + run: | + # Stop and remove the container only if it exists to avoid noisy errors. + if docker ps -a --format '{{.Names}}' | grep -qx app; then + docker stop app && docker rm app || true + fi build-intel: if: inputs.INTEL runs-on: ubuntu-24.04 timeout-minutes: ${{ inputs.TIMEOUT }} + env: + DOCKER_TAG: u24-intel-ompi-latest + CONFIG_FILE: ${{ inputs.INTEL_CONFIG }} steps: - - name: Checkout org workflows - uses: actions/checkout@v6 - with: - repository: scritical/.github - ref: feature-migrate_azure2gha - path: .org-dotgithub - - - name: Build and test (Intel) - uses: ./.org-dotgithub/.github/actions/build-and-test - with: - docker_tag: u24-intel-ompi-latest - config_file: ${{ inputs.INTEL_CONFIG }} - build_script: ${{ inputs.BUILD_SCRIPT }} - test_script: ${{ inputs.TEST_SCRIPT }} - build: ${{ inputs.BUILD }} - test: ${{ inputs.TEST }} - mypy: ${{ inputs.MYPY }} - docker_user: ${{ secrets.DOCKER_USER }} - docker_oat: ${{ secrets.DOCKER_OAT }} + *build_steps From 0a227721b676f03028a3e9a79af0c7008a682b41 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 11 Feb 2026 23:49:19 +0000 Subject: [PATCH 41/50] Fixing yaml anchor --- .github/workflows/build.yaml | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 24bbd85..9753f54 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -61,9 +61,8 @@ jobs: env: DOCKER_TAG: u24-gcc-ompi-latest CONFIG_FILE: ${{ inputs.GCC_CONFIG }} - steps: - - &build_steps - name: Checkout repository + steps: &build_steps + - name: Checkout repository uses: actions/checkout@v6 - name: Set repo environment variables @@ -127,5 +126,5 @@ jobs: DOCKER_TAG: u24-intel-ompi-latest CONFIG_FILE: ${{ inputs.INTEL_CONFIG }} steps: - *build_steps + - *build_steps From 8b24addade5a901faf0c8acf20b589ccd6bd3bd8 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Wed, 11 Feb 2026 23:51:08 +0000 Subject: [PATCH 42/50] Fixing yaml anchors pt 2 --- .github/workflows/build.yaml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 9753f54..03a4e89 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -125,6 +125,5 @@ jobs: env: DOCKER_TAG: u24-intel-ompi-latest CONFIG_FILE: ${{ inputs.INTEL_CONFIG }} - steps: - - *build_steps + steps: *build_steps From 4adc7d82f0b6b8cd399dd3539734fc1ae5646e83 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Fri, 13 Feb 2026 00:46:04 +0000 Subject: [PATCH 43/50] Addressing new round of comments pt 1 --- .github/workflows/README.md | 3 ++- .github/workflows/build.yaml | 8 ++------ README.md | 21 +++++++++++++-------- 3 files changed, 17 insertions(+), 15 deletions(-) diff --git a/.github/workflows/README.md b/.github/workflows/README.md index c37a341..337e11e 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -27,9 +27,10 @@ Runs separate GCC and Intel jobs when the matching config input is provided. | `TIMEOUT` | number | `120` | Runtime allowed for the job, in minutes | | `GCC_CONFIG` | string | `""` | Path to GCC configuration file (from repository root) | | `INTEL_CONFIG` | string | `""` | Path to Intel configuration file (from repository root) | +| `INTEL` | boolean | `false` | Whether to run Intel-specific build and test steps | +| `GCC` | boolean | `true` | Whether to run GCC-specific build and test steps | | `BUILD_SCRIPT` | string | `.github/build_real.sh` | Path to build script. Empty string skips this step | | `TEST_SCRIPT` | string | `.github/test_real.sh` | Path to test script. Empty string skips this step | -| `BUILD` | boolean | `true` | Whether to run the build step | | `TEST` | boolean | `true` | Whether to run the test step | | `MYPY` | boolean | `false` | Whether to run mypy type checking | diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 03a4e89..b631164 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -29,10 +29,6 @@ on: type: string default: ".github/test_real.sh" description: "Path to test script" - BUILD: - type: boolean - default: true - description: "Whether to run the build step" TEST: type: boolean default: true @@ -88,7 +84,7 @@ jobs: docker exec app /bin/bash -c "rm -rf $DOCKER_WORKING_DIR && mkdir -p $DOCKER_WORKING_DIR && cp -a $DOCKER_MOUNT_DIR/. $DOCKER_WORKING_DIR/" - name: Build - if: inputs.BUILD_SCRIPT != '' && inputs.BUILD + if: inputs.BUILD_SCRIPT != '' run: | docker exec \ -e CONFIG_FILE="${{ env.CONFIG_FILE }}" \ @@ -123,7 +119,7 @@ jobs: runs-on: ubuntu-24.04 timeout-minutes: ${{ inputs.TIMEOUT }} env: - DOCKER_TAG: u24-intel-ompi-latest + DOCKER_TAG: u24-intel-impi-latest CONFIG_FILE: ${{ inputs.INTEL_CONFIG }} steps: *build_steps diff --git a/README.md b/README.md index 6738095..1672029 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,18 @@ # .github This repo stores the following shared repository settings/configurations/templates: - Issue and pull request templates on GitHub -- Shared configurations for - - `pre-commit` - - `ruff` - - `isort` - - `pylint` - - `codecov` - - `mypy` - Shared configurations for GitHub Actions + - `ruff` + - formatting + - linting + - isort + - McCabe Complexity + - `clang_format` + - `build` + - Docker build + - Integration testing + - `mypy` + - `fprettify` + - `tapenade` + - `branch-name-check` - Issue/PR labels shared across the organization -- Configuration script for the build bot From b6571d61702f1a3fd31350013ea68f8e947693fc Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Fri, 13 Feb 2026 21:02:34 +0000 Subject: [PATCH 44/50] Restoring fprettify shell script --- fprettify.sh | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 fprettify.sh diff --git a/fprettify.sh b/fprettify.sh new file mode 100644 index 0000000..ea941b8 --- /dev/null +++ b/fprettify.sh @@ -0,0 +1,35 @@ +#!/bin/bash + +FPRETTIFY_CONFIG_FILE="" +LOCAL_CONFIG_FILE=".fprettify.rc" +GLOBAL_CONFIG_FILE="./.fprettify.rc" + +if [[ ! -z "$1" ]]; then + # Offer the option of supplying config file through an argument + echo "Using user supplied fprettify config file: $1" + FPRETTIFY_CONFIG_FILE="$1" +elif [[ -f "$LOCAL_CONFIG_FILE" ]]; then + # Use local config file + echo "Using local fprettify config file: $LOCAL_CONFIG_FILE" + FPRETTIFY_CONFIG_FILE="$LOCAL_CONFIG_FILE" +elif [[ -f "$GLOBAL_CONFIG_FILE" ]]; then + # Use global config file + echo "Using .github shared fprettify config file: $GLOBAL_CONFIG_FILE" + FPRETTIFY_CONFIG_FILE="$GLOBAL_CONFIG_FILE" +else + echo "No fprettify config was found. Exiting" + exit 2 +fi + +# Run fprettify on non-differentiated Fortran 90 source files +find . -type f -iname '*.f90' ! -iname '*_b.f90' ! -iname '*_d.f90' -print0 | while read -d $'\0' fileName +do + echo "Checking $fileName" + fprettify --config-file "$FPRETTIFY_CONFIG_FILE" "$fileName" + + # Check if this file changed and print a message if it did + git diff --summary --exit-code $fileName + if [ $? -ne 0 ]; then + echo "$fileName was formatted" + fi +done From 8217298c30e512ee11a18863a7573dbcffe4d93a Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Fri, 13 Feb 2026 21:03:30 +0000 Subject: [PATCH 45/50] Fixing secrets docs in readme --- .github/workflows/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 337e11e..91b6cd7 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -170,7 +170,7 @@ testflo -v . -n 1 ### Step 4: Add Secrets -In your repository settings, add the required secrets: +In your orginaization settings, add the required secrets: - `DOCKER_USER` - Docker organization name - `DOCKER_OAT` - Docker Organization Access Token for pulling private images From 97e58379a66bacd52613461bb1c7735eed059802 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Fri, 13 Feb 2026 21:29:03 +0000 Subject: [PATCH 46/50] Updating workflow options section --- .github/workflows/README.md | 30 +++++------------------------- 1 file changed, 5 insertions(+), 25 deletions(-) diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 91b6cd7..a170dc8 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -17,6 +17,8 @@ Reusable GitHub Actions workflows for Supercritical repositories. These workflow ## Workflow Options +Configuration inheritance/overrides are documented in each workflow section below. + ### build.yaml Docker-based build and test workflow using the `scritical/private-dev` image. @@ -40,6 +42,8 @@ Runs separate GCC and Intel jobs when the matching config input is provided. | `DOCKER_USER` | Docker registry username | | `DOCKER_OAT` | Docker registry Organization Access Token | +If these secrets are configured at the organization level, callers can use `secrets: inherit` instead of listing each secret. + --- ### ruff.yaml @@ -128,9 +132,7 @@ jobs: build: uses: scritical/.github/.github/workflows/build.yaml@main with: - GCC_CONFIG: config/defaults/config.LINUX_GFORTRAN.mk - BUILD_SCRIPT: .github/build_real.sh - TEST_SCRIPT: .github/test_real.sh + GCC_CONFIG: config/defaults/config.LINUX_GFORTRAN.mk # If necessary secrets: DOCKER_USER: ${{ secrets.DOCKER_USER }} DOCKER_OAT: ${{ secrets.DOCKER_OAT }} @@ -205,28 +207,6 @@ jobs: --- -## Configuration Inheritance - -Several workflows support configuration inheritance/overrides: - -| Tool | Method | Local Config File | -| :--- | :----- | :---------------- | -| Ruff | Native `extend` keyword | `ruff.toml` | -| clang-format | Local file takes precedence | `.clang-format` | -| fprettify | Local file takes precedence | `.fprettify.rc` | - -### Example: Ruff Override - -```toml -# ruff.toml -extend = "~/.config/ruff/ruff.toml" - -[lint] -ignore = ["N802", "N803"] # Allow uppercase function/argument names -``` - ---- - ## Adding Status Badge Add a status badge to your README: From fd0814e24e763ee48c0b0fa525866d4a48cd1a41 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Fri, 13 Feb 2026 22:51:56 +0000 Subject: [PATCH 47/50] Readme update and accounting for local/global ruff toml configs --- .github/workflows/README.md | 4 ++-- .github/workflows/ruff.yaml | 17 +++++++++++++++-- 2 files changed, 17 insertions(+), 4 deletions(-) diff --git a/.github/workflows/README.md b/.github/workflows/README.md index a170dc8..ee3fc43 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -22,7 +22,7 @@ Configuration inheritance/overrides are documented in each workflow section belo ### build.yaml Docker-based build and test workflow using the `scritical/private-dev` image. -Runs separate GCC and Intel jobs when the matching config input is provided. +Runs GCC and/or Intel jobs based on `GCC` and `INTEL` input flags. | Name | Type | Default | Description | | :--- | :--- | :------ | :---------- | @@ -49,7 +49,7 @@ If these secrets are configured at the organization level, callers can use `secr ### ruff.yaml Python formatting and linting using Ruff. -The workflow checks out the org-wide Ruff configuration from `scritical/.github` and uses `ruff.toml` from that repo as the base config. +The workflow checks out the org-wide Ruff configuration from `scritical/.github` and uses it for format, lint, and isort checks. | Name | Type | Default | Description | | :--- | :--- | :------ | :---------- | diff --git a/.github/workflows/ruff.yaml b/.github/workflows/ruff.yaml index c84dc17..4fa509c 100644 --- a/.github/workflows/ruff.yaml +++ b/.github/workflows/ruff.yaml @@ -16,8 +16,6 @@ jobs: defaults: run: working-directory: project - env: - RUFF_CONFIG: ../.org-dotgithub/ruff.toml steps: - name: Checkout repository uses: actions/checkout@v6 @@ -35,6 +33,21 @@ jobs: uses: actions/setup-python@v6 with: python-version: "3.11" + + - name: Setup Ruff Configuration + run: | + # Get the org-wide ruff config and copy it to ~/.config/ruff/ruff.toml + mkdir -p ~/.config/ruff + cp .org-dotgithub/ruff.toml ~/.config/ruff/ruff.toml + + if [-f ruff.toml]; then + echo "Found repo-specific ruff.toml" + RUFF_CONFIG=ruff.toml + cat ruff.toml + else + echo "No repo-specific ruff.toml found, using org-wide config" + RUFF_CONFIG=~/.config/ruff/ruff.toml + fi - name: Install Ruff run: | From 3286a73b11122877f652110aed13cead86796e60 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Fri, 13 Feb 2026 22:56:38 +0000 Subject: [PATCH 48/50] Fixing ruff config again --- .github/workflows/ruff.yaml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/workflows/ruff.yaml b/.github/workflows/ruff.yaml index 4fa509c..d697baf 100644 --- a/.github/workflows/ruff.yaml +++ b/.github/workflows/ruff.yaml @@ -38,15 +38,15 @@ jobs: run: | # Get the org-wide ruff config and copy it to ~/.config/ruff/ruff.toml mkdir -p ~/.config/ruff - cp .org-dotgithub/ruff.toml ~/.config/ruff/ruff.toml + cp ../.org-dotgithub/ruff.toml ~/.config/ruff/ruff.toml - if [-f ruff.toml]; then + if [[ -f ruff.toml ]]; then echo "Found repo-specific ruff.toml" - RUFF_CONFIG=ruff.toml + echo "RUFF_CONFIG=ruff.toml" >> "$GITHUB_ENV" cat ruff.toml else echo "No repo-specific ruff.toml found, using org-wide config" - RUFF_CONFIG=~/.config/ruff/ruff.toml + echo "RUFF_CONFIG=~/.config/ruff/ruff.toml" >> "$GITHUB_ENV" fi - name: Install Ruff @@ -75,4 +75,4 @@ jobs: run: | echo "=== McCabe Complexity Report ===" echo "Checking for functions with complexity > 10..." - ruff check --select=C90 . \ No newline at end of file + ruff check --select=C90 --config $RUFF_CONFIG . \ No newline at end of file From f328a2516c5414c6afe76932d38b3233e4a6b060 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Fri, 13 Feb 2026 23:12:20 +0000 Subject: [PATCH 49/50] Adding a standalone mypy workflow --- .github/workflows/build.yaml | 12 ------- .github/workflows/mypy.yaml | 65 ++++++++++++++++++++++++++++++++++++ 2 files changed, 65 insertions(+), 12 deletions(-) create mode 100644 .github/workflows/mypy.yaml diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index b631164..49497d1 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -33,10 +33,6 @@ on: type: boolean default: true description: "Whether to run the test step" - MYPY: - type: boolean - default: false - description: "Whether to run mypy type checking" secrets: DOCKER_USER: required: true @@ -98,14 +94,6 @@ jobs: -e BUILD_REASON="${{ github.event_name }}" \ app /bin/bash -c ". $BASHRC && cd $DOCKER_WORKING_DIR && source ~/MPIOversubscribe.sh && /bin/bash ${{ inputs.TEST_SCRIPT }}" - - name: MyPy type checking - if: inputs.MYPY - run: | - docker exec \ - -e AGENT_NAME="${{ runner.name }}" \ - -e BUILD_REASON="${{ github.event_name }}" \ - app /bin/bash -c ". $BASHRC && cd $DOCKER_WORKING_DIR && mypy ." - - name: Cleanup if: always() run: | diff --git a/.github/workflows/mypy.yaml b/.github/workflows/mypy.yaml new file mode 100644 index 0000000..eb31ad0 --- /dev/null +++ b/.github/workflows/mypy.yaml @@ -0,0 +1,65 @@ +on: + workflow_call: + inputs: + TIMEOUT: + type: number + default: 30 + description: "Timeout for the job in minutes" + secrets: + DOCKER_USER: + required: true + description: "Docker registry username" + DOCKER_OAT: + required: true + description: "Docker registry Organization Access Token" + +env: + SCRITICAL_HOMEDIR: /home/scriticaluser + BASHRC: /home/scriticaluser/.bashrc_scritical + +jobs: + mypy: + runs-on: ubuntu-24.04 + timeout-minutes: ${{ inputs.TIMEOUT }} + env: + DOCKER_TAG: u24-gcc-ompi-latest + steps: + - name: Checkout repository + uses: actions/checkout@v6 + + - name: Set repo environment variables + run: | + repo_name="${{ github.event.repository.name }}" + echo "REPO_NAME=$repo_name" >> "$GITHUB_ENV" + echo "DOCKER_WORKING_DIR=${{ env.SCRITICAL_HOMEDIR }}/repos/$repo_name" >> "$GITHUB_ENV" + echo "DOCKER_MOUNT_DIR=${{ env.SCRITICAL_HOMEDIR }}/mount/$repo_name" >> "$GITHUB_ENV" + + - name: Login to Docker Hub + uses: docker/login-action@v3 + with: + username: ${{ secrets.DOCKER_USER }} + password: ${{ secrets.DOCKER_OAT }} + + - name: Start Docker container + run: | + docker run -t -d --name app \ + --mount "type=bind,src=${{ github.workspace }},target=${{ env.DOCKER_MOUNT_DIR }}" \ + scritical/private-dev:${{ env.DOCKER_TAG }} /bin/bash + + # Copy repo to working directory + docker exec app /bin/bash -c "rm -rf $DOCKER_WORKING_DIR && mkdir -p $DOCKER_WORKING_DIR && cp -a $DOCKER_MOUNT_DIR/. $DOCKER_WORKING_DIR/" + + - name: MyPy type checking + run: | + docker exec \ + -e AGENT_NAME="${{ runner.name }}" \ + -e BUILD_REASON="${{ github.event_name }}" \ + app /bin/bash -c ". $BASHRC && cd $DOCKER_WORKING_DIR && mypy ." + + - name: Cleanup + if: always() + run: | + # Stop and remove the container only if it exists to avoid noisy errors. + if docker ps -a --format '{{.Names}}' | grep -qx app; then + docker stop app && docker rm app || true + fi From ddbaa7d88a72ced8e5ed8b18c78076b598566cb2 Mon Sep 17 00:00:00 2001 From: Andrew Lamkin Date: Fri, 13 Feb 2026 23:17:57 +0000 Subject: [PATCH 50/50] Updating readme with mypy workflow --- .github/workflows/README.md | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/.github/workflows/README.md b/.github/workflows/README.md index ee3fc43..8b65332 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -11,6 +11,7 @@ Reusable GitHub Actions workflows for Supercritical repositories. These workflow | `clang_format.yaml` | C/C++ formatting checks | | `fprettify.yaml` | Fortran 90 formatting checks | | `ruff.yaml` | Python formatting and linting with Ruff | +| `mypy.yaml` | Python type checking with MyPy | | `branch-name-check.yaml` | Enforce branch naming conventions | --- @@ -34,7 +35,6 @@ Runs GCC and/or Intel jobs based on `GCC` and `INTEL` input flags. | `BUILD_SCRIPT` | string | `.github/build_real.sh` | Path to build script. Empty string skips this step | | `TEST_SCRIPT` | string | `.github/test_real.sh` | Path to test script. Empty string skips this step | | `TEST` | boolean | `true` | Whether to run the test step | -| `MYPY` | boolean | `false` | Whether to run mypy type checking | **Required Secrets:** | Name | Description | @@ -67,6 +67,24 @@ ignore = ["N802"] --- +### mypy.yaml + +Runs MyPy type checking in the GCC OpenMPI Docker image. + +| Name | Type | Default | Description | +| :--- | :--- | :------ | :---------- | +| `TIMEOUT` | number | `30` | Runtime allowed for the job, in minutes | + +**Required Secrets:** +| Name | Description | +| :--- | :---------- | +| `DOCKER_USER` | Docker registry username | +| `DOCKER_OAT` | Docker registry Organization Access Token | + +If these secrets are configured at the organization level, callers can use `secrets: inherit` instead of listing each secret. + +--- + ### tapenade.yaml Run Tapenade automatic differentiation and check for uncommitted changes.