CCM-12869: add config -> DynamoDB publisher tooling + release-bundled GitHub Action by m-houston · Pull Request #50 · NHSDigital/nhs-notify-supplier-config

m-houston · 2026-02-27T13:39:05Z

Description

Add packages/file-store for loading supplier config from a local directory containing json files
Add packages/ddb-publisher CLI to publish config records directly into a single DynamoDB table (no event envelope)
- supports --dry-run (no AWS calls/creds; schema validation only)
- pre-load audit scans DDB; blocks upload if non-DISABLED records exist in DDB but not in local config unless --force
- strict pk/sk handling: invalid/missing keys throw with pk/sk included for debug
add a GitHub action that downloads ddb-publish-bundle.tgz from the matching GitHub Release and runs index.cjs (avoids committing multi‑MB dist directory)
Add release workflow to bundle and upload the CLI artifact to Releases, and run the bundling + smoke-test in stage-3-build to keep the release build healthy

Context

Provides CLI and actions for publishing supplier config to a dynamodb store.

Example of the action being invoked from the notify-internal repo includes currently up to date supplier config.

Type of changes

Refactoring (non-breaking change)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would change existing functionality)
Bug fix (non-breaking change which fixes an issue)

Checklist

I am familiar with the contributing guidelines
I have followed the code style of the project
I have added tests to cover my changes
I have updated the documentation accordingly
This PR is a result of pair or mob programming
This PR includes code generated by a coding agent

Sensitive Information Declaration

To ensure the utmost confidentiality and protect your and others privacy, we kindly ask you to NOT including PII (Personal Identifiable Information) / PID (Personal Identifiable Data) or any other sensitive data in this PR (Pull Request) and the codebase changes. We will remove any PR that do contain any sensitive information. We really appreciate your cooperation in this matter.

I confirm that neither PII/PID nor sensitive data are included in this PR and the codebase changes.

… GitHub Action - Add @supplier-config/file-store loader/validator for local config store layout store = subdir per domain entity, JSON file per record zod-based parsing/validation; fail fast on parse errors - Add @supplier-config/ddb-publisher CLI (supplier-config-publish) to publish config records directly into a single DynamoDB table (no event envelope) -supports --dry-run (no AWS calls/creds; schema validation only) pre-load audit scans DDB; blocks upload if non-DISABLED records exist in DDB but not in local config unless --force - strict pk/sk handling: invalid/missing keys throw with pk/sk included for debug - add a lightweight composite action that downloads ddb-publish-bundle.tgz from the matching GitHub Release and runs index.cjs (avoids committing multi‑MB dist bundles; action contains no reimplemented logic) - Add release workflow to bundle and upload the CLI artifact to Releases, and run the bundling + smoke-test in stage-3-build to keep the release build healthy

…ript

…ge imports

…of imports

…d compatibility

… packages

…nloads and improve error handling

…l runs and improve error handling

…configuration

packages/file-store/src/loader/config-store-loader.ts

nhsd-david-wass

Can we have a README with instructions for running the cli please

…fig store - add a short README for the ddb-publisher CLI - document running the CLI against a local DynamoDB container - add AWS CLI instructions to create the local supplier config table - add a minimal example config store under tests/example-config-store - rename the package script from start to cli for clearer CLI usage - allow localhost DynamoDB endpoints to use default local fake credentials - rename audit result field from blocking to blockingRecords for clarity - update audit and run tests to match the new audit shape - add coverage for local vs non-local endpoint credential handling

… config store loading

…dation tests

…onfiguration

…update validation logic

m-houston · 2026-03-12T11:41:34Z

packages/ddb-publisher/src/ddb/publish.ts

+  const base = persistedRecordShapeSchema.parse(record.data);
+
+  return {
+    pk: pkForEntity(record.entity),


@nhsd-david-wass - does this schema work for you?

(it only adds pk and sk to the base domain model schema)

…template

Copilot

Pull request overview

This PR introduces a local “supplier config store” file loader/validator plus a DynamoDB publishing CLI, then wires that CLI into a composite GitHub Action that runs from a release-bundled artifact (rather than committing a large dist directory). It also adds CI workflow support to bundle and publish the artifact and to run integration tests.

Changes:

Add @supplier-config/file-store for loading config JSON records from disk and validating them against the supplier-config event schemas.
Add @supplier-config/ddb-publisher CLI to audit a target DynamoDB table and batch-publish validated config records (plus unit + integration tests and a release bundling script).
Add composite action actions/ddb-publish and CI workflows to bundle/upload the CLI artifact and run integration tests.

Reviewed changes

Copilot reviewed 45 out of 46 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
tests/example-config-store/volume-group/vg-1.json	Adds example volume-group record for local/integration testing.
tests/example-config-store/supplier/sup-1.json	Adds example supplier record for local/integration testing.
tests/example-config-store/supplier-pack/sp-1.json	Adds example supplier-pack record for local/integration testing.
tests/example-config-store/supplier-allocation/alloc-1.json	Adds example supplier-allocation record for local/integration testing.
tests/example-config-store/pack-specification/pack-spec-1.json	Adds example pack-specification record for local/integration testing.
tests/example-config-store/letter-variant/lv-1.json	Adds example letter-variant record for local/integration testing.
tests/example-config-store/README.md	Documents the example config-store directory structure.
scripts/config/vale/styles/config/vocabularies/words/accept.txt	Updates Vale vocabulary to accept “repo”.
scripts/config/pre-commit.yaml	Cleans up local hook configuration layout.
packages/file-store/tsconfig.json	Adds TS config for new file-store package.
packages/file-store/tsconfig.build.json	Adds build TS config for new file-store package.
packages/file-store/src/validation/config-store-validator.ts	Adds schema-based validation for loaded config records.
packages/file-store/src/types.ts	Adds shared types for loaded records and validation results.
packages/file-store/src/loader/config-store-loader.ts	Adds loader to read JSON records from persisted entity folders.
packages/file-store/src/index.ts	Exposes public exports for file-store package.
packages/file-store/src/tests/config-store-validator.test.ts	Unit tests for validator behavior and error reporting.
packages/file-store/src/tests/config-store-loader.test.ts	Unit tests for loader behavior and error cases.
packages/file-store/package.json	Introduces file-store workspace package manifest.
packages/file-store/jest.config.ts	Adds Jest config for file-store unit tests and coverage.
packages/ddb-publisher/tsconfig.json	Adds TS config for new publisher CLI package.
packages/ddb-publisher/src/types.ts	Adds CLI plan/audit types.
packages/ddb-publisher/src/run.ts	Implements CLI execution flow: load → validate → audit → publish.
packages/ddb-publisher/src/ddb/publish.ts	Implements DynamoDB BatchWrite publishing with retries.
packages/ddb-publisher/src/ddb/keys.ts	Defines pk/sk formatting helpers for persisted items.
packages/ddb-publisher/src/ddb/audit.ts	Implements DynamoDB scan-based audit to find blocking records.
packages/ddb-publisher/src/cli.ts	Adds yargs-based CLI entrypoint.
packages/ddb-publisher/src/tests/run.test.ts	Unit tests for run flow (dry-run, validation failure, audit blocking, endpoint handling).
packages/ddb-publisher/src/tests/publish.test.ts	Unit tests for batching/retries/item shape in publish logic.
packages/ddb-publisher/src/tests/audit.test.ts	Unit tests for audit scan paging and blocking record detection.
packages/ddb-publisher/src/integration/publish-action.integration.test.ts	Integration test that bundles the CLI and publishes into DynamoDB Local.
packages/ddb-publisher/src/integration/dynamodb-local.ts	Testcontainers DynamoDB Local helpers for integration tests.
packages/ddb-publisher/scripts/bundle-release.mjs	Adds esbuild bundling script for release/action artifact generation.
packages/ddb-publisher/package.json	Introduces ddb-publisher workspace package manifest and scripts.
packages/ddb-publisher/jest.integration.config.ts	Adds separate Jest config for integration tests.
packages/ddb-publisher/jest.config.ts	Adds Jest config for ddb-publisher unit tests and coverage.
packages/ddb-publisher/README.md	Documents CLI usage, local DynamoDB setup, and integration test guidance.
packages/ddb-publisher/.gitignore	Ignores generated bundle artifacts.
package.json	Updates tooling deps/scripts and expands workspaces list.
actions/ddb-publish/action.yml	Adds composite action to download and run the bundled CLI from Releases/CI artifacts.
.tool-versions	Updates the Node.js tool version used by tooling/workflows.
.github/workflows/stage-3-build.yaml	Adds job to bundle and upload the ddb-publish tarball as a workflow artifact.
.github/workflows/stage-2-test.yaml	Adds integration test job to CI.
.github/workflows/release-ddb-publish.yml	Adds workflow to bundle and upload tarball to GitHub Releases (or as workflow artifact).
.github/actions/bundle-ddb-publish/action.yml	Adds reusable composite action to bundle + smoke-test + package the tarball.
.github/PULL_REQUEST_TEMPLATE.md	Adds checkbox indicating code generated by a coding agent.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

packages/file-store/jest.config.ts

packages/ddb-publisher/src/ddb/publish.ts

packages/ddb-publisher/src/ddb/audit.ts

packages/ddb-publisher/src/__integration__/dynamodb-local.ts

packages/ddb-publisher/scripts/bundle-release.mjs

package.json

packages/file-store/src/types.ts

…n and config files

…sion to 1.0.2

m-houston added 12 commits February 27, 2026 13:36

Refactor Node.js setup in CI workflows and enhance dependency management

fc3a7f4

Ensure workspace packages are generated before bundling in release sc…

7b82e85

…ript

Enhance bundling process in release script to resolve workspace packa…

74fd573

…ge imports

Use absolute path for alias in bundling to ensure correct resolution …

79879b0

…of imports

Update dependencies in package.json and package-lock.json for improve…

2d41988

…d compatibility

Improve debug logging and update bundling configuration for workspace…

ff73805

… packages

Add ddb-publish bundling action and update release workflows

c2f003c

Add workflow artifact upload and improve ddb-publish bundle resolution

9234b39

Refactor ddb-publish action to use action repository for artifact dow…

b30ec6f

…nloads and improve error handling

Enhance ddb-publish action to search multiple workflows for successfu…

70ff42e

…l runs and improve error handling

Enhance runPublisher to log detailed steps and summarize loaded records

daecddc

m-houston marked this pull request as ready for review March 3, 2026 14:25

m-houston requested review from a team as code owners March 3, 2026 14:25

m-houston added 4 commits March 3, 2026 14:55

Add integration tests for DynamoDB Local and support custom endpoint …

1f2a4bd

…configuration

Remove redundant console log eslint disables in bundle-release.mjs

6d3216e

Use testcontainers for ddb-local container management

365fe60

Refactor pre-commit hooks to remove redundant local repo entries

288f9b2

nhsd-david-wass reviewed Mar 10, 2026

View reviewed changes

packages/file-store/src/loader/config-store-loader.ts Show resolved Hide resolved

nhsd-david-wass reviewed Mar 10, 2026

View reviewed changes

m-houston added 5 commits March 11, 2026 18:37

feat(tests): enhance integration tests for DynamoDB Local and improve…

67e9305

… config store loading

refactor(tests): replace 'kind' fields with 'id' in config store vali…

53ff4e0

…dation tests

feat(tests): add test for explicit AWS env values in local DynamoDB c…

eed3b67

…onfiguration

refactor(publish): remove 'entity' and 'env' fields from records and …

423ba1d

…update validation logic

m-houston commented Mar 12, 2026

View reviewed changes

m-houston added 2 commits March 12, 2026 11:52

test(run): manage AWS environment variables during DynamoDB local tests

5a82841

chore(template): add note for code generated by a coding agent in PR …

6054b00

…template

m-houston requested a review from Copilot March 13, 2026 12:45

Copilot started reviewing on behalf of m-houston March 13, 2026 12:46 View session

Copilot AI reviewed Mar 13, 2026

View reviewed changes

m-houston added 3 commits March 13, 2026 16:05

feat(lint): add linting and type checking scripts; update package.jso…

2052c7a

…n and config files

fix(bundle): use fileURLToPath for packageRoot resolution; update ver…

07296d3

…sion to 1.0.2

feat(package): add pretypecheck script and update dependencies

2f9126b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CCM-12869: add config -> DynamoDB publisher tooling + release-bundled GitHub Action#50

CCM-12869: add config -> DynamoDB publisher tooling + release-bundled GitHub Action#50
m-houston wants to merge 26 commits intomainfrom
CCM-12869-dynamodb-publish-action

m-houston commented Feb 27, 2026 •

edited

Loading

Uh oh!

Uh oh!

nhsd-david-wass left a comment

Uh oh!

m-houston Mar 12, 2026

Uh oh!

m-houston Mar 12, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

m-houston commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Context

Type of changes

Checklist

Sensitive Information Declaration

Uh oh!

Uh oh!

nhsd-david-wass left a comment

Choose a reason for hiding this comment

Uh oh!

m-houston Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

m-houston Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

m-houston commented Feb 27, 2026 •

edited

Loading