Tagging for TF modules in charm repos

[MichaelThamm]

Summary

What do we want to do with tagging charm repos to match the CC008 spec? The spec gives us freedom to decide as long as we meet SemVer requirements. We want this solution to:

• have friendly UX for our users
• be extendible to product modules
• be automatable: tagging the repo and releasing to the TF registry

The core of the problem

Since we want the charm to have channel validation, we want to hint that in the tag. How can we map charm tracks to TF SemVer?

Tip

I think the best option is C. It works well for product modules as well. The user's confusion regarding what workload the tag is for can be reduced with the module's README. The runner-up is B. Please see the pros and cons for each.

Statements of truth

Charm tracks

1. All tracks are feature frozen, except dev.
2. It is never ok to make breaking changes in-track, only on dev; the source of new tracks.
3. If upstream major bumps, we make an effort to bump our track at the end of the current cycle, unless time pressure/capacity does not permit.
4. Some charms have tracks major.minor if they have an upstream workload.
   • Otherwise, they have major following COS' track: reference

Terraform

1. We must bump the major version on breaking changes, the minor for non-breaking changes, and the patch for trivial fixes (e.g., updating descriptions) and non-breaking bug fixes.
2. A breaking change can be either:
   • an API change to the TF layer e.g., removing an endpoint in the outputs.tf.
   • an endpoint interface changes without updating the endpoint name i.e., no change to the TF layer, but a side-effect in the module.
3. Terraform registry requires tags following a subset of SemVer i.e., {major}.{minor}.{patch} without features like metadata or pre-releases.
4. The registry will not pick up our rev123 tags because they are not valid semver.

Decision making

The main blockers for each strategy are identified with ❗️.

Rubric

• A: Charm revisions
• B: Track(major.minor), TF .patch SemVer
• C: Pure TF SemVer
• D: Track(major.minor) SemVer

Identifier	Tag scheme UX	Terrareg support	CC008 Compatible
A	❌	❌	❌
B	❌	✅	✅
C	❓️	✅	✅
D	✅	✅	❌

Tagging strategies

Upstream	Track	Cycle	Event	B	C	D
3.6.7	3.6	26.04		3.6.0	1.0.0	3.6 3.6-tf.0
			A TF endpoint added to track 3.6	3.6.1	1.1.0	3.6 3.6-tf.1
			A TF endpoint added to previous track e.g., 2.9	2.9.n+1	n-1.n+1.0 or n.n+1.0	2.9 2.9-tf.n+1
3.6.7	3.6	26.10	We consciously decide not to track bump (to 3.7.0) across cycles	3.6.1	1.1.0	3.6 3.6-tf.1
4.1.0	4.1	27.04	We time a breaking change with 27.04	4.1.0	2.0.0	4.1 4.1-tf.0
			We decide to make a breaking change mid-cycle	❗️	❗️	3.0.0

A: Charm revisions

Good:

• We already have the tags and automation.

Bad:

• ❗️Does not conform to CC008 spec.
• Tags are opaque e.g., rev123 is specific to track 2, but rev124 is specific to track dev.
• We do not release with (only) TF changes

B: Track(major.minor), TF .patch SemVer

Good:

• Provides insight on which track the tag is for.
• Terrareg allows mapping a git tag 1.2-tf.0 -> 1.2.0 in Terrareg; obvious in git, but confusing in Terrareg UI.

Bad:

• ❗️The TF-specific patch can confuse the user that it is the workload version patch.
• Workload-less charm tracks use the stack track i.e., just a major, not major.minor
  • We could map to major.minor e.g., 3 -> 3.0 and a TF version of 3.0.TF
  • This would not map well to product modules since they have no upstream e.g., major.TF.TF
  • Would documentation be enough for users to understand?

C: Pure TF SemVer

Good:

• Aligns most closely to the CC008 spec.
• Major changes are not gated per cycle.

Bad:

• ❗️Tags are opaque to charm-tracks e.g., 1.0.0 is specific to track 2.
  • A user needs to search release notes / module docs to understand what tracks are allowed.
• We have to manage a VERSION file for tagging CI.

D: Track(major.minor) SemVer

Good:

• Similar to sourcing product modules with track branches, but with tags.
  • E.g., in 26.04 source ?ref=track/3.6.
  • E.g., in 26.10 source ?ref=3.6.
• Breaking (major) changes are gated for 6 months (per cycle).
• The workload version is the tag.
• Having a second tag (with metadata and no TF changes) acts as a source of truth for users.
  • metadata identifies the tag creation order e.g., 3.6+tf.1 vs. 3.6+tf.2.

Bad:

• ❗️Does not fully conform to the CC008 spec.
• ❗️We lose reproducibility.
  • e.g., day 1 the module documentation says it supports endpoint X, then day 2 it supports both endpoints X & Y
  • i.e., user needs to refresh the documentation vs. going to the new version.
  • e.g., day 1 terraform apply, then day 2 terraform apply is broken!
• In the event that we (unlikely) commit a regression and overwrite the tag, the admin might not know how to (or that they even can) switch between a registry ref to a GitHub ref.
  • e.g., 3.6 is broken so they must use the ?ref=3.6+metadata tag, which is not published in the registry.
• Hard to scale the fixes if we make a breaking change e.g., ManSol needs to make many changes to switch to the -tf.n tags.

TODOs

• Comment on registry TF spec: both registry and module specs
• Move this to the teams ADRs or to a spec and close this issue

[MichaelThamm]

Testing Terrareg

sudo apt install podman
export SECRET_KEY=$(python3 -c 'import secrets; print(secrets.token_hex())')
podman run -d \
    --name terrareg \
    -p 5000:5000 \
    -e PUBLIC_URL=http://localhost:5000 \
    -e MIGRATE_DATABASE=True \
    -e SECRET_KEY=$SECRET_KEY \
    -e ADMIN_AUTHENTICATION_TOKEN=MySuperSecretPassword \
    --restart always \
    ghcr.io/matthewjohn/terrareg:latest

[MichaelThamm]

Channel/revision validation in the TF module?

Do we care about how much the user-requested charm diverges from its TF?

Yes:

• If we do not guide the user with TF validation they will have more apply-time (Juju client) failures.
  • Revision mismatches are not covered by our TF modules.
• Diko (ManSol) prefers this

flowchart TB
    classDef tf fill:#bbf,stroke:#333,stroke-width:1px,color:black;
    classDef charm fill:#fdb,stroke:#333,stroke-width:1px,color:black;
    classDef match fill:#dfd,stroke:#333,stroke-width:1px,color:black;
    classDef mismatch fill:#fbb,stroke:#333,stroke-width:2px,color:black,stroke-dasharray: 5 5;
    classDef consumer fill:#e8e8ff,stroke:#333,stroke-width:1px,color:black;

    Consumer[Example Charm<br/>requires: send-otlp]:::consumer
    Consumer -- "juju integrate" --> TFOut3
    subgraph TF ["Terraform Module (outputs.tf)"]
        direction TB
        TFOut2[output: metrics-endpoint]
        TFOut3[output: send-otlp]:::mismatch
        TFNote["'logging' removed\n'send-otlp' added"]
    end

    subgraph Charm ["Deployed Charm (charmcraft.yaml) -> User-selected channel/revision"]
        direction TB
        CharmEp2[provides: metrics-endpoint]
        CharmEp3[requires: logging]:::mismatch
        CharmNote["Still has 'logging'\nDoes not have 'send-otlp'"]
    end

    TFOut2 -. "ok" .-> CharmEp2
    TFOut3 -. "No matching endpoint\nin charm" .-x CharmEp3

    class TFOut1,TFOut2,CharmEp1,CharmEp2 match;
    class TF tf;
    class Charm charm;

Loading

No:

• This idea was implemented in the stack because we were not able to have smooth cross-track upgrades. It may no longer be necessary with howto/manage-charms/#update-a-charm-when-a-relation-would-break implemented in:
  • feat: Cross-track upgrade documentation observability-stack#174
• There is no concept of a "breaking change" for a charm module, except in the context of other component, product, or solution modules, all of which are not the concern of a charm module.

Implementation

We can warn users when using incompatible channel input like we do in the stack using the tagging strategy:

Valid cases:

module "catalogue" {
  source     = "git::https://github.com/MichaelThamm/catalogue-k8s-operator//terraform?ref=2.0.0"
  channel    = "3.9/stable"
}

module "catalogue" {
  source     = "git::https://github.com/MichaelThamm/catalogue-k8s-operator//terraform?ref=3.0.0"
  channel    = "4.1/stable"
}

Invalid case:

module "catalogue" {
  source     = "git::https://github.com/MichaelThamm/catalogue-k8s-operator//terraform?ref=2.0.0"
  channel    = "4.1/stable"
}

│ Error: Invalid value for variable
│ 
│   on root.tf line 17, in module "catalogue":
│   17:   channel    = "4.1/stable"
│     ├────────────────
│     │ var.channel is "4.1/stable"
│ 
│ The track of the channel must be '3.9/'. e.g. '3.9/edge'.

[MichaelThamm]

Strategy C workflow

flowchart TD
    %% Define styles for clarity
    classDef decision fill:#f9f,stroke:#333,stroke-width:2px,color:black;
    classDef process fill:#bbf,stroke:#333,stroke-width:1px,color:black;
    classDef artifact fill:#dfd,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5,color:black;
    classDef automatic fill:#fdb,stroke:#333,stroke-width:1px,font-weight:bold,color:black;
    classDef endnode fill:#ddd,stroke:#333,stroke-width:1px,color:black;

    %% 1. Process
    Start([Start: TF Module Change Needed]) --> IdentifyTrack[1. Identify Target Charm Track <br/> Checkout Corresponding Git Branch]
    IdentifyTrack --> MakeChanges[2. Make Changes to TF Source on Branch]

    %% 2. Context
    subgraph RepoContext ["Source of Truth (Independent per Branch)"]
        direction TB
        TrackBranch[Git Branch<br/>ex: 'main' or 'track/3.6']
        VersionFile[terraform/VERSION file]
        TrackBranch -.-> VersionFile
    end
    IdentifyTrack -. "maps to" .-> TrackBranch

    %% 3. Decision
    MakeChanges --> AssessChange{3. Assess Change Type}

    AssessChange -- "Breaking Change<br/>(API or Side-effect)" --> BumpMajor[Calculate Major Bump<br/>ex: 1.1.0 -> 2.0.0]
    AssessChange -- "New Feature<br/>(Non-breaking)" --> BumpMinor[Calculate Minor Bump<br/>ex: 1.1.0 -> 1.2.0]
    AssessChange -- "Bug Fix / Trivial" --> BumpPatch[Calculate Patch Bump<br/>ex: 1.1.0 -> 1.1.1]

    %% 4. Update
    BumpMajor --> UpdateVersion[4. Update VERSION File on Branch]
    BumpMinor --> UpdateVersion
    BumpPatch --> UpdateVersion
    UpdateVersion -. "modifies" .-> VersionFile
    UpdateVersion --> CommitPush[5. Commit and Push Changes]

    %% 5. Automation subgraph
    subgraph CICD ["CI/CD Automation Pipeline (Strategy C)"]
        direction TB
        Verify["A. Verify VERSION File Changed<br/>(Linter check)"]
        CreateTag[B. Create Git Tag<br/>matching VERSION file<br/>ex: 'v1.2.0']
        Publish[C. Publish Module to TF Registry]
        Fail([CI Fails: VERSION update required])

        Verify -- "Yes" --> CreateTag
        Verify -- "No" --> Fail
        CreateTag --> Publish
    end

    CommitPush -. "Triggers" .-> Verify
    Publish --> End([End: Module Released to Registry])

    %% Assign Classes (Moved outside subgraphs to fix lexical errors)
    class AssessChange decision;
    class TrackBranch,VersionFile artifact;
    class UpdateVersion process;
    class Verify,CreateTag,Publish automatic;
    class End,Fail endnode;

Loading

[lucabello]

My two cents.

We're seeing a number of problems from trying to have semver coexist with tracks.

Given how the Canonical ecosystems (charms, snaps, rocks) are founded on tracks, wouldn't it make sense to use/develop a Terraform registry that supports tracks rather than semver?

Some problems I see with this solution

The current solution means that when someone wants to use one of our module, after deciding they want (for example) Loki 3.6, they would have to:

1. Open some documentation to see which Terraform module version would give them Loki 3.6 (because of the validation we add that restricts the channel)
2. Import the module from that version, and set the Loki channel to 3.6/risk (because of the validation)

This makes the channel output somewhat useless: having only one allowed track is a bit misleading - no misconfiguration can happen because of validation, but if a user decides to bump to a new track 4 they would:

1. Try to change the channel to 4.0/stable
2. See the validation error
3. Go back to some docs to figure out the correct Terraform module version

Another idea

A different approach, already floated by @sed-i, is to use GitHub semver tags as tracks. The tag 3.6 would target both the Terraform module and the charm, and it would follow the track/3.6 branch.

Good

This would follow the CC008 spec, it would support Terrareg as a Terraform registry, and would be very simple to maintain.

• Breaking changes still never happen in-track (as they're feature-frozen)
• No VERSION file to maintain/update in different tracks
• Leverage our current release CI
  • If charm tests pass on main (which include Terraform lint), we also tag with the track name both the charm and the Terraform module
  • If we move integration tests to a quality gate rather than on PR, tagging the Terraform module would also automatically move there
• Auto-get Terraform fixes when the user refreshes the Terraform module, instead of needing to go through the docs

Bad

• Tags are floating and not fixed: git tags are already mutable, so it's not a guarantee regardless (and probably not an expectation either)
• Pinning to a specific Terraform module at a point in time can be done via the commit hash, which technically goes against CC008

[sed-i]

Pinning to a specific Terraform module at a point in time can be done via the commit hash, which technically goes against CC008

While terrareg takes only semver, our tags on github could be richer. We can have two tags on the same commit, e.g.

1. 3.6
2. 3.6-some-metadata

The first is floating, while the second we make a point of not bumping. This way the floating tag fulfills the spec and the "fixed" tag gives users what they want.

I wonder though if terrareg would let us push different content into the same tag.

[MichaelThamm]

I wonder though if terrareg would let us push different content into the same tag.

What do you mean by this? 3.6-some-metadata cannot be uploaded to Terrareg, although 3.6 can.

If we publish a tag to Terrareg from a GitHub repo tag e.g., 3.6 and then delete that tag in GitHub, that tag persists in the registry. We can then create that same tag 3.6 again with modified content and publish it again to Terrareg. This will overwrite the module which was in the registry. So the proposal of having a tag major.minor is a valid workflow.

However, I would argue that users do expect a tag to remain consistent and not float like we are suggesting.

[lucabello]

The thing is, even if the Terraform module doesn't change, the underlying charms will, because we don't do pinning of charm revision at this stage.

Users won't be able to get the same exact results regardless (for now), as the charms they will get are different. We won't pin charm modules to a specific charm revisions.

For product modules, assuming at some point in the future we do the give tests to SolQA -> release tf module with pinned revisions, then we can release product modules that will always produce the same results.

Having a non-floating tag for charm modules only gives the illusion of repeatable deployment imho.

[marcusboden]

One thing regarding strategy D: I expect my deployments to go the same every time.
With option D, you could have 10 deployments that went perfectly, then someone introduced a bug into the module and suddenly it stops working. The problem is: You did not change anything, and you suddenly get an error. Even if the whole -patch-1-thing documented and you can switch to that, at the moment it breaks, I will not remember that. I didn't change anything, and I therefore don't expect anything changing in my module. (And If, for that, I always have to add -patch versions that I deploy directly from a repo, we might as well just skip the registry altogether.)

With option C, there are no unexpected changes. If I upgrade to 1.1.0 and something breaks, I know that I just did a minor upgrade and I can just reverse it to 1.0.0.

So, from a user/operator perspective, I'd favor reproducibility of my deployments over readability of the versions.

[MichaelThamm]

@lucabello

Users won't be able to get the same exact results regardless (for now), as the charms they will get are different. We won't pin charm modules to a specific charm revisions.

To provide a specific example to the problem you are highlighting:

1. Loki charm code (previously rev 123) features a new endpoint (foo) in the charmcraft.yaml, released to rev124

Using Strategy C

2. The TF module (previously 1.0.0) adds this endpoint to its outputs.tf file, released to 1.1.0
3. Users can deploy the charm module from either 1.0.0 or 1.1.0; pinning to either revision (123/124).
4. A COS product module maintainer:
   1. Bumps the Loki module version to 1.1.0, but leaves the rev123 pin.
   2. Adds a TF integration for Loki over the foo endpoint; exists in TF, but rev123 doesn't support it yet.

This was a product module maintainer error because they need to know that the charm revision needs to bump with the tag.

Using Strategy D

2. The TF module (previously 3.6 / 3.6+patch-0) adds this endpoint to its outputs.tf file, released to 3.6 / 3.6+patch-1
3. Users can deploy the charm module from 3.6; pinning to either revision (123/124).
4. A COS product module maintainer:
   1. Doesn't change the Loki module version e.g., 3.6 and leaves the rev123 pin.
   2. Adds a TF integration for Loki over the foo endpoint; exists in TF, but rev123 doesn't support it yet.

This is the problem @marcusboden is highlighting and is partially a product module maintainer error.

[dparv]

A.. no. D will give you unpredictable results, so, down-voting that option. Both B and C look good to me, either of one will suffice in my use case - but it depends how the majority of people using this will suffer.

I was thinking about something of a mix between B and C - where we have terraform modules/repository for each upstream product version and then follow the C schema inside for versions of the terraform itself. So in terrareg we'll have product-, for example:

• catalogue-k8s-operator-3.9
• catalogue-k8s-operator-4.1
• postgresql-14
• postgresql-16

This way we are splitting the product version from the terraform itself - so people will know what product version they are deploying, and will have a good versioning of the terraform module, so deploying X.Y.Z or X.Y.Z+1 from - is always going to be the same product version. CC007 doesn't say we can't do that :)

But if that is not a possibility - then I would prefer C, so that we don't mix product and terraform - I think it's going to be very messy if we go the route of having B and people start confusing product and terraform versions.

[MichaelThamm]

@dparv possible implementation:

• We can fork the main charm every cycle-end and add the upstream tag e.g. CHARM_REPO-major.minor in the fork input
  • Each fork is specific to the source repo's (e.g., CHARM_REPO) branch related to that track and only relevant for TF.
• Add this to a GH workflow; tiggered at cycle-end
  • E.g., using GH CLI gh repo fork [<repository>] [-- <gitflags>...] [flags]

Counter arguments:

• This still requires user to read docs to know which upstream channel input is supported
  • since we agreed validating the channel input is a good idea
• This adds A LOT of repos to the canonical org; 1 fork per charm per cycle -> maintenance effort & developer confusion
• This would impose changes to how we develop in charm repos i.e., we have branches per track per repo, but you want 1 track per repo.
• "feature frozen" and "saving breaking changes for the next track" means TF SemVer has little utility

Summary:
I think this is not a valid option because the TF tagging solution should not impose changes on the structure/development of charm repos.