CI: Add test that checks that old releases can verify

[jku]

Making sure that old clients can verify bundles created by current code (and current public good infrastructure) is not too difficult and would be very useful.

This is an initial attempt, keeping it draft for now:

• I don't like that the tested versions are hard coded... but I don't think there is much alternative here, testing all tags is silly. It could be easy to forget to add releases here but I don't have better solutions
• I think we only want to test last patch release of every minor release that we still care about
• I would like to say we still care about v3.5.3 (released Nov 2024) but that release no longer handles current trusted root correctly and we did not patch that in 3.5.x: Backport trust root fixes to 3.5.x #1578 is a potential fix for that
• It might be useful to run this on cron as well (since trust root changes affect the results) ... but then this would need a file-an-issue job as well. Filed Improve testing on schedule #1583 to follow up

Example run: https://github.com/jku/sigstore-python/actions/runs/18432104146

Fixes #1573

[di]

Overall seems reasonable to me!

I don't like that the tested versions are hard coded... but I don't think there is much alternative here, testing all tags is silly. Still, this feels like it will get forgotten and then after a few releases this test is useless

We could make it parse all the tags but just test the latest from each of the major releases instead of all tags? I also think we could just hard-code the 1.x, 2.x, 3.x latest releases and test those against the latest tag (and only need to update this when we make a new major release).

I think we only want to test last patch release of every minor release that we still care about

Why not just every major release?

I would like to say we still care about v3.5.3 (released Nov 2024) but that release no longer handles current trusted root correctly and we did not patch that in 3.5.x

Should we patch it, at least for the sake of these tests?

[jku]

We could make it parse all the tags but just test the latest from each of the major releases instead of all tags? I also think we could just hard-code the 1.x, 2.x, 3.x latest releases and test those against the latest tag (and only need to update this when we make a new major release).

I think it's a little more nuanced:

• 1.x, 2.x are old: I don't think we should give the impression that we have the resources to support them
• 3.5.3 feels like an edge case -- only a year old but there is a 3.6.x series in the same major release... I think we could at least make a final release to get it included here if it's not too difficult (Backport trust root fixes to 3.5.x #1578)
• I don't see an issue testing multiple minor versions like 4.0 and 4.1 -- we can always drop the older one if it becomes problematic: the point is to know when we do become "incompatible"

I'll see if I can come up with something that doesn't require remembering to update the workflow, but still tests only a reasonable number of releases.

[jku]

I'll see if I can come up with something that doesn't require remembering to update the workflow, but still tests only a reasonable number of releases.

I think this is not worth the trouble:

• the workflow becomes much more complicated if the version list is built dynamically as the workflow matrix can't be used anymore
• while it's possible to build the version list with shell, it's not pretty: git tag | grep '^v[0-9]\+\.[0-9]\+\.[0-9]\+$' | awk -F'[.v]' '$2 > 3 || ($2 == 3 && $3 >= 5)' | sort -V -r | awk -F'[.v]' '!seen[$2"."$3]++'. So we'd likely want to do this with python instead... meaning more code to maintain

I think hard coding might be fine: it is after all the oldest versions (that we are adding now) that are likeliest to break when changes are done in future

[di]

That makes sense to me, thanks for looking into it!

[jku]

I'm marking this ready for review: If #1578 happens, we can add 3.5.3 to the test matrix