feat: ops.testing autoload support for charmcraft extensions

[tonyandrewmeyer]

When using a charmcraft extension (such as the 12-factor ones), the charm's charmcraft.yaml` has metadata, config, and actions added before being written out to the individual YAML files in the packed charm. Scenario doesn't understand this, so tests fail to make the right events available, and so on.

The PR adds a script that extracts the changes that each extension makes by running the charmcraft CLI tool, and saves that as a series of dictionaries in a private Scenario module. When Scenario autoloads, if there is an extension specified then it starts from the extension data and merges in the local data, simulating what charmcraft does when packing.

Changes to the extensions are infrequent, so we can start by manually regenerating the module when needed. If necessary, we could add a workflow that would automate this in the future.

Fixes #1699

[tonyandrewmeyer]

@james-garner-canonical , @dimaqq please see #1699 for some thoughts on how the issue could be solved. If you think that the one in this PR isn't the right one, let's have a team chat at daily about what we want to do. I feel like this is the right approach for the moment, and it's not locking us in too much (we could definitely import from charmcraft or some vendored charmcraft code without changing the API, we could definitely automate getting the data without changing the API, if we added a "run charmcraft expand-extensions" system we'd probably want to keep what's here as a fallback, etc).

[james-garner-canonical]

I think this is a nice approach to supporting Charmcraft extensions, and I agree with your decision to defer updating automatically until we need to.

I'm marking my review as 'request changes' since I'd like these specific concerns to be addressed before merging:

1. On the code generation side, I think we should drop all the manual formatting code since we run tox -e format at the end.
2. On the Scenario side, _apply_extensions should either mutate in place, or return a new dict, but not both.
3. Should it be an error (or at least a logging error or warning) if the extension defines a field and the charm overrides it with a different type (dict vs list)? How does Charmcraft handle this?

I've made a number of other comments and suggestions as I went, just putting into writing my thoughts as I reviewed. I think they'd improve readability going forward, but if you disagree then don't consider them blocking.

[tonyandrewmeyer]

3. Should it be an error (or at least a logging error or warning) if the extension defines a field and the charm overrides it with a different type (dict vs list)? How does Charmcraft handle this?

Charmcraft does have some validation that prevents some clashes. I'm not particularly keen on copying that into Scenario, and I think people will run into it at packing time, so it shouldn't really be an issue.

If you do something simple like redefine an config option then you get an error (not just in pack, but also e.g. in clean) that says that there are overlapping keys and you need to remove or remove that key. I think it'd be reasonable to make this an error in Scenario as well, even though you're going to quickly hit it elsewhere anyway.

[james-garner-canonical]

3. Should it be an error (or at least a logging error or warning) if the extension defines a field and the charm overrides it with a different type (dict vs list)? How does Charmcraft handle this?

Charmcraft does have some validation that prevents some clashes. I'm not particularly keen on copying that into Scenario, and I think people will run into it at packing time, so it shouldn't really be an issue.

If you do something simple like redefine an config option then you get an error (not just in pack, but also e.g. in clean) that says that there are overlapping keys and you need to remove or remove that key. I think it'd be reasonable to make this an error in Scenario as well, even though you're going to quickly hit it elsewhere anyway.

Agreed on not copying that logic. I'm happy with documenting that Scenario won't catch metadata that Charmcraft will consider invalid and/or adding a simple error case -- e.g. should the final else branch when we combine the metadata be an error?

[dimaqq]

.github/generate_charmcraft_extensions.py:205:5: F821 Undefined name `sys`
    |
204 | if __name__ == '__main__':
205 |     sys.exit(main())
    |     ^^^ F821
    |

Found 1 error.

[james-garner-canonical]

Looks good!