Architecture coverage test

[sofiia-chorna]

• collect coverage for the architectures, make them visible in codecov.io
• for threshold, use only utils & cli
• visual badge for coverage of the architectures in the doc

Please go the the Preview tab and select the appropriate PR template:

• Default template
• Adding a new architecture

---

📚 Documentation preview 📚: https://metatrain--803.org.readthedocs.build/en/803/

[sofiia-chorna]

seems to work
https://app.codecov.io/gh/metatensor/metatrain/tree/architecture-coverage-test/src%2Fmetatrain

[pfebrer]

Is it possible to do this in tox.ini instead of here in the workflow file?

[pfebrer]

Then it would generate the coverage also when running locally

[sofiia-chorna]

Hmm worth trying then, thanks for suggestion

[Luthaf]

seems to work [...]

The llpr/gap/nanopet parts of the code should also have tests, it is weird there is 0 coverage reported for these.

[pfebrer]

It's very nice to show this here. Right now they show as codecov: unknown, they will work once the PR is merged to main?

It would be better to add the badge automatically instead of manually in each documentation.py file. The string could be added here:

metatrain/docs/src/architectures/generate.py

Lines 220 to 222 in 91ee3e3

	docstring = (
	f".. _architecture-{template_variables['architecture']}:" + "\n\n" + docstring
	)

after f".. _architecture-{template_variables['architecture']}:" + "\n\n". You can try to do it and let me know if you have doubts :)

[sofiia-chorna]

it is unknown as it is not in main. here is the example of badge pointing to this branch:
https://codecov.io/gh/metatensor/metatrain/branch/architecture-coverage-test/graph/badge.svg?flag=coverage_nanopet

I wouldn't do it atomatically as it should be also set up in .codecov.yml

Let me know if you see better approach please

[pfebrer]

Hmm I see, but if we do it automatically and it is not set in .codecov.yml it will just show "Unknown" no? Seems fine.

Also I would then add the instruction to add your architecture in .codecov.yml here: https://github.com/metatensor/metatrain/blob/main/docs/src/dev-docs/new-architecture.rst

[sofiia-chorna]

Ah okay, seems fine indeed. Thanks

[Luthaf]

So I'm not 100% sure the way we partition the coverage between architectures is working, it looks like we would have to merge first to know about this.

Looking around codecov, this seems very relevant: https://docs.codecov.com/docs/components, should we use it instead of multiple projects?

[sofiia-chorna]

So I'm not 100% sure the way we partition the coverage between architectures is working, it looks like we would have to merge first to know about this.

Indeed, very good point. Checking the components

[sofiia-chorna]

Converting to draft to play with it (sorry, forking does not work with codecov)