[DO NOT MERGE] Try out allure for test reporting

[MatMoore]

Description

Previously we enabled test reporting using dorny/test-reporter. This updates the job summary of the Github actions job. However, this is not shown to users who are not logged into github, which defeats the point, as we wanted this to make the test results visible outside of the team.

This branch uses Allure - a tool that generates HTML reports, and uploads to Github Pages.
This tool supports tests written in multiple languages including python and javascript.

The resulting report is available at https://nhsdigital.github.io/dtos-manage-breast-screening/

To keep things simple, I've restricted this to the UI tests only, since (I assume) granular reporting of unit tests is not useful to anyone who is not a developer on this team. We also have Sonar which can report on test coverage.

The reporting is currently triggered by the CI job, but in practice we could run it from either the main CI/CD, or from a manually triggered workflow (in the latter case we could provide additional metadata identifying a logical "release" we are about to do, e.g. switching on a feature flag)

Running allure

• allure run <test command>
  • the test command uses the appropriate plugin for the test framework to generate metadata for allure to process
  • it's possible to split this into multiple stages if you want to combine test results from multiple commands
• allure generate is another way to generate the report, assuming you have previously run the test command
• allure open opens the report

Linking to Jira tickets

I've annotated the tests with links to Jira tickets. This only shows up when you drill down into the test though.

In order to make the Jira ticket visible at the top level, I've included it in the "epic" name.

Issues with deploying to github pages from github actions

• This model assumes we have one website always showing the latest results. When the CI/CD pipeline runs again, it will overwrite the report.
• It is possible to preserve history of prior runs. If you have this, then at an individual test level you can see whether it passed/failed on prior runs, and it can determine if your test is flaky or not.
  • However, even if we enable history, the report is still showing the current state, rather than allowing you to see historic reports or multiple reports at once.
  • If we wanted to be able to link to a report that is associated with a specific tag or commit, I expect we would have to build a static site where the reports are saved to subdirectories and never deleted.
    • We could possibly use Azure Storage for this
  • If we wanted to split a report into multiple reports we'd have to hardcode them in the config and apply filters to each one

Alternatives to hosting a static site

The output above is what we get using the default "awesome" plugin.

There is also a jira plugin which I haven't tested, that would upload the reports directly to Jira.

Other configuration options

Firstly, we can add arbitrary metadata to improve readability.

The hierarchy of the tests is also customisable - I've gone with epic / feature / story.

We can attach screenshots and files to tests. This could potentially include playwright traces as well. See the allure playwright guide for more info.

There is also a configuration file, currently allurerc.json. The default 'awesome' plugin has these options.

Note: I noticed when I installed allure globally, it didn't pick up my configuration, but after I installed it to node_modules it does.

Jira link

Review notes

TODO:

• Share with team for general feedback
• Annotate all UI tests with epics/features
• Fix linter errors

Review checklist

• Check database queries are correctly scoped to current_provider

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud