Pipeline PR integration tests for FlowEHR & FlowEHR Data Pipeline repos

[jjgriff93]

Is your feature request related to a problem? Please describe.
Currently on PRs, we have a simple check that the configured pipelines run successfully. This doesn't give us any insight into whether the pipelines performed the expected transformations/produced the expected results. We're also unable to specify only specific pipelines for PR tests vs running all of the pipelines that are configured to be deployed once the PR is merged. Finally, there is no easy way to test pipelines in a pipeline repo (i.e. a data pot/seedling) on PR. These first have to be merged and then referenced by a FlowEHR deployment by SHA.

Describe the solution you'd like
Within a FlowEHR pipeline, it would be useful to be able to define an optional activity (or multiple) that are ran when a flag is enabled, which runs after the main pipeline(s) has ran, which can perform assertions and other tests to make sure the results of the pipeline(s) are as expected.

FlowEHR should then set a flag and run this activity on deployments and fail the deployment if the activity fails. It should, however, NOT run this activity on standard invocations of the pipeline(s) (i.e. when triggered by TumblingWindowTrigger Data Factory triggers or whatever triggers are configured within the pipelines).

Also for pipeline repositories themselves (in the data pot templates), it would be useful to have a mechanism for configuring a FlowEHR instance to target, then be able to test the pipeline(s) on PR by running them on the instance, and failing if the pipelines or the final testing activity fails, so that we can iron out any e2e pipeline bugs before merging and then setting up to deploy to FlowEHR.

fyi @tanya-borisova if you have anything to add