diff --git a/docs/advanced-test-features/index.md b/docs/advanced-test-features/index.md index 97ce23c..66505bc 100644 --- a/docs/advanced-test-features/index.md +++ b/docs/advanced-test-features/index.md @@ -51,4 +51,8 @@ Inferno Core supports displaying a custom banner for your Test Kit. This banner The Inferno DSL supports associating discrete requirements with suites, test groups, and tests. Test developers can track which requirements Inferno is and is not verifying and Inferno displays these associations to users within the -[user interface](/docs/user-interface.html#viewing-verified-requirements). \ No newline at end of file +[user interface](/docs/user-interface.html#viewing-verified-requirements). + +## [Scripting Suite Execution](/docs/advanced-test-features/scripting-execution.html) +The Inferno CLI supports executing suites using a yaml configuration format +instead of the UI using the `inferno execute_script` command. \ No newline at end of file diff --git a/docs/advanced-test-features/scripting-execution.md b/docs/advanced-test-features/scripting-execution.md new file mode 100644 index 0000000..7d8b13a --- /dev/null +++ b/docs/advanced-test-features/scripting-execution.md @@ -0,0 +1,438 @@ +--- +title: Scripting Suite Execution +nav_order: 9 +parent: Advanced Features +layout: docs +section: docs +--- +# Scripting Suite Execution + +## Overview + +Inferno supports the scripting of test suite execution through the `inferno execute_script` +[CLI command](/docs/getting-started/inferno-cli#complex-scripted-execution). This command takes as input a +[configuration file](#creating-script-configuration-files) that tells Inferno when and how to perform +execution steps both inside and outside of Inferno. The results of the session are compared +against the results of a known-good execution to identify any regressions. The script +can be executed against the local Inferno instance or a specific remote instance. These scripts +can be built into [CI/CD Pipelines](/docs/ci-cd-usage#using-the-inferno-execute_script-cli-for-complex-orchestration). + +## Execution + +Three standard options for executing scripts include: +- Executing a specific script using the `execute_script` CLI: See the [CLI documentation](/docs/getting-started/inferno-cli#complex-scripted-execution) + for details on the command. +- Executing scripts defined in the test kit's `execution_scripts` directory using + the `bundle exec rake execute_scripts:run_all` command. Two environment variables may be set in addition: + - `INFERNO_BASE_URL=`: set to run the scripts against a specific Inferno host. + If not provided, then the local Inferno host will be used. + - `FILTER=execution_scripts//*.yaml`: provide a different filter to use to identify + the scripts to execute. If not provided, the default is `execution_scripts/**/*.yaml`. +- Executing all scripts defined in the test kit's `execution_scripts` directory within GitHub, + either in response to a commit or a manual trigger. See + [CI / CD Usage](/docs/ci-cd-usage#using-the-inferno-execute_script-cli-for-complex-orchestration) + for additional details. + +### Execution Process and Output + +When a script is executed, the following phases are performed: +1. Sessions are created +2. Steps are executed +3. Results are checked + +During execution, Inferno will print to the terminal details on its polling, matching, and actions during +execution and provide a summary of the results comparison performed. + +Inferno will use exit code 3 when an error is encountered. Error cases include the following, +separated into cases where a detailed results comparison is performed and those +where it is not: +- Reults comparison performed + - Compared results for one or more sessions did not match. + - Script execution reached an unmatched stable state (no action specified for a `done` or `waiting` state). +- No results comparison performed + - The exepected results file for one or more sessions did not exist. + - Script execution had to be interrupted due to a timeout. + - The results contained an `error` result, indicating a problem in the test logic + +Otherwise, the CLI command will end with exit code 0 indicating success. + +#### Session Creation + +Test suite sessions for the execution are detailed in the [`sessions`](#sessions) section of the script +configuration file. Creation is performed using the +[`inferno session create` CLI](/docs/getting-started/inferno-cli#manage-sessions). + +#### Step Execution + +The steps taken during execution are detailed in the [`steps`](#steps) section of the script +configuration file. + +Inferno checks the status of sessions for next step polling using the [`inferno session status` CLI](/docs/getting-started/inferno-cli#manage-sessions). It may also use the [`inferno session cancel` CLI](/docs/getting-started/inferno-cli#manage-sessions) if it needs to cancel a session. + +To start runs, Inferno uses the [`inferno session start_run` CLI](/docs/getting-started/inferno-cli#manage-sessions). + +#### Check Results + +For each session in the script, Inferno will compare the results in the expected results +file (see [Expected Results File and Alternates](#expected-results-file-and-alternates)) +to the results of the completed session execution. Configuration of the comparison is +detailed in the [`comparison_config`](#result-comparison-config) section of the script +configuration file. + +There is no requirement that the tests must pass for the results check to succeed. +The exepected results file can indicate that some or all entries have a failing result +(e.g., `fail` or `skip`) with particular details in their `result_message` and `messages` +fields. This allows scripts that verify that particular tests fail under certain conditions +and indicate this with expected messages. However, if any tests end with a `error` result, +indicating a problem in the test execution logic, the script execution will fail and result +comparison will be skipped. + +Comparison is performed using the [`inferno session compare` CLI](/docs/getting-started/inferno-cli#manage-sessions) +using the `-f` flag to specify the expected file. Comparison involves the following steps: +- normalize the expected and actual results using the configured [`normalized_strings`](#normalized-strings) +- match individual result entries using the runnable id (`test_id`, `test_group_id`, or `test_suite_id`) +- compare the matched result entries, looking at the status (e.g., pass, fail), the result message, and the + individual messages within the result + +When the actual and expected results do not match, two files will be written in the same location as the expected file: +- actual results: a file with the results json (prior to normalization) +- comparison analysis: a csv file with details of results that differed, including normalized details +These files are named using the same prefix as the executed file and include a timestamp for the execution so that +they will be unique for each script execution. + +When the target expected results file does not exist it will be created using the output of the +[`inferno session results` CLI](/docs/getting-started/inferno-cli#manage-sessions). The `execute_script` +CLI will exit with an error status in this case. After creation of a expected results +file in this way, users should review the results using the file and/or the Inferno +UI to make sure that they are the expected results as subsequent script executions +will perform the comparison using that file as the expected results. + +## Creating Script Configuration Files + +At a high-level, the script configuration file contains instructions for +- The session(s) to create, including any suite options and presets to use +- The steps to take to execute the script, each including an action, when to perform it, + and which session will act next if there are multiple Inferno sessions involved. +- Normalizations of the results to make sure that executions are comparable across runs + and Inferno instances. + +The easiest way to think about creating a script configuration file is that it mirrors +the manual steps taken in the UI to execute an Inferno test session of a particular suite +against a specific system. Go through the manual execution steps and note each time there +is an interaction with the Inferno UI or a state in Inferno that triggers an interaction +with the tested system: each of these will become a step within the script. + +The rest of this section provides details on how to turn the manual execution into a script file. +See the documentation at the top of the [`execute_script.rb`](https://github.com/inferno-framework/inferno-core/blob/main/lib/inferno/apps/cli/execute_script.rb) +file for a complete format reference. + +### Sessions + +Each script configuration file will indicate a list (sequence) of one or more Inferno sessions to create under +the `sessions:` top-level key. Each sequence includes +- **Suite** (`suite:`): *Required* - the internal id for the suite or the title selected in the UI. +- **Name** (`name:`): *Conditional* - a short key used to identify the session in steps. + Required when multiple sessions are defined, otherwise optional. +- **Preset** (`preset:`): *Optional* - the internal id or title of a preset to use for the session. +- **Suite Options** (`suite_options:`): *Optional* - pairs of `key: value` entries each corresponding to a suite + option. The key can be either the internal id or the title from the UI. The value can be either + the internal value or the label from the UI. + +### Result Comparison Config + +After completing the script, the results of the executed session will be compared for each +session to the expected results recorded in the session's expected results file. Because +direct comparison of results from different runs across different systems (e.g., local vs +deployed) will not always be directly comparable, two mechanisms are provided to tweak +the actual and/or expected results to facilitate like-to-like comparison. + +All configurations around results comparison are nested under the `comparison_config:` top-level +key. + +#### Normalized Strings + +Before comparing the actual results to the expected results, Inferno will replace strings +indicated in the `normalized_strings:` key. This prevents these values, which are expected +to be different between runs and across different Inferno hosts, from triggering an execution +failure. Each entry in the sequence can be +- A raw string: the exact string and its URL-encoded form are replaced with the string `` +- A raw regex: strings matching the regex are replaced with the string `` +- An object with two keys that provides specific replacement strings (useful for debugging): + - `patterns:` a sequence of strings or regexes used to identify the strings to replace, + which behave like the raw forms. (for single-entry lists, the `pattern:` can be used + instead with the single value) + - `replacement:` the string that replaces the patterns (e.g., ``) + +For example, the following configuration normalizes, the inferno host, the reference server +url, pkce code challenges, and UUIDs, each with a specific replacement string +for ease of debugging. + +``` +comparison_config: + normalized_strings: + - replacement: + patterns: + - http://localhost:4567/inferno # local inferno core ruby + - http://localhost:4567 # local ruby + - http://localhost # local docker + - https://inferno.healthit.gov/suites # prod + - https://inferno-qa.healthit.gov/suites # qa + - replacement: + patterns: + - https://inferno.healthit.gov/reference-server # prod reference server + - https://inferno-qa.healthit.gov/reference-server # qa reference server + - replacement: code_challenge= + pattern: /code_challenge=[A-Za-z0-9+\/=_-]{43}/ + - replacement: + pattern: /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/i +``` + +#### Expected Results File and Alternates + +When comparing results, Inferno first identifies the file that contains the expected +results. These are formatted as the output of the `session results` CLI command. Note +that Inferno will generate the target file the first time a script is run. + +By default, the expected results file will be `_expected.json`. For +example, for a script configuration file named `g10.yaml`, the default expected file +would be `g10_expected.json`. A different file to use as the default can be +specified in the `expected_results_file:` key. Inferno will interpret relative paths +as relative to the script configuration file's directory. + +Some suites will expected differences between session results that go beyond string +normalizations. For example, suites that verify a server's TLS setup will fail +when evaluating a local server without TLS setup, which may be the expected result +when running in a CI/CD environment. For this situation, an alternate expected +results file can be indicated for certain executions. These are indicated under +the `alternate_expected_files:` key which will contain a list (sequence) of entries +each with the following keys: +- **Alternate File** (`file:`): *Required* - the file containing the expected results, relative + to the script configuration file's directory. +- **Condition** (`when:`): *Required* - a list of conditions which must all match. Each condition + contains keys for + - **Target field** (`field:`): *Required* - the value to check. Can be one of + - `inputs.` to check the value of an input used during the session. + - `configuration_messages` to check the `messages` field of entries (one must match) + in `test_suite.configuration_messages`. + - `inferno_base_url` to check the Inferno host. + - **Match Criteria** (`matches:` or `not_matches`): *Required* - a regex to use to check the value(s) + either match or don't match. + +For example, the following configuration tells Inferno to use an alternate expected file +the expects terminology failures when +1. It isn't running on a public Inferno host, and +2. There is a configuration message indicating that there is a terminology build problem. + +``` +comparison_config: + expected_results_file: g10_results_expected.json + alternate_expected_files: + - file: g10_no_terminology_expected.json + when: + - field: inferno_base_url + not_matches: inferno(-qa)?\.healthit\.gov + - field: configuration_messages + matches: ^There is a problem with the terminology resources +``` + +If there are multiple matching `alternate_expected_file:` entries, the one listed first will be used. + +When there are multiple sessions defined, both the `expected_results_file:` and `alternate_expected_files:` +keys must be nested under a `sessions:` object with keys corresponding to the session name. If the +above example was in a multi-session script associated with the session named `g10`, it would +look like this: + +``` +comparison_config: + sessions: + g10: + expected_results_file: g10_results_expected.json + alternate_expected_files: + - file: g10_no_terminology_expected.json + when: + - field: inferno_base_url + not_matches: inferno(-qa)?\.healthit\.gov + - field: configuration_messages + matches: ^There is a problem with the terminology resources +``` + +Note that the `normalized_strings:` key never appears under the `sessions:` key. + +### Steps + +Each script configuration file will indicate a list (sequence) of steps that are taken by the script under +the `steps:` top-level key. There will be at least 2 in every script: +- an initial `created` state that will be the first step executed. +- a final `END_SCRIPT` action that will be the last step executed. + +While steps are executed when they are matched by Inferno's state, only one step is executed at a time +and loops are detected and not allowed. By convention, steps should be listed in the order they will +be executed during script execution. + +Each step consists of keys pertaining to the following details +- When to take the step +- What action to take +- How to look for the next step + +The following subsections detail the keys related to each of these areas + +#### When to take the step + +Each step indicates when it will be take through three keys +- `status:` *Required* - specifies the status to match with options: + - `done` when a test run has completed + - `waiting` when a test run is in progress but waiting for external input + - `created` when a test session has been created, but no runs started yet +- `last_completed:` *Conditional* - specifies a runnable in the form of an + internal id or short numeric id from the UI. + - for `done`, will be the test, group, or suite that was executed when the run was started + - for `waiting`, will be the test that initiated the wait + - for `created`, will be omitted (nothing executed yet) +- `session:` *Conditional* - when multiple sessions, the name (not suite id) of the + session to match on, otherwise omitted. + +Additionally, the optional `state_description:` key can be used to provide documentation +describing the state. This has no functional effect, but is echoed during script execution +for debugging purposes. + +Note that Inferno can be have other status (e.g., `running` and `cancelling`), but these +do not represent stable states where Inferno is waiting for an action, so they will never +be used in scripts. + +Conversely, Inferno needs to know what to do in any stable state that is reached. If no +step matches a `done` or `created` state that Inferno reaches, then the script will end +with an error (results not checked). If no step matches a `waiting` state, then Inferno +will cancel the current run (ending the wait) and attempt to continue with the next +matching `done` state. + +#### What action to take + +There are three kinds of actions that can be take by a step each specified by a different +key. Only one can be defined for each step: +- `start_run:` Used to execute an Inferno test, group, or suite on one of the sessions + started by this script. The following sub keys are used. + - `runnable:` *Required* - the test, group, or suite to execute, specified as a short ID + from the UI or an internal id. + - `inputs:` *Optional* - key-value pairs indicating inputs to be merged into those already + stored in the session (from the preset or previous run inputs or outputs). Each key must + be the internal name for the input (from the UI, use the yaml or json view of the inputs + to find the internal name). Values can be + - *a single value*: used as the literal input value; or + - *an object or sequence*: the json representation will be used as the input value; or + - *a file path prefixed with `@`*: the contents of the file will be used as the input value. + Relative paths are resolved from the directory containing this script file. + - `session:` *Conditional* - when multiple sessions are defined the name (not the suite) + of the session on which the run will be executed, otherwise omitted. +- `command:` Specifies a shell command to execute. For example, a `curl` command to navigate + to an attestation URL or an external script that automates manipulation of the tested system. + See [these considerations](#defining-executing-and-securing-complex-commands) + for creating, executing, and securing scripts with complex commands. +- `action:` Script-specific actions to bridge between steps. Defined actions include: + - `END_SCRIPT` used on the distinguished final step. The execution will end successfully + after this step. + - `NOOP` used on steps that don't require an action, but indicate that another step is + unblocked. See details in the [discussion of multiple session scripts](#scripts-with-multiple-sessions) + below. + - `WAIT` used on transitory `waiting` states that are expected to end without external + action. Note that unlike other actions, this one does not cause Inferno to break the + polling loop, so it can be matched multiple times (not a loop) but does not change + next step details (see the details on [finding the next step](#how-to-look-for-the-next-step) + below). + +The specifics of `start_run` and `command` actions may depend on dynamic session details. +The following template tokens are allowed in fields of these actions which will be replaced +with session-specific values at execution time. For most, there is both a named version for +scripts with multiple sessions and a raw version for those with only a single session: +- **Session Id**: `{session_id}` (`{.session_id}` when multiple sessions) +- **Result Message** (from waiting tests only): `{result_message}` + (`{.result_message}` when multiple sessions) +- **Output Value** (from waiting tests only): `{wait_outputs.KEY}` + (`{.wait_outputs.KEY}` when multiple sessions) +- **Inferno URL**: `{inferno_base_url}` (no session-specific version - always the same for all sessions) + +Additionally, the optional `action_description:` key can be used to provide documentation +describing the action. This has no functional effect, but is echoed during script execution +for debugging purposes. + +#### How to look for the next step + +After taking an action, script execution returns to a polling mode where it checks the status +of a session to determine if it has reached a stable state to match against to identify the +next step. The step that was just executed can control two aspects of this process using +the following keys: +- `timeout:` *Optional* - specifies a number of sections to wait before aborting by canceling + the run. The default is 120 seconds. After cancelling, Inferno will try to continue matching + other states, but will still ultimately indicate failure for the script as a whole. +- `next_poll_session:` *Optional* - the name (not suite id) of the session to poll for the next + step. Inferno only ever polls one session at a time, so in scripts with multiple sessions, + it must be told which session to poll next. The first session configured is used to start. + After executing a step, Inferno will either poll the session indicated in this key or the + session polled after the previous step if none is specified. + +## Scripts with Multiple Sessions + +Scripts with multiple sessions involve additional complexity but are useful for scripting the execution of +Inferno suites against each other. + +Considerations to keep in mind when creating script configuration files with multiple sessions: +- Include the `name:` key for each session for use as a key to refer to the session in the steps. +- The biggest change in that steps are specific to sessions and Inferno needs to know which session + to poll next after each session. This means that steps + - Will always declare the name of the session they match in the `session:` key. + - Will sometimes declare the name of the next session to poll in the `next_poll_session:` key if + diferent from the session of the matched step. This corresponds to steps in the manual execution + where the tester must change over to the tab of the another session's UI. +- Templates for dynamic values in executed actions (commands and Inferno runs) will use the session-specific + form. +- `NOOP` actions can be used to help clarify sequencing. For example, if two + sessions both have long running tests that interact but complete in a particular + order due to wait interactions and don't require any external interactions. You + could poll the session that will end last and look only for the `done` state, + which would require `WAIT` actions on any `waiting` states so that Inferno + recognizes them, or you could define states for each `waiting` state with `NOOP` + actions to make clear how the active execution passes back and forth. + +## Defining, Executing, and Securing Complex Commands + +The `command` action option allows execution of arbitrary logic as a part of running a script. +The executed command can be as simple as a http request to trigger the end of an Inferno wait +dialog to a complex scripted interaction with the tested system, for example visiting and +manipulating the UI of the Inferno FHIR Reference Server using Selenium. The following +are important considerations when using this feature: +- The executed logic must be available to the local Inferno instance. If distributed with a test + kit for use in CI/CD pipelines, this will mean including external scripts in the test kit. For + additional gems required only for scripts, it is recommended to add the additional gems either + - As a development dependency in the gemspec, or + - Within the `:test` and `:development` sections of the Gemfile and handle `LoadError` exception + explicitly to warn users that use the script that they need to add additional gem dependencies. +- Scripts that use `command` actions must be executed with the `--allow-commands` option of the + `execute_script` CLI enabled. This guard forces users to opt-in to arbitrary commands. +- Be careful executing scripts that have commands which you have not reviewed. + +## Example Scripts + +The following Test Kits include scripts that can be executed. To execute any of them +1. Checkout the repository locally and set it up to be run in developer mode +2. Either + - Run all the defined scripts under the by executing `bundle exec rake execute_scripts:run_all`, or + - Locate the a single target script configuration file under the `execution_scripts` directory + and run `bundle exec inferno execute_script execution_scripts/path/to/script_config_file.yaml` + pointing to the desired configuration file. + +- [**Inferno Template**](https://github.com/inferno-framework/inferno-template): + New Inferno Test Kits created from the [inferno-template repository](https://github.com/inferno-framework/inferno-template) + or the [`inferno new` CLI command](/docs/getting-started/inferno-cli.html#creating-a-new-test-kit) + include a [simple execution configuration file](https://github.com/inferno-framework/inferno-template/blob/main/execution_scripts/inferno_template_script.yaml) + for the example test suite included in the template. +- [**Inferno Core**](https://github.com/inferno-framework/inferno-core): Inferno Core contains + [execution scripts](https://github.com/inferno-framework/inferno-core/tree/main/execution_scripts) + for [sample development suites](https://github.com/inferno-framework/inferno-core/tree/main/dev_suites) + that demonstrate features of Inferno and the `execute_script` CLI. +- [**SMART App Launch**](https://github.com/inferno-framework/smart-app-launch-test-kit): + The SMART App Launch test kit includes both client and server tests and includes [execution scripts](https://github.com/inferno-framework/smart-app-launch-test-kit/tree/main/execution_scripts) + demonstrating how multi-session scripts can be used to execute these tests against each other. +- [**(g)(10) Certification**](https://github.com/onc-healthit/onc-certification-g10-test-kit): + The (g)(10) Certification test kit (and [US Core Core test kit](https://github.com/inferno-framework/us-core-test-kit/tree/main/execution_scripts)) + contains [scripts](https://github.com/onc-healthit/onc-certification-g10-test-kit/tree/main/execution_scripts) + that demonstrate one approach to scripting external interactions with tested systems, in this + case the Inferno Reference Server. \ No newline at end of file diff --git a/docs/ci-cd-usage.md b/docs/ci-cd-usage.md index b07999e..ed73300 100644 --- a/docs/ci-cd-usage.md +++ b/docs/ci-cd-usage.md @@ -7,11 +7,39 @@ section: docs # Using Test Kits in CI/CD -Since [inferno -execute](/docs/getting-started/inferno-cli.html#running-a-test-kit-in-command-line) -enables running Inferno tests in CLI, it may be used within a Continuous -Integration or Continuous Deployment (CI/CD) pipeline. The generic steps around -automatically running a Test Kit are: +Inferno provides two CLI options for executing Inferno tests via the command line: +1. The [`inferno execute_script`](/docs/getting-started/inferno-cli.html#complex-scripted-execution) CLI command for more complex executions that can involve multiple runs + and sessions as well as scripting of external systems. +2. The [`inferno execute`](/docs/getting-started/inferno-cli.html#simple-execution) CLI command for execution of a single suite, group, or test. + +## Using the `inferno execute_script` CLI for Complex Orchestration + +The [`inferno execute_script`](/docs/getting-started/inferno-cli.html#complex-scripted-execution) +command is well-suited for use in CI/CD pipelines. To enable this use, +the [Inferno Template Test Kit](https://github.com/inferno-framework/inferno-template) provides +a [standard Github Workflow file](https://github.com/inferno-framework/inferno-template/blob/main/.github/workflows/run_inferno_execution_scripts.yml) +that finds [script configuration files](/docs/advanced-test-features/scripting-execution.html#creating-script-configuration-files) +defined under the `execution_scripts` directory and runs each of them against a freshly +created Inferno instance running the test kit defined in the repository. This script is +trigged automatically on pull requests opened against the main branch and can also be +manually triggered. + +Key considerations when using this GitHub workflow: +- [Script configuration files](/docs/advanced-test-features/scripting-execution.html#creating-script-configuration-files) + defined anywhere under the `execution_scripts` directory will be identified and executed. +- All executed scripts are expected to complete successfully with the expected results. +- Scripts that end in `_with_commands.yaml` will be run with the `--allow-commands` flag of the + `execute_script` CLI set. Use that naming convention whenever your scripts include `command` + actions. +- Any `command` action scripts must be runnable on a `ubuntu-latest` using the contents of the + repository with Inferno's standard Ruby setup. You can modify the workflow if your commands + require additional setup to execute. + +## Using the `inferno execute` CLI for Simple Execution + +To run Inferno using the [inferno +execute](/docs/getting-started/inferno-cli.html#simple-execution) +CLI command, follow these steps: 1. Setup a container or environment with Ruby and Docker 2. Obtain your target Test Kit version and make it your working directory @@ -21,7 +49,7 @@ command works on top of docker compose and daemonizes the processes for you. 5. Launch the system under test and confirm that it is ready to receive requests. 6. Select the Tests, Test Groups, and Test Suites you want in CI and run them with `inferno execute`. See -[Running a Test Kit in Command Line](/docs/getting-started/inferno-cli.html#running-a-test-kit-in-command-line) +[Running a Test Kit in Command Line](/docs/getting-started/inferno-cli.html#simple-execution) for more information. The command will exit with status 0 if the Inferno Summarizer deems it a pass. + You can run multiple inferno execute commands, but remember that each call represents @@ -31,6 +59,5 @@ its own test session. Test Suite to pass. 7. Clean up. You can stop Inferno services with `bundle exec inferno services stop`. -If you want to use GitHub Actions for your CI/CD pipeline, or run the -ONC Certification (g)(10) Test Kit in CI/CD, you can view our [workflow file](https://github.com/inferno-framework/inferno-reference-server/blob/5e0d06ad5414efa93499fd3de093e29cf5e6d9d1/.github/workflows/inferno_ci.yml) -on the Inferno Reference Server. +To see an example of these steps, see [this Github workflow file](https://github.com/inferno-framework/inferno-reference-server/blob/5e0d06ad5414efa93499fd3de093e29cf5e6d9d1/.github/workflows/inferno_ci.yml) +which is used to test the Inferno Reference Server against the (g)(10) certification test kit. diff --git a/docs/getting-started-users.md b/docs/getting-started-users.md index 4a26db0..a556a35 100644 --- a/docs/getting-started-users.md +++ b/docs/getting-started-users.md @@ -8,7 +8,9 @@ layout: docs Start here if you're interested in testing a FHIR server against one or more existing Test Kits. -## Running an Existing Test Kit +## Running Test Kits + +### Running an Existing Test Kit Most Test Kits are developed using the [Inferno Template](https://github.com/inferno-framework/inferno-template) repository, which provides scripts for standing up an instance of Inferno to run a selected Test Kit. Running a Test Kit usually follows the below steps: @@ -32,7 +34,7 @@ cd us-core-test-kit Always check the documentation for an individual Test Kit, however, since there may be additional installation steps. -## Running Multiple Test Kits +### Running Multiple Test Kits There may be times when you wish to run multiple Test Kits in a single Inferno instance. You can load and run two or more separate Test Kits by using [Inferno Template](https://github.com/inferno-framework/inferno-template). @@ -86,3 +88,13 @@ cd inferno_template ``` _Note: Example Test Suites, Groups, Tests and IGs in the template can be removed._ + +## Executing Tests + +Suites hosted by a running test kit can be executed: +1. Interactively, using the web-based [Inferno Testing Interface UI](/docs/user-interface) +2. Automatically, using [scripted execution](/docs/advanced-test-features/scripting-execution). + +Advanced users can also use the lower-level [`session` CLI](/docs/getting-started/inferno-cli#manage-sessions) +or the [JSON API](https://inferno-framework.github.io/inferno-core/api-docs/) to create and interact +with Inferno sessions. \ No newline at end of file diff --git a/docs/getting-started/inferno-cli.md b/docs/getting-started/inferno-cli.md index 13dfe16..f9b6fe3 100644 --- a/docs/getting-started/inferno-cli.md +++ b/docs/getting-started/inferno-cli.md @@ -21,17 +21,22 @@ The commands available include: | Command | Description | |--------------|-------------| | `inferno console` | Starts an interactive console session with Inferno. More information can be found on the [Debugging Page](debugging.html#interactive-console). | +| `inferno execute` | Execute tests in the command line. | +| `inferno execute_script` | Execute tests in the command line instead of web UI. | +| `inferno evaluate IG_PATH [--options]` | Run the FHIR evaluator in the command line. Does not require `bundle exec`. | | `inferno help [COMMAND]` | Describes the available commands, or one specific command if specified. | | `inferno migrate` | Runs database migrations. | | `inferno new TEST_KIT_NAME` | Create a new test kit. Run `inferno new --help` for additional options. Does not require `bundle exec`. | +| `inferno requirements SUBCOMMAND [...ARGS]` | Manipulates and checks Inferno requirements. The available subcommands are `export_csv`, `check`, `coverage`, and `check_coverage`. See the [requirements page](/docs/advanced-test-features/requirements) for details of these subcommands. | +| `inferno requirements help SUBCOMMAND` | View details on the requirements subcommands. | | `inferno services start` | Starts the background services (nginx, Redis, etc.) for Inferno. | | `inferno services stop` | Stops the background services for Inferno. | +| `inferno session SUBCOMMAND [...ARGS]` | Manages Inferno test sessions. The available subcommands are `create`, `start_run`, `status`, `cancel_run`, `data`, `results`, and `compare`. | +| `inferno session help SUBCOMMAND` | View details on the session subcommands. | | `inferno start` | Starts Inferno web UI. Do not use `bundle exec` if the Inferno Core version is prior to 0.4.39. | | `inferno suite SUBCOMMAND [...ARGS]` | Performs suite-based operations. The available subcommands are `describe`, `help`, `input_template`, and `lock_short_ids`.| | `inferno suite help SUBCOMMAND` | View details on the suite subcommands. | | `inferno suites` | Lists available test suites. | -| `inferno execute` | Execute tests in the command line instead of web UI. | -| `inferno evaluate IG_PATH [--options]` | Run the FHIR evaluator in the command line. Does not require `bundle exec`. | | `inferno version` | Outputs the version of Inferno Core (not the Test Kit). Does not require `bundle exec`. | {: .grid.command-table} @@ -65,11 +70,192 @@ It will create a `my_test_kit.gemspec` file, and set the names of any authors su using the `-a` option, and set the name, summary and description values to your Test Kit name. You should manually inspect this file if you want to [publish or distribute the Test Kit](/docs/distributing-tests.html). - The `--implementation-guide` or `-i` option will look at an absolute file path if its prefixed by `file:///` or lookup the name in the [official FHIR package registry](https://packages.fhir.org/). +## Manage Sessions + +The `inferno session` command provides a programmatic interface for creating and managing +Inferno test sessions from the command line. It is useful for automating test execution, +especially in CI/CD pipelines. See the [CI/CD Usage](docs/ci-cd-usage) page for additional +information and tools for the CI/CD use case. + +All commands leverage the [JSON web API](https://inferno-framework.github.io/inferno-core/api-docs/) +to interact with Inferno. By default, these commands interact with the +local running Inferno instance. Additionally, all session subcommands +support the `--inferno_base_url` (`-I`) option to specify the URL of a remote running Inferno +instance to interact with instead. All commands write JSON to stdout and exit `0` on success or +`3` on error (`1` and `2` are reserved for Thor and Bash respectively). + +Subcommands include: + +- **`create SUITE_ID`** + + Create a new test session for the given suite. + + **Options**: + - `-I`, `--inferno_base_url`: URL of the target Inferno service. + - `-o`, `--suite_options`: Suite options as `key:value` pairs (e.g., + `--suite_options us_core_version:us_core_6 smart_app_launch_version:smart_app_launch_2`). + Keys and values can use either the display name seen in the Inferno UI or the internal name. + - `-p`, `--preset`: Apply a named preset when creating the session. Both the title or the id + can be used to identity the target preset. + + **Output**: JSON representation of the created session, including the session `id` used by all subsequent subcommands. + See the output of the [`POST /test_sessions`](https://inferno-framework.github.io/inferno-core/api-docs/#/Test%20Session/post_test_sessions) API for details on the output. + + ```sh + bundle exec inferno session create my_test_suite -I https://inferno.healthit.gov/suites -p my_preset + ``` + +- **`start_run SESSION_ID RUNNABLE_ID`** + + Initiate a test run on an existing session. The `RUNNABLE_ID` is + a short or internal ID of a specific group or test to run, or `suite` + to run the entire suite. + + **Options**: + - `-I`, `--inferno_base_url`: URL of the target Inferno service. + - `-i`, `--inputs`: Input values as `key:value` pairs (e.g., + `--inputs url:https://example.com patient_id:123`). These merge with and override + the session's previously stored inputs. The internal name for the input must be used + as the key. This can be found from the [Inferno testing interface UI](docs/user-interface#main-test-interface) by looking at the json or yaml representation of + the inputs. + + **Output**: JSON representation of the created test run. See the output of the [`POST /test_runs`](https://inferno-framework.github.io/inferno-core/api-docs/#/Test%20Run/post_test_runs) API for details on the output. + + ```sh + bundle exec inferno session start_run 7uLiz9qX4og my_group -I https://inferno.healthit.gov/suites \ + -i url:https://example.com patient_id:123 + ``` + +- **`status SESSION_ID`** + + Get the current run status of a session. + + **Options**: + - `-I`, `--inferno_base_url`: URL of the target Inferno service. + + **Output**: JSON including a `status` field (`created`, `queued`, `running`, `waiting`, `done`, + etc.) and the ID of the `last_test_executed`. See the output of the + [`GET /test_runs/{test_run_id}`](https://inferno-framework.github.io/inferno-core/api-docs/#/Test%20Run/get_test_runs__test_run_id_) + API for details on the output. Additionally, When `status` is `waiting`, fields + `wait_outputs` and `wait_result_message` are added to this base object using the + `outputs` and `result_message` fields from the result of the waiting test (last test that was executed) + which can be used to script the continuation of the session. + + ```sh + bundle exec inferno session status 7uLiz9qX4og -I https://inferno.healthit.gov/suites + ``` + +- **`cancel_run SESSION_ID`** + + Cancel the currently active run for a session. Applies when the run is in `queued`, + `running`, or `waiting` status. + + **Options**: + - `-I`, `--inferno_base_url`: URL of the target Inferno service. + + **Output**: JSON with `run_id` and `cancelled: true`, or an error object if no cancellable + run is active. + + ```sh + bundle exec inferno session cancel_run 7uLiz9qX4og -I https://inferno.healthit.gov/suites + ``` + +- **`data SESSION_ID`** + + Get the input values currently stored for a session. + + **Options**: + - `-I`, `--inferno_base_url`: URL of the target Inferno service. + + **Output**: JSON array of objects with `name` and `value` keys representing the session's current inputs. + See the output of the [`GET /test_sessions/{test_session_id}/session_data`](https://inferno-framework.github.io/inferno-core/api-docs/#/Session%20Data/get_test_sessions__test_session_id__session_data) + API for details on the output. + + ```sh + bundle exec inferno session data 7uLiz9qX4og -I https://inferno.healthit.gov/suites + ``` + +- **`results SESSION_ID`** + + Get the test results for a session. + + **Options**: + - `-I`, `--inferno_base_url`: URL of the target Inferno service. + + **Output**: JSON array of result objects for each test, group, or suite that has been executed. + See the output of the [`GET /test_sessions/{test_session_id}/results`](https://inferno-framework.github.io/inferno-core/api-docs/#/Result/get_test_sessions__test_session_id__results) + API for details on the output. + + ```sh + bundle exec inferno session results 7uLiz9qX4og -I https://inferno.healthit.gov/suites + ``` + +- **`compare SESSION_ID`** + + Compare a session's results against expected results from a file or another session. + Exits `0` when results match; exits `3` when they do not. + + **Options**: + - `-I`, `--inferno_base_url`: URL of the target Inferno service. + - `-f`, `--expected_results_file`: Path to a JSON file containing the expected results. When the session + results do not match the expected results in the file, generated comparison files will be placed in + the same directory. + - `-s`, `--expected_results_session`: Session ID on the same server. The results of this indicated + session will be used as the expected results. When the compared session's results do not match + comparison details will not be written to file (use the `-f` option). + - `-m`, `--compare_messages`: Also compare per-test messages (not only pass/fail result). Default: `false`. + - `-r`, `--compare_result_message`: Also compare each test's `result_message` string. Default: `false`. + - `-n`, `--normalized_strings`: List of literal strings and regexes which will be used to normalize + strings prior to comparison. For literal strings, the URL-encoded version will be replaced as well. + + **Output**: JSON with a top-level `matched` boolean and a `results` array. Each entry + contains `id`, `type` (`Compared` / `Missing` / `Additional`), `matched`, + `expected_result`, and `actual_result`. When `-m` or `-r` are set, the corresponding + message fields are included as well. + + When `-f` is provided and results do not match, two files are written to the same + directory as the expected results file. If the provided file name ends with + `expected.json`, the `` will be everything before that. Otherwise, it will be empty: + - `actual_results_.json` — the session's actual results + - `compared_results_.csv` — a CSV diff of the mismatched results + + ```sh + bundle exec inferno session compare 7uLiz9qX4og -I https://inferno.healthit.gov/suites -f expected.json -m -n + ``` + ## Running a Test Kit in Command Line +Inferno provides two CLI options for executing Inferno tests via the command line: +1. The `inferno execute_script` CLI command for complex executions that can involve multiple runs + and sessions as well as scripting of external systems. +2. The `inferno execute` CLI command for simple executions that don't involve any interaction beyond + starting the test runs. + +### Complex Scripted Execution + +The `inferno execute_script` CLI command can be used to execute complex Inferno runs from the command line +which are specified in a [script configuration file](/docs/advanced-test-features/scripting-execution#creating-script-configuration-files). +By default, scripts are executed against the local running Inferno instance (both background services +and the test kit must be running). Execution can be performed against a remote Inferno instance by providing +the root Inferno url using the `-I` option. To reduce the risk of running unsafe commands +unintentionally, the `--allow-commands` flag must be passed when the executed script +uses `command:` actions to execute separately defined scripts. See [Defining, Executing, and Securing Complex Commands](/docs/advanced-test-features/scripting-execution#defining-executing-and-securing-complex-commands) for additional details. + +For example, the following command would run the script defined in file `g10_with_commands.yaml`, +including `command:` actions, on the instance of +Inferno deployed to [inferno.healthit.gov](https://inferno.healthit.gov/suites): +``` +bundle exec inferno execute_script g10_with_commands.yaml -I https://inferno.healthit.gov/suites --allow-commands +``` + +See [Scripting Suite Execution](/docs/advanced-test-features/scripting-execution) for details on how to +define script configuration files and [CI / CD Usage](/docs/ci-cd-usage) for how to integrate scripts +into development pipelines. + +### Simple Execution + The `inferno execute` command allows you to execute [runnables](https://inferno-framework.github.io/docs/writing-tests/#test-suite-structure) from a Test Kit in the shell. The command requires: - The Test Kit as the working directory. @@ -78,7 +264,6 @@ from a Test Kit in the shell. The command requires: - All inputs and any [Suite Options](https://inferno-framework.github.io/docs/advanced-test-features/test-configuration.html#suite-options) necessary for execution. - Optionally, the short ids of select Tests and Test Groups you want to run. - For example, to run the [US Core Test Kit](https://github.com/inferno-framework/us-core-test-kit) v3.1.1 Single Patient API group, you would: @@ -123,9 +308,10 @@ bundle exec inferno execute --help for more information. -Please note that inferno execute is still low maturity and has limitations. Inferno -currently cannot support SMART launch tests or tests that receive client requests -via CLI execution. Some tests are also expected to [run as a group](https://inferno-framework.github.io/docs/writing-tests/properties.html#run-as-group), +The `execute` command does not support SMART launch tests or tests that receive client requests +via CLI execution. Use the [`execute_script`](#complex-scripted-execution) functionality +for these complex execution cases. Some tests are also expected to +[run as a group](https://inferno-framework.github.io/docs/writing-tests/properties.html#run-as-group), and may error or fail erroneously when run individually through CLI. ## Suite Subcommands diff --git a/docs/getting-started/repo-layout-and-organization.md b/docs/getting-started/repo-layout-and-organization.md index 2ce17e9..d1b9f93 100644 --- a/docs/getting-started/repo-layout-and-organization.md +++ b/docs/getting-started/repo-layout-and-organization.md @@ -15,6 +15,8 @@ running `inferno new`, you will have a directory structure that looks something like this: ``` +├── .github +│   └── workflows ├── Dockerfile ├── Gemfile ├── config @@ -26,6 +28,7 @@ like this: │   └── ... ├── docker-compose.yml ├── docker-compose.background.yml +├── execution_scripts ├── inferno_template.gemspec ├── lib │   ├── inferno_template @@ -39,6 +42,8 @@ like this: │   ├── ... └── worker.rb ``` +- `.github/workflows` - This directory tells github how to build and test Inferno + when updates are committed to a GitHub repository. - `Dockerfile` - This file controls how the Docker image for your tests is built. - `Gemfile` - This file is where you add extra Ruby dependencies. - `config` - This folder contains configuration for the database and web @@ -51,6 +56,10 @@ like this: needs. - `docker-compose.background.yml` - This file coordinates and runs the background services needed for running Inferno. +- `execution_scripts` - This folder is for scrips that execute test suites against + running Inferno services like the HL7 Validator. The template includes an example + script for the sample test suite. See [CI/CD Usage](docs/ci-cd-usage.html#using-the-inferno-execute_script-cli-for-complex-orchestration) + for additional information. - `inferno_template.gemspec` - This file controls how your tests are packaged up as a distributable Ruby gem. This is also where you can add additional Ruby gems if you need them. @@ -68,7 +77,7 @@ like this: for collecting discrete requirements to use with Inferno's optional requirement management infrastructure. See [Requirements](/docs/advanced-test-features/requirements.html) for more information on how to use the requirements tools. -- `spec` - This folder is for unit tests. +- `spec` - This folder is for unit tests. The template includes an example unit test for the sample test suite written in rspec. - `worker.rb` - This is the main file for Inferno's test runner process. ## Test Organization