From 74bba45412f3b1d39a16c154c6528169e59a299d Mon Sep 17 00:00:00 2001 From: Karl Naden Date: Tue, 31 Mar 2026 07:55:53 -0400 Subject: [PATCH 01/11] session CLI documentation --- docs/getting-started/inferno-cli.md | 150 +++++++++++++++++++++++++++- 1 file changed, 148 insertions(+), 2 deletions(-) diff --git a/docs/getting-started/inferno-cli.md b/docs/getting-started/inferno-cli.md index 13dfe16..a992a24 100644 --- a/docs/getting-started/inferno-cli.md +++ b/docs/getting-started/inferno-cli.md @@ -21,17 +21,21 @@ The commands available include: | Command | Description | |--------------|-------------| | `inferno console` | Starts an interactive console session with Inferno. More information can be found on the [Debugging Page](debugging.html#interactive-console). | +| `inferno execute` | Execute tests in the command line instead of web UI. | +| `inferno evaluate IG_PATH [--options]` | Run the FHIR evaluator in the command line. Does not require `bundle exec`. | | `inferno help [COMMAND]` | Describes the available commands, or one specific command if specified. | | `inferno migrate` | Runs database migrations. | | `inferno new TEST_KIT_NAME` | Create a new test kit. Run `inferno new --help` for additional options. Does not require `bundle exec`. | +| `inferno requirements SUBCOMMAND [...ARGS]` | Manipulates and checks Inferno requirements. The available subcommands are `export_csv`, `check`, `coverage`, and `check_coverage`. See the [requirements page](/docs/advanced-test-features/requirements) for details of these subcommands. | +| `inferno requirements help SUBCOMMAND` | View details on the requirements subcommands. | | `inferno services start` | Starts the background services (nginx, Redis, etc.) for Inferno. | | `inferno services stop` | Stops the background services for Inferno. | +| `inferno session SUBCOMMAND [...ARGS]` | Manages Inferno test sessions programmatically. The available subcommands are `create`, `start_run`, `status`, `cancel_run`, `data`, `results`, and `compare`. | +| `inferno session help SUBCOMMAND` | View details on the session subcommands. | | `inferno start` | Starts Inferno web UI. Do not use `bundle exec` if the Inferno Core version is prior to 0.4.39. | | `inferno suite SUBCOMMAND [...ARGS]` | Performs suite-based operations. The available subcommands are `describe`, `help`, `input_template`, and `lock_short_ids`.| | `inferno suite help SUBCOMMAND` | View details on the suite subcommands. | | `inferno suites` | Lists available test suites. | -| `inferno execute` | Execute tests in the command line instead of web UI. | -| `inferno evaluate IG_PATH [--options]` | Run the FHIR evaluator in the command line. Does not require `bundle exec`. | | `inferno version` | Outputs the version of Inferno Core (not the Test Kit). Does not require `bundle exec`. | {: .grid.command-table} @@ -68,6 +72,148 @@ You should manually inspect this file if you want to [publish or distribute the The `--implementation-guide` or `-i` option will look at an absolute file path if its prefixed by `file:///` or lookup the name in the [official FHIR package registry](https://packages.fhir.org/). +## Manage Sessions + +The `inferno session` command provides a programmatic interface for creating and managing +Inferno test sessions from the command line. It is useful for automating test execution, +especially in CI/CD pipelines. See the [CI/CD Usage](docs/ci-cd-usage) page for additional +information and tools for the CI/CD use case. + +All commands leverage the [JSON web API](https://inferno-framework.github.io/inferno-core/api-docs/) +to interact with Inferno. By default, these commands interact with the +local running Inferno instance. Additionally, all session subcommands +support the `--inferno_base_url` (`-I`) option to specify the URL of a remote running Inferno instance to interact +with instead. All commands write JSON to stdout and exit `0` on success or `3` on error (`1` and `2` are reserved +for Thor and Bash respectively). + +Subcommands include: + +### **`create SUITE_ID`** + + Create a new test session for the given suite. + + **Options**: + - `-I`, `--inferno_base_url`: URL of the target Inferno service. + - `-o`, `--suite_options`: Suite options as `key:value` pairs (e.g., `--suite_options us_core_version:us_core_6 smart_app_launch_version:smart_app_launch_2`). + - `-p`, `--preset_id`: Apply a named preset when creating the session. + + **Output**: JSON representation of the created session, including the session `id` used by all subsequent subcommands. + See the output of the [`POST /test_sessions`](https://inferno-framework.github.io/inferno-core/api-docs/#/Test%20Session/post_test_sessions) API for details on the output. + + ```sh + bundle exec inferno session create my_test_suite -I https://inferno.healthit.gov/suites -p my_preset + ``` + +### **`start_run SESSION_ID`** + + Initiate a test run on an existing session. + + **Options**: + - `-I`, `--inferno_base_url`: URL of the target Inferno service. + - `-r`, `--runnable`: Short or internal ID of a specific group or test to run. If omitted, the entire suite is run. + - `-i`, `--inputs`: Input values as `key:value` pairs (e.g., `--inputs url:https://example.com patient_id:123`). These merge with and override the session's previously stored inputs. + + **Output**: JSON representation of the created test run. See the output of the [`POST /test_runs`](https://inferno-framework.github.io/inferno-core/api-docs/#/Test%20Run/post_test_runs) API for details on the output. + + ```sh + bundle exec inferno session start_run abc123 -I https://inferno.healthit.gov/suites -r my_group \ + -i url:https://example.com patient_id:123 + ``` + +- **`status SESSION_ID`** + + Get the current run status of a session. + + **Options**: + - `-I`, `--inferno_base_url`: URL of the target Inferno service. + + **Output**: JSON including a `status` field (`created`, `queued`, `running`, `waiting`, `done`, + etc.) and the ID of the `last_test_executed`. See the output of the + [`GET /test_runs/{test_run_id}`](https://inferno-framework.github.io/inferno-core/api-docs/#/Test%20Run/get_test_runs__test_run_id_) + API for details on the output. Additionally, When `status` is `waiting`, fields + `wait_outputs` and `wait_result_message` are added to this base object using the + `outputs` and `result_message` fields from the result of the waiting test (last test that was executed). + + ```sh + bundle exec inferno session status abc123 -I https://inferno.healthit.gov/suites + ``` + +- **`cancel_run SESSION_ID`** + + Cancel the currently active run for a session. Applies when the run is in `queued`, + `running`, or `waiting` status. + + **Options**: + - `-I`, `--inferno_base_url`: URL of the target Inferno service. + + **Output**: JSON with `run_id` and `cancelled: true`, or an error object if no cancellable + run is active. + + ```sh + bundle exec inferno session cancel_run abc123 -I https://inferno.healthit.gov/suites + ``` + +- **`data SESSION_ID`** + + Get the input values currently stored for a session. + + **Options**: + - `-I`, `--inferno_base_url`: URL of the target Inferno service. + + **Output**: JSON array of `{ name, value }` objects representing the session's current inputs. + See the output of the [`GET /test_sessions/{test_session_id}/results`](https://inferno-framework.github.io/inferno-core/api-docs/#/Result/get_test_sessions__test_session_id__results) + API for details on the output. + + ```sh + bundle exec inferno session data abc123 -I https://inferno.healthit.gov/suites + ``` + +- **`results SESSION_ID`** + + Get the test results for a session. + + **Options**: + - `-I`, `--inferno_base_url`: URL of the target Inferno service. + + **Output**: JSON array of result objects for each test, group, or suite that has been executed. + See the output of the [`GET /test_sessions/{test_session_id}/session_data`](https://inferno-framework.github.io/inferno-core/api-docs/#/Session%20Data/get_test_sessions__test_session_id__session_data) + API for details on the output. + + ```sh + bundle exec inferno session results abc123 -I https://inferno.healthit.gov/suites + ``` + +- **`compare SESSION_ID`** + + Compare a session's results against expected results from a file or another session. + Exits `0` when results match; exits `3` when they do not. + + **Options**: + - `-I`, `--inferno_base_url`: URL of the target Inferno service. + - `-f`, `--expected_results_file`: Path to a JSON file containing the expected results. + - `-s`, `--expected_results_session`: Session ID on the same server whose results serve as expected. + - `-m`, `--compare_messages`: Also compare per-test messages (not only pass/fail result). Default: `false`. + - `-r`, `--compare_result_message`: Also compare each test's `result_message` string. Default: `false`. + - `-n`, `--normalize`: Before comparing strings, replace dynamic values with stable + placeholders: ISO 8601 datetimes → ``, UUIDs → ``, random base64 + strings → ``. Default: `false`. + + **Output**: JSON with a top-level `matched` boolean and a `results` array. Each entry + contains `id`, `type` (`Compared` / `Missing` / `Additional`), `matched`, + `expected_result`, and `actual_result`. When `-m` or `-r` are set, the corresponding + message fields are included as well. + + When `-f` is provided and results do not match, two files are written to the same + directory as the expected results file. If the provided file name ends with + `expected.json`, the `` will be everything before that. Otherwise, it will be empty: + - `actual_results_.json` — the session's actual results + - `compared_results_.csv` — a CSV diff of the mismatched results + + ```sh + bundle exec inferno session compare abc123 -I https://inferno.healthit.gov/suites -f expected.json -m -n + ``` + + ## Running a Test Kit in Command Line The `inferno execute` command allows you to execute [runnables](https://inferno-framework.github.io/docs/writing-tests/#test-suite-structure) From a71e843676f0c09a105d0ab1370163f4c8367e5f Mon Sep 17 00:00:00 2001 From: Karl Naden Date: Tue, 31 Mar 2026 07:55:53 -0400 Subject: [PATCH 02/11] WIP ci-cd doc updates --- docs/ci-cd-usage.md | 292 +++++++++++++++++- .../repo-layout-and-organization.md | 7 +- 2 files changed, 291 insertions(+), 8 deletions(-) diff --git a/docs/ci-cd-usage.md b/docs/ci-cd-usage.md index b07999e..7024e2a 100644 --- a/docs/ci-cd-usage.md +++ b/docs/ci-cd-usage.md @@ -7,11 +7,16 @@ section: docs # Using Test Kits in CI/CD -Since [inferno +Inferno provides two CLI options for executing Inferno tests via the command line: +1. The `inferno execute` CLI command for execution of a single suite, group, or test. +2. The `inferno execute_script` CLI command for more complex executions that can involve multiple runs + and sessions as well as scripting of external systems. + +## Using the `inferno execute` CLI for Simple Execution + +To run Inferno using the [inferno execute](/docs/getting-started/inferno-cli.html#running-a-test-kit-in-command-line) -enables running Inferno tests in CLI, it may be used within a Continuous -Integration or Continuous Deployment (CI/CD) pipeline. The generic steps around -automatically running a Test Kit are: +CLI command, follow these steps: 1. Setup a container or environment with Ruby and Docker 2. Obtain your target Test Kit version and make it your working directory @@ -31,6 +36,279 @@ its own test session. Test Suite to pass. 7. Clean up. You can stop Inferno services with `bundle exec inferno services stop`. -If you want to use GitHub Actions for your CI/CD pipeline, or run the -ONC Certification (g)(10) Test Kit in CI/CD, you can view our [workflow file](https://github.com/inferno-framework/inferno-reference-server/blob/5e0d06ad5414efa93499fd3de093e29cf5e6d9d1/.github/workflows/inferno_ci.yml) -on the Inferno Reference Server. +To see an example of these steps, see [this Github workflow file](https://github.com/inferno-framework/inferno-reference-server/blob/5e0d06ad5414efa93499fd3de093e29cf5e6d9d1/.github/workflows/inferno_ci.yml) +which is used to test the Inferno Reference Server against the (g)(10) certification test kit. + +## Using the `inferno execute_script` CLI for Complex Orchestration + +The `inferno execute` command covers straightforward single-run scenarios. However many +Test Suites require more complex orchestration to execute completely. +For example, some suites pause in a `waiting` state until an external action is performed. + +For these and other complex cases that require coordination, Inferno provides the `inferno execute_script` CLI command, +which runs a Bash orchestration library available in the [inferno-core repository](https://github.com/inferno-framework/inferno-core/blob/main/execution_scripts/session_runner.sh). +This script leverages the [`inferno session` CLI commands](/docs/getting-started/inferno-cli.html#manage-sessions) +to create, manage, and compare session results. Users define a [configuration file](#yaml-configuration-format) +that tells the runner when and how to perform execution steps inside and outside of Inferno. + +### Prerequisites + +To run the `inferno execute_script` CLI command, the following utilities must be available to execute: + +- `bash` (v4+) +- [`jq`](https://stedolan.github.io/jq/) +- [`yq`](https://github.com/mikefarah/yq) (v4+) + +Additional utilities may be needed to execute commands found in [execution configuration files](#yaml-configuration-format). + +Additionally, the command requires a running instance (local or remote) of the target Inferno Test Kit(s) +and their services. + +### Execution + +For basic execution via the command line, the CLI command takes the target [execution configuration file](#yaml-configuration-format) as an input. + +``` +bundle exec inferno execute_script path/to/script_config_file.yaml +``` + +By default, the execution targets the local running instance of Inferno, which must be up. A remote Inferno +instance can be targeted by passing the `-I` option with the url of the target running Inferno instance, +e.g., the public Inferno instance at `https://inferno.healthit.gov/suites`. + +#### Github Workflow Execution + +Inferno provides a standard Github Workflow file that finds script configuration files +defined under the `execution_scripts` directory and runs each of them against a freshly +created Inferno instance running the test kit defined in the repository. + +#### Environment Variables + +The following environment variables control specific script behavior and can be set prior to executing the +`inferno execute_script` command: + +| Variable | Default | Description | +|---|---|---| +| `POLL_INTERVAL` | `5` | Seconds between status checks while a run is active | +| `DEFAULT_POLL_TIMEOUT` | `30` | Default timeout in seconds for each polling step | +| `COMPARE_NORMALIZE` | `true` | Normalize UUIDs and base64 values before comparing results (`-n`) | +| `COMPARE_MESSAGES` | `true` | Include the `messages` array in result comparison (`-m`) | +| `COMPARE_RESULT_MESSAGE` | `true` | Include `result_message` in result comparison (`-r`) | +| `ACTIONS_FILE` | _(derived from script path)_ | Override the YAML config file path when sourcing the script as a library | + +#### Example Scripts to Execute + +The following Test Kits include scripts that can be executed. To execute any of them +1. Checkout the repository locally and set it up to be run in developer mode +2. Locate the target script configuration file under the `execution_scripts` directory + and run `bundle exec inferno execute_script execution_scripts/path/to/script_config_file.yaml` pointing to the desired configuration file. + +- [**Inferno Template**](https://github.com/inferno-framework/inferno-template): + New Inferno Test Kits created from the [inferno-template repository](https://github.com/inferno-framework/inferno-template) + or the [`inferno new` CLI command](/docs/getting-started/inferno-cli.html#inferno-commands) + include a simple execution configuration file for the example test suite included in the + template. +- [**Inferno Core**](): TODO +- [**(g)(10) Certification**](): TODO +- [**DaVinci PAS**](): TODO + +### Creating Script Configuration Files + +Script configuration files define two things +- The Inferno session(s) executed, including any suite options or presets to be used +- The individual steps to take including + - When to take the step in the form of an Inferno session status and last executed test. + - What action to take in the form of an Inferno `session start_run` CLI command or general shell command. + +#### Process + +Execution scripts are typically used to automate how a user would interact with the Inferno +UI. Starting from a UI execution makes planning the sessions and steps of an execution +script simple. When mapping a UI execution onto a execution script, each session created +(typically one, sometimes two if running Inferno suites against each other) will correspond +to a `sessions` entry, and each interaction with the Inferno UI (to start a test run, or respond +to a wait dialog) will correspond with a `steps` entry. + +The tricky piece is to determine the specific values to put in the script configuration file for internal +identifiers like +- suite option keys and values +- input keys and values +- test full ids for use in specifying the last test + + +To use the script and the workflow, +1. Create a YAML config file (see format below) in the `execution_script/` + directory. +2. Make sure that the containing test kit is running in Ruby developer mode, including + both the background services and the test kit itself. +3. From the root test kit directory, run the script + +```bash +bash execution_scripts/session_runner.sh execution_scripts/my_suite.yaml +``` + +4. The script will print to the terminal details on its execution, including matched + test states and executed actions. If the test reaches an unmatched state or never + reaches an `END_SCRIPT` action, it will fail, providing details on the issue so + that updates can be made to fix the problem. +5. Once the script completes for the first time, unless you manually defined an expected result + file (`my_suite_expected.yaml`), the command will generate the expected file based on + the results of the executed session. Review the results to ensure that they represent + the expected results. If they don't adjust the script yaml file to correct any issues. +6. Run the script once more locally to make sure that the results of the next run match + the expected. If they don't there may be non-deterministic behavior or other problematic + details in you test kit (see debugging details below). +7. Commit the script yaml file and the expected results json file to the test kit repository + and now the execution will be checked whenever changes are made to the test kit. + +#### YAML Configuration Format + +The config file has two top-level keys: `sessions` and `steps`. + +```yaml +sessions: + - suite_id: my_suite # required + name: my_session # optional; used to reference this session in step/tokens + preset_id: my-preset # optional + suite_options: # optional + option_key: option_value + expected_results_file: expected.json # optional; relative to the yaml file + # default: _expected.json + +steps: # ordered; first matching step wins + - status: done # session status to match: created, done, or waiting + last_test: my_suite-Group01-Test01 # optional; full id of the last test executed (exact match) + # must be present unless the status is created, in which case it must be absent + session: my_session # optional; in multi-session runs, scopes step to this session + start_run: # structured alternative to 'command' for inferno session start_run calls + runnable: "1.01" # optional; short_id or full id of the group/test to run (-r flag) + inputs: # optional; key/value map of inputs (-i flag) + input_name: "value" + timeout: 300 # optional; seconds before polling this step times out. If not provided + # falls back to the session_runner.sh default (30s or the value of the + # DEFAULT_POLL_TIMEOUT environment variable) + next_poll_session: other_session # optional; switch polling focus to the named session after action + state_description: "..." # optional; logged when the step is matched + action_description: "..." # optional; logged when the step is matched +``` + +Each step must have either a `start_run:` block or a `command:` string (see below). + +#### Step: `command` vs `start_run` + +Use `start_run:` when you want to launch a test run using the standard +`bundle exec inferno session start_run` command. For anything else (e.g. making an +external HTTP request, running a shell command, or passing data to a waiting test), +use the `command:` string field. The `command` string is evaluated by `eval`, so +full shell expressions are supported. + +```yaml +steps: + # Start a run using the structured start_run block + - status: created + start_run: + runnable: "1" + inputs: + url: "https://example.com/fhir" + + # Start a run using a raw command string + - status: done + last_test: my_suite-Group01-Test01 + command: "bundle exec inferno session start_run '{session_id}' -r 2" + + # Perform an external action to satisfy a waiting test + - status: waiting + last_test: my_suite-Group01-wait_test + command: "curl -s '{wait_outputs.redirect_url}'" + + # Execute a defined ruby script + - status: waiting + last_test: my_suite-Group01-launch_test + command: "bundle exec ruby execution_scripts/launch_helper.rb {wait_output.launch_url}" +``` + +### Template Tokens + +Template tokens in `command` strings (and `start_run` input values) are substituted +at runtime: + +| Token | Description | +|---|---| +| `{session_id}` | ID of the current session (single-session only) | +| `{session_id.NAME}` | ID of the session with the given name (required in multi-session) | +| `{result_message}` | The wait result message from the current session | +| `{NAME.result_message}` | Wait result message from the named session (triggers a status fetch) | +| `{wait_outputs.KEY}` | Value of the named wait output from the current session | +| `{NAME.wait_outputs.KEY}` | Named wait output from a different session (triggers a status fetch) | + +### Special `command` Values + +| Value | Behaviour | +|---|---| +| `END_SCRIPT` | Stop execution for this session and proceed to result comparison | +| `NOOP` | Match the step but take no action; keep polling with the current timeout | + +### Automatic Status Handling + +You do not need steps for every possible status. The script applies these defaults +automatically: + +| Status | Default behaviour | +|---|---| +| `running`, `queued`, `cancelling` | Poll again | +| `done` (no step matched) | Warn, run result comparison, exit non-zero | +| `waiting` (no step matched) | Cancel the current run and resume polling | +| `created` (no step matched) | Warn, run result comparison, exit non-zero | + +### Expected Results + +After all runs complete, the script compares results against an expected results JSON +file. If the file does not yet exist, the current results are written to it, creating +a baseline for future runs. The expected file path is resolved as follows: + +- If `expected_results_file` is set in the YAML, that path is used (relative to the + YAML file). +- For single-session configs with no explicit path: `_expected.json`. +- For multi-session configs with no explicit path: `__expected.json`. + +### Multi-Session Example + +Some test flows require two sessions to run in parallel — for example, a client test +that makes requests to a server session. Use multiple entries under `sessions:` and +scope steps with `session:` and `next_poll_session:` to coordinate between them. + +```yaml +sessions: + - suite_id: my_server_suite + name: server + - suite_id: my_client_suite + name: client + +steps: + # Start the server session first + - status: created + session: server + start_run: + + # Once server is waiting, start the client session + - status: waiting + session: server + next_poll_session: client + command: "NOOP" + + - status: created + session: client + start_run: + inputs: + server_url: "https://example.com" + + # When client is done, resume the server + - status: waiting + session: client + next_poll_session: server + command: "curl -s '{wait_outputs.redirect_url}'" + + - status: done + session: server + command: "END_SCRIPT" +``` \ No newline at end of file diff --git a/docs/getting-started/repo-layout-and-organization.md b/docs/getting-started/repo-layout-and-organization.md index 2ce17e9..cf9ad85 100644 --- a/docs/getting-started/repo-layout-and-organization.md +++ b/docs/getting-started/repo-layout-and-organization.md @@ -26,6 +26,7 @@ like this: │   └── ... ├── docker-compose.yml ├── docker-compose.background.yml +├── execution_scripts ├── inferno_template.gemspec ├── lib │   ├── inferno_template @@ -51,6 +52,10 @@ like this: needs. - `docker-compose.background.yml` - This file coordinates and runs the background services needed for running Inferno. +- `execution_scripts` - This folder is for scrips that execute test suites against + running Inferno services like the HL7 Validator. The template includes an example + script for the sample test suite. See [CI/CD Usage](/docs/getting-started/ci-cd-usage.html#using-session_runnersh-for-complex-orchestration) + for additional information. - `inferno_template.gemspec` - This file controls how your tests are packaged up as a distributable Ruby gem. This is also where you can add additional Ruby gems if you need them. @@ -68,7 +73,7 @@ like this: for collecting discrete requirements to use with Inferno's optional requirement management infrastructure. See [Requirements](/docs/advanced-test-features/requirements.html) for more information on how to use the requirements tools. -- `spec` - This folder is for unit tests. +- `spec` - This folder is for unit tests. The template includes an example unit test for the sample test suite written in rspec. - `worker.rb` - This is the main file for Inferno's test runner process. ## Test Organization From e2e549fc280e0493b2cf9577260e7e7adb8dfcca Mon Sep 17 00:00:00 2001 From: Karl Naden Date: Tue, 31 Mar 2026 07:55:53 -0400 Subject: [PATCH 03/11] updated documentation --- docs/advanced-test-features/index.md | 6 +- .../scripting-execution.md | 237 +++++++++++++++ docs/ci-cd-usage.md | 271 +----------------- docs/getting-started/inferno-cli.md | 48 +++- 4 files changed, 280 insertions(+), 282 deletions(-) create mode 100644 docs/advanced-test-features/scripting-execution.md diff --git a/docs/advanced-test-features/index.md b/docs/advanced-test-features/index.md index 97ce23c..7c8524d 100644 --- a/docs/advanced-test-features/index.md +++ b/docs/advanced-test-features/index.md @@ -51,4 +51,8 @@ Inferno Core supports displaying a custom banner for your Test Kit. This banner The Inferno DSL supports associating discrete requirements with suites, test groups, and tests. Test developers can track which requirements Inferno is and is not verifying and Inferno displays these associations to users within the -[user interface](/docs/user-interface.html#viewing-verified-requirements). \ No newline at end of file +[user interface](/docs/user-interface.html#viewing-verified-requirements). + +## [Scripting Suite Execution](/docs/advanced-test-features/script-execution.html) +The Inferno CLI supports executing suites using a yaml configuration format +instead of the UI using the `inferno execute_script` command. \ No newline at end of file diff --git a/docs/advanced-test-features/scripting-execution.md b/docs/advanced-test-features/scripting-execution.md new file mode 100644 index 0000000..27fb4c4 --- /dev/null +++ b/docs/advanced-test-features/scripting-execution.md @@ -0,0 +1,237 @@ +--- +title: Scripting Suite Execution +nav_order: 9 +parent: Advanced Features +layout: docs +section: docs +--- +# Scripting Suite Execution + +## Overview + +Inferno supports the scripting of test suite execution through the `inferno execute_script` +[CLI command](/docs/getting-started/inferno-cli). This command takes as input a +[configuration file](#yaml-configuration-format) that tells Inferno when and how to perform +execution steps both inside and outside of Inferno. The results of the session are compared +against the results of a known-good execution to help identify any regressions. The script +can be executed against the local Inferno instance or a specific remote instance. These scripts +can be built into [CI/CD Pipelines](/docs/ci-cd-usage). + +## Execution + +See the [CLI documentation](/docs/getting-started/inferno-cli#complex-scripted-execution) for details +on executing scripts from the command line. See [CI / CD Usage](/docs/ci-cd-usage.md) for how to +execute scripts within development workflows. + +## Creating Script Configuration Files + +At a high-level, the script configuration file contains instructions for +- The session(s) to create, including any suite options and presets to use +- The steps to take to execute the script, each including an action, when to perform it, and when there + are multiple Inferno sessions involved which session will act next. +- Normalizations of the results to make sure that executions are comparable across runs and Inferno instances. + +The easiest way to think about creating a script configuration file is that it mirrors the manual steps taken +in the UI to execute an Inferno test session of a particular suite against a specific system. Go through +the manual execution steps and note each time there is an interaction with the Inferno UI or a state in Inferno that +triggers an interaction with the tested system: each of these will become a step within the script. + +The rest of this section provides details on how to turn the manual execution into a script file. +See the documentation at the top of the [`execute_script.rb`](https://github.com/inferno-framework/inferno-core/blob/main/lib/inferno/apps/cli/execute_script.rb) file for a complete format reference. + +### Sessions + +Each script configuration file will indicate a list (sequence) of one or more Inferno sessions to create under +the `sessions:` top-leve key. Each sequence includes +- **Suite** (`suite:`): *Required* - the internal id for the suite or the title selected in the UI. +- **Name** (`name:`): *Optional* - a short key used to identify the session in steps. Required when multiple sessions + are defined. +- **Preset** (`preset:`): *Optional* - the internal id or title of a preset to use for the session. +- **Suite Options** (`suite_options:`): *Optional* - pairs of `key: value` entries each corresponding to a suite + option. The key can be either the internal id or the title from the UI. The value can be either + the internal value or the label from the UI. +- **Expected Results** (`expected_results_file:`): *Optional* - the path to the file containing the expected results, + relative to the executed yaml file name. If not provided defaults to _expected.json. + +### Steps + +Each script configuration file will indicate a list (sequence) of steps that are taken by the script under +the `steps:` top-level key. There will be at least 2 in every script: +- an initial `created` state that will be the first step executed. +- a final `END_SCRIPT` action that will be the last step executed. + +While steps are executed when they are matched by Inferno's state, only one step is executed at a time +and loops are detected and not allowed. By convention, steps should be listed in the order they will +be executed in practice. + +Each step consists of keys pertaining to the following details +- When to take the step +- What action to take +- How to look for the next step + +The following subsections detail the keys related to each of these areas + +#### When to take the step + +Each step indicates when it will be take through three keys +- `status:` *Required* - specifies the status to match with options: + - `done` when a test run has completed + - `waiting` when a test run is in progress but waiting for external input + - `created` when a test session has been created, but no runs started yet +- `last_completed:` *Required* unless the status is `created` - specifies a runnable in the form of an + internal id or short numeric id from the UI. + - for `done`, will be the test, group, or suite that was executed when the run was started + - for `waiting`, will be the test that initiated the wait + - for `created`, will be omitted (nothing executed yet) +- `session:` *Required* when multiple sessions, otherwise omitted - the name (not suite id) of the + session to poll for the next step. + +Additionally, the optional `state_description:` key can be used to provide documentation describing the +state. This has no functional effect, but is echoed during script execution for debugging purposes. + +Note that Inferno can be have other status (e.g., `running` and `cancelled`), but these do not +represent stable states where Inferno is waiting for an action, so they will never be used in +scripts. + +Conversely, Inferno needs to know what to do in any stable state that is reached. If no step matches +a `done` or `created` state that Inferno reaches, then the script will end with an error (results not checked). +If no step matches a `waiting` state, then Inferno will cancel the current run (ending the wait) and attempt +to continue with the next matching `done` state. + +#### What action to take + +There are three kinds of actions that can be take by a step each specified by a different key. Only +one can be defined for each step: +- `start_run:` Used to execute a test, group, or suite on one of the sessions started by this script. The following + sub keys are used. + - `runnable:` *Required* - the test, group, or suite to execute, specified as a short ID from the UI or an internal id. + - `inputs:` *Optional* - key-value pairs indicating inputs to be merged into those already stored in the session (from the + preset or previous run inputs or outputs). Each key must be the internal name for the input (from the UI, use the yaml or json view of the inputs to find the internal name). + - `session:` *Required* when the script has multiple sessions, otherwise omitted - the name (not the suite) of the session on which the run will be executed. +- `command:` Specifies a shell command to execute. For example, a `curl` command to navigate to an attestation URL or + an external script that automates manipulation of the tested system. +- `action:` Script-specific actions to bridge between steps. Defined actions include: + - `END_SCRIPT` used on the distinguished final step. The execution will end successfully after this step. + - `NOOP` used on steps that don't require an action, but indicate that another step is unblocked. See details in + the discussion of multiple session scripts below. + - `WAIT` used on transitory `waiting` states that are expected to end without external action. Note that unlike other + actions, this one does not cause Inferno to break the polling loop, so it can be matched multiple times (not a loop) + but does not change next step details (see next subsection). + +The specifics of `start_run` and `command` actions may depend on dynamic session details. The following template +tokens are allowed in fields of these actions which will be replaced with session-specific values at execution time. For +most, there is both a named version for scripts with multiple sessions and a raw version for those with only a single session: +- **Session Id**: `{session_id}` (`{.session_id}` when multiple sessions) +- **Result Message** (from waiting tests only): `{result_message}` (`{.result_message}` when multiple sessions) +- **Output Value** (from waiting tests only): `{wait_outputs.KEY}` (`{.wait_outputs.KEY}` when multiple sessions) +- **Inferno URL**: `{inferno_base_url}` (no session-specific version - always the same for all sessions) + +Additionally, the optional `action_description:` key can be used to provide documentation describing the +action. This has no functional effect, but is echoed during script execution for debugging purposes. + +#### How to look for the next step + +After taking an action, script execution returns to a polling mode where it checks the status of a session +to determine if it has reached a stable state to match against to identify the next step. The previous step +can control two aspects of this process using the following keys: +- `timeout:` *Optional* - specifies a number of sections to wait before aborting by canceling the run. The default is 30 seconds. + After cancelling, Inferno will try to continue matching other states, but will still ultimately indicate failure. +- `next_poll_session:` *Optional* - the name (not suite id) of the session to poll for the next step. Inferno only ever polls + one session at a time, so in scripts with multiple sessions, it must be told which session to poll next. + The first session configured is used to start. After executing a step, Inferno will either poll the session indicated + in this key or the session polled after the previous step if none is specified. + +#### Normalizations for Results Comparison + +After completing the script, the results of the executed session will be compared for each session to the expected +results recorded in the session's expected results file. When that comparison is performed, +Inferno will replace strings indicated in the `normalized_strings:` top-level key to normalize values that +are expected to be different between runs and across different Inferno hosts. Each entry in the sequence can be +- A raw string: the exact string and its URL-encoded form are replaced with the string `` +- A raw regex: strings matching the regex are replaced with the string `` +- An object with two keys that provides specific replacement strings (useful for debugging): + - `patterns:` a sequence of strings or regexes used to identify the strings to replace, which behave like the raw forms. (for single-entry lists, the `pattern:` can be used instead with the single value) + - `replacement:` the string that replaces the patterns + +## Execution Process and Output + +When a script is executed, the following phases are performed: +1. Sessions are created +2. Steps are executed +3. Results are checked + +During execution, Inferno will print to the terminal details on its polling, matching, and actions during +execution and provide a summary of the results comparison performed. + +Inferno will use exit code 3 when an error is encountered. Error cases include: +- The exepected results file for one or more sessions did not exist. +- Compared results for one or more sessions did not match. +- Script execution did not end at the `END_SCRIPT` step. +- Script execution had to be interrupted due to a timeout. + +Otherwise, the CLI command will end with exit code 0 indicating success. + +### Session Creation + +Internally, this uses the [`inferno session create` CLI](/docs/getting-started/inferno-cli#manage-sessions). + +### Step Execution + +Inferno checks the status of sessions for next step polling using the [`inferno session status` CLI](/docs/getting-started/inferno-cli#manage-sessions). It may also use the [`inferno session cancel` CLI](/docs/getting-started/inferno-cli#manage-sessions) if it needs to cancel a session. + +To start runs, Inferno uses the [`inferno session start_run` CLI](/docs/getting-started/inferno-cli#manage-sessions). + +### Check Results + +For each session in the script, Inferno will compare the results in the expected results file (see [sessions configuration](#sessions)) to the results of the completed session execution. To perform the comparison, it will use the [`inferno session compare` CLI](/docs/getting-started/inferno-cli#manage-sessions), which involves the following steps: +- normalize the expected and actual results using the configured [`normalized_strings`](#normalizations-for-results-comparison) +- match individual result entries using the runnable id (`test_id`, `test_group_id`, or `test_suite_id`) +- compare the matched result entries, looking at the status (e.g., pass, fail), the result message, and the + individual messages within the result + +When the actual and expected results do not match, two files will be written in the same location as the expected file: +- actual results: a file with the results json (prior to normalization) +- comparison analysis: a csv file with details of results that differed, including normalized details +These files are named using the same prefix as the executed file and include a timestamp for the execution so that +they will be unique for each script execution. + +When the target expected results file does not exist it will be created using the output of the +[`inferno session results` CLI](/docs/getting-started/inferno-cli#manage-sessions). The `execute_script` +CLI will exit with an error status in this case. + +## Scripts with Multiple Sessions + +Scripts with multiple sessions involve additional complexity but are useful for scripting the execution of +Inferno suites against each other. + +Considerations to keep in mind when creating script configuration files with multiple sessions: +- Include the `name:` key for each session for use as a key to refer to the session in the steps. +- The biggest change in that steps are specific to sessions and Inferno needs to know which session + to poll next after each session. This means that steps + - Will always declare the name of the session they match in the `session:` key. + - Will sometimes declare the name of the next session to poll in the `next_poll_session:` key if + diferent from the session of the matched step. This corresponds to steps in the manual execution + where the tester must change over to the tab of the another session's UI. +- Templates for dynamic values in executed actions (commands and Inferno runs) will use the session-specific + form. +- `NOOP` actions can be used to help clarify sequencing. For example, if two sessions both have long running + tests that interact but complete in a particular order due to wait interactions, but don't require any external + interactions. You could poll the session that will end last and look only for the `done` state, which would require + `WAIT` actions on any `waiting` states so that Inferno recognizes them, or you could define states for each `waiting` + state with `NOOP` actions to make clear how the active execution passes back and forth. + +## Example Scripts + +TODOThe following Test Kits include scripts that can be executed. To execute any of them +1. Checkout the repository locally and set it up to be run in developer mode +2. Locate the target script configuration file under the `execution_scripts` directory + and run `bundle exec inferno execute_script execution_scripts/path/to/script_config_file.yaml` pointing to the desired configuration file. + +- [**Inferno Template**](https://github.com/inferno-framework/inferno-template): + New Inferno Test Kits created from the [inferno-template repository](https://github.com/inferno-framework/inferno-template) + or the [`inferno new` CLI command](/docs/getting-started/inferno-cli.html#inferno-commands) + include a simple execution configuration file for the example test suite included in the + template. +- [**Inferno Core**](): TODO +- [**(g)(10) Certification**](): TODO +- [**DaVinci PAS**](): TODO \ No newline at end of file diff --git a/docs/ci-cd-usage.md b/docs/ci-cd-usage.md index 7024e2a..9280cf2 100644 --- a/docs/ci-cd-usage.md +++ b/docs/ci-cd-usage.md @@ -41,274 +41,7 @@ which is used to test the Inferno Reference Server against the (g)(10) certifica ## Using the `inferno execute_script` CLI for Complex Orchestration -The `inferno execute` command covers straightforward single-run scenarios. However many -Test Suites require more complex orchestration to execute completely. -For example, some suites pause in a `waiting` state until an external action is performed. - -For these and other complex cases that require coordination, Inferno provides the `inferno execute_script` CLI command, -which runs a Bash orchestration library available in the [inferno-core repository](https://github.com/inferno-framework/inferno-core/blob/main/execution_scripts/session_runner.sh). -This script leverages the [`inferno session` CLI commands](/docs/getting-started/inferno-cli.html#manage-sessions) -to create, manage, and compare session results. Users define a [configuration file](#yaml-configuration-format) -that tells the runner when and how to perform execution steps inside and outside of Inferno. - -### Prerequisites - -To run the `inferno execute_script` CLI command, the following utilities must be available to execute: - -- `bash` (v4+) -- [`jq`](https://stedolan.github.io/jq/) -- [`yq`](https://github.com/mikefarah/yq) (v4+) - -Additional utilities may be needed to execute commands found in [execution configuration files](#yaml-configuration-format). - -Additionally, the command requires a running instance (local or remote) of the target Inferno Test Kit(s) -and their services. - -### Execution - -For basic execution via the command line, the CLI command takes the target [execution configuration file](#yaml-configuration-format) as an input. - -``` -bundle exec inferno execute_script path/to/script_config_file.yaml -``` - -By default, the execution targets the local running instance of Inferno, which must be up. A remote Inferno -instance can be targeted by passing the `-I` option with the url of the target running Inferno instance, -e.g., the public Inferno instance at `https://inferno.healthit.gov/suites`. - -#### Github Workflow Execution - +The `inferno execute_script` command is well-suited for use in CI/CD pipelines. To enable this use, Inferno provides a standard Github Workflow file that finds script configuration files defined under the `execution_scripts` directory and runs each of them against a freshly -created Inferno instance running the test kit defined in the repository. - -#### Environment Variables - -The following environment variables control specific script behavior and can be set prior to executing the -`inferno execute_script` command: - -| Variable | Default | Description | -|---|---|---| -| `POLL_INTERVAL` | `5` | Seconds between status checks while a run is active | -| `DEFAULT_POLL_TIMEOUT` | `30` | Default timeout in seconds for each polling step | -| `COMPARE_NORMALIZE` | `true` | Normalize UUIDs and base64 values before comparing results (`-n`) | -| `COMPARE_MESSAGES` | `true` | Include the `messages` array in result comparison (`-m`) | -| `COMPARE_RESULT_MESSAGE` | `true` | Include `result_message` in result comparison (`-r`) | -| `ACTIONS_FILE` | _(derived from script path)_ | Override the YAML config file path when sourcing the script as a library | - -#### Example Scripts to Execute - -The following Test Kits include scripts that can be executed. To execute any of them -1. Checkout the repository locally and set it up to be run in developer mode -2. Locate the target script configuration file under the `execution_scripts` directory - and run `bundle exec inferno execute_script execution_scripts/path/to/script_config_file.yaml` pointing to the desired configuration file. - -- [**Inferno Template**](https://github.com/inferno-framework/inferno-template): - New Inferno Test Kits created from the [inferno-template repository](https://github.com/inferno-framework/inferno-template) - or the [`inferno new` CLI command](/docs/getting-started/inferno-cli.html#inferno-commands) - include a simple execution configuration file for the example test suite included in the - template. -- [**Inferno Core**](): TODO -- [**(g)(10) Certification**](): TODO -- [**DaVinci PAS**](): TODO - -### Creating Script Configuration Files - -Script configuration files define two things -- The Inferno session(s) executed, including any suite options or presets to be used -- The individual steps to take including - - When to take the step in the form of an Inferno session status and last executed test. - - What action to take in the form of an Inferno `session start_run` CLI command or general shell command. - -#### Process - -Execution scripts are typically used to automate how a user would interact with the Inferno -UI. Starting from a UI execution makes planning the sessions and steps of an execution -script simple. When mapping a UI execution onto a execution script, each session created -(typically one, sometimes two if running Inferno suites against each other) will correspond -to a `sessions` entry, and each interaction with the Inferno UI (to start a test run, or respond -to a wait dialog) will correspond with a `steps` entry. - -The tricky piece is to determine the specific values to put in the script configuration file for internal -identifiers like -- suite option keys and values -- input keys and values -- test full ids for use in specifying the last test - - -To use the script and the workflow, -1. Create a YAML config file (see format below) in the `execution_script/` - directory. -2. Make sure that the containing test kit is running in Ruby developer mode, including - both the background services and the test kit itself. -3. From the root test kit directory, run the script - -```bash -bash execution_scripts/session_runner.sh execution_scripts/my_suite.yaml -``` - -4. The script will print to the terminal details on its execution, including matched - test states and executed actions. If the test reaches an unmatched state or never - reaches an `END_SCRIPT` action, it will fail, providing details on the issue so - that updates can be made to fix the problem. -5. Once the script completes for the first time, unless you manually defined an expected result - file (`my_suite_expected.yaml`), the command will generate the expected file based on - the results of the executed session. Review the results to ensure that they represent - the expected results. If they don't adjust the script yaml file to correct any issues. -6. Run the script once more locally to make sure that the results of the next run match - the expected. If they don't there may be non-deterministic behavior or other problematic - details in you test kit (see debugging details below). -7. Commit the script yaml file and the expected results json file to the test kit repository - and now the execution will be checked whenever changes are made to the test kit. - -#### YAML Configuration Format - -The config file has two top-level keys: `sessions` and `steps`. - -```yaml -sessions: - - suite_id: my_suite # required - name: my_session # optional; used to reference this session in step/tokens - preset_id: my-preset # optional - suite_options: # optional - option_key: option_value - expected_results_file: expected.json # optional; relative to the yaml file - # default: _expected.json - -steps: # ordered; first matching step wins - - status: done # session status to match: created, done, or waiting - last_test: my_suite-Group01-Test01 # optional; full id of the last test executed (exact match) - # must be present unless the status is created, in which case it must be absent - session: my_session # optional; in multi-session runs, scopes step to this session - start_run: # structured alternative to 'command' for inferno session start_run calls - runnable: "1.01" # optional; short_id or full id of the group/test to run (-r flag) - inputs: # optional; key/value map of inputs (-i flag) - input_name: "value" - timeout: 300 # optional; seconds before polling this step times out. If not provided - # falls back to the session_runner.sh default (30s or the value of the - # DEFAULT_POLL_TIMEOUT environment variable) - next_poll_session: other_session # optional; switch polling focus to the named session after action - state_description: "..." # optional; logged when the step is matched - action_description: "..." # optional; logged when the step is matched -``` - -Each step must have either a `start_run:` block or a `command:` string (see below). - -#### Step: `command` vs `start_run` - -Use `start_run:` when you want to launch a test run using the standard -`bundle exec inferno session start_run` command. For anything else (e.g. making an -external HTTP request, running a shell command, or passing data to a waiting test), -use the `command:` string field. The `command` string is evaluated by `eval`, so -full shell expressions are supported. - -```yaml -steps: - # Start a run using the structured start_run block - - status: created - start_run: - runnable: "1" - inputs: - url: "https://example.com/fhir" - - # Start a run using a raw command string - - status: done - last_test: my_suite-Group01-Test01 - command: "bundle exec inferno session start_run '{session_id}' -r 2" - - # Perform an external action to satisfy a waiting test - - status: waiting - last_test: my_suite-Group01-wait_test - command: "curl -s '{wait_outputs.redirect_url}'" - - # Execute a defined ruby script - - status: waiting - last_test: my_suite-Group01-launch_test - command: "bundle exec ruby execution_scripts/launch_helper.rb {wait_output.launch_url}" -``` - -### Template Tokens - -Template tokens in `command` strings (and `start_run` input values) are substituted -at runtime: - -| Token | Description | -|---|---| -| `{session_id}` | ID of the current session (single-session only) | -| `{session_id.NAME}` | ID of the session with the given name (required in multi-session) | -| `{result_message}` | The wait result message from the current session | -| `{NAME.result_message}` | Wait result message from the named session (triggers a status fetch) | -| `{wait_outputs.KEY}` | Value of the named wait output from the current session | -| `{NAME.wait_outputs.KEY}` | Named wait output from a different session (triggers a status fetch) | - -### Special `command` Values - -| Value | Behaviour | -|---|---| -| `END_SCRIPT` | Stop execution for this session and proceed to result comparison | -| `NOOP` | Match the step but take no action; keep polling with the current timeout | - -### Automatic Status Handling - -You do not need steps for every possible status. The script applies these defaults -automatically: - -| Status | Default behaviour | -|---|---| -| `running`, `queued`, `cancelling` | Poll again | -| `done` (no step matched) | Warn, run result comparison, exit non-zero | -| `waiting` (no step matched) | Cancel the current run and resume polling | -| `created` (no step matched) | Warn, run result comparison, exit non-zero | - -### Expected Results - -After all runs complete, the script compares results against an expected results JSON -file. If the file does not yet exist, the current results are written to it, creating -a baseline for future runs. The expected file path is resolved as follows: - -- If `expected_results_file` is set in the YAML, that path is used (relative to the - YAML file). -- For single-session configs with no explicit path: `_expected.json`. -- For multi-session configs with no explicit path: `__expected.json`. - -### Multi-Session Example - -Some test flows require two sessions to run in parallel — for example, a client test -that makes requests to a server session. Use multiple entries under `sessions:` and -scope steps with `session:` and `next_poll_session:` to coordinate between them. - -```yaml -sessions: - - suite_id: my_server_suite - name: server - - suite_id: my_client_suite - name: client - -steps: - # Start the server session first - - status: created - session: server - start_run: - - # Once server is waiting, start the client session - - status: waiting - session: server - next_poll_session: client - command: "NOOP" - - - status: created - session: client - start_run: - inputs: - server_url: "https://example.com" - - # When client is done, resume the server - - status: waiting - session: client - next_poll_session: server - command: "curl -s '{wait_outputs.redirect_url}'" - - - status: done - session: server - command: "END_SCRIPT" -``` \ No newline at end of file +created Inferno instance running the test kit defined in the repository. TODO: pointer \ No newline at end of file diff --git a/docs/getting-started/inferno-cli.md b/docs/getting-started/inferno-cli.md index a992a24..aa85d31 100644 --- a/docs/getting-started/inferno-cli.md +++ b/docs/getting-started/inferno-cli.md @@ -21,7 +21,8 @@ The commands available include: | Command | Description | |--------------|-------------| | `inferno console` | Starts an interactive console session with Inferno. More information can be found on the [Debugging Page](debugging.html#interactive-console). | -| `inferno execute` | Execute tests in the command line instead of web UI. | +| `inferno execute` | Execute tests in the command line. | +| `inferno execute_script` | Execute tests in the command line instead of web UI. | | `inferno evaluate IG_PATH [--options]` | Run the FHIR evaluator in the command line. Does not require `bundle exec`. | | `inferno help [COMMAND]` | Describes the available commands, or one specific command if specified. | | `inferno migrate` | Runs database migrations. | @@ -30,7 +31,7 @@ The commands available include: | `inferno requirements help SUBCOMMAND` | View details on the requirements subcommands. | | `inferno services start` | Starts the background services (nginx, Redis, etc.) for Inferno. | | `inferno services stop` | Stops the background services for Inferno. | -| `inferno session SUBCOMMAND [...ARGS]` | Manages Inferno test sessions programmatically. The available subcommands are `create`, `start_run`, `status`, `cancel_run`, `data`, `results`, and `compare`. | +| `inferno session SUBCOMMAND [...ARGS]` | Manages Inferno test sessions. The available subcommands are `create`, `start_run`, `status`, `cancel_run`, `data`, `results`, and `compare`. | | `inferno session help SUBCOMMAND` | View details on the session subcommands. | | `inferno start` | Starts Inferno web UI. Do not use `bundle exec` if the Inferno Core version is prior to 0.4.39. | | `inferno suite SUBCOMMAND [...ARGS]` | Performs suite-based operations. The available subcommands are `describe`, `help`, `input_template`, and `lock_short_ids`.| @@ -69,7 +70,6 @@ It will create a `my_test_kit.gemspec` file, and set the names of any authors su using the `-a` option, and set the name, summary and description values to your Test Kit name. You should manually inspect this file if you want to [publish or distribute the Test Kit](/docs/distributing-tests.html). - The `--implementation-guide` or `-i` option will look at an absolute file path if its prefixed by `file:///` or lookup the name in the [official FHIR package registry](https://packages.fhir.org/). ## Manage Sessions @@ -88,7 +88,7 @@ for Thor and Bash respectively). Subcommands include: -### **`create SUITE_ID`** +- **`create SUITE_ID`** Create a new test session for the given suite. @@ -104,7 +104,7 @@ Subcommands include: bundle exec inferno session create my_test_suite -I https://inferno.healthit.gov/suites -p my_preset ``` -### **`start_run SESSION_ID`** +- **`start_run SESSION_ID`** Initiate a test run on an existing session. @@ -132,7 +132,8 @@ Subcommands include: [`GET /test_runs/{test_run_id}`](https://inferno-framework.github.io/inferno-core/api-docs/#/Test%20Run/get_test_runs__test_run_id_) API for details on the output. Additionally, When `status` is `waiting`, fields `wait_outputs` and `wait_result_message` are added to this base object using the - `outputs` and `result_message` fields from the result of the waiting test (last test that was executed). + `outputs` and `result_message` fields from the result of the waiting test (last test that was executed) + which can be used to script the continuation of the session. ```sh bundle exec inferno session status abc123 -I https://inferno.healthit.gov/suites @@ -160,7 +161,7 @@ Subcommands include: **Options**: - `-I`, `--inferno_base_url`: URL of the target Inferno service. - **Output**: JSON array of `{ name, value }` objects representing the session's current inputs. + **Output**: JSON array of objects with `name` and `value` keys representing the session's current inputs. See the output of the [`GET /test_sessions/{test_session_id}/results`](https://inferno-framework.github.io/inferno-core/api-docs/#/Result/get_test_sessions__test_session_id__results) API for details on the output. @@ -194,9 +195,8 @@ Subcommands include: - `-s`, `--expected_results_session`: Session ID on the same server whose results serve as expected. - `-m`, `--compare_messages`: Also compare per-test messages (not only pass/fail result). Default: `false`. - `-r`, `--compare_result_message`: Also compare each test's `result_message` string. Default: `false`. - - `-n`, `--normalize`: Before comparing strings, replace dynamic values with stable - placeholders: ISO 8601 datetimes → ``, UUIDs → ``, random base64 - strings → ``. Default: `false`. + - `-n`, `--normalized_strings`: List of literal strings and regexes which will be used to normalize + strings prior to comparison. For literal strings, the URL-encoded version will be replaced as well. **Output**: JSON with a top-level `matched` boolean and a `results` array. Each entry contains `id`, `type` (`Compared` / `Missing` / `Additional`), `matched`, @@ -213,9 +213,34 @@ Subcommands include: bundle exec inferno session compare abc123 -I https://inferno.healthit.gov/suites -f expected.json -m -n ``` - ## Running a Test Kit in Command Line +Inferno provides two CLI options for executing Inferno tests via the command line: +1. The `inferno execute_script` CLI command for complex executions that can involve multiple runs + and sessions as well as scripting of external systems. +2. The `inferno execute` CLI command for simple executions that don't involve any interaction beyond + starting the test runs. + +### Complex Scripted Execution + +The `inferno execute_script` CLI command can be used to execute complex Inferno runs from the command line +which are specified in a [script configuration file](/docs/advanced-test-features/scripting-execution#creating-script-configuration-files). +By default, scripts are executed against the local running Inferno instance (both background services +and the test kit must be running). Execution can be performed against a remote Inferno instance by providing +the root Inferno url using the `-I` option. + +For example, the following command would run the script defined in file `g10.yaml` on the instance of +Inferno deployed to [inferno.healthit.gov](https://inferno.healthit.gov): +``` +bundle exec inferno execute_script g10.yaml -I https://inferno.healthit.gov +``` + +See [Scripting Suite Execution](/docs/advanced-test-features/scripting-execution) for details on how to +define script configuration files and [CI / CD Usage](/docs/ci-cd-usage.md) for how to integrate scripts +into development pipelines. + +### Simple Execution + The `inferno execute` command allows you to execute [runnables](https://inferno-framework.github.io/docs/writing-tests/#test-suite-structure) from a Test Kit in the shell. The command requires: - The Test Kit as the working directory. @@ -224,7 +249,6 @@ from a Test Kit in the shell. The command requires: - All inputs and any [Suite Options](https://inferno-framework.github.io/docs/advanced-test-features/test-configuration.html#suite-options) necessary for execution. - Optionally, the short ids of select Tests and Test Groups you want to run. - For example, to run the [US Core Test Kit](https://github.com/inferno-framework/us-core-test-kit) v3.1.1 Single Patient API group, you would: From 1197283a2fcc93ccfd7f9b3c46dee237808cd0e4 Mon Sep 17 00:00:00 2001 From: Karl Naden Date: Tue, 31 Mar 2026 23:30:02 -0400 Subject: [PATCH 04/11] completed initial documentation pass --- .../scripting-execution.md | 349 +++++++++++++----- docs/ci-cd-usage.md | 38 +- docs/getting-started/inferno-cli.md | 47 ++- .../repo-layout-and-organization.md | 6 +- 4 files changed, 320 insertions(+), 120 deletions(-) diff --git a/docs/advanced-test-features/scripting-execution.md b/docs/advanced-test-features/scripting-execution.md index 27fb4c4..f5c6c72 100644 --- a/docs/advanced-test-features/scripting-execution.md +++ b/docs/advanced-test-features/scripting-execution.md @@ -13,45 +13,175 @@ Inferno supports the scripting of test suite execution through the `inferno exec [CLI command](/docs/getting-started/inferno-cli). This command takes as input a [configuration file](#yaml-configuration-format) that tells Inferno when and how to perform execution steps both inside and outside of Inferno. The results of the session are compared -against the results of a known-good execution to help identify any regressions. The script +against the results of a known-good execution to identify any regressions. The script can be executed against the local Inferno instance or a specific remote instance. These scripts can be built into [CI/CD Pipelines](/docs/ci-cd-usage). ## Execution -See the [CLI documentation](/docs/getting-started/inferno-cli#complex-scripted-execution) for details -on executing scripts from the command line. See [CI / CD Usage](/docs/ci-cd-usage.md) for how to -execute scripts within development workflows. +Three standard options for executing scripts include: +- Executing a specific script using the `execute_script` CLI: See the [CLI documentation](/docs/getting-started/inferno-cli#complex-scripted-execution) + for details on the command. +- Executing scripts defined in the test kit's `execution_scripts` directory using + the `bundle exec rake execute_scripts:run_all` command. Two environment variables may be set in addition: + - `INFERNO_BASE_URL=`: set to run the scripts against a specific Inferno host. + If not provided, then the local Inferno host will be used. + - `FILTER=execution_scripts//*.yaml`: provide a different filter to use to identify + the scripts to execute. If not provided, the default is `execution_scripts/**/*.yaml`. +- Executing all scripts defined in the test kit's `execution_scripts` directory within GitHub, + either in response to a commit or a manual trigger. See [CI / CD Usage](/docs/ci-cd-usage.md) + for additional details. ## Creating Script Configuration Files At a high-level, the script configuration file contains instructions for - The session(s) to create, including any suite options and presets to use -- The steps to take to execute the script, each including an action, when to perform it, and when there - are multiple Inferno sessions involved which session will act next. -- Normalizations of the results to make sure that executions are comparable across runs and Inferno instances. +- The steps to take to execute the script, each including an action, when to perform it, + and which session will act next if there are multiple Inferno sessions involved. +- Normalizations of the results to make sure that executions are comparable across runs + and Inferno instances. -The easiest way to think about creating a script configuration file is that it mirrors the manual steps taken -in the UI to execute an Inferno test session of a particular suite against a specific system. Go through -the manual execution steps and note each time there is an interaction with the Inferno UI or a state in Inferno that -triggers an interaction with the tested system: each of these will become a step within the script. +The easiest way to think about creating a script configuration file is that it mirrors +the manual steps taken in the UI to execute an Inferno test session of a particular suite +against a specific system. Go through the manual execution steps and note each time there +is an interaction with the Inferno UI or a state in Inferno that triggers an interaction +with the tested system: each of these will become a step within the script. The rest of this section provides details on how to turn the manual execution into a script file. -See the documentation at the top of the [`execute_script.rb`](https://github.com/inferno-framework/inferno-core/blob/main/lib/inferno/apps/cli/execute_script.rb) file for a complete format reference. +See the documentation at the top of the [`execute_script.rb`](https://github.com/inferno-framework/inferno-core/blob/main/lib/inferno/apps/cli/execute_script.rb) +file for a complete format reference. ### Sessions Each script configuration file will indicate a list (sequence) of one or more Inferno sessions to create under the `sessions:` top-leve key. Each sequence includes - **Suite** (`suite:`): *Required* - the internal id for the suite or the title selected in the UI. -- **Name** (`name:`): *Optional* - a short key used to identify the session in steps. Required when multiple sessions - are defined. +- **Name** (`name:`): *Conditional* - a short key used to identify the session in steps. + Required when multiple sessions are defined, otherwise optional. - **Preset** (`preset:`): *Optional* - the internal id or title of a preset to use for the session. - **Suite Options** (`suite_options:`): *Optional* - pairs of `key: value` entries each corresponding to a suite option. The key can be either the internal id or the title from the UI. The value can be either the internal value or the label from the UI. -- **Expected Results** (`expected_results_file:`): *Optional* - the path to the file containing the expected results, - relative to the executed yaml file name. If not provided defaults to _expected.json. + +### Result Comparison Config + +After completing the script, the results of the executed session will be compared for each +session to the expected results recorded in the session's expected results file. Because +direct comparison of results from different runs across different systems (e.g., local vs +deployed) will not always be directly comparable, two mechanisms are provided to tweak +the actual and/or expected results to facilitate like-to-like comparison. + +All configurations around results comparison are nested under the `comparison_config:` top-level +key. + +#### Normalized String + +Before comparing the actual results to the expected results, Inferno will replace strings +indicated in the `normalized_strings:` key. This prevents these values, which are expected +to be different between runs and across different Inferno hosts, from triggering an execution +failure. Each entry in the sequence can be +- A raw string: the exact string and its URL-encoded form are replaced with the string `` +- A raw regex: strings matching the regex are replaced with the string `` +- An object with two keys that provides specific replacement strings (useful for debugging): + - `patterns:` a sequence of strings or regexes used to identify the strings to replace, + which behave like the raw forms. (for single-entry lists, the `pattern:` can be used + instead with the single value) + - `replacement:` the string that replaces the patterns (e.g., ``) + +For example, the following configuration normalizes, the inferno host, the reference server +url, pkce code challenges, and UUIDs, each with a specific replacement string +for ease of debugging. + +``` +comparison_config: + normalized_strings: + - replacement: + patterns: + - http://localhost:4567/inferno # local inferno core ruby + - http://localhost:4567 # local ruby + - http://localhost # local docker + - https://inferno.healthit.gov/suites # prod + - https://inferno-qa.healthit.gov/suites # qa + - replacement: + patterns: + - https://inferno.healthit.gov/reference-server # prod reference server + - https://inferno-qa.healthit.gov/reference-server # qa reference server + - replacement: code_challenge= + pattern: /code_challenge=[A-Za-z0-9+\/=_-]{43}/ + - replacement: + pattern: /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/i +``` + +#### Expected Results File and Alternates + +When comparing results, Inferno first identifies the file that contains the expected +results. These are formatted as the output of the `session results` CLI command. Note +that Inferno will generate the target file the first time a script is run. + +By default, the expected results file will be `_expected.json`. For +example, for a script configuration file named `g10.yaml`, the default expected file +would be `g10_expected.json`. A different file to use as the default can be +specified in the `expected_results_file:` key. Inferno will interpret relative paths +as relative to the script configuration file's directory. + +Some suites will expected differences between session results that go beyond string +normalizations. For example, suites that verify a server's TLS setup will fail +when evaluating a local server without TLS setup, which may be the expected result +when running in a CI/CD environment. For this situation, an alternate expected +results file can be indicated for certain executions. These are indicated under +the `alternate_expected_files:` key which will contain a list (sequence) of entries +each with the following keys: +- **Alternate File** (`file:`): *Required* - the file containing the expected results, relative + to the script configuration file's directory. +- **Condition** (`when:`): *Required* - a list of conditions which must all match. Each condition + contains keys for + - **Target field** (`field:`): *Required* - the value to check. Can be one of + - `inputs.` to check the value of an input used during the session. + - `configuration_messages` to check the `messages` field of entries (one must match) + in `test_suite.configuration_messages`. + - `inferno_base_url` to check the Inferno host. + - **Match Criteria** (`matches:` or `not_matches`): *Required* - a regex to use to check the value(s) + either match or don't match. + +For example, the following configuration tells Inferno to use an alternate expected file +the expects terminology failures when +1. It isn't running on a public Inferno host, and +2. There is a configuration message indicating that there is a terminology build problem. + +``` +comparison_config: + expected_results_file: g10_results_expected.json + alternate_expected_files: + - file: g10_no_terminology_expected.json + when: + - field: inferno_base_url + not_matches: inferno(-qa)?\.healthit\.gov + - field: configuration_messages + matches: ^There is a problem with the terminology resources +``` + +If there are multiple matching `alternate_expected_file:` entries, the one listed first will be used. + +When there are multiple sessions defined, both the `expected_results_file:` and `alternate_expected_files:` +keys must be nested under a `sessions:` object with keys corresponding to the session name. If the +above example was in a multi-session script associated with the session named `g10`, it would +look like this: + +``` +comparison_config: + sessions: + g10: + expected_results_file: g10_results_expected.json + alternate_expected_files: + - file: g10_no_terminology_expected.json + when: + - field: inferno_base_url + not_matches: inferno(-qa)?\.healthit\.gov + - field: configuration_messages + matches: ^There is a problem with the terminology resources +``` + +Note that the `normalized_string:` key never appears under the `sessions:` key. ### Steps @@ -62,7 +192,7 @@ the `steps:` top-level key. There will be at least 2 in every script: While steps are executed when they are matched by Inferno's state, only one step is executed at a time and loops are detected and not allowed. By convention, steps should be listed in the order they will -be executed in practice. +be executed during script execution. Each step consists of keys pertaining to the following details - When to take the step @@ -78,80 +208,87 @@ Each step indicates when it will be take through three keys - `done` when a test run has completed - `waiting` when a test run is in progress but waiting for external input - `created` when a test session has been created, but no runs started yet -- `last_completed:` *Required* unless the status is `created` - specifies a runnable in the form of an +- `last_completed:` *Conditional* - specifies a runnable in the form of an internal id or short numeric id from the UI. - for `done`, will be the test, group, or suite that was executed when the run was started - for `waiting`, will be the test that initiated the wait - for `created`, will be omitted (nothing executed yet) -- `session:` *Required* when multiple sessions, otherwise omitted - the name (not suite id) of the - session to poll for the next step. +- `session:` *Conditional* - when multiple sessions, the name (not suite id) of the + session to match on, otherwise omitted. -Additionally, the optional `state_description:` key can be used to provide documentation describing the -state. This has no functional effect, but is echoed during script execution for debugging purposes. +Additionally, the optional `state_description:` key can be used to provide documentation +describing the state. This has no functional effect, but is echoed during script execution +for debugging purposes. -Note that Inferno can be have other status (e.g., `running` and `cancelled`), but these do not -represent stable states where Inferno is waiting for an action, so they will never be used in -scripts. +Note that Inferno can be have other status (e.g., `running` and `cancelled`), but these +do not represent stable states where Inferno is waiting for an action, so they will never +be used in scripts. -Conversely, Inferno needs to know what to do in any stable state that is reached. If no step matches -a `done` or `created` state that Inferno reaches, then the script will end with an error (results not checked). -If no step matches a `waiting` state, then Inferno will cancel the current run (ending the wait) and attempt -to continue with the next matching `done` state. +Conversely, Inferno needs to know what to do in any stable state that is reached. If no +step matches a `done` or `created` state that Inferno reaches, then the script will end +with an error (results not checked). If no step matches a `waiting` state, then Inferno +will cancel the current run (ending the wait) and attempt to continue with the next +matching `done` state. #### What action to take -There are three kinds of actions that can be take by a step each specified by a different key. Only -one can be defined for each step: -- `start_run:` Used to execute a test, group, or suite on one of the sessions started by this script. The following - sub keys are used. - - `runnable:` *Required* - the test, group, or suite to execute, specified as a short ID from the UI or an internal id. - - `inputs:` *Optional* - key-value pairs indicating inputs to be merged into those already stored in the session (from the - preset or previous run inputs or outputs). Each key must be the internal name for the input (from the UI, use the yaml or json view of the inputs to find the internal name). - - `session:` *Required* when the script has multiple sessions, otherwise omitted - the name (not the suite) of the session on which the run will be executed. -- `command:` Specifies a shell command to execute. For example, a `curl` command to navigate to an attestation URL or - an external script that automates manipulation of the tested system. +There are three kinds of actions that can be take by a step each specified by a different +key. Only one can be defined for each step: +- `start_run:` Used to execute an Inferno test, group, or suite on one of the sessions + started by this script. The following sub keys are used. + - `runnable:` *Required* - the test, group, or suite to execute, specified as a short ID + from the UI or an internal id. + - `inputs:` *Optional* - key-value pairs indicating inputs to be merged into those already + stored in the session (from the preset or previous run inputs or outputs). Each key must + be the internal name for the input (from the UI, use the yaml or json view of the inputs + to find the internal name). + - `session:` *Conditional* - when multiple sessions are defined the name (not the suite) + of the session on which the run will be executed, otherwise omitted. +- `command:` Specifies a shell command to execute. For example, a `curl` command to navigate + to an attestation URL or an external script that automates manipulation of the tested system. + See [these considerations](#defining-executing-and-securing-complex-commands) + for creating, executing, and securing scripts with complex commands. - `action:` Script-specific actions to bridge between steps. Defined actions include: - - `END_SCRIPT` used on the distinguished final step. The execution will end successfully after this step. - - `NOOP` used on steps that don't require an action, but indicate that another step is unblocked. See details in - the discussion of multiple session scripts below. - - `WAIT` used on transitory `waiting` states that are expected to end without external action. Note that unlike other - actions, this one does not cause Inferno to break the polling loop, so it can be matched multiple times (not a loop) - but does not change next step details (see next subsection). - -The specifics of `start_run` and `command` actions may depend on dynamic session details. The following template -tokens are allowed in fields of these actions which will be replaced with session-specific values at execution time. For -most, there is both a named version for scripts with multiple sessions and a raw version for those with only a single session: + - `END_SCRIPT` used on the distinguished final step. The execution will end successfully + after this step. + - `NOOP` used on steps that don't require an action, but indicate that another step is + unblocked. See details in the [discussion of multiple session scripts](#scripts-with-multiple-sessions) + below. + - `WAIT` used on transitory `waiting` states that are expected to end without external + action. Note that unlike other actions, this one does not cause Inferno to break the + polling loop, so it can be matched multiple times (not a loop) but does not change + next step details (see the details on [finding the next step](#how-to-look-for-the-next-step) + below). + +The specifics of `start_run` and `command` actions may depend on dynamic session details. +The following template tokens are allowed in fields of these actions which will be replaced +with session-specific values at execution time. For most, there is both a named version for +scripts with multiple sessions and a raw version for those with only a single session: - **Session Id**: `{session_id}` (`{.session_id}` when multiple sessions) -- **Result Message** (from waiting tests only): `{result_message}` (`{.result_message}` when multiple sessions) -- **Output Value** (from waiting tests only): `{wait_outputs.KEY}` (`{.wait_outputs.KEY}` when multiple sessions) +- **Result Message** (from waiting tests only): `{result_message}` + (`{.result_message}` when multiple sessions) +- **Output Value** (from waiting tests only): `{wait_outputs.KEY}` + (`{.wait_outputs.KEY}` when multiple sessions) - **Inferno URL**: `{inferno_base_url}` (no session-specific version - always the same for all sessions) -Additionally, the optional `action_description:` key can be used to provide documentation describing the -action. This has no functional effect, but is echoed during script execution for debugging purposes. +Additionally, the optional `action_description:` key can be used to provide documentation +describing the action. This has no functional effect, but is echoed during script execution +for debugging purposes. #### How to look for the next step -After taking an action, script execution returns to a polling mode where it checks the status of a session -to determine if it has reached a stable state to match against to identify the next step. The previous step -can control two aspects of this process using the following keys: -- `timeout:` *Optional* - specifies a number of sections to wait before aborting by canceling the run. The default is 30 seconds. - After cancelling, Inferno will try to continue matching other states, but will still ultimately indicate failure. -- `next_poll_session:` *Optional* - the name (not suite id) of the session to poll for the next step. Inferno only ever polls - one session at a time, so in scripts with multiple sessions, it must be told which session to poll next. - The first session configured is used to start. After executing a step, Inferno will either poll the session indicated - in this key or the session polled after the previous step if none is specified. - -#### Normalizations for Results Comparison - -After completing the script, the results of the executed session will be compared for each session to the expected -results recorded in the session's expected results file. When that comparison is performed, -Inferno will replace strings indicated in the `normalized_strings:` top-level key to normalize values that -are expected to be different between runs and across different Inferno hosts. Each entry in the sequence can be -- A raw string: the exact string and its URL-encoded form are replaced with the string `` -- A raw regex: strings matching the regex are replaced with the string `` -- An object with two keys that provides specific replacement strings (useful for debugging): - - `patterns:` a sequence of strings or regexes used to identify the strings to replace, which behave like the raw forms. (for single-entry lists, the `pattern:` can be used instead with the single value) - - `replacement:` the string that replaces the patterns +After taking an action, script execution returns to a polling mode where it checks the status +of a session to determine if it has reached a stable state to match against to identify the +next step. The step that was just executed can control two aspects of this process using +the following keys: +- `timeout:` *Optional* - specifies a number of sections to wait before aborting by canceling + the run. The default is 120 seconds. After cancelling, Inferno will try to continue matching + other states, but will still ultimately indicate failure for the script as a whole. +- `next_poll_session:` *Optional* - the name (not suite id) of the session to poll for the next + step. Inferno only ever polls one session at a time, so in scripts with multiple sessions, + it must be told which session to poll next. The first session configured is used to start. + After executing a step, Inferno will either poll the session indicated in this key or the + session polled after the previous step if none is specified. ## Execution Process and Output @@ -183,7 +320,7 @@ To start runs, Inferno uses the [`inferno session start_run` CLI](/docs/getting- ### Check Results -For each session in the script, Inferno will compare the results in the expected results file (see [sessions configuration](#sessions)) to the results of the completed session execution. To perform the comparison, it will use the [`inferno session compare` CLI](/docs/getting-started/inferno-cli#manage-sessions), which involves the following steps: +For each session in the script, Inferno will compare the results in the expected results file (see [Expected Results File and Alternates](#expected-results-file-and-alternates)) to the results of the completed session execution. To perform the comparison, it will use the [`inferno session compare` CLI](/docs/getting-started/inferno-cli#manage-sessions), which involves the following steps: - normalize the expected and actual results using the configured [`normalized_strings`](#normalizations-for-results-comparison) - match individual result entries using the runnable id (`test_id`, `test_group_id`, or `test_suite_id`) - compare the matched result entries, looking at the status (e.g., pass, fail), the result message, and the @@ -197,7 +334,10 @@ they will be unique for each script execution. When the target expected results file does not exist it will be created using the output of the [`inferno session results` CLI](/docs/getting-started/inferno-cli#manage-sessions). The `execute_script` -CLI will exit with an error status in this case. +CLI will exit with an error status in this case. After creation of a expected results +file in this way, users should review the results using the file and/or the Inferno +UI to make sure that they are the expected results as subsequent script executions +will perform the comparison using that file as the expected results. ## Scripts with Multiple Sessions @@ -214,24 +354,55 @@ Considerations to keep in mind when creating script configuration files with mul where the tester must change over to the tab of the another session's UI. - Templates for dynamic values in executed actions (commands and Inferno runs) will use the session-specific form. -- `NOOP` actions can be used to help clarify sequencing. For example, if two sessions both have long running - tests that interact but complete in a particular order due to wait interactions, but don't require any external - interactions. You could poll the session that will end last and look only for the `done` state, which would require - `WAIT` actions on any `waiting` states so that Inferno recognizes them, or you could define states for each `waiting` - state with `NOOP` actions to make clear how the active execution passes back and forth. +- `NOOP` actions can be used to help clarify sequencing. For example, if two + sessions both have long running tests that interact but complete in a particular + order due to wait interactions, but don't require any external interactions. You + could poll the session that will end last and look only for the `done` state, + which would require `WAIT` actions on any `waiting` states so that Inferno + recognizes them, or you could define states for each `waiting` state with `NOOP` + actions to make clear how the active execution passes back and forth. + +## Defining, Executing, and Securing Complex Commands + +The `command` action option allows execution of arbitrary logic as a part of running a script. +The executed command can be as simple as a http request to trigger the end of an Inferno wait +dialog to a complex scripted interaction with the tested system, for example visiting and +manipulating the UI of the Inferno FHIR Reference Server using Selenium. The following +are important considerations when using this feature: +- The executed logic must be available to the local Inferno instance. If distributed with a test + kit for use in CI/CD pipelines, this will mean including external scripts in the test kit. For + additional gems required only for scripts, it is recommended to add the additional gems either + - As a development dependency in the gemspec, or + - Within the `:test` and `:development` sections of the Gemfile and handle `LoadError` exception + explicitly to warn users that use the script that they need to add additional gem dependencies. +- Scripts that use `command` actions must be executed with the `--allow-commands` option of the + `execute_script` CLI enabled. This guard forces users to opt-in to arbitrary commands. +- Be careful executing scripts that have commands which you have not reviewed. ## Example Scripts -TODOThe following Test Kits include scripts that can be executed. To execute any of them +The following Test Kits include scripts that can be executed. To execute any of them 1. Checkout the repository locally and set it up to be run in developer mode -2. Locate the target script configuration file under the `execution_scripts` directory - and run `bundle exec inferno execute_script execution_scripts/path/to/script_config_file.yaml` pointing to the desired configuration file. +2. Either + - Run all the defined scripts under the by executing `bundle exec rake execute_scripts:run_all`, or + - Locate the a single target script configuration file under the `execution_scripts` directory + and run `bundle exec inferno execute_script execution_scripts/path/to/script_config_file.yaml` + pointing to the desired configuration file. - [**Inferno Template**](https://github.com/inferno-framework/inferno-template): New Inferno Test Kits created from the [inferno-template repository](https://github.com/inferno-framework/inferno-template) or the [`inferno new` CLI command](/docs/getting-started/inferno-cli.html#inferno-commands) - include a simple execution configuration file for the example test suite included in the - template. -- [**Inferno Core**](): TODO -- [**(g)(10) Certification**](): TODO -- [**DaVinci PAS**](): TODO \ No newline at end of file + include a [simple execution configuration file](https://github.com/inferno-framework/inferno-template/blob/main/execution_scripts/inferno_template_script.yaml) + for the example test suite included in the template. +- [**Inferno Core**](https://github.com/inferno-framework/inferno-core): Inferno Core contains + [execution scripts](https://github.com/inferno-framework/inferno-core/tree/main/execution_scripts) + for [sample development suites](https://github.com/inferno-framework/inferno-core/tree/main/dev_suites) + that demonstrate features of Inferno and the `execute_script` CLI. +- [**(g)(10) Certification**](https://github.com/onc-healthit/onc-certification-g10-test-kit): + The (g)(10) Certification test kit (and [US Core Core test kit](https://github.com/inferno-framework/us-core-test-kit/tree/main/execution_scripts)) + contains [scripts](https://github.com/onc-healthit/onc-certification-g10-test-kit/tree/main/execution_scripts) + that demonstrate one approach to scripting external interactions with tested systems, in this + case the Inferno Reference Server. +- [**Da Vinci CRD**](https://github.com/inferno-framework/davinci-crd-test-kit): + The Da Vinci CRD test kit includes both client and server tests and includes [execution scripts](https://github.com/inferno-framework/davinci-crd-test-kit/tree/main/execution_scripts) demonstrating + how multi-session scripts can be used to execute these tests against each other. \ No newline at end of file diff --git a/docs/ci-cd-usage.md b/docs/ci-cd-usage.md index 9280cf2..ed73300 100644 --- a/docs/ci-cd-usage.md +++ b/docs/ci-cd-usage.md @@ -8,14 +8,37 @@ section: docs # Using Test Kits in CI/CD Inferno provides two CLI options for executing Inferno tests via the command line: -1. The `inferno execute` CLI command for execution of a single suite, group, or test. -2. The `inferno execute_script` CLI command for more complex executions that can involve multiple runs +1. The [`inferno execute_script`](/docs/getting-started/inferno-cli.html#complex-scripted-execution) CLI command for more complex executions that can involve multiple runs and sessions as well as scripting of external systems. +2. The [`inferno execute`](/docs/getting-started/inferno-cli.html#simple-execution) CLI command for execution of a single suite, group, or test. + +## Using the `inferno execute_script` CLI for Complex Orchestration + +The [`inferno execute_script`](/docs/getting-started/inferno-cli.html#complex-scripted-execution) +command is well-suited for use in CI/CD pipelines. To enable this use, +the [Inferno Template Test Kit](https://github.com/inferno-framework/inferno-template) provides +a [standard Github Workflow file](https://github.com/inferno-framework/inferno-template/blob/main/.github/workflows/run_inferno_execution_scripts.yml) +that finds [script configuration files](/docs/advanced-test-features/scripting-execution.html#creating-script-configuration-files) +defined under the `execution_scripts` directory and runs each of them against a freshly +created Inferno instance running the test kit defined in the repository. This script is +trigged automatically on pull requests opened against the main branch and can also be +manually triggered. + +Key considerations when using this GitHub workflow: +- [Script configuration files](/docs/advanced-test-features/scripting-execution.html#creating-script-configuration-files) + defined anywhere under the `execution_scripts` directory will be identified and executed. +- All executed scripts are expected to complete successfully with the expected results. +- Scripts that end in `_with_commands.yaml` will be run with the `--allow-commands` flag of the + `execute_script` CLI set. Use that naming convention whenever your scripts include `command` + actions. +- Any `command` action scripts must be runnable on a `ubuntu-latest` using the contents of the + repository with Inferno's standard Ruby setup. You can modify the workflow if your commands + require additional setup to execute. ## Using the `inferno execute` CLI for Simple Execution To run Inferno using the [inferno -execute](/docs/getting-started/inferno-cli.html#running-a-test-kit-in-command-line) +execute](/docs/getting-started/inferno-cli.html#simple-execution) CLI command, follow these steps: 1. Setup a container or environment with Ruby and Docker @@ -26,7 +49,7 @@ command works on top of docker compose and daemonizes the processes for you. 5. Launch the system under test and confirm that it is ready to receive requests. 6. Select the Tests, Test Groups, and Test Suites you want in CI and run them with `inferno execute`. See -[Running a Test Kit in Command Line](/docs/getting-started/inferno-cli.html#running-a-test-kit-in-command-line) +[Running a Test Kit in Command Line](/docs/getting-started/inferno-cli.html#simple-execution) for more information. The command will exit with status 0 if the Inferno Summarizer deems it a pass. + You can run multiple inferno execute commands, but remember that each call represents @@ -38,10 +61,3 @@ Test Suite to pass. To see an example of these steps, see [this Github workflow file](https://github.com/inferno-framework/inferno-reference-server/blob/5e0d06ad5414efa93499fd3de093e29cf5e6d9d1/.github/workflows/inferno_ci.yml) which is used to test the Inferno Reference Server against the (g)(10) certification test kit. - -## Using the `inferno execute_script` CLI for Complex Orchestration - -The `inferno execute_script` command is well-suited for use in CI/CD pipelines. To enable this use, -Inferno provides a standard Github Workflow file that finds script configuration files -defined under the `execution_scripts` directory and runs each of them against a freshly -created Inferno instance running the test kit defined in the repository. TODO: pointer \ No newline at end of file diff --git a/docs/getting-started/inferno-cli.md b/docs/getting-started/inferno-cli.md index aa85d31..c7ed097 100644 --- a/docs/getting-started/inferno-cli.md +++ b/docs/getting-started/inferno-cli.md @@ -82,9 +82,9 @@ information and tools for the CI/CD use case. All commands leverage the [JSON web API](https://inferno-framework.github.io/inferno-core/api-docs/) to interact with Inferno. By default, these commands interact with the local running Inferno instance. Additionally, all session subcommands -support the `--inferno_base_url` (`-I`) option to specify the URL of a remote running Inferno instance to interact -with instead. All commands write JSON to stdout and exit `0` on success or `3` on error (`1` and `2` are reserved -for Thor and Bash respectively). +support the `--inferno_base_url` (`-I`) option to specify the URL of a remote running Inferno +instance to interact with instead. All commands write JSON to stdout and exit `0` on success or +`3` on error (`1` and `2` are reserved for Thor and Bash respectively). Subcommands include: @@ -94,8 +94,11 @@ Subcommands include: **Options**: - `-I`, `--inferno_base_url`: URL of the target Inferno service. - - `-o`, `--suite_options`: Suite options as `key:value` pairs (e.g., `--suite_options us_core_version:us_core_6 smart_app_launch_version:smart_app_launch_2`). - - `-p`, `--preset_id`: Apply a named preset when creating the session. + - `-o`, `--suite_options`: Suite options as `key:value` pairs (e.g., + `--suite_options us_core_version:us_core_6 smart_app_launch_version:smart_app_launch_2`). + Keys and values can use either the display name seen in the Inferno UI or the internal name. + - `-p`, `--preset`: Apply a named preset when creating the session. Both the title or the id + can be used to identity the target preset. **Output**: JSON representation of the created session, including the session `id` used by all subsequent subcommands. See the output of the [`POST /test_sessions`](https://inferno-framework.github.io/inferno-core/api-docs/#/Test%20Session/post_test_sessions) API for details on the output. @@ -110,13 +113,18 @@ Subcommands include: **Options**: - `-I`, `--inferno_base_url`: URL of the target Inferno service. - - `-r`, `--runnable`: Short or internal ID of a specific group or test to run. If omitted, the entire suite is run. - - `-i`, `--inputs`: Input values as `key:value` pairs (e.g., `--inputs url:https://example.com patient_id:123`). These merge with and override the session's previously stored inputs. + - `-r`, `--runnable`: Short or internal ID of a specific group or test to run. + If omitted, the entire suite is run. + - `-i`, `--inputs`: Input values as `key:value` pairs (e.g., + `--inputs url:https://example.com patient_id:123`). These merge with and override + the session's previously stored inputs. The internal name for the input must be used + as the key. This can be found from the [Inferno testing interface UI](docs/user-interface#main-test-interface) by looking at the json or yaml representation of + the inputs. **Output**: JSON representation of the created test run. See the output of the [`POST /test_runs`](https://inferno-framework.github.io/inferno-core/api-docs/#/Test%20Run/post_test_runs) API for details on the output. ```sh - bundle exec inferno session start_run abc123 -I https://inferno.healthit.gov/suites -r my_group \ + bundle exec inferno session start_run 7uLiz9qX4og -I https://inferno.healthit.gov/suites -r my_group \ -i url:https://example.com patient_id:123 ``` @@ -136,7 +144,7 @@ Subcommands include: which can be used to script the continuation of the session. ```sh - bundle exec inferno session status abc123 -I https://inferno.healthit.gov/suites + bundle exec inferno session status 7uLiz9qX4og -I https://inferno.healthit.gov/suites ``` - **`cancel_run SESSION_ID`** @@ -151,7 +159,7 @@ Subcommands include: run is active. ```sh - bundle exec inferno session cancel_run abc123 -I https://inferno.healthit.gov/suites + bundle exec inferno session cancel_run 7uLiz9qX4og -I https://inferno.healthit.gov/suites ``` - **`data SESSION_ID`** @@ -166,7 +174,7 @@ Subcommands include: API for details on the output. ```sh - bundle exec inferno session data abc123 -I https://inferno.healthit.gov/suites + bundle exec inferno session data 7uLiz9qX4og -I https://inferno.healthit.gov/suites ``` - **`results SESSION_ID`** @@ -181,7 +189,7 @@ Subcommands include: API for details on the output. ```sh - bundle exec inferno session results abc123 -I https://inferno.healthit.gov/suites + bundle exec inferno session results 7uLiz9qX4og -I https://inferno.healthit.gov/suites ``` - **`compare SESSION_ID`** @@ -210,7 +218,7 @@ Subcommands include: - `compared_results_.csv` — a CSV diff of the mismatched results ```sh - bundle exec inferno session compare abc123 -I https://inferno.healthit.gov/suites -f expected.json -m -n + bundle exec inferno session compare 7uLiz9qX4og -I https://inferno.healthit.gov/suites -f expected.json -m -n ``` ## Running a Test Kit in Command Line @@ -230,13 +238,13 @@ and the test kit must be running). Execution can be performed against a remote I the root Inferno url using the `-I` option. For example, the following command would run the script defined in file `g10.yaml` on the instance of -Inferno deployed to [inferno.healthit.gov](https://inferno.healthit.gov): +Inferno deployed to [inferno.healthit.gov](https://inferno.healthit.gov/suites): ``` -bundle exec inferno execute_script g10.yaml -I https://inferno.healthit.gov +bundle exec inferno execute_script g10.yaml -I https://inferno.healthit.gov/suites ``` See [Scripting Suite Execution](/docs/advanced-test-features/scripting-execution) for details on how to -define script configuration files and [CI / CD Usage](/docs/ci-cd-usage.md) for how to integrate scripts +define script configuration files and [CI / CD Usage](/docs/ci-cd-usage) for how to integrate scripts into development pipelines. ### Simple Execution @@ -293,9 +301,10 @@ bundle exec inferno execute --help for more information. -Please note that inferno execute is still low maturity and has limitations. Inferno -currently cannot support SMART launch tests or tests that receive client requests -via CLI execution. Some tests are also expected to [run as a group](https://inferno-framework.github.io/docs/writing-tests/properties.html#run-as-group), +The `execute` command does not support SMART launch tests or tests that receive client requests +via CLI execution. Use the [`execute_script`](#complex-scripted-execution) functionality +for these complex execution cases. Some tests are also expected to +[run as a group](https://inferno-framework.github.io/docs/writing-tests/properties.html#run-as-group), and may error or fail erroneously when run individually through CLI. ## Suite Subcommands diff --git a/docs/getting-started/repo-layout-and-organization.md b/docs/getting-started/repo-layout-and-organization.md index cf9ad85..d1b9f93 100644 --- a/docs/getting-started/repo-layout-and-organization.md +++ b/docs/getting-started/repo-layout-and-organization.md @@ -15,6 +15,8 @@ running `inferno new`, you will have a directory structure that looks something like this: ``` +├── .github +│   └── workflows ├── Dockerfile ├── Gemfile ├── config @@ -40,6 +42,8 @@ like this: │   ├── ... └── worker.rb ``` +- `.github/workflows` - This directory tells github how to build and test Inferno + when updates are committed to a GitHub repository. - `Dockerfile` - This file controls how the Docker image for your tests is built. - `Gemfile` - This file is where you add extra Ruby dependencies. - `config` - This folder contains configuration for the database and web @@ -54,7 +58,7 @@ like this: services needed for running Inferno. - `execution_scripts` - This folder is for scrips that execute test suites against running Inferno services like the HL7 Validator. The template includes an example - script for the sample test suite. See [CI/CD Usage](/docs/getting-started/ci-cd-usage.html#using-session_runnersh-for-complex-orchestration) + script for the sample test suite. See [CI/CD Usage](docs/ci-cd-usage.html#using-the-inferno-execute_script-cli-for-complex-orchestration) for additional information. - `inferno_template.gemspec` - This file controls how your tests are packaged up as a distributable Ruby gem. This is also where you can add additional Ruby From 17fb8476f906175ad6b6f25860bf0abaed46b257 Mon Sep 17 00:00:00 2001 From: Karl Naden Date: Fri, 3 Apr 2026 14:23:31 -0400 Subject: [PATCH 05/11] update example set --- docs/advanced-test-features/scripting-execution.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/advanced-test-features/scripting-execution.md b/docs/advanced-test-features/scripting-execution.md index f5c6c72..0235fd4 100644 --- a/docs/advanced-test-features/scripting-execution.md +++ b/docs/advanced-test-features/scripting-execution.md @@ -398,11 +398,11 @@ The following Test Kits include scripts that can be executed. To execute any of [execution scripts](https://github.com/inferno-framework/inferno-core/tree/main/execution_scripts) for [sample development suites](https://github.com/inferno-framework/inferno-core/tree/main/dev_suites) that demonstrate features of Inferno and the `execute_script` CLI. +- [**SMART App Launch**](https://github.com/inferno-framework/smart-app-launch-test-kit): + The SMART App Launch test kit includes both client and server tests and includes [execution scripts](https://github.com/inferno-framework/smart-app-launch-test-kit/tree/main/execution_scripts) + demonstrating how multi-session scripts can be used to execute these tests against each other. - [**(g)(10) Certification**](https://github.com/onc-healthit/onc-certification-g10-test-kit): The (g)(10) Certification test kit (and [US Core Core test kit](https://github.com/inferno-framework/us-core-test-kit/tree/main/execution_scripts)) contains [scripts](https://github.com/onc-healthit/onc-certification-g10-test-kit/tree/main/execution_scripts) that demonstrate one approach to scripting external interactions with tested systems, in this - case the Inferno Reference Server. -- [**Da Vinci CRD**](https://github.com/inferno-framework/davinci-crd-test-kit): - The Da Vinci CRD test kit includes both client and server tests and includes [execution scripts](https://github.com/inferno-framework/davinci-crd-test-kit/tree/main/execution_scripts) demonstrating - how multi-session scripts can be used to execute these tests against each other. \ No newline at end of file + case the Inferno Reference Server. \ No newline at end of file From 7d29894ed4caa6989b9dfde496fe2915362a2794 Mon Sep 17 00:00:00 2001 From: Karl Naden Date: Fri, 3 Apr 2026 15:13:09 -0400 Subject: [PATCH 06/11] typo fixes --- docs/advanced-test-features/index.md | 2 +- .../scripting-execution.md | 6 +++--- docs/getting-started/inferno-cli.md | 19 +++++++++++-------- 3 files changed, 15 insertions(+), 12 deletions(-) diff --git a/docs/advanced-test-features/index.md b/docs/advanced-test-features/index.md index 7c8524d..66505bc 100644 --- a/docs/advanced-test-features/index.md +++ b/docs/advanced-test-features/index.md @@ -53,6 +53,6 @@ groups, and tests. Test developers can track which requirements Inferno is and i verifying and Inferno displays these associations to users within the [user interface](/docs/user-interface.html#viewing-verified-requirements). -## [Scripting Suite Execution](/docs/advanced-test-features/script-execution.html) +## [Scripting Suite Execution](/docs/advanced-test-features/scripting-execution.html) The Inferno CLI supports executing suites using a yaml configuration format instead of the UI using the `inferno execute_script` command. \ No newline at end of file diff --git a/docs/advanced-test-features/scripting-execution.md b/docs/advanced-test-features/scripting-execution.md index 0235fd4..e695893 100644 --- a/docs/advanced-test-features/scripting-execution.md +++ b/docs/advanced-test-features/scripting-execution.md @@ -54,7 +54,7 @@ file for a complete format reference. ### Sessions Each script configuration file will indicate a list (sequence) of one or more Inferno sessions to create under -the `sessions:` top-leve key. Each sequence includes +the `sessions:` top-level key. Each sequence includes - **Suite** (`suite:`): *Required* - the internal id for the suite or the title selected in the UI. - **Name** (`name:`): *Conditional* - a short key used to identify the session in steps. Required when multiple sessions are defined, otherwise optional. @@ -171,7 +171,7 @@ look like this: comparison_config: sessions: g10: - expected_results_file: g10_results_expected.json + expected_results_file: g10_results_expected.json alternate_expected_files: - file: g10_no_terminology_expected.json when: @@ -220,7 +220,7 @@ Additionally, the optional `state_description:` key can be used to provide docum describing the state. This has no functional effect, but is echoed during script execution for debugging purposes. -Note that Inferno can be have other status (e.g., `running` and `cancelled`), but these +Note that Inferno can be have other status (e.g., `running` and `cancelling`), but these do not represent stable states where Inferno is waiting for an action, so they will never be used in scripts. diff --git a/docs/getting-started/inferno-cli.md b/docs/getting-started/inferno-cli.md index c7ed097..bf1c635 100644 --- a/docs/getting-started/inferno-cli.md +++ b/docs/getting-started/inferno-cli.md @@ -107,14 +107,14 @@ Subcommands include: bundle exec inferno session create my_test_suite -I https://inferno.healthit.gov/suites -p my_preset ``` -- **`start_run SESSION_ID`** +- **`start_run SESSION_ID RUNNABLE_ID`** - Initiate a test run on an existing session. + Initiate a test run on an existing session. The `RUNNABLE_ID` is + a short or internal ID of a specific group or test to run, or `suite` + to run the entire suite. **Options**: - `-I`, `--inferno_base_url`: URL of the target Inferno service. - - `-r`, `--runnable`: Short or internal ID of a specific group or test to run. - If omitted, the entire suite is run. - `-i`, `--inputs`: Input values as `key:value` pairs (e.g., `--inputs url:https://example.com patient_id:123`). These merge with and override the session's previously stored inputs. The internal name for the input must be used @@ -124,7 +124,7 @@ Subcommands include: **Output**: JSON representation of the created test run. See the output of the [`POST /test_runs`](https://inferno-framework.github.io/inferno-core/api-docs/#/Test%20Run/post_test_runs) API for details on the output. ```sh - bundle exec inferno session start_run 7uLiz9qX4og -I https://inferno.healthit.gov/suites -r my_group \ + bundle exec inferno session start_run 7uLiz9qX4og my_group -I https://inferno.healthit.gov/suites \ -i url:https://example.com patient_id:123 ``` @@ -235,12 +235,15 @@ The `inferno execute_script` CLI command can be used to execute complex Inferno which are specified in a [script configuration file](/docs/advanced-test-features/scripting-execution#creating-script-configuration-files). By default, scripts are executed against the local running Inferno instance (both background services and the test kit must be running). Execution can be performed against a remote Inferno instance by providing -the root Inferno url using the `-I` option. +the root Inferno url using the `-I` option. To reduce the risk of running unsafe commands +unintentionally, the `--allow-commands` flag must be passed when the executed script +uses `command:` actions to execute separately defined scripts. See [Defining, Executing, and Securing Complex Commands](/docs/advanced-test-features/scripting-execution#defining-executing-and-securing-complex-commands) for additional details. -For example, the following command would run the script defined in file `g10.yaml` on the instance of +For example, the following command would run the script defined in file `g10_with_commands.yaml`, +including `command:` actions, on the instance of Inferno deployed to [inferno.healthit.gov](https://inferno.healthit.gov/suites): ``` -bundle exec inferno execute_script g10.yaml -I https://inferno.healthit.gov/suites +bundle exec inferno execute_script g10_with_commands.yaml -I https://inferno.healthit.gov/suites --allow-commands ``` See [Scripting Suite Execution](/docs/advanced-test-features/scripting-execution) for details on how to From 3ede2cbcd3925caab786050351501ce3d36b68a8 Mon Sep 17 00:00:00 2001 From: Karl Naden Date: Fri, 10 Apr 2026 12:24:37 -0400 Subject: [PATCH 07/11] update documentation --- .../scripting-execution.md | 123 ++++++++++-------- docs/getting-started-users.md | 16 ++- docs/getting-started/inferno-cli.md | 8 +- 3 files changed, 88 insertions(+), 59 deletions(-) diff --git a/docs/advanced-test-features/scripting-execution.md b/docs/advanced-test-features/scripting-execution.md index e695893..bfe5ac3 100644 --- a/docs/advanced-test-features/scripting-execution.md +++ b/docs/advanced-test-features/scripting-execution.md @@ -10,12 +10,12 @@ section: docs ## Overview Inferno supports the scripting of test suite execution through the `inferno execute_script` -[CLI command](/docs/getting-started/inferno-cli). This command takes as input a -[configuration file](#yaml-configuration-format) that tells Inferno when and how to perform +[CLI command](/docs/getting-started/inferno-cli#complex-scripted-execution). This command takes as input a +[configuration file](#creating-script-configuration-files) that tells Inferno when and how to perform execution steps both inside and outside of Inferno. The results of the session are compared against the results of a known-good execution to identify any regressions. The script can be executed against the local Inferno instance or a specific remote instance. These scripts -can be built into [CI/CD Pipelines](/docs/ci-cd-usage). +can be built into [CI/CD Pipelines](/docs/ci-cd-usage#using-the-inferno-execute_script-cli-for-complex-orchestration). ## Execution @@ -29,9 +29,71 @@ Three standard options for executing scripts include: - `FILTER=execution_scripts//*.yaml`: provide a different filter to use to identify the scripts to execute. If not provided, the default is `execution_scripts/**/*.yaml`. - Executing all scripts defined in the test kit's `execution_scripts` directory within GitHub, - either in response to a commit or a manual trigger. See [CI / CD Usage](/docs/ci-cd-usage.md) + either in response to a commit or a manual trigger. See + [CI / CD Usage](/docs/ci-cd-usage#using-the-inferno-execute_script-cli-for-complex-orchestration) for additional details. +### Execution Process and Output + +When a script is executed, the following phases are performed: +1. Sessions are created +2. Steps are executed +3. Results are checked + +During execution, Inferno will print to the terminal details on its polling, matching, and actions during +execution and provide a summary of the results comparison performed. + +Inferno will use exit code 3 when an error is encountered. Error cases include: +- The exepected results file for one or more sessions did not exist. +- Compared results for one or more sessions did not match. +- Script execution did not end at the `END_SCRIPT` step. +- Script execution had to be interrupted due to a timeout. + +Otherwise, the CLI command will end with exit code 0 indicating success. + +#### Session Creation + +Test suite sessions for the execution are detailed in the [`sessions`](#sessions) section of the script +configuration file. Creation is performed using the +[`inferno session create` CLI](/docs/getting-started/inferno-cli#manage-sessions). + +#### Step Execution + +The steps taken during execution are detailed in the [`steps`](#steps) section of the script +configuration file. + +Inferno checks the status of sessions for next step polling using the [`inferno session status` CLI](/docs/getting-started/inferno-cli#manage-sessions). It may also use the [`inferno session cancel` CLI](/docs/getting-started/inferno-cli#manage-sessions) if it needs to cancel a session. + +To start runs, Inferno uses the [`inferno session start_run` CLI](/docs/getting-started/inferno-cli#manage-sessions). + +#### Check Results + +For each session in the script, Inferno will compare the results in the expected results +file (see [Expected Results File and Alternates](#expected-results-file-and-alternates)) +to the results of the completed session execution. Configuration of the comparison is +detailed in the [`comparison_config`](#result-comparison-config) section of the script +configuration file. + +Comparison is performed using the [`inferno session compare` CLI](/docs/getting-started/inferno-cli#manage-sessions) +using the `-f` flag to specify the expected file. Comparison involves the following steps: +- normalize the expected and actual results using the configured [`normalized_strings`](#normalized-strings) +- match individual result entries using the runnable id (`test_id`, `test_group_id`, or `test_suite_id`) +- compare the matched result entries, looking at the status (e.g., pass, fail), the result message, and the + individual messages within the result + +When the actual and expected results do not match, two files will be written in the same location as the expected file: +- actual results: a file with the results json (prior to normalization) +- comparison analysis: a csv file with details of results that differed, including normalized details +These files are named using the same prefix as the executed file and include a timestamp for the execution so that +they will be unique for each script execution. + +When the target expected results file does not exist it will be created using the output of the +[`inferno session results` CLI](/docs/getting-started/inferno-cli#manage-sessions). The `execute_script` +CLI will exit with an error status in this case. After creation of a expected results +file in this way, users should review the results using the file and/or the Inferno +UI to make sure that they are the expected results as subsequent script executions +will perform the comparison using that file as the expected results. + ## Creating Script Configuration Files At a high-level, the script configuration file contains instructions for @@ -74,7 +136,7 @@ the actual and/or expected results to facilitate like-to-like comparison. All configurations around results comparison are nested under the `comparison_config:` top-level key. -#### Normalized String +#### Normalized Strings Before comparing the actual results to the expected results, Inferno will replace strings indicated in the `normalized_strings:` key. This prevents these values, which are expected @@ -290,55 +352,6 @@ the following keys: After executing a step, Inferno will either poll the session indicated in this key or the session polled after the previous step if none is specified. -## Execution Process and Output - -When a script is executed, the following phases are performed: -1. Sessions are created -2. Steps are executed -3. Results are checked - -During execution, Inferno will print to the terminal details on its polling, matching, and actions during -execution and provide a summary of the results comparison performed. - -Inferno will use exit code 3 when an error is encountered. Error cases include: -- The exepected results file for one or more sessions did not exist. -- Compared results for one or more sessions did not match. -- Script execution did not end at the `END_SCRIPT` step. -- Script execution had to be interrupted due to a timeout. - -Otherwise, the CLI command will end with exit code 0 indicating success. - -### Session Creation - -Internally, this uses the [`inferno session create` CLI](/docs/getting-started/inferno-cli#manage-sessions). - -### Step Execution - -Inferno checks the status of sessions for next step polling using the [`inferno session status` CLI](/docs/getting-started/inferno-cli#manage-sessions). It may also use the [`inferno session cancel` CLI](/docs/getting-started/inferno-cli#manage-sessions) if it needs to cancel a session. - -To start runs, Inferno uses the [`inferno session start_run` CLI](/docs/getting-started/inferno-cli#manage-sessions). - -### Check Results - -For each session in the script, Inferno will compare the results in the expected results file (see [Expected Results File and Alternates](#expected-results-file-and-alternates)) to the results of the completed session execution. To perform the comparison, it will use the [`inferno session compare` CLI](/docs/getting-started/inferno-cli#manage-sessions), which involves the following steps: -- normalize the expected and actual results using the configured [`normalized_strings`](#normalizations-for-results-comparison) -- match individual result entries using the runnable id (`test_id`, `test_group_id`, or `test_suite_id`) -- compare the matched result entries, looking at the status (e.g., pass, fail), the result message, and the - individual messages within the result - -When the actual and expected results do not match, two files will be written in the same location as the expected file: -- actual results: a file with the results json (prior to normalization) -- comparison analysis: a csv file with details of results that differed, including normalized details -These files are named using the same prefix as the executed file and include a timestamp for the execution so that -they will be unique for each script execution. - -When the target expected results file does not exist it will be created using the output of the -[`inferno session results` CLI](/docs/getting-started/inferno-cli#manage-sessions). The `execute_script` -CLI will exit with an error status in this case. After creation of a expected results -file in this way, users should review the results using the file and/or the Inferno -UI to make sure that they are the expected results as subsequent script executions -will perform the comparison using that file as the expected results. - ## Scripts with Multiple Sessions Scripts with multiple sessions involve additional complexity but are useful for scripting the execution of @@ -391,7 +404,7 @@ The following Test Kits include scripts that can be executed. To execute any of - [**Inferno Template**](https://github.com/inferno-framework/inferno-template): New Inferno Test Kits created from the [inferno-template repository](https://github.com/inferno-framework/inferno-template) - or the [`inferno new` CLI command](/docs/getting-started/inferno-cli.html#inferno-commands) + or the [`inferno new` CLI command](/docs/getting-started/inferno-cli.html#creating-a-new-test-kit) include a [simple execution configuration file](https://github.com/inferno-framework/inferno-template/blob/main/execution_scripts/inferno_template_script.yaml) for the example test suite included in the template. - [**Inferno Core**](https://github.com/inferno-framework/inferno-core): Inferno Core contains diff --git a/docs/getting-started-users.md b/docs/getting-started-users.md index 4a26db0..a556a35 100644 --- a/docs/getting-started-users.md +++ b/docs/getting-started-users.md @@ -8,7 +8,9 @@ layout: docs Start here if you're interested in testing a FHIR server against one or more existing Test Kits. -## Running an Existing Test Kit +## Running Test Kits + +### Running an Existing Test Kit Most Test Kits are developed using the [Inferno Template](https://github.com/inferno-framework/inferno-template) repository, which provides scripts for standing up an instance of Inferno to run a selected Test Kit. Running a Test Kit usually follows the below steps: @@ -32,7 +34,7 @@ cd us-core-test-kit Always check the documentation for an individual Test Kit, however, since there may be additional installation steps. -## Running Multiple Test Kits +### Running Multiple Test Kits There may be times when you wish to run multiple Test Kits in a single Inferno instance. You can load and run two or more separate Test Kits by using [Inferno Template](https://github.com/inferno-framework/inferno-template). @@ -86,3 +88,13 @@ cd inferno_template ``` _Note: Example Test Suites, Groups, Tests and IGs in the template can be removed._ + +## Executing Tests + +Suites hosted by a running test kit can be executed: +1. Interactively, using the web-based [Inferno Testing Interface UI](/docs/user-interface) +2. Automatically, using [scripted execution](/docs/advanced-test-features/scripting-execution). + +Advanced users can also use the lower-level [`session` CLI](/docs/getting-started/inferno-cli#manage-sessions) +or the [JSON API](https://inferno-framework.github.io/inferno-core/api-docs/) to create and interact +with Inferno sessions. \ No newline at end of file diff --git a/docs/getting-started/inferno-cli.md b/docs/getting-started/inferno-cli.md index bf1c635..03c3e21 100644 --- a/docs/getting-started/inferno-cli.md +++ b/docs/getting-started/inferno-cli.md @@ -199,8 +199,12 @@ Subcommands include: **Options**: - `-I`, `--inferno_base_url`: URL of the target Inferno service. - - `-f`, `--expected_results_file`: Path to a JSON file containing the expected results. - - `-s`, `--expected_results_session`: Session ID on the same server whose results serve as expected. + - `-f`, `--expected_results_file`: Path to a JSON file containing the expected results. When the session + results do not match the expected results in the file, generated comparison files will be placed in + the same directory. + - `-s`, `--expected_results_session`: Session ID on the same server. The results of this indicated + session will be used as the expected results. When the compared session's results do not match + comparison details will not be written to file (use the `-f` option). - `-m`, `--compare_messages`: Also compare per-test messages (not only pass/fail result). Default: `false`. - `-r`, `--compare_result_message`: Also compare each test's `result_message` string. Default: `false`. - `-n`, `--normalized_strings`: List of literal strings and regexes which will be used to normalize From 890098da43444d9122ea743d81a13ef666e4579c Mon Sep 17 00:00:00 2001 From: Karl Naden Date: Mon, 13 Apr 2026 08:48:10 -0400 Subject: [PATCH 08/11] review fixes --- docs/advanced-test-features/scripting-execution.md | 8 ++++++-- docs/getting-started/inferno-cli.md | 4 ++-- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/advanced-test-features/scripting-execution.md b/docs/advanced-test-features/scripting-execution.md index bfe5ac3..e9a7057 100644 --- a/docs/advanced-test-features/scripting-execution.md +++ b/docs/advanced-test-features/scripting-execution.md @@ -243,7 +243,7 @@ comparison_config: matches: ^There is a problem with the terminology resources ``` -Note that the `normalized_string:` key never appears under the `sessions:` key. +Note that the `normalized_strings:` key never appears under the `sessions:` key. ### Steps @@ -303,7 +303,11 @@ key. Only one can be defined for each step: - `inputs:` *Optional* - key-value pairs indicating inputs to be merged into those already stored in the session (from the preset or previous run inputs or outputs). Each key must be the internal name for the input (from the UI, use the yaml or json view of the inputs - to find the internal name). + to find the internal name). Values can be + - *a single value*: used as the literal input value; or + - *an object or sequence*: the json representation will be used as the input value; or + - *a file path prefixed with `@`*: the contents of the file will be used as the input value. + Relative paths are resolved from the directory containing this script file. - `session:` *Conditional* - when multiple sessions are defined the name (not the suite) of the session on which the run will be executed, otherwise omitted. - `command:` Specifies a shell command to execute. For example, a `curl` command to navigate diff --git a/docs/getting-started/inferno-cli.md b/docs/getting-started/inferno-cli.md index 03c3e21..f9b6fe3 100644 --- a/docs/getting-started/inferno-cli.md +++ b/docs/getting-started/inferno-cli.md @@ -170,7 +170,7 @@ Subcommands include: - `-I`, `--inferno_base_url`: URL of the target Inferno service. **Output**: JSON array of objects with `name` and `value` keys representing the session's current inputs. - See the output of the [`GET /test_sessions/{test_session_id}/results`](https://inferno-framework.github.io/inferno-core/api-docs/#/Result/get_test_sessions__test_session_id__results) + See the output of the [`GET /test_sessions/{test_session_id}/session_data`](https://inferno-framework.github.io/inferno-core/api-docs/#/Session%20Data/get_test_sessions__test_session_id__session_data) API for details on the output. ```sh @@ -185,7 +185,7 @@ Subcommands include: - `-I`, `--inferno_base_url`: URL of the target Inferno service. **Output**: JSON array of result objects for each test, group, or suite that has been executed. - See the output of the [`GET /test_sessions/{test_session_id}/session_data`](https://inferno-framework.github.io/inferno-core/api-docs/#/Session%20Data/get_test_sessions__test_session_id__session_data) + See the output of the [`GET /test_sessions/{test_session_id}/results`](https://inferno-framework.github.io/inferno-core/api-docs/#/Result/get_test_sessions__test_session_id__results) API for details on the output. ```sh From cae1f84ebed8bc64ca3eff1c65e9da7c54164275 Mon Sep 17 00:00:00 2001 From: Karl Naden Date: Wed, 15 Apr 2026 09:05:47 -0400 Subject: [PATCH 09/11] clarify that tests can be expected to fail but not error --- docs/advanced-test-features/scripting-execution.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/advanced-test-features/scripting-execution.md b/docs/advanced-test-features/scripting-execution.md index e9a7057..f9bc758 100644 --- a/docs/advanced-test-features/scripting-execution.md +++ b/docs/advanced-test-features/scripting-execution.md @@ -48,6 +48,7 @@ Inferno will use exit code 3 when an error is encountered. Error cases include: - Compared results for one or more sessions did not match. - Script execution did not end at the `END_SCRIPT` step. - Script execution had to be interrupted due to a timeout. +- The results contained an `error` result, indicating a problem in the test logic Otherwise, the CLI command will end with exit code 0 indicating success. @@ -74,6 +75,14 @@ to the results of the completed session execution. Configuration of the comparis detailed in the [`comparison_config`](#result-comparison-config) section of the script configuration file. +There is no requirement that the tests must pass for the results check to succeed. +The exepected results file can indicate that some or all entries have a failing result +(e.g., `fail` or `skip`) with particular details in their `result_message` and `messages` +fields. This allows scripts that verify that particular tests fail under certain conditions +and indicate this with expected messages. However, if any tests end with a `error` result, +indicating a problem in the test execution logic, the script execution will fail and result +comparison will be skipped. + Comparison is performed using the [`inferno session compare` CLI](/docs/getting-started/inferno-cli#manage-sessions) using the `-f` flag to specify the expected file. Comparison involves the following steps: - normalize the expected and actual results using the configured [`normalized_strings`](#normalized-strings) From 03a795987d43b23eedbc64bf60c783fecaec05bb Mon Sep 17 00:00:00 2001 From: Karl Naden Date: Wed, 15 Apr 2026 09:47:09 -0400 Subject: [PATCH 10/11] clarify error cases where results comparison is not performed --- .../scripting-execution.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/docs/advanced-test-features/scripting-execution.md b/docs/advanced-test-features/scripting-execution.md index f9bc758..8238dfc 100644 --- a/docs/advanced-test-features/scripting-execution.md +++ b/docs/advanced-test-features/scripting-execution.md @@ -43,12 +43,16 @@ When a script is executed, the following phases are performed: During execution, Inferno will print to the terminal details on its polling, matching, and actions during execution and provide a summary of the results comparison performed. -Inferno will use exit code 3 when an error is encountered. Error cases include: -- The exepected results file for one or more sessions did not exist. -- Compared results for one or more sessions did not match. -- Script execution did not end at the `END_SCRIPT` step. -- Script execution had to be interrupted due to a timeout. -- The results contained an `error` result, indicating a problem in the test logic +Inferno will use exit code 3 when an error is encountered. Error cases include the following, +separated into cases where a detailed results comparison is performed and those +where it is not: +- Reults comparison performed + - Compared results for one or more sessions did not match. + - Script execution reached an unmatched stable state (no action specified for a `done` or `waiting` state). +- No results comparison performed + - The exepected results file for one or more sessions did not exist. + - Script execution had to be interrupted due to a timeout. + - The results contained an `error` result, indicating a problem in the test logic Otherwise, the CLI command will end with exit code 0 indicating success. From f72a4737d1831c6f93157887bee7ba99f83b1853 Mon Sep 17 00:00:00 2001 From: Karl Naden Date: Wed, 15 Apr 2026 10:25:31 -0400 Subject: [PATCH 11/11] typo --- docs/advanced-test-features/scripting-execution.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/advanced-test-features/scripting-execution.md b/docs/advanced-test-features/scripting-execution.md index 8238dfc..7d8b13a 100644 --- a/docs/advanced-test-features/scripting-execution.md +++ b/docs/advanced-test-features/scripting-execution.md @@ -386,7 +386,7 @@ Considerations to keep in mind when creating script configuration files with mul form. - `NOOP` actions can be used to help clarify sequencing. For example, if two sessions both have long running tests that interact but complete in a particular - order due to wait interactions, but don't require any external interactions. You + order due to wait interactions and don't require any external interactions. You could poll the session that will end last and look only for the `done` state, which would require `WAIT` actions on any `waiting` states so that Inferno recognizes them, or you could define states for each `waiting` state with `NOOP`