qubership-pipelines-modules-ops

Description

This Python package provides functions to handle data for Qubership pipelines/modules.

Operations

Variables

Job variables transformation

Nested structures for certain <root> variables can be passed in two formats:

1. string representation of yaml;
2. hierarchical key representation.

Example:

variables:
  <root>: |
    <string in yaml format>
  <root>__[<key>__...]PARAM: VALUE

Processing:

1. Variable <root> is parsed as yaml.
2. All variables with prefix <root> are combined:
   • Hierarchical keys are processed, using dot . or double underscore __ as a delimiter.
     Note: GitLab does not allow dots in variable names.
   • Elements of hierarchical keys can be:
     • strings - keys in mappings:
       keyA.keyB and keyA__keyB where keyB is string, are treated as a hierarchical key for structure

       keyA:
         keyB: value

     • integers - indexes in lists:
       keyA.N and keyA__N where N is integer, are treated as a hierarchical key for structure

       keyA:
         ...
         - value  # N'th position in the list, in 0-based numeration
         ...

3. Then these two structures are merged as nested mappings.
   Lists are treated as solid objects in this merge operation.
   Priority is defined by the value of env variable QUBERSHIP_PIPELINES_MODULES_OPS_PREFIXED_VARS_PRIORITY. If true (default), prefixed parameters have greater priority.

Type casting

In GitLab, variable values ​​must be strings.

When collecting data, qubership_pipelines_modules_ops automatically applies type casting:

• 'true' and 'false' are converted into booleans;
• integer and float numbers are detected.

Variables substitution

GitLab substitutes variables given in notations $VAR and ${VAR} when starting a job. Only initially defined and predefined variables can be used in these substitutions.

Postponed substitution is implemented:

1. Escape variable in expression by doubling dollar symbol: $$VAR or $${VAR}. GitLab replaces double dollar sign with a single one, without substituting the variable: $$VAR -> $VAR, $${VAR} -> ${VAR}
2. qubership_pipelines_modules_ops performs substitution of these variables:
   • The following values are available for substitution:
     • Initially defined and predefined variables,
     • Parameters defined in the stored job data;
   • Nested variables substitution is implemented.

variables:
  VAR_A: ${A}  # variable 'A' will be substituted by GitLab
  VAR_B: postponed $${B}  # variable 'B' will be substituted by qubership_pipelines_modules_ops

Modules data

Input modules data

To compose input data (input_params.yaml and input_params_secure.yaml) for a module:

1. Stored data from previous jobs (parameters and files) is collected and combined. The order of the job stages is taken into account.
2. Then current job variables are added.

Variable input is processed as described in Job variables transformation section.

Additionally, the following shorthands can be used for prefixed variables:

<job_name>:
  variables:
    params__[<key>__...]PARAM: VALUE  # short for input__params__params__[<key>__...]PARAM: VALUE
    systems__[<key>__...]PARAM: VALUE  # short for input__params__systems__[<key>__...]PARAM: VALUE

I.e. prefix input__params__ is optional in input__params__params__ and input__params__systems__ hierarchical keys.

Example:

<job_name>:
  variables:
    input: |  # string in yaml format
      files:
        FILE_KEY: ... # <path, relative to input_files/>|<empty or null: means without renaming>
      params:
        params:
          PARAM: VALUE
        systems:
          PARAM: VALUE
      params_secure: {...}

    input__params__params__PARAM: VALUE
    params__PARAM: VALUE  # short for input__params__params__PARAM: VALUE
    systems__PARAM: VALUE  # short for input__params__systems__PARAM: VALUE

Output modules data

Module output parameters and files must be stored as artifacts, and be available in jobs in subsequent stages.

Variable output defines parameters and files that will be stored as module output. It is processed as described in Job variables transformation section.

<job_name>:
  variables:
    output: | # <string, in yaml format>
      params:
        NEW_PARAM_1: <hierarchical key in output_params.yaml or output_params_secure.yaml>|BUILD_STATUS|BUILD_URL|BUILD_DATE
        NEW_PARAM_2: ...
        PATH_IN_OUTPUT_YAML: '*'
      files:
        FILE_KEY_1: <file path relative to output_files/>
        FILE_KEY_2: ...
        PATH_IN_OUTPUT_YAML: '*'

    output__params__P: VALUE_OR_ASTERISK
    output__files__FILE_KEY: PATH_OR_ASTERISK
    ...

Under output.params mapping:

• Key defines parameter name.

• Value is the source for the parameter value. It can be:

  • a hierarchical key in output_params.yaml or output_params_secure.yaml;
  • one of the special environment variables: BUILD_STATUS, BUILD_URL, BUILD_DATE (must be provided)

• Special case: If the value is *, then the key is considered to be a path (i.e. hierarchical key) to a mapping in output_params.yaml or output_params_secure.yaml to get all values from, without renaming. For example, output__params__params: '*' instructs to import all parameters listed in output_params.yaml and in output_params_secure.yaml, under params section.

Under output.files mapping:

• Key defines file key (file identifier that can be used in later jobs as input parameter).
• Value is the file path, relative to output_files directory.
• Special case: If the value is *, then the key is considered to be a path (i.e. hierarchical key) to a mapping in output_params.yaml or output_params_secure.yaml to get all values from. For example, output__files__files: '*' instructs to import and copy all file keys listed in output_params.yaml and in output_params_secure.yaml, under files section.

Pipeline output

If output__pipeline_output parameter is present, then:

• Module output data is copied to rt/pipeline/output directory:
  • output params file,
  • output params secure file,
  • output files directory;
• Context descriptor rt/pipeline/output/context.yaml is created. Path in this context descriptor are relative to rt/pipeline/output directory.

Report

If variables contain report section and/or report__ prefixed parameters, an additional artifact will be generated in module input data: pipeline execution report.

During module input data preparation, variable report is processed as described in Job variables transformation section.

Example of configuration:

<job name>:
  variables:
    report: |
      version: v1  # Optional, default is 'v1'
      config:
        - name: PIPELINE_DATA
          value: ...
        ...

or equivalent in prefixed format:

<job name>:
  variables:
    report__version: v1
    report__config__0__name: PIPELINE_DATA
    report__config__0__value: ...
    ...

Minimal configuration to activate report function:

<job name>:
  variables:
    report: '{}'

pipeline_execution_report.yaml is generated and placed into input_files/ directory.

Conditions

During module input data preparation, variable when is processed as described in Job variables transformation section.

If variables contain when.condition (when__condition):

• its value is treated as a Python expression;
• the following entities can be used as variables in the expression:
  • parameters stored in previous jobs,
  • environment variables;
• if the expression evaluation gives Python falsy value (i.e. bool(result) == False), then:
  • an empty file logs/.when_condition_false is created (logs dir is in the same directory as context descriptor),
  • module_ops exits;
• Exit code is 0 by default, but can be changed by setting QUBERSHIP_PIPELINES_MODULES_OPS_FALSE_CONDITION_EXIT_CODE environment variable.

Example:

<job_name>:
  variables:
    when:
      condition: rate > 95

Development notes

Storage layout

rt/
  job_data/    # not a part of public contract
    input_files/
    output_files/
    context.yaml
    input_params.yaml
    input_params_secure.yaml
    output_params.yaml
    output_params_secure.yaml
  stored_data/    # not a part of public contract
    <stage>/
      <job>/
        files/
        logs/
        files.yaml
        params.yaml
      stage.yaml
  pipeline/
    output/
      output_files/
      context.yaml
      output_params.yaml
      output_params_secure.yaml

Testing

For testing purposes, a special handling of input files section is implemented:

<job name>
  input:
    files:
      TEMP_FILE1: |
        file1.txt:
        file content

If the value of input files entry contains semicolon :, then it is treated as <file_path>:<file_content>.

• A new file is created in the input files directory, with the given path and content;
• One leading line break in the content is trimmed, if presents;
• The file key (TEMP_FILE1 in the example above) is ignored.