Open-source python pipelines orchestrator and executor.
This application is distributed as a library on PyPI, Docker image on GitHub Packages, and as a Reusable Workflow for your existing GitHub workflows. You can use this application in any of the provided forms that best suits your needs.
Provided Docker image with this application includes:
- Pipelines Declarative Executor app itself
- Sample "Python Modules" to execute commands in your "Atlas Pipeline"
- SOPS binaries to securely process configuration files and output parameters
Sample "Python Modules", included in distributed Docker image, are only showcasing the intended way of working via Execution Commands and Execution Context. You might need to have your custom commands executed in "Atlas Pipelines" - then you will need to create and distribute your own implementation. There's a Development Guide available in CLI Samples repository.
A few ways you can include your own "Python Modules" into executor Docker image:
- Building new image with required modules, and putting path to them to
PIPELINES_DECLARATIVE_EXECUTOR_PYTHON_MODULE_PATHenv variable - Mounting directory with your modules into the image, while also upgrading
PIPELINES_DECLARATIVE_EXECUTOR_PYTHON_MODULE_PATHenv variable to mounted path
Multiple "Python Modules" are supported via stage.path property (check syntax guide for more information)
You can also use this application as a python dependency, by installing it from PyPI:
pip install qubership-pipelines-declarative-executorOr adding it to your dependency list:
qubership-pipelines-declarative-executor = "^2.0.0"Alternative usage scenario (although much less configurable than creating your own workflow, since we can't pass custom env variables into it) is using provided Reusable Workflow in your repository. There is an example of how it can be invoked from GitHub Workflow: pipeline.yml
There's also an example of a similar implementation for GitLab
When you've created your own Docker image with necessary "Python Modules" (non-sample ones), you might want to create your own workflows using that image. Provided workflows serve as a starting point, but custom workflows will allow full control and ability to pass env variables and local files.
"AtlasPipelines" are intended to work via Execution Commands, packed into "Python Modules".
Pipeline itself describes data flow between sequentially executed stages, while also supporting invoking nested pipelines (for reusing configuration) and parallel stages.
This repository uses actual "AtlasPipelines" in its tests, you can check them here.
Separate article on syntax with examples is available here.
Executor collects and can upload report (intended for UI representation) of currently executed pipeline.
This feature is configured via env variables in Report section.
You can select REPORT_SEND_MODE (either ON_COMPLETION or PERIODIC), send intervals, and endpoint configs:
Report configuration example is here
Orchestrator can fetch remote AtlasPipeline and AtlasConfig files from various sources. To access private repositories or authenticated endpoints, you can configure authentication rules.
Example Auth Rules are present here
Rules are processed in order they are defined, and first applicable rule will be used (in case when multiple would've matched).
If no rules match, requests are made without authentication.
You can pass your CI (GitHub, GitLab, Jenkins, etc.) execution instance parameters (user who triggered pipeline, pipeline's URL) to make them available in the report via environment variables
Executor decrypts input files (pipelines and configs) if they are encrypted with SOPS, and can also encrypt any output secure files. This feature is configured via a set of environment variables
Other parameters are available and documented in the General section
Performance tests and comparisons of different ways to invoke imported "Python Modules" will be available here