Skip to content

CONTRIBUTING.rst

Latest commit

 

History

History
253 lines (153 loc) · 8.31 KB

CONTRIBUTING.rst

File metadata and controls

253 lines (153 loc) · 8.31 KB

Contributing to the klio ecosystem

Requirements

  • Python 3.5, 3.6, 3.7, and 3.8 (suggestion: use pyenv to install required versions)
  • tox (suggestion: use pipx to install tox)
  • protoc

Project Overview

Each directory is its own Python package (except for docs, examples, and integration):

├── audio/
├── cli/
├── core/
├── devtools/
├── docs/
├── examples/
├── exec/
├── integration/
└── lib/

The klio-cli (user-facing) commands essentially wrap klio-exec (non-user-facing) commands. klio-cli $CMD sets up the run context needed to correctly run klio-exec $CMD inside the job's Docker container (i.e. mounts the job directory, sets up env vars, copies over credentials). In general, changes to klio-cli require a corresponding change in klio-exec, but there are a few klio-cli commands self-contained and therefore do not need any changes to klio-exec (i.e. klio message publish).

klio-core is a non-user-facing library of common utilities, including protobuf definitions. The klio library contains the base transform for users to implement, and the main message processing logic (i.e. triggering parents, skipping already processed work, etc). klio-devtools is just a collection of utilities to help aid the development of klio.

Run Tests

Tests can be invoked in two ways: pytest and tox.

Run tests via tox

This is the recommended way to run tests since tox will manage its own virtualenvs and is also what will be run as part of the CI checks.

tox is a tool to easily run tests and do other development tasks in a Python project, kind of like a Makefile for Python. It is configured with a tox.ini file in the root folder of a project. Because Klio is divided into several Python projects each one has its own tox.ini.

Caution!

Occasionally tox may cache an outdated version of a dependency. If you find yourself getting errors that aren't reproducible by others or by CI, try deleting the .tox folder in your project's directory.

Before you begin, be sure your shell is not using a virtualenv.

First, install tox:

$ pipx install tox

Now you can navigate to one of the Klio sub-project directories and use tox to run unit tests. Currently tox is configured to run all tests using several different Python versions. This can take a while, so it's much faster when testing to just run with one environment:

$ tox -e py37

This might take a while the first time as tox sets up its envs and downloads dependencies, but after that it should be pretty fast.

To run a specific test:

$ tox -e py37 -- tests/unit/test_message_handler.py::test_preprocess_klio_message_non_klio

You can also use

$ tox -e format

to run black and auto-format your code, and

$ tox -e lint

to run flake8 which enforces a number of syntactic and formatting conventions. The CI will run both of these (with black in non-edit mode), so be sure to run these yourself!

Run tests via pytest and pyenv for Multi-project Dev

The above testing with tox will not work if you need to makes changes across several projects at once, instead you will need to create your own virtualenv and install your checked-out code as a local dependency. Then you'll bypass tox and run pytest directly.

Ensure that you have followed the above instructions and installed pyenv. You will also need to install pyenv-virtualenv. Now we'll use pyenv to create a new virtualenv called klio-dev based on Python 3.7.7:

$ pyenv virtualenv 3.7.7 klio-dev

We then have to activate it in order to use it:

$ pyenv activate klio-dev

Your console's prompt should now include (klio-dev) to help you keep track of which venv you're currently using. Now we will install your local klio-core as a library:

$ cd <klio-core-directory>
$ pip install -e ".[dev]"

Next, navigate to the klio-cli dir and do the same thing:

$ cd <klio-cli-directory>
$ pip install -e ".[dev]"

Now you can run pytest directly (do _not_ run tox)

# in klio-cli directory
$ pytest

You can now make changes in either project and they will be picked up immediately, no need to re-install each time.

If you want to try running the klio-cli command directly yourself, you will have to run one more command:

$ pyenv rehash

This is necessary to make sure your terminal is pointing to the local version.

Integration Testing with klio-devtools

The easiest way we've found to do local integration testing of changes to any Klio library before making PRs is:

  1. Identify or create a simple job that you can use to test your changes locally.
  2. Temporarily update the package versions locally for klio-exec, klio-core, and klio libraries (found in their top-level __init__.py). A helpful convention is to bump the patch version and add a suffix of .devN. For instance, if the current version is 1.2.3, update it to 1.2.4.dev1. This is to ensure the correct versions are installed, and not reusing an already-released version. Once you're satisfied with your changes, undo these changes then run bumpversion <part> as usual.
  3. Make sure the virtualenv for the klio-cli is activated, and that the klio-devtools package is installed (usually installed via pip install -e ".[dev]").
  4. In the directory of your simple job, run klio-dev develop --klio-path $PATH_TO_REPO. This will launch you into the job's container with klio, klio-core, and klio-exec installed as an editable Python package, and will pick up on any changes you make locally.

Note

When using klio-dev develop, the path to the root of the Klio repo can be either relative or absolute.

  1. Now that you're inside the job's container, you can run klioexec $CMD or whatever else is needed to do any manual integration testing.

Protobuf

klio uses protobuf for messages between transforms via Google Pub/Sub. The protobuf definition is located in the klio-core library.

Compile Protobuf

Attention!

If creating a new protobuf version, be sure to create a new directory and update the command/ tox config below.

Manually compile protobuf:

# Within top-level of repo
$ protoc \
    --proto_path src/klio/proto/v1beta1 klio.proto \
    --python_out src/klio/proto/v1beta1

Or via tox:

# outside a virtualenv
$ tox -e protoc

Documentation

All documentation related to the Klio ecosystem is in the top-level docs/ directory/

Generate Documentation

Create a new virtual environment for documentation & install dependencies:

$ pyenv virtualenv 3.7.7 klio-docs
$ pyenv activate klio-docs
(klio-docs) $ cd docs
(klio-docs) $ pip install -r requirements.txt -r klio-requirements.txt

Note

You may see some errors of version conflicts when installing documentation dependencies. However, these should be benign and should not get in the way of generating documentation.

Once the environment is setup, documentation can be generated and viewed via make:

# in the docs/ directory
(klio-docs) $ make clean && make html

To view them locally

# in the docs/ directory
(klio-docs) $ make clean && make livehtml

Then navigate to http://localhost:8888 in your browser.