From 5a13cf3ba454ab7caad65a11c53f9b9b5db458ca Mon Sep 17 00:00:00 2001 From: Anton Gulenko Date: Wed, 27 May 2020 15:17:15 +0200 Subject: [PATCH] Update index.md --- docs/index.md | 65 ++++++++++++++++++--------------------------------- 1 file changed, 23 insertions(+), 42 deletions(-) diff --git a/docs/index.md b/docs/index.md index ba4659b..e6c3e9d 100644 --- a/docs/index.md +++ b/docs/index.md @@ -4,62 +4,43 @@ [![Reliability](https://ci.bitflow.team/sonarqube/api/project_badges/measure?project=python-bitflow&metric=reliability_rating)](http://wally144.cit.tu-berlin.de/sonarqube/dashboard?id=python-bitflow) # python-bitflow -**python-bitflow** is a python library for sending, receiving and transforming streams of data. Read the documentation under [bitflow.rtfd.io](https://bitflow.readthedocs.io/en/latest/ "Bitflow documentation") to learn more about bitflow and the correlated projects. +**python-bitflow** is a plugin for [`go-bitflow`](https://github.com/bitflow-stream/go-bitflow) that allows writing and executing datastream operators in Python. +Python operators can be used inside the Bitflowscript executed by `go-bitflow`. +The main `go-bitflow` dataflow graph runs as a single Go process, while each `python-bitflow` operator is executed in a separate child process that receives input data over the standard input and produces results on the standard output. -Python-bitflow is current running in python3.7. It may run under other python3.x version but is not tested yet. +Tested Python version: 3.8 -## Installation -You can install python-bitflow by running the following command. This will put the exection script python-bitflow into your $PATH and will make the bitflow classes availble within your current python environmnet. -``` -python setup.py install -``` -## Python-Bitflow script and usage examples -Bitflow script is a powerfull scripting language. It allows to define pipelines, processing steps (order), forks and more. +# Installation (building Docker container) -#### Help! -Get help: -``` -python-bitflow --help -``` +Dockerfiles are provided for the amd64 platform (based on Alpine Linux) and arm32v7. +Select one of these two as the build TARGET and run the following in the repository root: -#### Python-Bitflow, Capabilities -Get an overview of all available processing steps and forks ``` -python-bitflow -capabilities +TARGET=[alpine|arm32v7] +docker build -t IMAGE_NAME -f build/$TARGET.Dockerfile . ``` -#### Script example 1. reading file into Noop processing step -``` -python-bitflow -script "testing/testing_file_in.txt -> Noop()"" -``` +# Dockerhub -#### Script example 2. reading file into PlotLinePlot processing step -This will generate in .png file in the current path -``` -python-bitflow -script "testing/in.csv-> PlotLinePlot(metric_names='ongoing_connections')" -``` +Docker container images are available in the [`bitflowstream/bitflow-pipeline-python`](https://hub.docker.com/repository/docker/bitflowstream/bitflow-pipeline-python) Dockerhub repository: -#### Script example 3. reading file into PlotLinePlot processing step -Reads testing file, if tag "filter" is set to "port_1935" the metric pkg_out_1300-1400 will be plottet, else if tag "filter" is set to "port_1936" the metric pkg_out_1400-1500. Afterwards both kind of samples got forwarded to a Noop() processing step. ``` -python-bitflow -script "testing/in.csv -> Fork_Tags(tag='filter'){ port_1935 -> PlotLinePlot(metric_names='pkg_out_1300-1400') ; port_1936 -> PlotLinePlot(metric_names='pkg_out_1400-1500') } -> Noop()" +docker pull bitflowstream/bitflow-pipeline-python +docker run bitflowstream/bitflow-pipeline-python --help ``` -#### Script example 4. load processing step from external file (requires installation) -``` -python-bitflow -script "testing/in.csv -> my_processing()" -p my_processing.py -``` -Current version does not close properly in all cases. Use strg-C to exit. +The Docker manifest will select the appropriate platform (amd64/arm32v7) automatically. -## Library Examples -**bitflow-example.py**: provides a short overview about how to setup a pipeline and initialize source,sink, and processing steps. +# Usage -**download-data.py**: downloads data from a given endpoint, filters samples +For the usage of the [`bitflowstream/bitflow-pipeline-python`](https://hub.docker.com/repository/docker/bitflowstream/bitflow-pipeline-python) container, see the [`go-bitflow`](https://github.com/bitflow-stream/go-bitflow) documentation. -**provide-data.py**: reads a file and provides this file via a listen port +`python-bitflow` allows to use the `python(step=NAME, args={ a1=v1, a2=v2 }, exe-args=PYTHON_ARGS)` operator. +When starting, `python-bitflow` scans for subclasses of [`bitflow.ProcessingStep`](bitflow/runner.py). +All non-abstract classes can be used as `NAME` in the `python()` operator, by default referred to through their class name. -## TODO -* closing python-bitflow properly +Example: -## Known Issues: -* Forks are currently not listed in -capabilities +``` +docker run -p 8888 -ti bitflowstream/bitflow-pipeline-python ':8888 -> python(step=echo, args={msg=hello}) -> text://-' +```