Skip to content

Update index.md #48

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
antongulenko wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Update index.md #48

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 23 additions & 42 deletions docs/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,62 +4,43 @@
[![Reliability](https://ci.bitflow.team/sonarqube/api/project_badges/measure?project=python-bitflow&metric=reliability_rating)](http://wally144.cit.tu-berlin.de/sonarqube/dashboard?id=python-bitflow)

# python-bitflow
**python-bitflow** is a python library for sending, receiving and transforming streams of data. Read the documentation under [bitflow.rtfd.io](https://bitflow.readthedocs.io/en/latest/ "Bitflow documentation") to learn more about bitflow and the correlated projects.
**python-bitflow** is a plugin for [`go-bitflow`](https://github.com/bitflow-stream/go-bitflow) that allows writing and executing datastream operators in Python.
Python operators can be used inside the Bitflowscript executed by `go-bitflow`.
The main `go-bitflow` dataflow graph runs as a single Go process, while each `python-bitflow` operator is executed in a separate child process that receives input data over the standard input and produces results on the standard output.

Python-bitflow is current running in python3.7. It may run under other python3.x version but is not tested yet.
Tested Python version: 3.8

## Installation
You can install python-bitflow by running the following command. This will put the exection script python-bitflow into your $PATH and will make the bitflow classes availble within your current python environmnet.
```
python setup.py install
```
## Python-Bitflow script and usage examples
Bitflow script is a powerfull scripting language. It allows to define pipelines, processing steps (order), forks and more.
# Installation (building Docker container)

#### Help!
Get help:
```
python-bitflow --help
```
Dockerfiles are provided for the amd64 platform (based on Alpine Linux) and arm32v7.
Select one of these two as the build TARGET and run the following in the repository root:

#### Python-Bitflow, Capabilities
Get an overview of all available processing steps and forks
```
python-bitflow -capabilities
TARGET=[alpine|arm32v7]
docker build -t IMAGE_NAME -f build/$TARGET.Dockerfile .
```

#### Script example 1. reading file into Noop processing step
```
python-bitflow -script "testing/testing_file_in.txt -> Noop()""
```
# Dockerhub

#### Script example 2. reading file into PlotLinePlot processing step
This will generate in .png file in the current path
```
python-bitflow -script "testing/in.csv-> PlotLinePlot(metric_names='ongoing_connections')"
```
Docker container images are available in the [`bitflowstream/bitflow-pipeline-python`](https://hub.docker.com/repository/docker/bitflowstream/bitflow-pipeline-python) Dockerhub repository:

#### Script example 3. reading file into PlotLinePlot processing step
Reads testing file, if tag "filter" is set to "port_1935" the metric pkg_out_1300-1400 will be plottet, else if tag "filter" is set to "port_1936" the metric pkg_out_1400-1500. Afterwards both kind of samples got forwarded to a Noop() processing step.
```
python-bitflow -script "testing/in.csv -> Fork_Tags(tag='filter'){ port_1935 -> PlotLinePlot(metric_names='pkg_out_1300-1400') ; port_1936 -> PlotLinePlot(metric_names='pkg_out_1400-1500') } -> Noop()"
docker pull bitflowstream/bitflow-pipeline-python
docker run bitflowstream/bitflow-pipeline-python --help
```

#### Script example 4. load processing step from external file (requires installation)
```
python-bitflow -script "testing/in.csv -> my_processing()" -p my_processing.py
```
Current version does not close properly in all cases. Use strg-C to exit.
The Docker manifest will select the appropriate platform (amd64/arm32v7) automatically.

## Library Examples
**bitflow-example.py**: provides a short overview about how to setup a pipeline and initialize source,sink, and processing steps.
# Usage

**download-data.py**: downloads data from a given endpoint, filters samples
For the usage of the [`bitflowstream/bitflow-pipeline-python`](https://hub.docker.com/repository/docker/bitflowstream/bitflow-pipeline-python) container, see the [`go-bitflow`](https://github.com/bitflow-stream/go-bitflow) documentation.

**provide-data.py**: reads a file and provides this file via a listen port
`python-bitflow` allows to use the `python(step=NAME, args={ a1=v1, a2=v2 }, exe-args=PYTHON_ARGS)` operator.
When starting, `python-bitflow` scans for subclasses of [`bitflow.ProcessingStep`](bitflow/runner.py).
All non-abstract classes can be used as `NAME` in the `python()` operator, by default referred to through their class name.

## TODO
* closing python-bitflow properly
Example:

## Known Issues:
* Forks are currently not listed in -capabilities
```
docker run -p 8888 -ti bitflowstream/bitflow-pipeline-python ':8888 -> python(step=echo, args={msg=hello}) -> text://-'
```