reader

reader is a workbench for analyzing experimental data and a library/CLI that provides supporting commands.

Contents

1. Overview
2. Installation
3. Quickstart
4. More documentation

---

Overview

1. Place raw data and metadata in an experiment directory.

   experiments/
       <exp>/
           config.yaml     # experiment config
           inputs/         # raw files + metadata (e.g., metadata.xlsx)
           outputs/        # pipeline outputs, plots, exports, notebooks

2. Run config-driven pipelines that process data and produce structured outputs.

   outputs/
     reader.log      # run log
     manifests/      # inventories what was produced
     artifacts/      # pipeline outputs (dataframes + metadata)
     plots/          # rendered figures (optional)
     exports/        # exported files (optional)
     notebooks/      # scaffolded marimo notebooks

3. Optionally generate plots and exports data.

4. Use notebooks for exploratory data analysis.

---

Installation

This repo is managed with uv:

1. pyproject.toml declares dependencies (runtime + optional extras).
2. uv.lock is the fully pinned dependency graph.
3. .venv/ is the project virtual environment.

Two key commands:

1. uv sync installs everything from the lockfile into .venv.
2. uv run <cmd> runs commands inside the project environment without requiring source .venv/bin/activate.

Install uv. (Below is for macOS/Linux, for other OSs see here.)

curl -LsSf https://astral.sh/uv/install.sh | sh
# ensure your uv bin dir is on PATH

Clone repo

git clone https://github.com/e-south/reader.git
cd reader

Create/sync the environment from the committed lockfile:

uv sync --locked

Sanity checks:

uv run python -c "import reader, pandas, pyarrow; print('ok')"

Dev tooling is opt-in via a dependency group.

uv sync --locked --group dev --group notebooks --group cytometry --extra dnadesign
uv run ruff --version
uv run ruff check .
uv run pytest -q

This project defines console scripts, which you can run via:

Option A: no .venv activation — use uv run

uv run reader --help
uv run reader ls

Option B: traditional — activate .venv

source .venv/bin/activate
reader --help
reader ls
deactivate

---

Quickstart

Try the guided walkthrough:

uv run reader demo

Create an experiment directory:

# From the reader root
mkdir -p experiments/<year>/my_experiment/{inputs,outputs}

Design and run a pipeline via config.yaml:

uv run reader explain experiments/my_experiment/config.yaml
uv run reader run     experiments/my_experiment/config.yaml   # pipeline (artifacts)

Generate plots/exports or scaffold a notebook:

uv run reader plot         experiments/my_experiment/config.yaml --list
uv run reader export       experiments/my_experiment/config.yaml --list
uv run reader notebook     experiments/my_experiment/config.yaml

---

More documentation

1. CLI reference
2. Configuring pipelines
3. Notebooks
4. Plugin development
5. Spec / architecture
6. End-to-end demo

---

@e-south