Python implementation of the Neuroscience Data Interface (NDI) — a framework for managing, querying, and analyzing neuroscience experimental data.
NDI provides a unified interface for working with multi-modal neuroscience data (electrophysiology, imaging, stimulation) across different acquisition systems, with built-in support for time synchronization, document-based metadata, and cloud storage.
- Document management — JSON-schema-backed documents for experiments, subjects, probes, epochs, and more
- Database — SQLite-backed storage with rich querying (regex, numeric, dependency graphs)
- Time synchronization — Clock types, time mappings, and sync graphs for aligning data across devices
- DAQ readers — Built-in readers for Intan, Blackrock, CED Spike2, and SpikeGadgets formats
- Element/Probe hierarchy — Represent electrodes, optical fibers, and other recording devices
- Session management — Directory-backed sessions with epoch discovery and file navigation
- App framework — Extensible application framework with spike extraction, sorting, and stimulus analysis
- Calculator framework — Reusable computation pipelines (tuning curves, orientation selectivity)
- Cloud API — REST client for NDI Cloud with sync, upload/download, and DOI administration
- Ontology providers — 13 providers (OLS, NCBITaxon, PubChem, RRID, UniProt, and more)
- Schema validation — JSON Schema validation with superclass chain walking
- OpenMINDS integration — Convert openMINDS metadata objects to NDI documents
git clone https://github.com/Waltham-Data-Science/NDI-python.git
cd NDI-python
python -m venv venv
source venv/bin/activate # Linux/macOS (venv\Scripts\activate on Windows)
python ndi_install.pyThe installer clones all dependencies, installs packages, and validates your setup. Run python -m ndi check at any time to verify your installation.
python ndi_install.py --updatepython tutorials/tutorial_67f723d574f5f79c6062389d.py # Dabrowska dataset
python tutorials/tutorial_682e7772cdf3f24938176fac.py # Jess Haley datasetSee tutorials/README.md for full setup and cloud credentials.
NDI-Python requires these VH-Lab packages (installed automatically by ndi_install.py):
| Package | Repository | Purpose |
|---|---|---|
| DID-python | VH-Lab/DID-python | Document database backend (SQLite, queries) |
| vhlab-toolbox-python | VH-Lab/vhlab-toolbox-python | Data utilities, file formats, signal processing |
Additional dependencies (installed automatically): numpy, networkx, jsonschema, requests, scipy, pandas, matplotlib.
Manual installation (advanced)
git clone https://github.com/Waltham-Data-Science/NDI-python.git
cd NDI-python
python -m venv venv
source venv/bin/activate
# Clone vhlab-toolbox-python (not yet on PyPI)
git clone https://github.com/VH-Lab/vhlab-toolbox-python.git ~/.ndi/tools/vhlab-toolbox-python
export PYTHONPATH="$HOME/.ndi/tools/vhlab-toolbox-python:$PYTHONPATH"
# Install NDI-python (DID-python is resolved automatically via pip)
pip install -e ".[dev,tutorials]"
pip install scipy pandas matplotlib opencv-python-headless portalocker openmindsfrom ndi import Document, Query, DirSession
from ndi.session import MockSession
# Create a document with schema validation
doc = Document('base')
print(f"Document ID: {doc.id}")
# Query documents using Pythonic operators
q = Query('base.name') == 'my_experiment'
q_type = Query('').isa('subject')
q_combined = q & q_type
# Use MockSession for quick experimentation
with MockSession('test') as session:
session.database_add(Document('base'))
results = session.database_search(Query.all())
print(f"Found {len(results)} documents")
# Use DirSession for persistent, directory-backed sessions
session = DirSession('my_experiment', '/path/to/data')
session.database_add(doc)
results = session.database_search(Query('').isa('base'))from ndi.cloud.api.datasets import get_published_datasets
# Option 1: Set env vars and call without a client
# export NDI_CLOUD_USERNAME="you@example.com"
# export NDI_CLOUD_PASSWORD="your-password"
datasets = get_published_datasets()
# Option 2: Pass an explicit client
from ndi.cloud import CloudClient, login
config = login('you@example.com', 'your-password')
client = CloudClient(config)
datasets = get_published_datasets(client=client)All ndi.cloud.api.* functions accept an optional client parameter. If omitted, a client is built automatically from environment variables.
src/ndi/
├── Core
│ ├── document.py Document class with JSON schema loading
│ ├── query.py Query builder with operator overloading (==, &, |)
│ ├── database.py SQLite database backend
│ ├── ido.py Unique identifier generation (UUID-based)
│ ├── validate.py JSON Schema validation with superclass chains
│ └── common/ PathConstants, timestamp, logging
│
├── Data Acquisition
│ ├── daq/ DAQ system abstraction and readers
│ │ ├── reader/mfdaq/ Intan, Blackrock, CED Spike2, SpikeGadgets
│ │ └── metadatareader/ NewStim, NielsenLab metadata readers
│ ├── epoch/ Epoch, EpochSet, EpochProbeMap
│ └── file/ FileNavigator, EpochDirNavigator
│
├── Neural Elements
│ ├── element/ Element base class and utilities
│ ├── probe/ Probe, ProbeTimeseries, ProbeTimeseriesMFDAQ
│ ├── element_timeseries.py ElementTimeseries (data access)
│ ├── neuron.py Neuron class
│ └── subject.py Subject class
│
├── Sessions & Datasets
│ ├── session/ Session, DirSession, MockSession, SessionTable
│ ├── dataset.py Multi-session Dataset container
│ └── cache.py FIFO/LIFO cache implementations
│
├── Applications
│ ├── app/ App framework, SpikeExtractor, SpikeSorter
│ │ └── stimulus/ StimulusDecoder, TuningResponse
│ ├── calc/ Calculator framework
│ │ ├── example/ SimpleCalc
│ │ └── stimulus/ TuningCurveCalc, TuningFit
│ └── calculator.py Calculator base class with run loop
│
├── Cloud & Sync
│ ├── cloud/ CloudClient, CloudConfig
│ │ ├── api/ REST endpoints (datasets, documents, files, users)
│ │ ├── sync/ Sync engine (push, pull, index management)
│ │ └── admin/ DOI generation, Crossref submission
│ └── ontology/ 13 ontology providers with LRU cache
│
├── Utilities
│ ├── fun/ Utility functions (doc, epoch, file, data, stimulus)
│ ├── mock/ Mock data generators for testing
│ ├── database_fun.py Database search/export utilities
│ ├── database_ingestion.py File ingestion/expulsion system
│ ├── openminds_convert.py OpenMINDS object conversion
│ └── documentservice.py DocumentService mixin
│
└── Shared Data (ndi_common/)
├── database_documents/ 84 JSON document schemas
├── schema_documents/ JSON Schema validation files
├── probe/ Probe type → class mapping
└── ... Configs, ontology, vocabulary data
This package is a complete Python port of the MATLAB NDI codebase. See MATLAB_MAPPING.md for the full function-by-function mapping.
| Concept | MATLAB | Python |
|---|---|---|
| Query creation | ndi.query('field', 'exact_string', 'val') |
Query('field') == 'val' |
| Type query | ndi.query('', 'isa', 'subject') |
Query('').isa('subject') |
| Combine queries | ndi.query(q1, '&', q2) |
q1 & q2 |
| Method names | camelCase (e.g., setSessionId) |
snake_case (e.g., set_session_id) |
| Properties | doc.id() (method call) |
doc.id (property access) |
| Session ID | session.id() |
session.id() (still a method) |
DocumentService (mixin)
├── Subject
└── Element ← Ido + EpochSet + DocumentService
├── Probe (measurement devices)
│ └── ProbeTimeseries
│ ├── ProbeTimeseriesMFDAQ
│ └── ProbeTimeseriesStimulator
├── ElementTimeseries (data access)
└── Neuron
App (DocumentService)
├── MarkGarbage
├── SpikeExtractor
├── SpikeSorter
├── StimulusDecoder
├── TuningResponse
└── Calculator (App + AppDoc)
├── SimpleCalc
├── TuningCurveCalc
└── TuningFit (abstract)
Session (abstract)
├── DirSession (directory-backed)
└── MockSession (in-memory, for testing)
Dataset (multi-session container)
# Run all tests (excludes symmetry tests by default)
pytest tests/ -v
# Run specific test module
pytest tests/test_document.py -v
# Run with coverage
pytest tests/ --cov=src/ndi --cov-report=term-missingSymmetry tests verify that NDI-python and NDI-matlab produce identical artifacts.
They live in tests/symmetry/ and are excluded from the default test run
because they require artifacts that may have been generated by a prior MATLAB session.
# Step 1 — Generate Python artifacts
pytest tests/symmetry/make_artifacts/ -v
# Step 2 — (Optional) Run MATLAB makeArtifacts to generate MATLAB artifacts
# In MATLAB: results = runtests('ndi.symmetry.makeArtifacts');
# Step 3 — Verify artifacts from both languages
pytest tests/symmetry/read_artifacts/ -v
# Run all symmetry tests at once (make + read)
pytest tests/symmetry/ -vTests that cannot find their expected artifact directory are skipped (not
failed), so the suite runs safely on machines with only one language installed.
See docs/developer_notes/symmetry_tests.md for the full framework description.
CI enforces formatting and lint on every push/PR:
# Format code (must pass `black --check` in CI)
black src/ tests/
# Lint (must pass `ruff check` in CI)
ruff check src/ tests/
# Auto-fix lint issues
ruff check --fix src/ tests/
# Type check (optional, not yet enforced in CI)
mypy src/ndi/pip install -e ".[docs]"
mkdocs build
mkdocs serve # Local preview at http://127.0.0.1:8000- 1,704 tests passing across 30+ test files (Python 3.10, 3.11, 3.12)
- Covers all modules: core, DAQ, time, session, app, cloud, ontology, validation
- ~71% line coverage across
src/ndi/
This project is licensed under CC BY-NC-SA 4.0.
Contact Brandeis University Office of Technology Licensing for commercial licensing.
- VH-Lab at Brandeis University — original MATLAB NDI codebase
- Audri Bhowmick / Waltham Data Science — Python port
- NDI Cloud — ndi-cloud.com for cloud storage and sharing