From 494212088bf828cb522ea1397551dd5d5622090a Mon Sep 17 00:00:00 2001 From: Frank Loesche Date: Wed, 26 Nov 2025 16:55:11 -0500 Subject: [PATCH 1/7] move page structure from js to json --- docs/developer-guide.md | 1869 ++++------------- docs/user-guide.md | 1222 +++++------ .../services/index_generator_service.py | 176 +- templates/data/neurons.js.template.jinja | 26 + templates/index.html.jinja | 154 +- .../static/js/neuron-search.js.template.jinja | 221 +- 6 files changed, 1370 insertions(+), 2298 deletions(-) create mode 100644 templates/data/neurons.js.template.jinja diff --git a/docs/developer-guide.md b/docs/developer-guide.md index c05d65f6..d65b1f25 100644 --- a/docs/developer-guide.md +++ b/docs/developer-guide.md @@ -1,27 +1,22 @@ # neuView Developer Guide -A comprehensive guide for developers working on the neuView neuron visualization platform. This guide covers architecture, development setup, implementation details, and contribution guidelines. +A comprehensive guide for developers working on the neuView neuron visualization platform. ## Table of Contents - [Project Overview](#project-overview) - [Architecture Overview](#architecture-overview) - [Getting Started](#getting-started) +- [Neuron Data System](#neuron-data-system) - [Core Components](#core-components) - [Service Architecture](#service-architecture) - [Data Processing Pipeline](#data-processing-pipeline) -- [Visualization System](#visualization-system) - [Template System](#template-system) - [Performance & Caching](#performance--caching) -- [Development Patterns](#development-patterns) - [Testing Strategy](#testing-strategy) -- [Dataset Aliases](#dataset-aliases) - [Configuration](#configuration) -- [API Reference](#api-reference) - [Dataset-Specific Implementations](#dataset-specific-implementations) -- [Feature Implementation Guides](#feature-implementation-guides) - [Troubleshooting](#troubleshooting) -- [Contributing](#contributing) ## Project Overview @@ -30,11 +25,11 @@ neuView is a modern Python CLI tool that generates beautiful HTML pages for neur ### Key Features - **🔌 NeuPrint Integration**: Direct data fetching with intelligent caching -- **📱 Modern Web Interface**: Responsive design with advanced filtering -- **⚡ High Performance**: Up to 97.9% speed improvement with persistent caching +- **📱 Modern Web Interface**: Responsive design with advanced filtering and search +- **⚡ High Performance**: Persistent caching with optimal data loading strategies - **🧠 Multi-Dataset Support**: Automatic adaptation for CNS, Hemibrain, Optic-lobe, FAFB - **🎨 Beautiful Reports**: Clean, accessible HTML pages with interactive features -- **🔍 Advanced Search**: Real-time filtering by cell count, neurotransmitter, brain regions +- **🔍 Advanced Search**: Real-time autocomplete search with synonym and FlyWire type support ### Technology Stack @@ -47,25 +42,13 @@ neuView is a modern Python CLI tool that generates beautiful HTML pages for neur ### Architecture Overview -neuView follows a layered architecture pattern with four distinct layers: +neuView follows a layered architecture pattern: - **Presentation Layer**: CLI Commands, Templates, Static Assets, HTML Generation - **Application Layer**: Services, Orchestrators, Command Handlers, Factories - **Domain Layer**: Entities, Value Objects, Domain Services, Business Logic - **Infrastructure Layer**: Database, File System, External APIs, Caching, Adapters -For detailed architecture implementation, see `src/neuview/` directory structure. - -### Key Architectural Principles - -- **Separation of Concerns**: Clear boundaries between layers -- **Dependency Inversion**: High-level modules don't depend on low-level modules -- **Single Responsibility**: Each component has one well-defined purpose -- **Open/Closed Principle**: Open for extension, closed for modification -- **Command/Query Separation**: Clear distinction between data modification and retrieval -- **Result Pattern**: Explicit error handling with Result types -- **Service Container**: Dependency injection for loose coupling - ## Getting Started ### Prerequisites @@ -77,1642 +60,558 @@ For detailed architecture implementation, see `src/neuview/` directory structure ### Development Setup -1. **Clone the repository:** -Clone the repository and navigate to the project directory. - -2. **Install dependencies:** -Install dependencies using `pixi install`. - -3. **Set up environment:** -Set up the environment using `pixi run setup-env` and edit the .env file with your NeuPrint token. - -4. **Verify setup:** -Test the connection using `pixi run neuview test-connection`. +1. **Clone the repository** +2. **Install dependencies**: `pixi install` +3. **Set up environment**: `pixi run setup-env` and edit `.env` with your NeuPrint token +4. **Verify setup**: `pixi run neuview test-connection` ### Development Commands -neuView uses pixi for task management with separate commands for different types of work: +**Testing:** +- `pixi run unit-test` - Run all unit tests +- `pixi run integration-test` - Run integration tests +- `pixi run test` - Run all tests -#### Testing Tasks +**Code Quality:** +- `pixi run lint` - Run ruff linter +- `pixi run format` - Format code with ruff -**Unit Tests** - Fast, isolated tests for individual components: -**Unit Test Commands** (defined in `pixi.toml`): -- `pixi run unit-test` - Run all unit tests -- `pixi run unit-test-verbose` - Detailed output with specific file/test targeting support +**Content Generation:** +- `pixi run neuview build` - Generate website +- `pixi run neuview inspect ` - Inspect neuron type data -**Integration Tests** - End-to-end tests for component interactions: -**Integration Test Commands** (defined in `pixi.toml`): -- `pixi run integration-test` - Run all integration tests -- `pixi run integration-test-verbose` - Detailed output with specific file targeting support +## Neuron Data System -**General Testing**: -**Combined Test Commands** (defined in `pixi.toml`): -- `pixi run test` - Run all tests (unit + integration) -- `pixi run test-verbose` - Detailed output for all tests -- `pixi run test-coverage` - Generate coverage reports +The neuron search functionality uses a dual-source data loading system that provides universal compatibility and external API access. -#### Code Quality Tasks +### Architecture -**Code Quality Commands** (defined in `pixi.toml`): -- `pixi run format` - Format code with ruff -- `pixi run check` - Run linting and quality checks +``` +output/ +├── data/ +│ ├── neurons.json # Primary: JSON for external services & HTTP(S) +│ └── neurons.js # Fallback: JavaScript wrapper for file:// access +└── static/ + └── js/ + └── neuron-search.js # Search logic (no embedded data) +``` -#### Content Generation Tasks +### Data Format (Version 2.0) -**Content Generation Commands** (defined in `pixi.toml`): -- `pixi run clean-output` - Clean generated output -- `pixi run fill-all` - Fill processing queue with all neuron types -- `pixi run pop-all` - Process all items in queue -- `pixi run create-list` - Generate index page -- `pixi run create-all-pages` - Complete workflow automation +**Structure:** +```json +{ + "names": ["AN07B013", "AOTU001", ...], + "neurons": [ + { + "name": "AN07B013", + "urls": { + "combined": "AN07B013.html", + "left": "AN07B013_L.html", + "right": "AN07B013_R.html" + }, + "types": { + "flywire": ["AN_GNG_60"], + "synonyms": ["Cachero 2010: aSP-j"] + } + } + ], + "metadata": { + "generated": "2025-01-26 15:04:31", + "total_types": 28, + "version": "2.0" + } +} +``` -Queue management implemented in `src/neuview/services/queue_service.py`. +**Key Points:** +- URLs stored WITHOUT `types/` prefix (added dynamically in JavaScript) +- `types.flywire` and `types.synonyms` are arrays +- `types` field only included when it has content +- Same structure for both `neurons.json` and `neurons.js` -#### Development Support Tasks +### Data Loading Flow -**Development Support Commands** (defined in `pixi.toml`): -- `pixi run setup-env` - Setup development environment -- `pixi run help` - CLI help system -- `pixi run subset-medium` / `pixi run subset-medium-no-index` - Generate medium-sized test datasets -- `pixi run subset-small` / `pixi run subset-small-no-index` - Generate small test datasets -- `pixi run extract-and-fill` - Batch processing from config files +**Web Server (HTTP/HTTPS):** +1. neuron-search.js loads +2. Attempts `fetch('data/neurons.json')` +3. Success → Uses JSON data (optimal) +4. neurons.js never downloaded -Implementation in `scripts/extract_and_fill.py` and CLI modules. +**Local Files (file://):** +1. neuron-search.js loads +2. Attempts `fetch('data/neurons.json')` +3. Fails (CORS restriction) +4. Dynamically loads `