From c5ca57a37f4f846ae5e653cb59983cdd4bc5a635 Mon Sep 17 00:00:00 2001 From: Frank Loesche Date: Mon, 1 Dec 2025 11:10:16 -0500 Subject: [PATCH 01/10] improve documentation of connections and synapses --- templates/help.html.jinja | 43 +++++++++++++++++++++------------------ 1 file changed, 23 insertions(+), 20 deletions(-) diff --git a/templates/help.html.jinja b/templates/help.html.jinja index 59d59de..e3645c5 100644 --- a/templates/help.html.jinja +++ b/templates/help.html.jinja @@ -15,7 +15,7 @@ - + {# Table of Contents #}

Contents

@@ -55,7 +55,7 @@
- + {# Overview #}

Overview

@@ -82,7 +82,7 @@

- + {# Types Page #}

Types Index Page

@@ -96,7 +96,7 @@ How to use the types filters. - + {# Using Filters #}

Using Filters

@@ -313,7 +313,7 @@

- + {# Individual Neuron Pages #}

Individual Neuron Pages

@@ -327,7 +327,7 @@ side of the page or the menu (☰) to switch between views.

- + {# Header Info #}

Header Information

@@ -338,7 +338,7 @@

- + {# Summary Stats #}

Summary Statistics

The summary section provides key quantitative information:

@@ -372,7 +372,7 @@

- + {# Layer Analysis #}

Layer Analysis

Detailed analysis of neuron distribution across layers in visual system neuropils:

@@ -407,7 +407,7 @@ Interactivity of the hexagonal eyemap plots.
- + {# Neuroglancer #}

Neuroglancer 3D Viewer

Interactive 3D visualization of neuron morphology:

@@ -458,7 +458,7 @@

- + {# ROI Innervation #}

ROI Innervation

Detailed breakdown of connectivity within specific brain regions:

@@ -470,7 +470,7 @@
- + {# Connectivity #}

Connectivity Data

Comprehensive analysis of synaptic connections with other neuron types:

@@ -537,7 +537,7 @@
- + {# Terminology #}

Terminology

Key terms and concepts used throughout the catalog:

@@ -556,20 +556,23 @@
Pre-synaptic / Post-synaptic
- Pre-synaptic refers to outputs from a neuron; post-synaptic refers to inputs to a neuron. + Post-synaptic refers to inputs to a neuron, pre-synaptic refers to outputs from a neuron.
Synapse Count
-
The number of synaptic connections. Higher counts indicate stronger or denser connectivity.
+
The number of synaptic connections. Higher counts indicate stronger or denser connectivity. neuView uses the properties from the neurons in the database.
Post-synaptic Sites
-
Postsynaptic receptor sites where a neuron receives inputs from presynaptic partners.
+
Postsynaptic receptor sites where a neuron receives inputs from presynaptic partners. In the data model, this is the `Neuron.post` property.
Pre-synaptic Sites (T-bars)
-
Presynaptic release sites where a neuron outputs signals. Also called t-bars. One presynaptic site can connect to multiple postsynaptic partners.
+
Presynaptic release sites where a neuron outputs signals. Also called t-bars. One presynaptic site can connect to multiple postsynaptic partners. In the neuprint data model, this is the `Neuron.pre` property.
+ +
Connections
+
Input connections are conceptually similar to post-synaptic sites while a group of post-synaptic sites form a single output connection. We calculate the number of connections by identifying the connecting neurons and then use the `weight` property from the `ConnectsTo` edge between the neurons.
Connections vs. Synapses
-
A single presynaptic site (t-bar) can form multiple connections to different postsynaptic partners. Therefore, the number of output connections can be greater than the number of presynaptic sites.
+
The numeric difference between post-synaptic sites and input connections stems from the difference in how the numbers are retrieves (see Synapse Count and Connections above). A single presynaptic site (t-bar) can form multiple connections to different postsynaptic partners. Therefore, the number of output connections can be greater than the number of presynaptic sites.
Inputs
All pre-synaptic connecting cell types upstream from a given cell type (neurons providing input to the cell type).
@@ -622,7 +625,7 @@
- + {# FAQ and Troubleshooting #}

FAQ and Troubleshooting

@@ -662,7 +665,7 @@
- + {# Contact and Feedback #}

Contact and Feedback

@@ -679,7 +682,7 @@

- + {# Footer #}
From fde7b1bbd61f89fc552d420f008443913d0959ae Mon Sep 17 00:00:00 2001 From: Frank Loesche Date: Mon, 1 Dec 2025 13:53:13 -0500 Subject: [PATCH 02/10] update summary statistics section --- templates/help.html.jinja | 39 ++++++++++++++++++++++++++------------- 1 file changed, 26 insertions(+), 13 deletions(-) diff --git a/templates/help.html.jinja b/templates/help.html.jinja index e3645c5..35d83b2 100644 --- a/templates/help.html.jinja +++ b/templates/help.html.jinja @@ -343,33 +343,46 @@

Summary Statistics

The summary section provides key quantitative information:

-

Total Neurons

+

Neurons

- The number of individual neurons of this type in the dataset. For bilateral neurons, counts and ratios may + The number of individual neurons of this type in the dataset. For bilateral neurons, counts per hemisphere and ratios may be shown across hemispheres.

-

Total Synapses

-

Combined pre-synaptic (outputs) and post-synaptic (inputs), with breakdowns where applicable. - For combined pages the breakdown shows the number of neurons assigned to each hemisphere. - Whereas for unilateral pages the breakdown shows the number of pre and post synapses across all neurons of the type.

+

Synapses

+

Combined post-synaptic (inputs) and pre-synaptic (outputs), with breakdowns and log ratios where applicable. Technically, this number is a pre-calculated property for each neuron inside the neuprint database.
+ For combined pages the breakdown shows the number of neurons assigned to each hemisphere and their ratio. + Whereas for unilateral pages the breakdown shows the number of post and pre synapses across all neurons of the type assigned to that side.

-

Mean Synapses

-

Average number of synapses per neuron of this type. Useful for comparing connectivity levels. +

Connections

+

+ Combined upstream and downstream connections. Here, connections are the number of anatomical synapses between neurons. Technically, this number comes from the number of connected synapses, which are combined to SynapseSets and part of Neurons inside the neuprint database.
+ For combined pages the breakdown shows the number of neurons assigned to each hemisphere and their ratio. + Whereas for unilateral pages the breakdown shows the number of upstream and downstream connections across all neurons of the type assigned to that side. +

+ +

Neurotransmitter

+

+ Predicted neurotransmitter for the type and the corresponding confidence level (CL). + For some types, the predicted neurotransmitter may be Unc for unclear. +

+ +

Synapses per Neuron

+

Average number of synapses per neuron of this type. Useful for comparing connectivity levels.
For the combined page, the breakdown shows the mean number of synapses for neurons on the right and left. Whereas for unilateral pages the breakdown shows the mean number of pre and post synapses per neuron assigned to that side.

+

Connections per Neuron

+

Average number of connections per neuron of this type. Useful for comparing connectivity levels.
+ For the combined page, the breakdown shows the mean number of connections for neurons on the right and left. + Whereas for unilateral pages the breakdown shows the mean number of pre and post connections per neuron assigned to that side.

+

Log Ratios

Log ratios (ld, base 2) indicate balance across hemispheres and inputs/outputs; values below 0 favor right/inputs and values above 0 favor left/outputs.

-

Neurotransmitter

-

- Predicted neurotransmitter for the type and the corresponding confidence level (CL). - For some types, the predicted neurotransmitter may be "Unc" for unclear. -

{# Layer Analysis #} From 1a3aa0a4975b5be773c8b4e0df439a8fc81efb52 Mon Sep 17 00:00:00 2001 From: Frank Loesche Date: Mon, 1 Dec 2025 14:40:20 -0500 Subject: [PATCH 03/10] make help fit types page --- templates/help.html.jinja | 37 +++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 18 deletions(-) diff --git a/templates/help.html.jinja b/templates/help.html.jinja index 35d83b2..2813f93 100644 --- a/templates/help.html.jinja +++ b/templates/help.html.jinja @@ -40,11 +40,10 @@
  • Terminology
    • @@ -385,12 +384,13 @@
    - {# Layer Analysis #} -
    -

    Layer Analysis

    -

    Detailed analysis of neuron distribution across layers in visual system neuropils:

    + {# Columnar Layer and Spatial Coverage (Eye Maps) #} +
    +

    Layer and Column Coverage

    +

    Note: These views are only present for visual system neuron types with columnar organization.

    +

    Detailed analysis of neuron distribution across layers in visual system neuropils. This section explains how to interpret the “Mean Synapse Count per Layer” tables shown on individual neuron type pages (e.g., Tm3).

    -

    Layer Tables

    +

    Mean Synapse Count per Layer

    • Layer names and anatomical locations: Lamina (LA), Medulla (ME), Lobula (LO), Accessory Medulla (AME), Lobula Plate (LOP) and Central Brain (CB).
      • @@ -407,8 +407,9 @@ the tables include only the counts for the ipsilateral synapses.
    -

    Hexagonal Grid Visualizations

    -

    Interactive hexagonal grids show the spatial distribution of neuron terminals across columnar structure of visual system regions.

    +

    Population Spatial Coverage

    +

    Note: These Population Spatial Coverage maps are only present for visual system neuron types with columnar organization.

    +

    This section visualizes how a neuron type covers the columnar structure of visual system regions using interactive hexagonal grids.

    The plots show the neural innervation of the ipsilateral medulla (ME), lobula (LO) and lobula plate (LOP).

    • Column Analysis: Which columns within the ROI the neurons innervate.
      • @@ -420,10 +421,10 @@ Interactivity of the hexagonal eyemap plots.
    - {# Neuroglancer #} -
    -

    Neuroglancer 3D Viewer

    -

    Interactive 3D visualization of neuron morphology:

    + {# Neuron Visualization #} +
    +

    Neuron Visualization

    +

    Interactive 3D visualization of neuron morphology using the embedded Neuroglancer viewer:

    Viewing the 3D Model

      @@ -456,7 +457,7 @@

      Adding Partners from the Connectivity table

    You can add connected partners to the 3D view directly from the Connectivity tables by selecting their - blue checkboxes (see Connectivity Data). Some connected cell types have filled pink checkboxes because there + blue checkboxes (see Connectivity section below). Some connected cell types have filled pink checkboxes because there are no neurons of that type that directly innervate the example neuron visible by default in the neuroglancer view. A random neuron of that type can be added by searching for the type in the search box within the neuroglancer view and adding it manually.

    @@ -484,8 +485,8 @@
    {# Connectivity #} -
    -

    Connectivity Data

    +
    +

    Connectivity

    Comprehensive analysis of synaptic connections with other neuron types:

    • Upstream Partners: Neurons that provide input (pre-synaptic partners)
      • From 5ccca826be2ab9bbb4093fa5ebab72aa6226d801 Mon Sep 17 00:00:00 2001 From: Frank Loesche Date: Mon, 1 Dec 2025 15:25:13 -0500 Subject: [PATCH 04/10] obvious fixes --- .env.example | 4 ---- docs/developer-guide.md | 24 ++++++++++-------------- 2 files changed, 10 insertions(+), 18 deletions(-) diff --git a/.env.example b/.env.example index 5792ba4..9335924 100644 --- a/.env.example +++ b/.env.example @@ -4,7 +4,3 @@ # NeuPrint authentication token # Get your token from https://neuprint.janelia.org/ NEUPRINT_TOKEN=your_neuprint_token_here - -# Optional: Override other configuration settings -# NEUPRINT_SERVER=neuprint.janelia.org -# NEUPRINT_DATASET=hemibrain:v1.2.1 diff --git a/docs/developer-guide.md b/docs/developer-guide.md index de2a27b..6ef15da 100644 --- a/docs/developer-guide.md +++ b/docs/developer-guide.md @@ -39,7 +39,7 @@ neuView is a modern Python CLI tool that generates beautiful HTML pages for neur ### Technology Stack - **Backend**: Python 3.11+, asyncio for async processing -- **Data Layer**: NeuPrint API, persistent caching with SQLite +- **Data Layer**: NeuPrint API, persistent file-based caching with pickle - **Frontend**: Modern HTML5, CSS3, vanilla JavaScript - **Templates**: Jinja2 with custom filters and extensions - **Testing**: pytest with comprehensive coverage @@ -96,30 +96,30 @@ neuView uses pixi for task management with separate commands for different types #### Testing Tasks **Unit Tests** - Fast, isolated tests for individual components: -**Unit Test Commands** (defined in `pixi.toml`): +**Unit Test Commands** (defined in `pyproject.toml`): - `pixi run unit-test` - Run all unit tests - `pixi run unit-test-verbose` - Detailed output with specific file/test targeting support **Integration Tests** - End-to-end tests for component interactions: -**Integration Test Commands** (defined in `pixi.toml`): +**Integration Test Commands** (defined in `pyproject.toml`): - `pixi run integration-test` - Run all integration tests - `pixi run integration-test-verbose` - Detailed output with specific file targeting support **General Testing**: -**Combined Test Commands** (defined in `pixi.toml`): +**Combined Test Commands** (defined in `pyproject.toml`): - `pixi run test` - Run all tests (unit + integration) - `pixi run test-verbose` - Detailed output for all tests - `pixi run test-coverage` - Generate coverage reports #### Code Quality Tasks -**Code Quality Commands** (defined in `pixi.toml`): +**Code Quality Commands** (defined in `pyproject.toml`): - `pixi run format` - Format code with ruff - `pixi run check` - Run linting and quality checks #### Content Generation Tasks -**Content Generation Commands** (defined in `pixi.toml`): +**Content Generation Commands** (defined in `pyproject.toml`): - `pixi run clean-output` - Clean generated output - `pixi run fill-all` - Fill processing queue with all neuron types - `pixi run pop-all` - Process all items in queue @@ -130,7 +130,7 @@ Queue management implemented in `src/neuview/services/queue_service.py`. #### Development Support Tasks -**Development Support Commands** (defined in `pixi.toml`): +**Development Support Commands** (defined in `pyproject.toml`): - `pixi run setup-env` - Setup development environment - `pixi run help` - CLI help system - `pixi run subset-medium` / `pixi run subset-medium-no-index` - Generate medium-sized test datasets @@ -143,7 +143,7 @@ Implementation in `scripts/extract_and_fill.py` and CLI modules. The project includes automated version management for releases: -**Version Management** (defined in `pixi.toml`): +**Version Management** (defined in `pyproject.toml`): - `pixi run increment-version` - Increment patch version and create git tag Implementation in `scripts/increment_version.py` with `--dry-run` support for testing. @@ -640,7 +640,7 @@ End-to-end tests that verify component interactions and real-world scenarios. #### Test Execution Commands -**Test execution tasks are defined in `pixi.toml`:** +**Test execution tasks are defined in `pyproject.toml`:** - Unit tests: `pixi run unit-test` (fast feedback) - Integration tests: `pixi run integration-test` (comprehensive testing) - All tests: `pixi run test` (complete test suite) @@ -787,11 +787,7 @@ See current utility scripts in `scripts/` directory for reference implementation Environment variable support for sensitive configuration: -- `NEUPRINT_APPLICATION_CREDENTIALS`: NeuPrint API token -- `NEUVIEW_CONFIG_PATH`: Custom configuration file path -- `NEUVIEW_CACHE_DIR`: Cache directory override -- `NEUVIEW_DEBUG`: Enable debug logging -- `NEUVIEW_PROFILE`: Enable performance profiling +- `NEUPRINT_TOKEN`: NeuPrint API token ### Configuration Validation From 274d2ad3199876d066e4feaff2a031bfa69fb6da Mon Sep 17 00:00:00 2001 From: Frank Loesche Date: Mon, 1 Dec 2025 15:33:06 -0500 Subject: [PATCH 05/10] removed wrong, added missing docu --- docs/developer-guide.md | 658 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 647 insertions(+), 11 deletions(-) diff --git a/docs/developer-guide.md b/docs/developer-guide.md index 6ef15da..d423029 100644 --- a/docs/developer-guide.md +++ b/docs/developer-guide.md @@ -787,7 +787,10 @@ See current utility scripts in `scripts/` directory for reference implementation Environment variable support for sensitive configuration: -- `NEUPRINT_TOKEN`: NeuPrint API token +#### Currently Implemented + +- `NEUPRINT_TOKEN` - NeuPrint API token (required) + - Checked in three locations: `.env` file, environment, config.yaml ### Configuration Validation @@ -845,6 +848,602 @@ Environment variable support for sensitive configuration: - Automatic missing citation logging and validation - Supports custom link text and output directory configuration +## CLI Reference + +### Overview + +neuView provides a comprehensive command-line interface for generating neuron type pages, managing the processing queue, and testing connections. All commands support the `--verbose` flag for detailed logging output. + +### Global Options + +Available for all commands: + +- `-c, --config TEXT` - Path to configuration file (default: `config.yaml`) +- `-v, --verbose` - Enable verbose output and DEBUG level logging +- `--version` - Show neuView version from git tags and exit +- `--help` - Show help message and exit + +### Commands + +#### generate + +Generate HTML pages for neuron types. + +**Usage:** +```bash +neuview generate [OPTIONS] +``` + +**Options:** +- `--neuron-type, -n TEXT` - Specific neuron type to generate page for +- `--output-dir TEXT` - Custom output directory (overrides config) +- `--image-format [svg|png]` - Format for hexagon grid images (default: svg) +- `--embed/--no-embed` - Embed images directly in HTML instead of saving to files (default: no-embed) +- `--minify/--no-minify` - Enable/disable HTML minification (default: minify) + +**Examples:** +```bash +# Generate page for specific neuron type +neuview generate --neuron-type Tm3 + +# Generate with PNG images and embedded content +neuview generate -n Dm4 --image-format png --embed + +# Generate without minification for debugging +neuview generate -n SAD103 --no-minify + +# Auto-discover and generate for multiple types (up to 20) +neuview generate +``` + +#### inspect + +Inspect detailed information about a specific neuron type including counts, soma sides, and synapse statistics. + +**Usage:** +```bash +neuview inspect NEURON_TYPE +``` + +**Arguments:** +- `NEURON_TYPE` - Name of the neuron type to inspect (required) + +**Examples:** +```bash +# Get detailed statistics for Tm3 +neuview inspect Tm3 + +# Inspect with verbose logging +neuview --verbose inspect Dm4 +``` + +**Output includes:** +- Total neuron count +- Soma side distribution (left/right/middle) +- Bilateral ratio +- Synapse statistics (average, median, std dev) +- Computation timestamp + +#### test-connection + +Test connection to the NeuPrint server and verify dataset access. + +**Usage:** +```bash +neuview test-connection [OPTIONS] +``` + +**Options:** +- `--detailed` - Show detailed dataset information +- `--timeout INTEGER` - Connection timeout in seconds (default: 30) + +**Examples:** +```bash +# Basic connection test +neuview test-connection + +# Detailed server and dataset information +neuview test-connection --detailed + +# With custom timeout +neuview test-connection --timeout 60 +``` + +#### fill-queue + +Create YAML queue files with generate command options and update JSON cache manifest. This is the first step in batch processing workflows. + +**Usage:** +```bash +neuview fill-queue [OPTIONS] +``` + +**Options:** +- `--neuron-type, -n TEXT` - Specific neuron type to add to queue +- `--all` - Create queue files for all discovered neuron types and update cache manifest +- `--output-dir TEXT` - Custom output directory +- `--image-format [svg|png]` - Format for hexagon grid images (default: svg) +- `--embed/--no-embed` - Embed images directly in HTML (default: no-embed) + +**Examples:** +```bash +# Add single neuron type to queue +neuview fill-queue --neuron-type Tm3 + +# Fill queue with all discovered types +neuview fill-queue --all + +# Fill queue with custom options +neuview fill-queue --all --image-format png --embed +``` + +**Queue Files:** +- Created in `output/.queue/` directory +- YAML format with command parameters +- Includes `.lock` mechanism to prevent concurrent processing +- Updates `output/.cache/cache_manifest.json` + +#### pop + +Pop and process a single queue file from the processing queue. Processes one item at a time using FIFO order. + +**Usage:** +```bash +neuview pop [OPTIONS] +``` + +**Options:** +- `--output-dir TEXT` - Custom output directory +- `--minify/--no-minify` - Enable/disable HTML minification (default: minify) + +**Examples:** +```bash +# Process one queue item +neuview pop + +# Process without minification +neuview pop --no-minify + +# Process all items in queue (using pixi task) +pixi run pop-all +``` + +**Processing:** +- Acquires file lock to prevent concurrent processing +- Reads YAML queue file +- Generates page with stored parameters +- Removes queue file on success +- Returns lock file to `.yaml` on error for retry + +#### create-list + +Generate an index page listing all available neuron types with ROI analysis and comprehensive neuron information. + +**Usage:** +```bash +neuview create-list [OPTIONS] +``` + +**Options:** +- `--output-dir TEXT` - Output directory to scan for neuron pages +- `--minify/--no-minify` - Enable/disable HTML minification (default: minify) + +**Examples:** +```bash +# Create index page +neuview create-list + +# Create without minification +neuview create-list --no-minify +``` + +**Generated Files:** +- `output/index.html` - Main index page +- Uses cached neuron type data to avoid database re-queries +- Includes ROI analysis across all neuron types +- Provides search and filter functionality + +### Verbose Logging + +The `--verbose` flag enables DEBUG level logging throughout the application. This provides detailed information about: + +- Cache operations (hits, misses, saves, expirations) +- Database query execution and results +- File operations and path resolutions +- Data processing steps and transformations +- ROI hierarchy loading and validation +- Template rendering and context preparation +- Performance timing for operations + +**Example with verbose output:** +```bash +neuview --verbose generate --neuron-type Tm3 +``` + +**Logger output includes:** +- Timestamp for each operation +- Logger name (module path) +- Log level (DEBUG, INFO, WARNING, ERROR) +- Detailed message with context + +**Key loggers to watch:** +- `neuview.cache` - Cache operations and expiration +- `neuview.services.*` - Service-level operations +- `neuview.visualization.*` - Data processing and visualization + +## Cache Implementation Details + +### Overview + +neuView uses a persistent file-based caching system with pickle serialization to store expensive query results and computed data across sessions. + +### Cache Architecture + +**Implementation:** `src/neuview/strategies/cache/file_cache.py` + +The cache system provides: +- **Persistent storage** - Data survives application restarts +- **TTL support** - Automatic expiration of stale data +- **Thread-safe operations** - Concurrent access protection with `threading.RLock()` +- **Metadata tracking** - Separate files for expiration information +- **MD5 key hashing** - Safe filename generation from cache keys + +### Cache Types + +#### File Cache Strategy + +**Primary cache backend** using pickle serialization: + +```python +class FileCacheStrategy(CacheStrategy): + def __init__(self, cache_dir: str, default_ttl: Optional[int] = None) +``` + +**Features:** +- Stores data in `{cache_dir}/{md5_hash}.cache` files +- Metadata stored in `{md5_hash}.meta` files +- Automatic directory creation +- Configurable TTL (default: 24 hours for neuron type cache) +- Thread-safe read/write operations + +#### Memory Cache Strategy + +**Fast in-memory cache** for frequently accessed data: + +```python +class MemoryCacheStrategy(CacheStrategy): +``` + +**Features:** +- LRU eviction policy +- No persistence across restarts +- Fastest access times +- Used for temporary computation results + +#### Composite Cache + +**Multi-level caching** combining memory and file strategies: + +```python +class CompositeCacheStrategy(CacheStrategy): +``` + +**Behavior:** +- Checks memory cache first (fastest) +- Falls back to file cache on miss +- Promotes file cache hits to memory +- Writes to all configured levels + +### Cache Directory Structure + +``` +output/ +├── .cache/ +│ ├── {md5_hash}.cache # Pickled data +│ ├── {md5_hash}.meta # TTL metadata (JSON) +│ ├── roi_hierarchy.json # ROI hierarchy cache +│ └── cache_manifest.json # Neuron type manifest +└── .queue/ + ├── {neuron_type}.yaml # Queue files + └── {neuron_type}.yaml.lock # Processing locks +``` + +### Cache Key Strategy + +**Key generation:** +```python +safe_key = hashlib.md5(key.encode()).hexdigest() +``` + +**Key components typically include:** +- Neuron type name +- Query parameters +- Dataset identifier +- Soma side filter +- ROI context + +**Example keys:** +- `neuron_type:Tm3:soma_side:left` +- `roi_hierarchy:male-cns:v0.9` +- `connectivity:Tm3:threshold:5` + +### Cache Lifecycle Management + +#### Saving Data + +```python +cache_manager.save_neuron_type_cache(cache_data) +``` + +**Process:** +1. Serialize data to JSON (for neuron type cache) or pickle (for general cache) +2. Generate MD5 hash from cache key +3. Write data to `{hash}.cache` file +4. Write metadata with TTL to `{hash}.meta` file +5. Log operation at DEBUG level + +#### Loading Data + +```python +cache_data = cache_manager.load_neuron_type_cache(neuron_type) +``` + +**Process:** +1. Generate cache key and hash +2. Check if cache file exists +3. Load metadata and check expiration +4. Return `None` if expired +5. Deserialize and return data if valid +6. Log cache hit/miss at DEBUG level + +#### Invalidation + +```python +cache_manager.invalidate_neuron_type_cache(neuron_type) +``` + +**Strategies:** +- Manual: `rm -rf output/.cache/` +- Programmatic: `cache_manager.invalidate_neuron_type_cache()` +- Automatic: TTL expiration (default: 24 hours) + +### Error-Resilient Caching + +**All cache operations use try/except:** +- Cache failures never block page generation +- Warnings logged for debugging +- Graceful degradation to database queries +- Missing cache treated as cache miss + +**Example pattern:** +```python +try: + # Attempt cache operation + cache_data = load_from_cache(key) +except Exception as e: + logger.warning(f"Cache operation failed: {e}") + cache_data = None # Proceed without cache +``` + +### Cache Performance + +**Expected behavior:** +- First generation: Cache miss, query database (~2-5s per type) +- Subsequent generations: Cache hit, no database query (~50-200ms) +- Index generation: Uses cached data, no re-queries +- Cache expiry: 24 hours default (configurable) + +**Cache hit rates:** +- Development: ~30-50% (frequent invalidation) +- Production: ~90-95% (stable data) +- Index generation: ~100% (relies on generation cache) + +### Container Integration Pattern + +**Service container provides cache service:** + +```python +@property +def cache_service(self) -> NeuronTypeCacheManager: + if not self._cache_service: + cache_dir = self.config.output.directory / ".cache" + self._cache_service = NeuronTypeCacheManager(cache_dir=str(cache_dir)) + return self._cache_service +``` + +**Benefits:** +- Single cache instance per container +- Lazy initialization +- Consistent cache location +- Easy dependency injection + +## Queue System Details + +### Overview + +The queue system enables batch processing of neuron types with parallel execution support, avoiding duplicate work and providing fault tolerance. + +### Queue Architecture + +**Implementation:** `src/neuview/services/queue_file_manager.py`, `src/neuview/services/queue_processor.py` + +### Queue File Format + +Queue files are YAML documents storing generation parameters: + +```yaml +neuron_type: "Tm3" +output_directory: "output" +image_format: "svg" +embed_images: false +minify: true +config_file: "config.yaml" +``` + +### Queue Directory Structure + +``` +output/.queue/ +├── Tm3.yaml # Ready to process +├── Dm4.yaml.lock # Currently processing +├── SAD103.yaml # Waiting +└── AOTU019.yaml # Waiting +``` + +### Lock File Mechanism + +**Purpose:** Prevent concurrent processing of the same neuron type + +**Process:** +1. `pop` command finds first `.yaml` file +2. Renames to `.yaml.lock` (atomic operation) +3. Processes the queue item +4. Deletes `.lock` file on success +5. Renames back to `.yaml` on error for retry + +**Benefits:** +- Atomic lock acquisition +- No orphaned locks (file-based) +- Simple error recovery +- Visible processing state + +### Queue Operations + +#### Fill Queue + +**Single neuron type:** +```bash +neuview fill-queue --neuron-type Tm3 +``` + +**All discovered types:** +```bash +neuview fill-queue --all +``` + +**Custom parameters:** +```bash +neuview fill-queue --all --image-format png --embed +``` + +#### Process Queue + +**Single item (manual):** +```bash +neuview pop +``` + +**All items (parallel with pixi):** +```bash +pixi run pop-all +``` + +This uses GNU Parallel to process multiple items concurrently: +```bash +yes pop | head -n $(find output/.queue -name '*.yaml' | wc -l) | parallel --no-notice neuview +``` + +#### Queue Status + +**Check queue size:** +```bash +ls -1 output/.queue/*.yaml | wc -l +``` + +**List pending items:** +```bash +ls -1 output/.queue/*.yaml +``` + +**Check for locked items:** +```bash +ls -1 output/.queue/*.lock +``` + +#### Clear Queue + +**Remove all queue files:** +```bash +rm -rf output/.queue/ +``` + +**Remove specific neuron type:** +```bash +rm output/.queue/Tm3.yaml* +``` + +### Cache Manifest Integration + +**File:** `output/.cache/cache_manifest.json` + +**Purpose:** Track all neuron types in the queue for index generation + +**Structure:** +```json +{ + "neuron_types": ["Tm3", "Dm4", "SAD103"], + "last_updated": "2024-01-15T10:30:00", + "total_count": 3 +} +``` + +**Updates:** +- `fill-queue --all` creates/updates manifest +- `create-list` uses manifest to discover available types +- Avoids database queries during index generation + +### Workflow Integration + +**Complete batch workflow:** +```bash +# 1. Clean previous output +pixi run clean-output + +# 2. Fill queue with all types +pixi run fill-all + +# 3. Process all items in parallel +pixi run pop-all + +# 4. Generate index page +pixi run create-list + +# 5. Increment version +pixi run increment-version +``` + +**Or use the combined task:** +```bash +pixi run create-all-pages +``` + +### Error Handling + +**Queue processing errors:** +- Lock file renamed back to `.yaml` +- Error message logged +- Item remains in queue for retry +- No data corruption + +**Common errors:** +- Database connection timeout → Retry queue item +- Invalid neuron type → Remove from queue manually +- Missing dependencies → Check environment setup +- Permission errors → Check output directory permissions + +### Performance Considerations + +**Serial processing:** +- ~2-5 seconds per neuron type (with cache misses) +- ~50-200ms per neuron type (with cache hits) +- Predictable, sequential + +**Parallel processing:** +- Use `pixi run pop-all` (GNU Parallel) +- Processes multiple items concurrently +- Limited by database connection pool +- 3-5x faster for large queues + ## Dataset Aliases ### Overview @@ -1656,26 +2255,63 @@ Citation logging is automatically enabled when an output directory is provided t #### Development Mode -Enable development mode by setting the `NEUVIEW_DEBUG` and `NEUVIEW_PROFILE` environment variables and running neuview with the `--verbose` flag. +Enable development mode by running neuview with the `--verbose` flag: + +```bash +neuview --verbose generate --neuron-type Tm3 +``` -This enables: -- Detailed operation logging -- Performance timing information -- Memory usage tracking -- Cache operation details -- Database query logging +This enables DEBUG level logging which provides: +- Detailed cache operation logging (hits, misses, saves, expirations) +- ROI hierarchy loading and validation details +- File operation and path resolution tracking +- Data processing steps and transformations +- Performance timing information for some operations +- Citation tracking and missing citation warnings + +**Note:** The environment variables `NEUVIEW_DEBUG` and `NEUVIEW_PROFILE` are documented but not currently implemented. Use the `--verbose` flag instead. ### Logging Architecture -neuView uses a multi-layer logging system for different concerns including main application logging and dedicated citation logging with isolated loggers. +neuView uses Python's standard `logging` module with a multi-logger architecture for different concerns. #### System Loggers -The system uses separate loggers for main application events and citation tracking. See the logging configuration in the service files for logger setup and configuration. +The system uses separate loggers throughout the codebase: + +**Primary loggers:** +- `neuview.cache` - Cache operations (NeuronTypeCacheManager) +- `neuview.services.*` - Service-level operations +- `neuview.visualization.*` - Data processing and visualization +- `neuview.missing_citations` - Dedicated citation tracking + +**Logger configuration:** +- Default level: WARNING (set in `cli.py`) +- Verbose mode: DEBUG (enabled with `--verbose` flag) +- Format: `%(asctime)s - %(name)s - %(levelname)s - %(message)s` + +**Key modules with active logging:** +- `src/neuview/cache.py` - 13 logger calls for cache operations +- `src/neuview/visualization/` - Multiple logger calls for data processing +- `src/neuview/services/citation_service.py` - Citation tracking #### Citation Logging Implementation -The citation logging system automatically tracks missing citations with dedicated logger setup, log directory creation, file rotation handling, and custom formatting. See the `_setup_citation_logger` method in the citation service for the complete implementation including rotating file handlers and UTF-8 encoding support. +The citation logging system uses a dedicated logger with rotating file handlers: + +**Implementation details:** +- **Logger name:** `neuview.missing_citations` +- **Handler:** `RotatingFileHandler` (1MB max size, 5 backups) +- **Level:** INFO (independent of main logger level) +- **Format:** Timestamped entries with context +- **Encoding:** UTF-8 for international character support + +**Setup location:** `src/neuview/services/citation_service.py` in `_setup_citation_logger()` method + +**Automatic triggers:** +- Text processing with synonyms (`TextUtils.process_synonyms()`) +- Citation link creation (`CitationService.create_citation_link()`) +- Missing citation references in templates #### Integration Points From 458b2568a83004f6269182a7314ff45b2e883c12 Mon Sep 17 00:00:00 2001 From: Frank Loesche Date: Mon, 1 Dec 2025 15:45:43 -0500 Subject: [PATCH 06/10] cleanup docu --- docs/developer-guide.md | 129 +++++++++++++++------------------------- docs/user-guide.md | 60 +++++++++++++++---- 2 files changed, 96 insertions(+), 93 deletions(-) diff --git a/docs/developer-guide.md b/docs/developer-guide.md index d423029..8231340 100644 --- a/docs/developer-guide.md +++ b/docs/developer-guide.md @@ -645,7 +645,7 @@ End-to-end tests that verify component interactions and real-world scenarios. - Integration tests: `pixi run integration-test` (comprehensive testing) - All tests: `pixi run test` (complete test suite) - Coverage reports: `pixi run test-coverage` -- Verbose output: Add `-verbose` suffix to any test command +- Verbose output: Add `--verbose` suffix to any test command **Selective execution supports targeting specific files, classes, or methods using pytest syntax.** @@ -656,7 +656,6 @@ End-to-end tests that verify component interactions and real-world scenarios. - `test_male_cns_integration.py` - Integration tests for end-to-end scenarios - `services/` - Service-specific test modules - `visualization/` - Visualization component tests -- `fixtures/` - Test data and fixture files ### Naming Conventions @@ -682,30 +681,6 @@ End-to-end tests that verify component interactions and real-world scenarios. - Integration tests use secure token injection for NeuPrint access - Parallel execution for efficient CI pipeline -### Test Data and Fixtures - -#### Unit Test Data -- Hardcoded test values for predictable behavior -- Use of mock objects for external dependencies -- Parameterized tests for multiple similar scenarios - -#### Integration Test Data -- Temporary configuration files created during test execution -- Real project configuration files when available -- Cleanup of temporary resources after tests - -### Dataset Alias Testing - -**Dataset Alias Testing**: Special focus on testing dataset alias functionality with comprehensive test cases for versioned aliases. See `test_male_cns_versioned_alias_resolution()` and `test_end_to_end_male_cns_workflow()` in test files for reference implementations. - -### Debugging Failed Tests - -#### Unit Test Failures -Run specific failing tests with verbose output using pytest selection syntax. Check test markers to understand test categorization. - -#### Integration Test Failures -Verify environment setup, token configuration, and run tests with verbose debugging and traceback options to diagnose integration test issues. - ### Adding New Tests When adding new features: @@ -716,13 +691,55 @@ When adding new features: 4. **Follow naming conventions** 5. **Ensure proper cleanup** of resources in integration tests -### Test Data Factory +### Test Fixtures + +neuView uses inline pytest fixtures defined within individual test files rather than a centralized factory pattern. + +**Fixture Pattern** (defined in test files): +- Unit tests: Use `@pytest.fixture` with `unittest.mock.Mock` objects +- Integration tests: Use temporary config files and real database connections +- Test-specific fixtures: Defined inline for clarity and simplicity + +**Example Fixture** (`test/services/test_neuron_selection_service.py`): +```python +@pytest.fixture +def selection_service(): + """Create NeuronSelectionService instance.""" + mock_config = Mock() + return NeuronSelectionService(mock_config) + +@pytest.fixture +def mock_connector(): + """Create mock connector for testing.""" + connector = Mock() + connector.dataset_adapter = Mock() + connector.dataset_adapter.dataset_info = Mock() + return connector +``` + +**Example Mock Data** (`test/services/test_soma_detection_service.py`): +```python +@pytest.fixture +def mock_query_result_bilateral(): + """Mock DataFrame result for bilateral neuron type.""" + mock_result = MagicMock() + mock_result.empty = False + mock_result.iloc = [ + {"leftCount": 10, "rightCount": 8, "middleCount": 0, "totalCount": 18} + ] + return mock_result +``` + +**Test Data Strategies:** +- **Unit tests**: Use `unittest.mock.Mock` and `MagicMock` for fast, isolated testing +- **Integration tests**: Connect to real NeuPrint database or use temporary config files +- **Config objects**: Use `Config.create_minimal_for_testing()` for test configurations -**TestDataFactory** (`test/fixtures/test_data_factory.py`): -- Centralized test data creation for consistent testing -- Key methods: `create_neuron_data()`, `create_connectivity_data()` -- Provides standardized test data structures for neuron and connectivity testing -- Parameterizable factory methods for different test scenarios +**Benefits:** +- Simple and maintainable (no central dependency) +- Test data visible next to tests that use it +- Easy to modify fixtures for specific test needs +- Minimal boilerplate and coupling between test files ### Script Management @@ -2034,15 +2051,6 @@ The filtering implementation can be tested with: - Visual feedback verification through DOM element counting - Console logging for debugging filter behavior -#### Future Enhancements - -Planned improvements: -1. **Filter Combinations**: Allow synonym AND Flywire filters simultaneously -2. **Filter Persistence**: Save filter state in URL parameters -3. **Advanced Search**: Boolean operators for complex queries -4. **Performance**: Virtual scrolling for large datasets - - #### CV Calculation **CV Implementation** (`src/neuview/services/partner_analysis_service.py`): @@ -2084,45 +2092,6 @@ Planned improvements: - Comprehensive test cases covering edge cases and statistical accuracy - Assertion-based validation for CV calculation correctness -### Neuroglancer Integration Fixes - -#### Problem -Neuroglancer JavaScript errors due to placeholder mismatches: - -**Problem**: Neuroglancer JavaScript errors due to placeholder type mismatches with expected array format. - -#### Solution -**Template Variable Correction** (`src/neuview/services/neuroglancer_js_service.py`): -- Correct placeholder types in template generation -- Empty array initialization instead of string placeholders -- Proper JSON array format for Neuroglancer compatibility - -#### Flexible Dataset Layer Detection - -**Multi-Dataset Layer Detection** (`templates/static/js/neuroglancer-url-generator.js.jinja`): -- Enhanced layer detection logic for multiple dataset types -- Flexible segmentation layer identification by type and properties -- Support for both CNS ("cns-seg") and FAFB ("flywire-fafb:v783b") layer names - -### HTML Tooltip System Implementation - -Rich tooltips for enhanced user experience: - -#### Basic Structure - -**HTML Tooltip Structure** (`templates/sections/tooltips.html.jinja`): -- Rich HTML content containers with formatted text, lists, and styling -- Tooltip content wrapper with trigger elements -- Flexible content structure supporting headings, paragraphs, and lists - -#### JavaScript Implementation - -**Tooltip JavaScript** (`templates/static/js/tooltip-system.js`): -- `initializeHtmlTooltips()` - Sets up event listeners for tooltip elements -- `positionTooltip()` - Dynamic positioning with screen boundary detection -- Mouseenter/mouseleave event handling for show/hide behavior -- Automatic positioning adjustment to prevent off-screen display - ## Troubleshooting ### Common Issues diff --git a/docs/user-guide.md b/docs/user-guide.md index ca9f6ba..0708362 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -52,7 +52,7 @@ Get up and running with neuView in minutes: **Option 1: Environment File (Recommended with pixi)** **Authentication Methods**: 1. **Environment file**: Run `pixi run setup-env`, then edit `.env` with your token -2. **Environment variable**: `export NEUPRINT_APPLICATION_CREDENTIALS="your-token-here"` +2. **Environment variable**: `export NEUPRINT_TOKEN="your-token-here"` **Getting Your Token**: 1. Visit [neuprint.janelia.org](https://neuprint.janelia.org/account) @@ -213,7 +213,7 @@ Get detailed information about processing: **Verbose Mode Commands**: - Enable verbose output: `pixi run neuview --verbose generate -n Dm4` -- Enable debug mode: Set `NEUVIEW_DEBUG=1` then run neuview commands +- Enable debug mode: Use the `--verbose` flag for detailed logging ## Generated Website Features @@ -619,7 +619,7 @@ Citation logs automatically rotate when they reach 1MB: **Authentication Problems** **Connection Troubleshooting Commands**: -- Verify token: `echo $NEUPRINT_APPLICATION_CREDENTIALS` +- Verify token: `echo $NEUPRINT_TOKEN` - Test connection: `pixi run neuview test-connection` - Verbose connection test: `pixi run neuview --verbose test-connection` @@ -633,7 +633,7 @@ Citation logs automatically rotate when they reach 1MB: **Performance Issue Commands**: - Check output directory for cached files: `ls -la output/.cache/` - Clear corrupted cache: `rm -rf output/.cache/` -- Enable performance monitoring: Set `NEUVIEW_PROFILE=1` +- Enable performance monitoring: Use `--verbose` flag for detailed operation logs **Missing Output** **Missing Output Troubleshooting**: @@ -670,7 +670,7 @@ For FAFB datasets: Enable detailed troubleshooting: -**Debug Mode Setup**: Set `NEUVIEW_DEBUG=1` and `NEUVIEW_PROFILE=1`, then run `pixi run neuview --verbose generate -n Dm4` +**Debug Mode Setup**: Run `pixi run neuview --verbose generate -n Dm4` This provides: - Detailed operation logging @@ -749,7 +749,42 @@ html: ### Command Reference -Available commands include `generate` for creating neuron type pages, `create-list` for generating index pages, `fill-queue` for creating queue entries, `pop` for processing queue files, `inspect` for examining neuron types, and `test-connection` for verifying NeuPrint access. All commands are run with the `pixi run neuview` prefix. +neuView provides a comprehensive command-line interface. All commands are run with the `pixi run neuview` prefix. + +#### Available Commands + +**`generate`** - Generate HTML pages for neuron types +- Options: `--neuron-type`, `--output-dir`, `--image-format`, `--embed/--no-embed`, `--minify/--no-minify` +- Example: `pixi run neuview generate --neuron-type Tm3` + +**`inspect`** - Inspect detailed information about a specific neuron type +- Shows: neuron counts, soma sides, synapse statistics, bilateral ratio +- Example: `pixi run neuview inspect Dm4` + +**`test-connection`** - Test connection to NeuPrint server +- Options: `--detailed`, `--timeout` +- Example: `pixi run neuview test-connection --detailed` + +**`fill-queue`** - Create YAML queue files for batch processing +- Options: `--neuron-type`, `--all`, `--output-dir`, `--image-format`, `--embed/--no-embed` +- Example: `pixi run neuview fill-queue --all` + +**`pop`** - Process a single queue file +- Options: `--output-dir`, `--minify/--no-minify` +- Example: `pixi run neuview pop` + +**`create-list`** - Generate index page with all neuron types +- Options: `--output-dir`, `--minify/--no-minify` +- Example: `pixi run neuview create-list` + +#### Global Options + +- `-c, --config` - Path to configuration file +- `-v, --verbose` - Enable verbose output and DEBUG logging +- `--version` - Show version and exit +- `--help` - Show help message + +For detailed command options and examples, see the developer guide. ### Performance Tips @@ -774,13 +809,12 @@ When using neuView-generated data in publications: ### Environment Variables -| Variable | Description | Example | -|----------|-------------|---------| -| `NEUPRINT_APPLICATION_CREDENTIALS` | NeuPrint API token | `your-token-string` | -| `NEUVIEW_CONFIG_PATH` | Custom config file path | `/path/to/config.yaml` | -| `NEUVIEW_CACHE_DIR` | Cache directory override | `/tmp/neuview_cache` | -| `NEUVIEW_DEBUG` | Enable debug logging | `1` or `true` | -| `NEUVIEW_PROFILE` | Enable performance profiling | `1` or `true` | +**Currently Implemented:** + +- **`NEUPRINT_TOKEN`** - NeuPrint API token (required) + - Example: `your-token-string` + - Set in `.env` file or as environment variable + - Required for all database operations ### Keyboard Shortcuts From a882cfdfa66754331840da78d4cf07b9352293fe Mon Sep 17 00:00:00 2001 From: Frank Loesche Date: Mon, 1 Dec 2025 15:55:36 -0500 Subject: [PATCH 07/10] remove old and new issues --- docs/developer-guide.md | 83 +++++------------------------------------ docs/user-guide.md | 8 +--- 2 files changed, 11 insertions(+), 80 deletions(-) diff --git a/docs/developer-guide.md b/docs/developer-guide.md index 8231340..b7ccaef 100644 --- a/docs/developer-guide.md +++ b/docs/developer-guide.md @@ -543,16 +543,7 @@ All caches are organized under the output directory to maintain consistency. Sta **Container Integration Pattern**: Register cache-aware services with output directory injection. See `roi_data_service_factory()` in `PageGenerationContainer` for reference implementation. -#### Migration Considerations -When updating cache locations: - -1. **Automatic Migration**: Cache files regenerate automatically from sources -2. **No Data Loss**: Old cache files can be safely removed -3. **Backward Compatibility**: Fallback paths maintain functionality -4. **Clean Transition**: Remove old cache files after verification - -**Migration Pattern**: Services support automatic migration from previous cache locations. The ROI Data Service demonstrates output directory adoption with automatic cleanup. ## Development Patterns @@ -1692,28 +1683,11 @@ Combined pages showed separate ROI entries: ### Dynamic ROI Data System -The Dynamic ROI Data System replaces hardcoded ROI arrays in Neuroglancer templates with live data fetched from Google Cloud Storage, ensuring ROI information is always up-to-date. - -#### Problem Statement - -Previously, the `neuroglancer-url-generator.js.jinja` template contained hardcoded arrays: - -```javascript -// Hardcoded ROI data (maintenance burden) -const ROI_IDS = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, ...]; -const ALL_ROIS = ["AL(L)", "AL(R)", "AME(L)", "AME(R)", ...]; -const VNC_IDS = [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, ...]; -const VNC_NAMES = ["CV-posterior", "LegNp(T1)(L)", ...]; -``` - -This approach had significant drawbacks: -- **Manual Maintenance**: Required manual updates when ROI data changed -- **Data Inconsistency**: Risk of hardcoded data becoming stale -- **Version Mismatch**: Potential conflicts between different dataset versions +The Dynamic ROI Data System fetches live ROI data from Google Cloud Storage, ensuring ROI information is always up-to-date. -#### Solution Architecture +#### Architecture -The system now uses a `ROIDataService` that: +The system uses a `ROIDataService` that: 1. **Fetches Live Data**: Retrieves ROI segment properties from GCS endpoints 2. **Caches Locally**: Stores data in `output/.cache/roi_data/` for performance @@ -1814,54 +1788,15 @@ for key, value in roi_data.items(): self.env.globals[key] = value ``` -#### Jinja Template Processing Fix - -A critical bug was resolved where Jinja placeholders weren't being processed correctly. - -**Problem**: The entire JavaScript template was wrapped in `{% raw %}` tags, preventing Jinja expression processing: - -```javascript -// Template variables (processed) -const NEUROGLANCER_BASE_URL = "{{ neuroglancer_base_url }}"; - -{% raw %} -// Everything here is literal text - Jinja expressions NOT processed -const ROI_IDS = {{ roi_ids|tojson }}; // ❌ Not processed! -{% endraw %} -``` - -**Solution**: Moved ROI data declarations outside the `{% raw %}` block: - -```javascript -// Template variables (processed by Jinja) -const NEUROGLANCER_BASE_URL = "{{ neuroglancer_base_url }}"; -const ROI_IDS = {{ roi_ids|tojson }}; -const ALL_ROIS = {{ all_rois|tojson }}; - -{% raw %} -// Static JavaScript code (not processed by Jinja) -// ... rest of JavaScript -{% endraw %} -``` - -#### ROI ID Collision Fix +#### ROI ID Collision Handling -A sophisticated fix was implemented to resolve ROI ID collisions between brain and VNC datasets. +The system handles ROI ID collisions between brain and VNC datasets. -**Problem**: ROI segment IDs overlap between datasets: +ROI segment IDs can overlap between datasets: - ID 7: "AOTU(L)" in brain dataset, "LegNp(T2)(L)" in VNC dataset -- Clicking brain ROI checkboxes incorrectly toggled VNC layers +- Without context tracking, clicking brain ROI checkboxes could toggle VNC layers -**Root Cause**: Layer assignment logic used only ID values without dataset context: - -```javascript -// Problematic code -if (VNC_IDS.includes(parseInt(roiId))) { - // Always assigned to VNC layer, even for brain ROIs with same ID -} -``` - -**Solution**: Introduced context tracking with `selectedRoiContexts` map: +**Implementation**: The system uses context tracking with `selectedRoiContexts` map: ```javascript // Context-aware ROI management @@ -2238,7 +2173,7 @@ This enables DEBUG level logging which provides: - Performance timing information for some operations - Citation tracking and missing citation warnings -**Note:** The environment variables `NEUVIEW_DEBUG` and `NEUVIEW_PROFILE` are documented but not currently implemented. Use the `--verbose` flag instead. + ### Logging Architecture diff --git a/docs/user-guide.md b/docs/user-guide.md index 0708362..c9f0872 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -368,10 +368,7 @@ Advanced filters activated by clicking colored tags within neuron type cards: - Check for browser console errors - Try refreshing the page -**Performance Considerations**: -- Large datasets (>1000 neuron types) may experience slower filter response -- Consider using basic filters first to reduce the working set -- Text search is debounced to prevent excessive filtering during typing + ##### Best Practices @@ -613,7 +610,6 @@ If `output/.log/missing_citations.log` doesn't exist: Citation logs automatically rotate when they reach 1MB: - Up to 5 backup files are kept (`.log.1`, `.log.2`, etc.) - Check for repeated missing citations that should be added to `citations.csv` -- Old backup files can be safely deleted if disk space is needed ### Common Issues @@ -790,7 +786,7 @@ For detailed command options and examples, see the developer guide. 1. **Use Caching**: Cache provides up to 97.9% speed improvement on subsequent runs 2. **Process in Batches**: Use queue system for multiple neuron types -3. **Clean Cache Periodically**: Remove old cache files with `rm -rf output/.cache/` when needed +3. **Clean Cache Periodically**: Remove cache files with `rm -rf output/.cache/` when needed 4. **Monitor Progress**: Use verbose mode for long-running operations 5. **Optimize Configuration**: Adjust cache settings based on available memory From 76aed09332a088cbb392b5053b528e0daa9efc93 Mon Sep 17 00:00:00 2001 From: leburnett Date: Mon, 1 Dec 2025 17:49:48 -0500 Subject: [PATCH 08/10] Update help.html.jinja --- templates/help.html.jinja | 34 +++++++++++++++++++++------------- 1 file changed, 21 insertions(+), 13 deletions(-) diff --git a/templates/help.html.jinja b/templates/help.html.jinja index 2813f93..2481242 100644 --- a/templates/help.html.jinja +++ b/templates/help.html.jinja @@ -344,19 +344,24 @@

      Neurons

      - The number of individual neurons of this type in the dataset. For bilateral neurons, counts per hemisphere and ratios may - be shown across hemispheres. + The number of individual neurons of this type in the dataset. For bilateral neurons, counts reflect the number per hemisphere and log ratios + show the balance across hemispheres.

      Synapses

      -

      Combined post-synaptic (inputs) and pre-synaptic (outputs), with breakdowns and log ratios where applicable. Technically, this number is a pre-calculated property for each neuron inside the neuprint database.
      - For combined pages the breakdown shows the number of neurons assigned to each hemisphere and their ratio. - Whereas for unilateral pages the breakdown shows the number of post and pre synapses across all neurons of the type assigned to that side.

      +

      Combined post-synaptic (inputs) and pre-synaptic (T-bars, outputs), with breakdowns and log ratios where applicable. + Technically, this number is a pre-calculated property for each neuron from the neuPrint database.
      + For combined pages the breakdown shows the total number of synapses assigned to each hemisphere and their ratio. + Whereas for unilateral pages the breakdown and log ratio shows the number of post and pre synapses across all neurons of the type assigned to that side.

      Connections

      - Combined upstream and downstream connections. Here, connections are the number of anatomical synapses between neurons. Technically, this number comes from the number of connected synapses, which are combined to SynapseSets and part of Neurons inside the neuprint database.
      - For combined pages the breakdown shows the number of neurons assigned to each hemisphere and their ratio. + Combined upstream and downstream connections, with breakdowns and log ratios where applicable. + Here, connections are the number of anatomical synapses between neurons. Technically, this number comes from the number of connected + synapses, which are combined to form SynapseSets and assigned to Neurons within the neuprint database. + It is possible that a single pre-synapse of a target cell can contain multiple anatomical synaptic contacts (downstream connections) + and so cells can have more downstream connections than pre-synapses.
      + For combined pages the breakdown shows the total number of connections assigned to each hemisphere and their ratio. Whereas for unilateral pages the breakdown shows the number of upstream and downstream connections across all neurons of the type assigned to that side.

      @@ -574,19 +579,22 @@
      Synapse Count
      -
      The number of synaptic connections. Higher counts indicate stronger or denser connectivity. neuView uses the properties from the neurons in the database.
      +
      The number of synaptic connections. Higher counts indicate denser connectivity. neuView uses the properties from the neurons in the database.
      Post-synaptic Sites
      -
      Postsynaptic receptor sites where a neuron receives inputs from presynaptic partners. In the data model, this is the `Neuron.post` property.
      +
      Postsynaptic receptor sites where a neuron receives inputs from presynaptic partners. In the neuPrint data model, this is the `Neuron.post` property.
      Pre-synaptic Sites (T-bars)
      -
      Presynaptic release sites where a neuron outputs signals. Also called t-bars. One presynaptic site can connect to multiple postsynaptic partners. In the neuprint data model, this is the `Neuron.pre` property.
      +
      Presynaptic release sites where a neuron outputs signals, also called t-bars. In the neuprint data model, this is the `Neuron.pre` property.
      Connections
      -
      Input connections are conceptually similar to post-synaptic sites while a group of post-synaptic sites form a single output connection. We calculate the number of connections by identifying the connecting neurons and then use the `weight` property from the `ConnectsTo` edge between the neurons.
      +
      Input `upstream` connections are conceptually similar to post-synaptic sites while one presynaptic site can connect to multiple `downstream` postsynaptic partners. + We calculate the number of connections by identifying the connecting neurons and then use the `weight` property from the `ConnectsTo` edge between the neurons.
      -
      Connections vs. Synapses
      -
      The numeric difference between post-synaptic sites and input connections stems from the difference in how the numbers are retrieves (see Synapse Count and Connections above). A single presynaptic site (t-bar) can form multiple connections to different postsynaptic partners. Therefore, the number of output connections can be greater than the number of presynaptic sites.
      +
      + Connections vs. Synapses + The numeric difference between post-synaptic sites and input connections stems from the difference in how the numbers are retrieved (see Synapse Count and Connections above). A single presynaptic site (t-bar) can form multiple connections to different postsynaptic partners. Therefore, the number of output connections can be greater than the number of presynaptic sites. +
      Inputs
      All pre-synaptic connecting cell types upstream from a given cell type (neurons providing input to the cell type).
      From 825d0e3f550b3f2c96dce687059a3c195a893c0d Mon Sep 17 00:00:00 2001 From: Frank Loesche Date: Tue, 9 Dec 2025 11:08:07 -0500 Subject: [PATCH 09/10] update dev docs --- docs/developer-guide.md | 50 ++++++++++++++++++++++++++++++++++++----- 1 file changed, 44 insertions(+), 6 deletions(-) diff --git a/docs/developer-guide.md b/docs/developer-guide.md index 9eadbca..4c9b6ef 100644 --- a/docs/developer-guide.md +++ b/docs/developer-guide.md @@ -33,7 +33,7 @@ neuView is a modern Python CLI tool that generates beautiful HTML pages for neur ### Technology Stack -- **Backend**: Python 3.11+, asyncio for async processing +- **Backend**: Python 3.11+ with built-in asyncio support - **Data Layer**: NeuPrint API, persistent file-based caching with pickle - **Frontend**: Modern HTML5, CSS3, vanilla JavaScript - **Templates**: Jinja2 with custom filters and extensions @@ -58,6 +58,44 @@ neuView follows a layered architecture pattern: - NeuPrint access token - Git for version control +### Key Dependencies + +The project relies on several key Python packages defined in `pyproject.toml`. These are installed and managed through `pixi`, e.g via running `pixi shell` or `pixi install`. + +**Core Dependencies:** +- `click>=8.0.0` - CLI framework +- `jinja2>=3.0.0` - Template engine +- `pyyaml>=6.0` - YAML configuration parsing +- `requests>=2.28.0` - HTTP requests +- `pandas>=1.5.0` - Data manipulation +- `neuprint-python>=0.4.0` - NeuPrint API client + +**Visualization & Processing:** +- `pycairo>=1.20.0` - Cairo graphics library +- `cairosvg>=2.5.0` - SVG processing +- `minify-html>=0.16.4,<0.17` - HTML minification + +**Development Tools:** +- `gitpython>=3.1.0` - Git integration for versioning +- `python-dotenv>=1.0.0` - Environment variable management +- `toml>=0.10.0` - TOML file parsing +- `cst-lsp>=0.1.3,<0.2` - Language server protocol support + +**Testing (dev environment):** +- `pytest` - Testing framework +- `pytest-cov` - Coverage reporting +- `pytest-asyncio>=1.2.0,<2` - Async test support + +**Code Quality (dev environment):** +- `ruff>=0.12.11,<0.13` - Fast Python linter and formatter + +**Workflow Tools:** +- `parallel>=20200322,<20250623` - GNU Parallel for batch processing +- `python-lsp-server>=1.13.1,<2` - Python language server +- `pyprojroot>=0.3.0,<0.4` - Project root detection +- `psutil>=7.1.0,<8` - System and process utilities +- `semver>=3.0.4,<4` - Semantic versioning + ### Development Setup 1. **Clone the repository** @@ -92,6 +130,7 @@ neuView uses pixi for task management with separate commands for different types **Code Quality Commands** (defined in `pyproject.toml`): - `pixi run format` - Format code with ruff +- `pixi run check` - Check code with ruff linter **Content Generation:** - `pixi run neuview generate` - Generate website for 10 random neuron types @@ -441,10 +480,9 @@ Custom Jinja2 filters for formatting data in templates. ### Cache Management -**Clear cache:** -```bash -pixi run neuview build --clear-cache -``` +**Cache Location:** +- Cache files are stored in `output/.cache/` directory +- Manually delete this directory to clear all caches: `rm -rf output/.cache/` **PerformanceMonitor** (`src/neuview/services/performance_monitor.py`): - Operation timing and metrics collection @@ -2124,7 +2162,7 @@ Citation logging is integrated into: 2. Make changes with tests 3. Run tests: `pixi run test` 4. Format code: `pixi run format` -5. Lint code: `pixi run lint` +5. Check code: `pixi run check` 6. Submit pull request ### Code Style From f196724808055024eb3e58180623a3ab4d4017ae Mon Sep 17 00:00:00 2001 From: Frank Loesche Date: Tue, 9 Dec 2025 11:17:11 -0500 Subject: [PATCH 10/10] update user docs --- docs/user-guide.md | 83 +++++++++++++++++++++++++--------------------- 1 file changed, 45 insertions(+), 38 deletions(-) diff --git a/docs/user-guide.md b/docs/user-guide.md index 645ca7d..c5be8fa 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -8,6 +8,7 @@ A comprehensive guide for users of the neuView neuron visualization platform. - [Installation](#installation) - [Configuration](#configuration) - [Basic Usage](#basic-usage) +- [Advanced Features](#advanced-features) - [Generated Website Features](#generated-website-features) - [Search Functionality](#search-functionality) - [Dataset-Specific Features](#dataset-specific-features) @@ -25,13 +26,16 @@ pixi install pixi run setup-env # Edit .env file with your NeuPrint token -# NEUPRINT_APPLICATION_CREDENTIALS=your_token_here +# NEUPRINT_TOKEN=your_token_here # Test connection pixi run neuview test-connection -# Generate website -pixi run neuview build +# Generate pages for random neuron types +pixi run neuview generate + +# Or use the complete workflow +pixi run create-all-pages ``` Your website will be in the `output/` directory. Open `output/index.html` in a browser. @@ -65,7 +69,7 @@ Your website will be in the `output/` directory. Open `output/index.html` in a b This creates a `.env` file. Edit it to add your NeuPrint token: ``` - NEUPRINT_APPLICATION_CREDENTIALS=your_token_here + NEUPRINT_TOKEN=your_token_here ``` 4. **Test connection:** @@ -76,16 +80,19 @@ Your website will be in the `output/` directory. Open `output/index.html` in a b ### Setting Up Authentication **Option 1: Environment File (Recommended with pixi)** -**Authentication Methods**: -1. **Environment file**: Run `pixi run setup-env`, then edit `.env` with your token -2. **Environment variable**: `export NEUPRINT_TOKEN="your-token-here"` +**Authentication Steps**: 1. Go to your NeuPrint server (e.g., https://neuprint.janelia.org) 2. Click "Account" → "Auth Token" 3. Copy the token -4. Add to `.env` file or set as environment variable: +4. Add to `.env` file: ```bash - export NEUPRINT_APPLICATION_CREDENTIALS="your_token_here" + NEUPRINT_TOKEN=your_token_here + ``` + + Or set as environment variable: + ```bash + export NEUPRINT_TOKEN="your_token_here" ``` ## Configuration @@ -144,9 +151,9 @@ subsets: generate_index: true ``` -Use subsets with: +Use the `extract-and-fill` script with subset configurations: ```bash -pixi run neuview build --subset small-test +pixi run extract-and-fill config.yaml subset-small ``` ## Basic Usage @@ -154,34 +161,35 @@ pixi run neuview build --subset small-test ### Essential Commands ```bash -# Generate complete website -pixi run neuview build +# Generate page for a specific neuron type +pixi run neuview generate --neuron-type Tm3 -# Generate with cache clearing -pixi run neuview build --clear-cache +# Generate pages for multiple random types (auto-discovery) +pixi run neuview generate # Inspect a specific neuron type -pixi run neuview inspect "Tm3" - -# Generate for specific neuron types -pixi run neuview build --types "Tm3,Mi1,T4" +pixi run neuview inspect Tm3 -# Use a subset from config -pixi run neuview build --subset small-test +# Generate complete website with all neurons +pixi run create-all-pages # Verbose output for debugging -pixi run neuview build --verbose +pixi run neuview --verbose generate --neuron-type Tm3 ``` ### Command Options -**`build` command:** -- `--clear-cache` - Clear all caches before building -- `--types ` - Comma-separated list of neuron types -- `--subset ` - Use a predefined subset from config -- `--parallel` - Enable parallel processing -- `--verbose` - Detailed logging output -- `--output ` - Custom output directory +**`generate` command:** +- `--neuron-type, -n TEXT` - Specific neuron type to generate page for +- `--output-dir TEXT` - Custom output directory +- `--image-format [svg|png]` - Format for hexagon grid images (default: svg) +- `--embed/--no-embed` - Embed images directly in HTML (default: no-embed) +- `--minify/--no-minify` - Enable/disable HTML minification (default: minify) + +**Global options** (use before command): +- `--verbose, -v` - Enable detailed logging output +- `--config, -c TEXT` - Path to configuration file +- `--version` - Show version and exit ### Neuron Type Inspection @@ -572,7 +580,7 @@ Citation logs automatically rotate when they reach 1MB: **Authentication Problems** **Connection Troubleshooting Commands**: -- Verify token: `echo $NEUPRINT_TOKEN` +- Verify token is set: `echo $NEUPRINT_TOKEN` - Test connection: `pixi run neuview test-connection` - Check network connectivity @@ -603,11 +611,9 @@ Citation logs automatically rotate when they reach 1MB: Enable detailed logging: -**Debug Mode Setup**: Run `pixi run neuview --verbose generate -n Dm4` - -# Or set environment variable -export NEUVIEW_LOG_LEVEL=DEBUG -pixi run neuview build +**Debug Mode Setup**: +```bash +pixi run neuview --verbose generate -n Dm4 ``` Check log output for: @@ -765,10 +771,11 @@ For detailed command options and examples, see the developer guide. ### Performance Tips 1. **Use Caching**: Cache provides up to 97.9% speed improvement on subsequent runs -2. **Process in Batches**: Use queue system for multiple neuron types +2. **Process in Batches**: Use queue system (`fill-queue` and `pop`) for multiple neuron types 3. **Clean Cache Periodically**: Remove cache files with `rm -rf output/.cache/` when needed -4. **Monitor Progress**: Use verbose mode for long-running operations -5. **Optimize Configuration**: Adjust cache settings based on available memory +4. **Monitor Progress**: Use `--verbose` flag for long-running operations +5. **Parallel Processing**: Use `pixi run pop-all` for concurrent queue processing +6. **Optimize Configuration**: Adjust cache settings based on available memory ### Data Citation