Skip to content

Home

Jump to bottom
Siger Ma edited this page Jun 4, 2025 · 10 revisions

Machine Intelligence for Neuron Detection 🧠 🚀

Introduction

Welcome to the Machine Intelligence for Neuron Detection (MIND) Wiki!

MIND is a platform that facilitates deep learning pipelines to automatically segment neuron somas in large-scale brain images. Its capabilities and how to use it are clearly documented in this wiki. The platform includes functions that are accessible as a Vaa3D plugin. All functions are run on an open Vaa3D image, either loaded directly from a 3D TIFF or a TeraFly image. Documentation on how to use TeraConverter to set up TeraFly can be found here.

A brief summary of each of the functions is described below:

Function Description
Isotropic Correction Adjust the 3D view of images for the true dimensions of a voxel based on the microscopy technique.
Soma Segmentation Automatically segments neuron somas that have been bounded within spheres around their center of mass with Vaa3D markers. Outputs a visualization of the segmentation, as well as the binary segmentation file, and a probability model of Neuron Soma shape obtained through Principal Component Analysis (PCA) on the binary segmentation of somas. The PCA results are also output in a CSV file.
PC Analysis Intensity-weighted Principal Component Analysis on the original image based on the spherical volume captured by the Vaa3D markers. Results are output in a CSV file.
Visualize PCA Visualize the results of PCA from the Soma Segmentation function in 3D with accurate lengths for each soma. The longest PC is shown in green, the second longest in red, and the shortest in cyan. This function also supports visualizing the results of the PC Analysis function, but the lengths of the PCs have no meaning.
Visualize Probability Model Visualize the probability model output by the Soma Segmentation function in 2D or 3D, as a heatmap with a legend.
Probability Model Legend View only the legend of the probability model.
Create Background Create a synthetic background for somas by analysing an open image. Gives access to the synthetic background without having to put somas in it.
Simulate Somas Must be run after the Soma Segmentation function. Generates synthetic somas in a synthetic background and a corresponding binary segmentation. Outputs these as images, along with a CSV file containing their PCA results, a marker file locating synthetic somas, and a visualization of the simulation results.

Functions

We can now describe how to run each plugin function in detail, including any necessary preprocessing steps, inputs, and where the outputs of each function are saved. This documentation was created with the help of generative AI.

Isotropic Correction

Purpose: Adjusts the scaling factors used by Vaa3D's 3D viewer to display the image with the correct aspect ratio, accounting for anisotropic voxel sizes common in microscopy. This does not resample or change the image data itself, only how it is viewed. It does affect calculations that use voxel dimensions to get results.

Inputs:

  • An image (3D TIFF or TeraFly) opened in Vaa3D.
  • Voxel resolutions (X, Y, Z) in µm, entered via the dialog.

How to Use:

  1. Open the image you want to view in Vaa3D.
  2. Navigate to the Vaa3D menu: Plugins -> MIND -> Isotropic Correction.
  3. A dialog box titled "Enter Resolutions" will appear.

  1. Enter the physical size of a single voxel in micrometers (µm) along the X, Y, and Z axes. These values depend on your microscope's objective, camera pixel size, and Z-step interval.
  2. Click OK.
  3. The 3D view in the Vaa3D window will update its scaling factors. Observe the change in the aspect ratio of the displayed volume.

Outputs:

  • Updates the view scaling factors (resolution metadata) of the currently active Vaa3D image window. No files are generated.

Soma Segmentation

Purpose: Segments neuron somas based on user-placed markers (landmarks) using a 3D region-growing algorithm. It also analyzes the shape of the segmented somas using PCA on their binary masks to generate a soma shape probability model.

Prerequisites:

  • Label the image with Vaa3D landmarks on the somas with a sphere englobing its entirety around its center of mass. There should be no additional overlapping somas inside the sphere.

Inputs:

  • An image (3D TIFF or TeraFly) opened in Vaa3D.
  • Vaa3D landmarks placed to envelope each labelled soma with a sphere around its center of mass.

How to Use:

  1. Load your image in Vaa3D.
  2. Place landmarks on the somas. Adjust the radius of each landmark sphere appropriately if using Local Otsu or Marker Constraint. An appropriate radius is when the sphere englobes the entire soma without it overlapping with another soma. The default radius can be permanently adjusted by going to File -> Adjust Preferences -> Default Marker Size.
  3. Navigate to the Vaa3D menu: Plugins -> MIND -> Soma Segmentation.
  4. A dialog box titled "Soma Segmentation" appears. Configure the parameters:

  • Color channel: Select the image channel containing the soma signal.
  • Geometry stat: (Primarily affects validation heuristics for initial segmentation)
    • Shape: Assume sphere-like (default) or cube-like for internal PCA shape checks.
    • Max anisotropic deviation: Tolerance for shape deviation checks (Default: 1).
    • Min/Max region size vs. exemplar ratio: Lower and upper bounds for the segmented region size relative to an internal estimate (Default: 0.1 and 20).
  • Exemplar definition: (This section can largely be ignored. It configures the out-of-date iterative thresholding algorithm)
    • Max movement from mass center: Maximum allowed squared Euclidean distance (pixels²) the region's center of mass can move from its position in the previous iteration during iterative thresholding. Default: 4 (i.e., 2 pixels).
    • Max movement from marker position: Maximum allowed squared Euclidean distance (pixels²) the region's final center of mass can be from the original user-placed marker during iterative thresholding. Default: 16 (i.e., 4 pixels).
  • Segmentation Preferences:
    • Segmentation Mode: Choose the thresholding method:
      • Local Otsu (Default): Calculates Otsu threshold within a cubic region defined by the marker's radius around each landmark.
      • Global Otsu: Calculates a single Otsu threshold based on the entire image's histogram.
      • Iterative Threshold: Starts region growing from the marker's intensity and lowers the threshold iteratively until movement constraints are violated.
    • Manual Thresholding: (Checkbox, default off) If checked, ignores the selected Segmentation Mode and instead reads the threshold from the comment field of each Vaa3D landmark if it exists (requires comments in the format threshold:VALUE, e.g., threshold:50).
    • Median Filtering: (Checkbox, default on) Apply 3D median filtering as a preprocessing step to the selected channel before segmentation.
    • Median Filtering Radius (Voxels): (Enabled if Median Filtering is checked) Sets the radius for the median filter (1–9 voxels, default 1).
    • Marker Constraint: (Checkbox, default on) If checked, region growing is constrained to stay within the sphere defined by the Vaa3D landmark's radius.
  1. Click Run.

Process:

  • Optionally applies 3D median filtering to the selected channel.
  • Iterates through each landmark.
  • For each landmark, it determines the segmentation threshold based on the selected mode (or manual value).
  • Performs 3D region-growing starting from the landmark, using the determined threshold and applying the marker radius constraint if enabled.
  • Applies heuristics based on region size and center of mass movement (compared to Exemplar definition constraints) to validate the segmentation.
  • For each successfully segmented soma:
    • Performs PCA on its binary mask to find principal axes and center of mass.
    • Aligns the binary mask to its center of mass and rotates it to align its principal axes with the coordinate axes.
    • Adds the aligned, rotated binary mask to a running sum (the probability model).
  • Saves the outputs.
  • Displays a segmentation visualization overlay.

Outputs:

  • Visualization: A new Vaa3D window is opened, displaying a 3-channel image:
    • Channel 1: Original image data (selected channel).
    • Channel 2: Binary segmentation result (segmented somas are white/colored).
    • Channel 3: Gradient image highlighting the boundaries of the binary segmentation.

  • Files (saved in the same directory as the input image, or a MIND/[ImageBaseName]/ subdirectory if using TeraFly. The MIND directory is created in the same location as the Vaa3D executable):
    • [ImageName]_binary_segmentation.tif: A 3D TIFF image containing the final binary segmentation mask (soma=255, background=0).
    • [ImageName]_probability_model.bin: A binary file storing the dimensions (X, Y, Z) and the 3D double array representing the probability model of aligned soma shapes.
    • [ImageName]_pca_binary_segmentation.csv: A CSV file detailing the PCA results for each successfully segmented soma's binary mask. Includes eigenvalues, eigenvectors, calculated center of mass, original marker coordinates, and radius.
    • [ImageName]_segmentation_summary.txt: A text file with a summary of global information, parameters, and statistics for the segmentation.
  • Updated Landmark Comments: In the original Vaa3D window, the comments for successfully segmented landmarks are updated to threshold:VALUE, indicating the final threshold used for that soma.

PC Analysis

Purpose: Performs Principal Component Analysis (PCA) on the original image intensity values within spherical regions around user-placed landmarks. This differs from the PCA performed during Soma Segmentation, which analyzes the binary shape of the resulting segmentation. This function analyzes the texture/intensity distribution within the sphere.

Inputs:

  • An image (3D TIFF or TeraFly) opened in Vaa3D.
  • Vaa3D landmarks placed on or near the features of interest. Spherical landmarks are recommended, as the analysis window size is determined by the landmark radius.

How to Use:

  1. Load your image in Vaa3D.
  2. Place landmarks on the features you want to analyze. Set the radius of the spherical markers to define the analysis window size (the actual analysis uses a sphere with diameter = 2 * radius, centered on the marker).
  3. Navigate to the Vaa3D menu: Plugins -> MIND -> PC Analysis.
  4. The analysis runs automatically for each landmark.

Process:

  • Iterates through each landmark in the current Vaa3D window.
  • For each landmark:
    • Defines a spherical analysis window (diameter = 2 * radius) centered at the landmark coordinates.
    • Calculates the intensity-weighted PCA based on the voxel intensity values within that sphere in the original image. Uses the compute_sphere_win3d_pca_eigVec function internally.
    • Appends the results to a CSV file.

Outputs:

  • Files (saved in the same directory as the input image, or a MIND/[ImageBaseName]/ subdirectory if using TeraFly):
    • [ImageName]_pca_intensity_weighted.csv: A CSV file containing the PCA results for each landmark. Includes eigenvalues, eigenvectors, calculated intensity-weighted center of mass, original marker coordinates, and radius. If the file exists, it will be overwritten.

Visualize PCA

Purpose: Loads PCA results from a .csv file (generated by either "Soma Segmentation" or "PC Analysis") and visualizes the principal components as lines overlaid on the corresponding image.

Prerequisites:

  • A previous run of the Soma Segmentation or PC Analysis function on the image, which will output a PCA result file (e.g., *_pca_binary_segmentation.csv or *_pca_intensity_weighted.csv).

Inputs:

  • The image corresponding to the PCA data must be open in Vaa3D.
  • PCA results .csv file (e.g., *_pca_binary_segmentation.csv or *_pca_intensity_weighted.csv).

How to Use:

  1. Open the image in Vaa3D that was used to generate the PCA results.
  2. Navigate to the Vaa3D menu: Plugins -> MIND -> Visualize PCA.
  3. A file dialog will prompt you to "Open PCA Results". Select the appropriate .csv file.
  4. A new Vaa3D window will open, displaying the visualization.

Process:

  • Reads the selected .csv file row by row.
  • For each row (representing one soma/landmark):
    • Reads the center of mass coordinates (CenterMassX, CenterMassY, CenterMassZ).
    • Reads the eigenvalues (eigenvalue1, eigenvalue2, eigenvalue3) and eigenvectors (eigenvectorN_x/y/z).
    • Calculates the scaled principal component vectors (endpoints): CenterOfMass ± Eigenvector * sqrt(Eigenvalue) * 2.35 (scaling approximates +/- 2 standard deviations).
    • Draws lines representing these scaled components onto a copy of the original image data.

Outputs:

  • Visualization: A new Vaa3D window is opened, displaying the original image data with colored lines representing the principal components overlaid at the center of mass for each entry in the CSV file:
    • Green Line: Longest principal component (associated with eigenvalue1).
    • Red Line: Second longest principal component (associated with eigenvalue2).
    • Cyan Line: Shortest principal component (associated with eigenvalue3).

Visualize Probability Model

Purpose: Loads and visualizes the 3D soma shape probability model (.bin file) generated by the "Soma Segmentation" function.

Prerequisites:

  • A previous run of the Soma Segmentation function on the original image, which will output a binary probability model file (e.g., *_probability_model.bin).

Prerequisites:

  • A binary probability model file named *_probability_model.bin.

How to Use:

  1. Navigate to the Vaa3D menu: Plugins -> MIND -> Visualize Probability Model.
  2. A file dialog will prompt you to "Open Probability Model". Select the desired .bin file.
  3. A new Vaa3D window will display the heatmap, and a separate legend window will appear.

Process:

  • Loads the dimensions and double array data from the specified .bin file.
  • Determines the minimum and maximum probability values in the data.
  • Creates a new 3-channel Vaa3D image.
  • For each voxel in the probability model:
    • Normalizes the probability value to the range [0, 1].
    • Applies the "Inferno" colormap to convert the normalized value to an RGB color.
    • Sets the corresponding voxel in the output image to this RGB color.
  • Displays the resulting heatmap image in a new Vaa3D window.
  • Automatically calls the "Probability Model Legend" function to display the legend.

Outputs:

  • Visualization: A new Vaa3D window displaying the 3D probability model as a volumetric heatmap using the Inferno colormap. Regions with higher probability (more consistently part of aligned/averaged somas) will appear brighter/yellower.
  • Legend: A separate dialog window showing the Inferno colormap legend (see next section).

Probability Model Legend

Purpose: Displays the Inferno colormap legend corresponding to the probability values shown in the "Visualize Probability Model" function.

How to Use:

  • Navigate to the Vaa3D menu: Plugins -> MIND -> Probability Model Legend.
  • This function is also called automatically when "Visualize Probability Model" is used.

Process:

  • Generates an image containing the Inferno colormap bar.
  • Adds numeric labels (0.00, 0.25, 0.50, 0.75, 1.00) below the bar, representing the probability of a voxel being a soma voxel instead of a background voxel.
  • Displays this image in a small dialog window.

Outputs:

  • Legend Window: A dialog window showing the Inferno colormap bar and corresponding probability values.

Create Background

Purpose: Generates a synthetic background image with statistical properties mimicking the background of a real image. This is useful for creating realistic environments for simulated data.

Inputs:

  • An image (3D TIFF or TeraFly) opened in Vaa3D that contains representative background regions.

How to Use:

  1. Open the representative image in Vaa3D.
  2. Navigate to the Vaa3D menu: Plugins -> MIND -> Create Background.
  3. The function runs automatically.

Process:

  • Divides the input image into multiple 3D chunks.
  • For each chunk:
    • Calculates the Otsu threshold to distinguish potential foreground (e.g., somas) from background.
    • Calculates the mean and standard deviation of the intensity values for voxels classified as background (below the threshold).
  • Creates a new blank image of the same dimensions as the input.
  • For each voxel in the new image:
    • Determines the appropriate mean and standard deviation by smoothly blending the statistics from nearby chunks based on distance.
    • Generates a random intensity value from a Gaussian distribution using the blended mean and standard deviation.
    • Clamps the value to the valid range [0, 255].
    • Assigns the value to the voxel.
  • Displays the resulting synthetic background image.

Outputs:

  • Visualization: A new Vaa3D window opens, displaying the generated synthetic background image. The window title will be [OriginalImageName]_background.
  • Files: No files are saved directly by this menu function. (The underlying generation logic is used by the "Simulate Somas" function).

Simulate Somas

Purpose: Generates a synthetic dataset consisting of realistically shaped and textured somas placed within a synthetic background. It also produces the corresponding ground truth binary segmentation mask. This requires prior analysis of real somas using the "Soma Segmentation" function.

Prerequisites:

  • A previous run of the "Soma Segmentation" function on a representative real image must exist in the expected location. Specifically, the plugin needs:
    • [ImageName]_binary_segmentation.tif (to get soma shapes from the original image).
    • [ImageName]_pca_binary_segmentation.csv (to get statistics on soma positions and orientations).
  • An image (3D TIFF or TeraFly) opened in Vaa3D. This image is primarily used to:
    • Get the soma intensities from the original image.
    • Define the simulation volume's dimensions (X, Y, Z).
    • Provide representative background statistics for the create_background step.

Inputs:

  • An image (3D TIFF or TeraFly) opened in Vaa3D.
  • A binary segmentation file of the current image named [ImageName]_binary_segmentation.tif at its expected location
  • A PCA statistics file of somas of the current image named [ImageName]_pca_binary_segmentation.csv at its expected location

How to Use:

  1. Ensure the prerequisite .tif and .csv files exist in the location expected by the plugin (usually the same directory as the image file used to generate them, or within the MIND/[ImageBaseName]/ subdirectory if TeraFly was used).
  2. Open the image in Vaa3D, which will serve as the dimensions and background statistics template.
  3. Navigate to the Vaa3D menu: Plugins -> MIND -> Simulate Somas.
  4. An input dialog asks, "Enter the number of synthetic somas to generate:".

  1. Enter the desired number (e.g., 50) and click OK.

Process:

  1. Loads the prerequisite binary segmentation (.tif) and PCA results (.csv).
  2. Calculates the mean and standard deviation of soma center coordinates and PCA eigenvector components from the loaded PCA data.
  3. Generates a synthetic background image using the create_background logic, analyzing the currently open image for background statistics.
  4. Initializes blank output images for the simulated intensity data and the binary segmentation ground truth.
  5. Repeats for the requested number of synthetic somas:
    • Randomly selects a real soma from the loaded segmentation data to use as a template.
    • Generates a random target position for the new soma based on the distribution of real soma centers, ensuring it's within image bounds and accounting for soma size.
    • Checks for overlaps with previously placed synthetic somas. If an overlap occurs or the position is invalid after multiple attempts, skip this soma.
    • Generates random rotation vectors (eigenvectors) based on the distribution calculated from the real soma PCA data (using QR decomposition to ensure orthogonality).
    • Extracts the 3D binary mask and the corresponding original intensity values for the template soma from the loaded segmentation and the original image data.
    • Applies the random rotation to both the extracted binary mask and the intensity profile.
    • Places the rotated binary mask into the ground truth output image.
    • Places the rotated intensity profile into the synthetic intensity output image (overwriting the synthetic background).
    • Records the position and radius of the successfully placed synthetic soma as a landmark.
  6. Performs PCA analysis on the binary shapes of the generated synthetic somas in the ground truth image.
  7. Saves all output files into a new timestamped directory.
  8. Displays a segmentation visualization overlay.

Outputs:

  • Timestamped Output Directory: A new directory named MIND/[ImageBaseName]_simulation_YYYYMMDD_HHMMSS/ (if using TeraFly path structure) or [ImageDirectory]/[ImageBaseName]_simulation_YYYYMMDD_HHMMSS/ is created, containing the following files:
    • simulated_intensity.tif: The final synthetic image with somas placed in the synthetic background.
    • simulated_segmentation.tif: The ground truth binary segmentation mask for the synthetic somas.
    • simulated_landmarks.marker: A Vaa3D marker file containing the locations and radii of the generated synthetic somas.
    • pca_simulated_segmentation.csv: PCA results (eigenvalues, eigenvectors, center of mass) calculated from the binary shapes of the generated synthetic somas in simulated_segmentation.tif.
    • simulation_summary.txt: A text file with a summary of global information, parameters, and statistics for the simulation.
  • Visualization: A new Vaa3D window opens displaying a 3-channel overlay:
    • Channel 1: Simulated intensity image (simulated_intensity.tif).
    • Channel 2: Simulated binary segmentation (simulated_segmentation.tif).
    • Channel 3: Gradient image highlighting the boundaries from the binary segmentation.

About

This project was primarily developed by ImagiNeuron, a group of 4 McGill University students.

  • Shidan Javaheri
  • Siger Ma
  • Athmane Benarous
  • Thibaut Baguette

ImagiNeuron is deeply grateful for the supervision and expertise of Dr. Amir Shmuel of the Neuro Institute and McGill throughout the project. In addition, the project's success would not have been possible without the open source community behind Vaa3D.

Clone this wiki locally