Zarr

[will-moore]

Previously this was #619, but that PR also included more changes (standalone deployment of OMERO.figure app etc) not related to OME-Zarr support. That PR has been split to make code-review and testing easier.

This PR only adds support for rendering OME-Zarr images in the web app, allowing you to "Add Image" by pasting in a zarr URL instead of an Image ID.

NB: Updates to the export python script to support zarr images are in #662

To test:

Use public OME-Zarr images, e.g. from https://idr.github.io/ome-ngff-samples/ or https://ome.github.io/ome2024-ngff-challenge/ or anywhere else.

• Copy Zarr URL to a single zarr Image (not a Plate or bioformats2raw container)
• Enter this value (or multiple comma-separated URLs) into the "Add Images" dialog
• Images should be added and rendered as normal
• Try various rendering settings, zoom/pan, Z/T, LUTs, etc.
• Check images render OK on main panel, ROI dialog, Crop dialog
• Pixel sizes should be read from the OME-Zarr, and timestamps (but we don't have good samples as discussed below)
• Save the figure to OMERO, reopen it etc. NB: won't see thumbnail on File Open dialog
• NOT supported: other OMERO-based operations: Labels from KVPs/Tags, loading ROIs etc.
• NOT yet supported: export of OME-Zarr images in script (see Export script handles OME-zarr images #662)

We use https://github.com/BioNGFF/ome-zarr.js for some of the rendering logic (rendering arrays to png data url).

The rendering logic for zarr images is the same as for OMERO images. For "small" images, 3k x 3k or less, we render the whole plane each time, and simply zoom and position it relative to the viewport. For bigger images, we render the required region (with some overlap outside the viewport).

We cache the zarr array data that is loaded via zarrita.js, using a key that encodes the range (slices) requested. So when we want to re-render the same region (e.g. adjust rendering settings) we don't have to re-fetch data.
This means that panning "small" images doesn't need to reload the zarr data, but panning bigger images does. Since we cache the whole region (not individual tiles/chunks), even a small amount of panning will re-load the zarr data for the whole rendered region. The amount of data loading is mitigated by rendering at low-ish resolutions.

We save the zarr data to the figure.json for each panel. The imageId is used to store the zarr URL:

{
  "imageId": "imageId": "https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0048A/9846152.zarr",
  "zarr": {
    "version": "0.5",
    "zarr_version": 3,
    "arrays": {
      "0": {
        "dtype": "uint16",
        "shape": [
          3, 91, 13350, 19120
        ]
      },
      "1": {
        "dtype": "uint16",
        "shape": [ 3, 91, 6675, 9560
        ]
      }
    },
    "multiscales": [
      {
        "axes": [
          {
            "name": "c",
            "type": "channel"
          },
          {
            "name": "z",
            "type": "space"
          },
          {
            "name": "y",
            "type": "space"
          },
          {
            "name": "x",
            "type": "space"
          }
        ],
        "datasets": [
          {
            "coordinateTransformations": [
              {
                "scale": [ 1, 1, 1, 1],
                "type": "scale"
              }
            ],
            "path": "0"
          },
          {
            "coordinateTransformations": [
              {
                "scale": [ 1, 1, 2, 2],
                "type": "scale"
              }
            ],
            "path": "1"
          }
        ]
      }
    ]
  }
}

The array dimensions allows us to choose the correct dataset array when rendering a particular region.

TODO:

• TBD: do we try to pick a higher resolution of the zarr pyramid when the panel is shown larger on the page?

[pwalczysko]

Export script #662 will be merged before this is released, so users will have it working. I guess for this PR, we could test for zarr urls and simply skip them from export.

Not sure about the latter. I guess the best would be to upload the script from #662 to merge-ci ?

Edit: I went ahead and uploaded the script from #662 to merge-ci. A problem reported on #662 (comment)

[pwalczysko]

We don't handle those yet. I guess we could add a check for that and provide a link to open in validator

@will-moore thanks for the confirmation. +1 for the check (i.e. the user does not get exposed to the techy error). But I guess the user will appreciate more a hint about how to get to single images which are viewable in Figure ? (maybe wanting too much, or maybe the validator will give a hint ?)

[will-moore]

@pwalczysko I added handling for bf2raw images, plates, 404s etc.

In ngff-validator, you can actually copy a single Image URL by clicking on the Mobie icon. I guess I could add a dedicated "Copy URL" button too.

For Labels from KVPs and Tags, the zarr images now behave the same as OMERO images without Tags/KVPs.

Thnaks for testing the export script. I'll have a look at that once this PR is looking good.

[snoopycrimecop]

Conflicting PR. Removed from build OMERO-plugins-push#38. See the console output for more details.
Possible conflicts:

• PR OMERO ROIs improvements : filtering visible and adding all #641 Rdornier 'OMERO ROIs improvements : filtering visible and adding all'
  • src/js/views/roi_modal_view.js

--conflicts Conflict resolved in build OMERO-plugins-push#41. See the console output for more details.

[pwalczysko]

Re-tested.

All works fine. But the script in the #662 has problems.

@Tom-TBT do you want to have a look at the improvements you asked for ? I can see that imho your insert-in-sequence prob with picking up the resolutions was solved. Also the picking of single images seems to have been solved by #663 (comment).

(btw, sorry for the X:0, Y:0 comment, I actually realized I do not understand this feature, it always shows 0,0, also on non-zarr images -> but the Width in pixels is being displayed okay for zarr)

[pwalczysko]

I would be happy to go ahead once the #662 is solved. Of course, a second look by @Tom-TBT would be nice too.

[Tom-TBT]

I gave it another go on merge-ci, I added one of the Zarr OMERO image to figure, and added the same image from its URL.

The difference here is quite explicit. I pushed the zoom to the max so I'm guessing that's why the effect is so pronounced.
This doesn't have to be a blocker, I think I'd also want to see it together with the export. So if it helps to go forward merging the PR, go ahead, and the sub-res selection logic could be in the next PR.

JSON of my figure

{
  "version": 9,
  "panels": [
    {
      "x": 186.46806324110673,
      "y": 27.16,
      "width": 168.06482213438736,
      "height": 167.91850975289174,
      "zoom": 1000,
      "dx": -175.82887700534752,
      "dy": 8.983957219251351,
      "labels": [
        {
          "text": "From OMERO",
          "size": "12",
          "position": "top",
          "color": "000000"
        }
      ],
      "deltaT": [],
      "rotation": 360,
      "selected": false,
      "pixel_size_x_symbol": "µm",
      "pixel_size_x_unit": "MICROMETER",
      "rotation_symbol": "°",
      "max_export_dpi": 1000,
      "vertical_flip": false,
      "horizontal_flip": false,
      "imageId": 2302,
      "name": "Patient_ImageStack_0000.ome.zarr",
      "sizeZ": 25,
      "theZ": 0,
      "sizeT": 1,
      "theT": 0,
      "channels": [
        {
          "emissionWave": null,
          "label": "protein",
          "color": "00FF00",
          "inverted": false,
          "reverseIntensity": false,
          "family": "linear",
          "coefficient": 1,
          "window": {
            "min": 0,
            "max": 65535,
            "start": 107,
            "end": 189
          },
          "active": true
        },
        {
          "emissionWave": null,
          "label": "cell",
          "color": "FF0000",
          "inverted": false,
          "reverseIntensity": false,
          "family": "linear",
          "coefficient": 1,
          "window": {
            "min": 0,
            "max": 65535,
            "start": 121,
            "end": 1892
          },
          "active": true
        }
      ],
      "orig_width": 1200,
      "orig_height": 1200,
      "datasetName": "http",
      "datasetId": 1801,
      "pixelsType": "uint16",
      "pixel_range": [
        0,
        65535
      ]
    },
    {
      "x": 10,
      "y": 27.16,
      "width": 168.06482213438736,
      "height": 167.9185097528918,
      "zoom": 1000,
      "dx": -175.82887700534755,
      "dy": 8.983957219251344,
      "labels": [
        {
          "text": "From URL",
          "size": "12",
          "position": "top",
          "color": "000000"
        }
      ],
      "deltaT": [],
      "rotation": 360,
      "selected": false,
      "pixel_size_x_symbol": "µm",
      "pixel_size_x_unit": "MICROMETER",
      "rotation_symbol": "°",
      "max_export_dpi": 1000,
      "vertical_flip": false,
      "horizontal_flip": false,
      "imageId": "https://uk1s3.embassy.ebi.ac.uk/bia-idr-integration/S-BIAD2216/microglia_zipped/HC/caudate/HC10/Patient_ImageStack_0000.ome.zarr",
      "name": "/",
      "sizeZ": 25,
      "theZ": 0,
      "sizeT": 1,
      "theT": 0,
      "channels": [
        {
          "active": true,
          "color": "00FF00",
          "label": "protein",
          "window": {
            "end": 189,
            "start": 107
          }
        },
        {
          "active": true,
          "color": "FF0000",
          "label": "cell",
          "window": {
            "end": 1892,
            "start": 121
          }
        }
      ],
      "orig_width": 1200,
      "orig_height": 1200,
      "pixelsType": "uint8",
      "zarr": {
        "multiscales": [
          {
            "axes": [
              {
                "name": "c",
                "type": "channel"
              },
              {
                "name": "z",
                "scale": 0.5,
                "type": "space",
                "unit": "micrometer"
              },
              {
                "name": "y",
                "scale": 0.11,
                "type": "space",
                "unit": "micrometer"
              },
              {
                "name": "x",
                "scale": 0.11,
                "type": "space",
                "unit": "micrometer"
              }
            ],
            "channel_names": [
              "protein",
              "cell"
            ],
            "datasets": [
              {
                "coordinateTransformations": [
                  {
                    "scale": [
                      1,
                      0.5,
                      0.11,
                      0.11
                    ],
                    "type": "scale"
                  }
                ],
                "path": "0"
              },
              {
                "coordinateTransformations": [
                  {
                    "scale": [
                      1,
                      1,
                      0.22,
                      0.22
                    ],
                    "type": "scale"
                  }
                ],
                "path": "1"
              },
              {
                "coordinateTransformations": [
                  {
                    "scale": [
                      1,
                      2,
                      0.44,
                      0.44
                    ],
                    "type": "scale"
                  }
                ],
                "path": "2"
              },
              {
                "coordinateTransformations": [
                  {
                    "scale": [
                      1,
                      4,
                      0.88,
                      0.88
                    ],
                    "type": "scale"
                  }
                ],
                "path": "3"
              },
              {
                "coordinateTransformations": [
                  {
                    "scale": [
                      1,
                      8,
                      1.76,
                      1.76
                    ],
                    "type": "scale"
                  }
                ],
                "path": "4"
              }
            ],
            "name": "/",
            "version": "0.4"
          }
        ],
        "arrays": {
          "0": {
            "shape": [
              2,
              25,
              1200,
              1200
            ],
            "dtype": "uint16"
          },
          "1": {
            "shape": [
              2,
              25,
              600,
              600
            ],
            "dtype": "uint16"
          },
          "2": {
            "shape": [
              2,
              25,
              300,
              300
            ],
            "dtype": "uint16"
          },
          "3": {
            "shape": [
              2,
              25,
              150,
              150
            ],
            "dtype": "uint16"
          },
          "4": {
            "shape": [
              2,
              25,
              75,
              75
            ],
            "dtype": "uint16"
          }
        },
        "zarr_version": 2,
        "version": "0.4"
      },
      "pixel_size_x": 0.11,
      "pixel_size_y": 0.11,
      "pixel_size_y_unit": "MICROMETER",
      "pixel_size_y_symbol": "µm",
      "pixel_size_z": 0.5,
      "pixel_size_z_unit": "MICROMETER",
      "pixel_size_z_symbol": "µm"
    }
  ],
  "paper_width": 364.53288537549406,
  "paper_height": 205.0785097528918,
  "page_size": "mm",
  "page_count": 1,
  "paper_spacing": 50,
  "page_col_count": 1,
  "height_mm": 72,
  "width_mm": 129,
  "orientation": "vertical",
  "legend": "",
  "legend_collapsed": true,
  "page_color": "ffffff"
}

[will-moore]

Thanks @Tom-TBT that's useful info...

There's a couple of issues there...

I've boosted the target resolution (target pixel size with respect to the size of the panel on the page).

So, now the targetSize for rendering is 672 pixels (instead of 336) and we pick the 600 x 600 zarr resolution level for rendering.

The other issue is that the panel is quite small on the figure.
If you resize the page to A4 and zoom out, then the panel appears smaller, so you don't see the pixels so much.

The combination of these fixes improves the impression:

Actually, I also added a further tweak - If the image isn't a big image (we render the whole plane each time, same as from OMERO), we can further boost the targetSize because caching will avoid us having to re-fetch zarr chunks on each re-render. With this change, there's now NO difference in resolution between the panels in your figure.

[will-moore]

NB: I noticed the Channel sliders weren't working for that Zarr image because the window min/max aren't set in the zarr json.
That would make the zarr "invalid" according to current schema https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-idr-integration/S-BIAD2216/microglia_zipped/HC/caudate/HC10/Patient_ImageStack_0000.ome.zarr
but we should handle that anyway, particularly as we plan to relax that requirement: ome/ngff#297

[Tom-TBT]

Looks good now, thanks Will

[will-moore]

The last commit handles any missing values for omero.channels filling in the min/max values by calculating them from the lowest resolution of the image if needed.

[pwalczysko]

Conflicting branch. Resolve conflicts please.