Proposed changes to ome-zarr-py

[will-moore]

Following on from #407

It is proposed that we deprecate the usage of ome-zarr-py for most of its existing functionality. Either use other existing tools (ngff-zarr or ome-zarr-models-py).

• For writing OME-Zarr data, propose that we use ngff-zarr
• For validating OME-Zarr data, use ome-zarr-models-py (command line?), ome-ngff-validator,
• For reading OME-Zarr data, most usage is to get the first dataset (full size) array for a single image, which is quite simple (or use ome-zarr-models-py)
  • For reading data into napari, we want a dask pyramid for an image or plate. I would like to deprecate the graph traversal logic in ome-zarr-py as I find it hard to understand. See investigate dropping ome-zarr dependency napari-ome-zarr#123 for a more concise re-working of the same logic in napari-ome-zarr. However, the Plate get_pyramid_lazy() code could be used by napari-ome-zarr.

TODOs:

• Review docs and tests in ome-zarr-py to check if the functionality can be ~easily supported by ngff-zarr. Write code examples for ngff-zarr somewhere.
  • Look at open tickets to see if issues can be solved with ngff-zarr
  • Are the ngff-zarr dependencies likely to be a blocker for anyone?
• complete investigate dropping ome-zarr dependency napari-ome-zarr#123 to handle Plates, Labels, etc.
• complete Remove ome-zarr-py dependency omero-cli-zarr#169 (remove ome-zarr-py dependency) and use ngff-zarr instead.
• review Zarr v3 #404 (zarr v3 reading support) - decide what to do with this (merge/close etc)?

cc @joshmoore

[psobolewskiPhD]

I understand and appreciate the motivation, but there is a lot of value in an integrated, "canonical" ome-zarr implementation under the ome/ banner, essentially in parallel to ome/ngff, much how zarr-python is for me at least the canonical zarr implementation. It gives newcomers an easier entry into the ecosystem and some confidence that they are on the right track.
I think that it's great there there is a thriving ecosystem of ome-zarr tools, but that's a lot of cognitive load to figure out what you need vs. just using ome-zarr-py.

One of the big points of feedback I got when talking about OME-Zarr at the QBI meeting last year was how confusing the ecosystem was as it related to zarr, ome-zarr, ngff. Honestly I think it's worse now with zarr v2/v3 ome-zarr 0.4/0.5 compatibility issues 😢. I feel like I've gotten a taste of the post-ome-zarr-py world lately, due to the status of #404 (apt PR number?) and as someone who is pretty plugged in but far from an expert, I've struggled to navigate it let alone advise others.

[joshmoore]

Thanks for the feedback, @psobolewskiPhD. I think everyone agrees that we need more clarity and stability. From my side, I think I'd want us to separate though where it needs to be this codebase and this name. Basically, we've got a lot of flexibility in how we achieve what you're talking about. Dream big. 😄

cc: @thewtex

[fitz-vivodyne]

+1 as a newcomer to the domain.
It's nice having an official batteries included project hosted by this github org.