List of file formats to evaluate

[gmaze]

Here it comes !

Considering #39 and #41, here is the list of the possible file formats to be evaluated (not ordered):

• Zarr
• GeoParquet - Parquet
• COG: Cloud-Optimized GeoTIFF
• JSON - geoJSON - Json
• "Indexed database"
• ?

They are many useful informations about some of these formats in here.

From my feeling and limited knowledge of existing codes, I attempted to place them in this decision guiding 2D map:

This map can be modified in here. Please tell me if this should be modified based on your knowledge and will of implementing a format tests.

Note: I listed "indexed database" above but I don't see how it could be compared with the other candidates. But this is my own feeling, so please tell me how to be precise and possibly insert or not an explicit "indexed database" candidate as a cloud file format in this exercice

[bkatiemills]

Thanks for putting this together, @gmaze. To your question about a database-driven approach, you're asking the right questions in #42; assuming evaluating those questions is what you mean in this thread by 'testing', Argovis can answer these questions quantitatively today for one example of a database and API implementation - just point apache bench (or whatever your favorite benchmarker is) at our API and off you go. I'm very happy to drive there once the metrics become clearer. Therefore I'd put a database-plus-API approach all the way to the right on the chart, in terms of ease of testing, as an example case is already implemented and many tools exist to benchmark it.

What makes something a prime or not-prime candidate at this point? I'd like to understand the Objectives, also from #42, a bit more before making this judgement call; for example, flat file formats are probably optimal for archival applications and users who want everything, whereas database-driven solutions are probably optimal for targeted search (though by all means, let's test!).

[gmaze]

What makes something a prime or not-prime candidate at this point?

My pragmatic answer:
File formats that our resources -best efforts - will allow us to reasonably explore/test before next ADMT

But I guess we should seek for more guidance from the ADMT/GDAC folks,
poke @mscanderbeg @cgourcuf @tcarval

[gmaze]

flat file formats are probably optimal for archival applications and users who want everything, whereas database-driven solutions are probably optimal for targeted search

Indeed, and that's why with I'm very concern that we will be comparing apples with oranges if we simply test for one use case, or use cases with kind of opposite requirements to perform well

so again, I think we should seek for more guidance from the ADMT/GDAC folks,
poke @mscanderbeg @cgourcuf @tcarval

[dirkslawinski]

Just a reminder not all formats are supported by all analysis tools. Zarr for instance would be ideal but generally considered Python only. There is no official Matlab Zarr access at the moment except via the Python bridge, JAVA and custom mex files

[gmaze]

Fully agreed, this is a big limitation for zarr at the moment

[gmaze]

But it is an active development subject, raised a while ago here:
zarr-developers/community#16

and considering the last post on this issue, Mathworks seems to be working on a zarr interface

no timeline available, but it's on a roadmap somewhere

ps: current zarr implementations listed here: https://github.com/zarr-developers/zarr_implementations