add /validate endpoint for md dataset

[vincentsarago]

Q:

• should we validate the entire dataset
• should we validate all the variables
• if a dataset if valide, does all the variables are?

My knowledge of Xarray dataset structure is limited so would love some help on this 🙏

[maxrjones]

Q:

• should we validate the entire dataset
• should we validate all the variables

My knowledge of Xarray dataset structure is limited so would love some help on this 🙏

Could specific data variable be included as a query parameter and all data variables are validated if not a specific one is not queried? E.g., if someone knows they will want to tile precipitation they would likely not want to go through the effort to validate all the DataArrays in the Dataset.

• if a dataset if valide, does all the variables are?

No, as one common example the CRS information when using rioxarray is often stored in a crs data variable which is not valid for titiler even though the dimensions and coordinates of the dataset are valid:

[vincentsarago]

🙏 @maxrjones

Right now the I have added the variable= option which will validate a specific variable, if not present it will try to validate the whole dataset.

If I understand well, when variable is not set, you'll iterate through all the variables and return something like

{
    "var1": {
        "compatible_with_titiler": true,
        "errors": []
    },
    "var2": {
        "compatible_with_titiler": false,
        "errors": [
            "Dataset does not have compatible spatial coordinates"
        ]
    },
    ...
}

[maxrjones]

This approach seems good for short-term use, though eventually it would be nice to use a schema validation approach (e.g., https://github.com/xarray-contrib/xarray-schema) rather than eagerly applying the operations to save time and memory.

My main concern is about the validation and io code becoming out of sync. Could one or more utility functions be defined for some of the code in https://github.com/developmentseed/titiler/blob/main/src/titiler/xarray/titiler/xarray/io.py#L96-L202 for use in both io and validate to mitigate that chance?

It also looks like the metadata and variable extensions are becoming redundant. Is the separate Variable extension still needed?