`rust future panicked: unknown error` when opening TIFF files

[maxrjones]

Here is an MVCE for rust panics I encountered when building virtual_tiff. The TIFFs are from GDAL's test suite. You can change file_id to switch which TIFF is being opened.

import obstore as obs
from async_tiff import TIFF
from async_tiff.store import LocalStore
import os
import asyncio


# Set which file to run the test on
file_id = 0

# Download file
store = obs.store.HTTPStore.from_url("https://dagshub.com")

panics = [
    ("gdal_gcore", "huge-number-strips.tif"),
    ("gdal_gcore", "byte_user_defined_geokeys.tif"),
    ("gdal_gcore", "minimum_tiff_tags_no_warning.tif"),
    ("gdal_gcore", "uint16_sgilog.tif"),
    ("gdal_gcore", "corrupted_gtiff_tags.tif"),
    ("gdal_gcore", "gtiff_towgs84_override.tif"),
    ("gdal_gcore", "minimum_tiff_tags_with_warning.tif"),
    ("gdal_gcore", "bigtiff_header_extract.tif"),
    ("gdal_gcore", "spaf27_markedcorrect.tif"),
    ("gdal_gcore", "huge-implied-number-strips.tif"),
    ("gdal_gcore", "excessive-memory-TIFFFillStrip2.tif"),
    ("gdal_gcore", "packbits-not-enough-data.tif"),
    ("gdal_gcore", "excessive-memory-TIFFFillStrip.tif"),
    ("gdal_gcore", "byte_bigtiff_invalid_slong8_for_stripoffsets.tif"),
    ("gdal_gcore", "weird_mercator_2sp.tif"),
    ("gdal_gcore", "many_blocks_truncated.tif"),
    ("gdal_autotest", "epsg_27563_allgeokeys.tif"),
    ("gdal_autotest", "nodata_precision_issue_float32.tif"),
    ("gdal_autotest", "byte_coord_epoch.tif"),
    ("gdal_autotest", "projected_GTCitationGeoKey_with_underscore_and_GeogTOWGS84GeoKey.tif"),
    ("gdal_autotest", "nodata_precision_issue_float32.tif"),
    ("gdal_autotest", "missing_tilebytecounts_and_offsets.tif"),
]
path = f"maxrjones/virtual-tiff/raw/73934fbfb31cd5bb099f225a47ac8ea34298f3ea/tests/dvc/{panics[file_id][0]}/{panics[file_id][1]}"


resp = obs.get(store, path)
with open(panics[file_id][1], "wb") as f:
    for chunk in resp:
        f.write(chunk)

# Attempt to open file using async_tiff
async def open_tiff(*, store, path):
    return await TIFF.open(path, store=store)

store = LocalStore()
filepath = f"{os.getcwd()}/{panics[file_id][1]}"


tiff = asyncio.run(open_tiff(store=store, path=filepath))
print(tiff)

[weiji14]

How do you plan on tackling these @maxrjones? Do you want this repo to include unit tests that cover part of GDAL's test suite, or just ensure that there are no panics (i.e. replace all the .unwrap()s in the Rust code)?

These are the main panics/errors I got from running your list above:

• photometric interpretation not found
• called Result::unwrap() on an Err value: IOError(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" })
• Unknown GeoKeyTag id: {key_id}: TryFromPrimitiveError { number: 2062 }
• samples_per_pixel not found
• Unknown GeoKeyTag id: {key_id}: TryFromPrimitiveError { number: 3059 }
• called Result::unwrap() on an Err value: InternalTIFFError(FormatError(UnsignedIntegerExpected(SignedBig(496))))

Some of these are GDAL/geo-specific (e.g. GeoKeyTag 2062 and 3059) and might better belong in https://github.com/georust/geotiff if we're trying to keep async-tiff as a generic TIFF reader. Edit: on reading the linked threads, I'm of the opinion that we should not implement reading those tags into the TIFF struct, unless they are specified in the GeoTIFF 1.2 spec, but we can at least not panic when encountering unknown tags.

Other panics like the photometric_interpretation and samples_per_pixel ones could certainly be handle here, but want to double check on how you prefer to organize the repo's unit tests before going ahead.

[kylebarron]

Just one quick note: I wonder if we could have a registry of extension tags. I'd love to remove geo specific tags from this repo, or at least remove them from their special status. Perhaps we could let users register custom tag codes plus how to parse them?

[kylebarron]

These are the main panics/errors I got from running your list above:

@weiji14 perhaps we should create a separate issue for each of these?

[maxrjones]

Do you want this repo to include unit tests that cover part of GDAL's test suite, or just ensure that there are no panics (i.e. replace all the .unwrap()s in the Rust code)?

Thanks both for your work on this! I don't have strong opinions about the test structure for async_tiff. I plan to continue to independently test virtualization over in virtual_tiff.

[kylebarron]

Where are you developing virtual_tiff?

[maxrjones]

Where are you developing virtual_tiff?

I just moved it to https://github.com/developmentseed/virtual-tiff from my individual account to make it more findable.

[maxrjones]

FYI I decided to try the OME-TIFF file from https://www.cellimagelibrary.org/images/40613 and encountered another panic on opening (xref virtual-zarr/virtual-tiff#32).