New working item for BigTIFF community standard

[joanma747]

BigTIFF has become more necessary that ever with lager images, multiband images and multiple resolutions (COG).
The Testbed 17 has a proposal for a new draft specification for BigTIFF: https://gitlab.ogc.org/ogc/T17-D046-COG-Specification-ER/-/tree/master/standardDrafts/BigTIFF

We would like to move it to an OGC community standard in this group. The process is a bit different form other community standards. There is only an "informal" web page that defines BigTIFF. IT is clear and comprehensible but not sufficiently formal, so we cannot adopt any preexisting document directly. It was needed to reformatted it into clear requirements.

The issue is identified in #12 where an analysis of the implications for BigTIFF tags in GeoTIFF keys could have is presented

[ogcscotts]

@joanma747 - two questions on the Community Standard.

1. Do the owners of BigTIFF agree to submit the document?
2. If the Community Standard is being written as a result of Testbed activity, we will need to get the owner of BigTIFF to agree to the text and that it represents an authoritative specification for BigTIFF before we could consider this for a Community Standard. It may also be worth considering whether the BigTIFF ownership could be transferred to OGC.

[cmheazel]

An alternate approach, add a BigTIFF conformance class where we document the differences between TIFF and BigTIFF. No need for an external standard.

[joanma747]

Let's advertise our initiative in the LibTIFF email list and look for their support. We should strongly underline that we are fully compatible with the previous documentation and implementation of the BigTIFF.

Consider to make BigTIFF a public repo

MOTION to establish an OGC BigTIFF standards development effort, using a public Git (gitlab or github) repository and invite participation of the BigTIFF community
Moved by Chuck Heazel
Second by Even Rouault
Discussion: We are going to use the LibTIFF email list. IT seems that it will be and OGC standard with support of the community. It is not clear if this is an "OGC community standard" or a "pure OGC standard with the support of the community" at this point.
NOTUC

We will ask Greg for a new repository for this working item. It makes configuration management easier

Let's draft some simple text to do the first contact to the LibTIFF list.

@cmheazel, the requirement class for BigTIFF is here:
https://gitlab.ogc.org/ogc/T17-D046-COG-Specification-ER/-/blob/master/standardDrafts/BigTIFF/standard/clause_7_normative_text.adoc

[joanma747]

The repository to work in this working item is here:
https://github.com/opengeospatial/BigTIFF

[joanma747]

The repository is now populated with the materials created from Testbed17. The intention is to keep them aligned (manually) until the end of Testbed 17, when the work will exclusively continue in the GeoTIFF.SWG

[joanma747]

Proposal of text email for the LibTIFF list:

A group of people in the Open Geospatial Consortium (OGC, www.ogc.org) would like to move the BigTIFF into the OGC standards process. The intention is to create a OGC standard that is fully compatible with the previous documentation and implementation of the BigTIFF. We would like to do this in a open process and make sure that the community that was behind the BigTIFF specification are invited to participate in the process and are well recognized for their previous work. The OGC will provide a process to maintain the standard in the future and will work to make GeoTIFF and Cloud Optimized GeoTIFF (COG) standards to support both TIFF and BigTIFF. All OGC standards are available for free in the OGC website.

The process will be done in a open GitHub repository and the first attempt can be found here: https://github.com/opengeospatial/BigTIFF
You are invited to submit issues or to create pull requests as part of the way to provided comments and improve the standard draft.

Is there any objection or comment before we start this process?

[joanma747]

Even Rouault will send this to the LibTIFF email list. Thanks for that!

[rouault]

email just sent

[rouault]

Good question from Rob Tillaart regaring TIFF tags in general:

Rob Tillaart: Will the OGC put a process in place for BigTiff tags?

me: > What do you call exactly as BigTIFF tags ?

Rob Tillaart: In the past there was a lot of discussion about (un)documented / (un)registered tags and how to handle them.
Will the OGC play a role in this process of documenting tags?
I can imagine that they only are interested in Geospatial related tags and not in e.g. medical tags

[rouault]

"""
Hi Even et al:

To my knowledge, the community that was behind the BigTIFF specification is now effectively Leica (medical, not Geosystems).
The initial specification came from Ole Eichhorn, then at Aperio, a digital pathology company which was acquired by Leica.
The involvement of Aware systems as far as I know was only hosting a discussion on their mailing list, and documenting essentially a copy of Ole's work. I don't believe the participants were from Aware.
Anyone do please correct me if my understanding is incomplete or incorrect.

However, this unofficial specification only covers a migration of TIFF to 64-bits, and says nothing in particular about tiling, multi-resolutions, multi-channels, bit depths, compression, TIFFtags/metadata, etc, all of which need to be defined per application at a higher level.
Various stabs at this for multiresolution imaging have been made over the years: Aperio.svs, COG, ZIF (my own), OME-TIFF (used ZIF as a reference point), and many 99% similar flavors of pyramidal tiled (big)TIFF. Shame we can't all get it together.

Given that GeoTIFF and COG specifications exist, what is OGC intending to add beyond that? Is it just the "64-bit"ness, or metadata?

There is no formal maintainer; the AWARE systems site does not maintain the BigTIFF specification, that is just a reference; another is at http://bigtiff.org, a site now owned by Leica.

And I'm not sure the wording on https://github.com/opengeospatial/BigTIFF is as intended?: "it becomes an OGC standard" sounds like OGC intends to take over ownership of BigTIFF? Leica might have something to say about that, although the specification has been put in the public domain.
Personally, I really wish Adobe as the owner of the TIFF specification would step up to the plate with this, and update the actual TIFF specification to cover BigTIFF. Their focus has been elsewhere for many years, but this would be a great community service.

Regards,

Kemp Watson
Objective Pathology Services
Toronto, Canada
"""

[rouault]

Some clarification given by Joris Van Damme on the non-central implication of Aperio in the BigTIFF design:
"""
Kemp,

To my knowledge, the community that was behind the BigTIFF specification is now effectively Leica (medical, not Geosystems).

Incorrect. Leica, as a company, was only involved in that they were
one of four sponsors of the BigTIFF implementation inside LibTIFF.
They were completely unrelated to the design. It's possible a number
of people that contributed to the design in the mailing list were
employed by Leica at the time, but if so they were not speaking for
that company.

The initial specification came from Ole Eichhorn, then at Aperio, a digital pathology company which was acquired by Leica.

No. Original specification was a community effort, on the mailing
list, and to this day you can read up on that and verify it in the
mailing list archive. Aperio, at that time, had an alternative design,
partly incompatible, that they put into an independent implementation,
without consulting the community. It made things quite awkward for a
while. They still own bigtiff.org. I hope that at least the stuff they
offer on that site is no longer incompatible with our own
specifications, but I haven't checked.

The involvement of Aware systems as far as I know was only hosting a discussion on their mailing list

Incorrect. AWare Systems is me, Joris Van Damme. I actively
contributed to the design phase quite a bit, though it truly was a
community effort. I also did the implementation in LibTIFF, sponsored
by four other companies.

Various stabs at this for multiresolution imaging have been made over the years

The SubIFDs tag allows for any number of any resolution child ifds.
It's a specification totally independent from BigTIFF, SubIFDs can and
should be used in classic TIFF and BigTIFF. You're not making sense.

I can go on correcting your other statements, but I fail to see how
that's worth the effort at this point. It's ancient history. Why spin
these things, what is your angle?

Given that GeoTIFF and COG specifications exist, what is OGC intending to add beyond that? Is it just the "64-bit"ness, or metadata?

Like the man said, it is the BigTIFF part, so, sure, the "64-bit"ness.
BigTIFF is just TIFF with 64bit offsets, by design.

I feel the current activity in the GeoTIFF community is a good thing
that will surely result in more widespread BigTIFF adoption, and more
widespread confidence in its validity and the fact that it's here to
stay. Even, thanks for letting us know.

Best regards,

Joris Van Damme
AWare Systems
"""

[joanma747]

It is so great that we have such support from Joris Van Damme. I love the tone of the last paragraph. This email clarifies the history and helps a lot.

[rouault]

Project of response to the reactions from the libtiff mailing list

Hi,
thanks to all who have provided feedback. Our group just met and we'd like to provide the following clarifications:
- we'll change the readme of the github repository to mention that the current BigTIFF specification is hosted on the awaresystems.be website rather than maintained.
- we do not plan, at least at that stage, to maintain a full TIFF tag registry. OGC only intends to maintain the GeoTIFF tags, as well as tags for geospatial metadata.
- the scope of the BigTIFF specification maintained by our OGC group will be exactly the same as the one mentioned above. As you may see from the draft, it is a relatively short specification, documenting only the differences between BigTIFF and classic TIFF, that is the evolution of the header, the directory structure, and more generally all the 64-bits indexing / offsets extension of TIFF6 specification. The TIFF 6 spec will be the reference for all aspects not modified by BigTIFF.
- "who is responsible for BigTIFF?" That's indeed the main issue that triggered our initiative. Our review of the current situation was that no one was formally a "owner" or in charge, or showing hints to be willing to be in charge, hence we've decided to step up. Our motivation to do that is clearly to be able to use BigTIFF in a GeoTIFF context. But BigTIFF having not a formal enough specification, this makes it hard to comply with the requirements of a standards process. We've considered to put BigTIFF as an annex of the GeoTIFF specification, but that would not have been a clean approach. We believe that by making BigTIFF a standalone specification, it might also be useful for other communities, beyond the geospatial one, and will make further adoption of the technology easier in contexts where a formal specification is needed.

Can we answer if there will be an ETS for BigTIFF ?