COG parameters and specifics

[davidfrantz]

In FORCE, you can use three different pre-configured output formats, plus using custom formats by providing your own options.

1. The default option is GeoTiff (though with pre-configured options that are similar to a Cloud Optiized Geotiff)
2. ENVI, which is flat binary datastream with an ENVI header.
3. Cloud Optimized GeoTiff

COG is only usable since GDAL 3.1 (see #373)

However, COG has some issues. It is meant to be used for partial file access. But apparently, this only concerns reading, not writing, ..huh

For Level 2 output generation, COG works fine. Even when merging two images, the output file is newly created.

For Higher Level output generation, COG is an issue as GDAL rejects updating existing files. As this is required for block-wise processing, this will result in output issues (see #345).

One solution is to skip block-wise processing. You can achieve this by setting BLOCK_SIZE to the size of the tiles, i.e., one "block" only. However, this requires you to have enough RAM to hold all necessary data of a tile in RAM.

Steps forward:

1. I will implement a check whether BLOCK_SIZE != TILE_SIZE if OUTPUT_FORMAT = COG. This will make the issue more transparent.

2. Secondly, the GDAL documentation says that there is a parameter IGNORE_COG_LAYOUT_BREAK, which seems worth testing.

Updating a COG file generally breaks part of the optimizations, but still produces a valid GeoTIFF file. Starting with GDAL 3.8, to avoid undesired loss of the COG characteristics, opening such a file in update mode will be rejected, unless the IGNORE_COG_LAYOUT_BREAK open option is also explicitly set to YES.

However, this does not seem to have any effect when I enable it. I am unsure whether this parameter is actually used. This brings me to another related issue:

---

I am currently trying to set these default parameters:

    case _FMT_COG_:
      copy_string(gdalopt->extension,   NPOW_04, "tif");
      copy_string(gdalopt->driver,      NPOW_04, "COG");
      copy_string(gdalopt->option[o++], NPOW_10, "BLOCKSIZE");
      copy_string(gdalopt->option[o++], NPOW_10, "256");
      copy_string(gdalopt->option[o++], NPOW_10, "COMPRESS");
      copy_string(gdalopt->option[o++], NPOW_10, "ZSTD");
      copy_string(gdalopt->option[o++], NPOW_10, "PREDICTOR");
      copy_string(gdalopt->option[o++], NPOW_10, "YES");
      copy_string(gdalopt->option[o++], NPOW_10, "INTERLEAVE");
      copy_string(gdalopt->option[o++], NPOW_10, "TILE");
      copy_string(gdalopt->option[o++], NPOW_10, "BIGTIFF");
      copy_string(gdalopt->option[o++], NPOW_10, "YES");
      copy_string(gdalopt->option[o++], NPOW_10, "TILED");
      copy_string(gdalopt->option[o++], NPOW_10, "YES");
      copy_string(gdalopt->option[o++], NPOW_10, "OVERVIEW_RESAMPLING");
      copy_string(gdalopt->option[o++], NPOW_10, "AVERAGE");
      copy_string(gdalopt->option[o++], NPOW_10, "IGNORE_COG_LAYOUT_BREAK");
      copy_string(gdalopt->option[o++], NPOW_10, "YES");
      break;

However, GDAL seems to ignore a few of them, e.g. the interleave:

Image Structure Metadata:
  COMPRESSION=ZSTD
  INTERLEAVE=BAND
  LAYOUT=COG
  PREDICTOR=2

This makes me wonder whether the IGNORE_COG_LAYOUT_BREAK parameter was actually used... The default interleave is PIXEL, so it seems like setting the parameter did have an effect. But I would expect it to display TILE.

@jakimow, as far as I understand, you have looked into the COGs before. Do you have any suggestions or experience in this regard?