Harmonization of elevation data sources

[zod]

BRouter currently has four elevation sources

• ConvertSrtmTile supports _1arc_v3_bil.zip which seem to support files in 1 deg tiles "Band Interleaved by Line" (BIL) format which is used by USGS for 1 arcsec global coverage. Data will be converted to 5 deg .bef tiles.
• SrtmData supports "Ersi ASCII" format in 5 deg tiles which is used by CSI CGIAR for the SRTM v4.1 90m global dataset. BRouter supports converting to 5 deg .bef tiles or reading it directly into SrtmRaster
• ConvertLidarTile supports "HGT" format in 1 deg tiles which is used by NASA for their SRTM Plus 30m global dataset or Sonny's LiDAR DTM. 1 deg 90m tiles can be converted into 5 deg .bef tiles and direct access via SrtmRaster allows 30m and 90m 1 deg tiles. Converted tiles need to use srtm_x_y.bef as filename

The .rd5 segments currently use two datasources

• SRTM CSI CGIAR for global coverage (lat -60 till +60)
• Sonny's LiDAR for northern europe (above lat +60)

As it's rather complicated (and undocumented) and uses some confusing names we should rename and remove some parts.

• SrtmRaster should be renamed to ElevationRaster because it supports elevation data from sources other than SRTM
• ConvertSrtmTile should be removed because the dataset it supports a dataset which is superseded by other SRTM datasets which are provided in different formats
• SrtmData should be renamed to something different which indicates the "Ersi ASCII" format. It's not related to SRTM and other datasources could be converted to "Ersi ASCII" using gdal
• ConvertLidarTile should support 30m resolution also for .bef conversion
• .bef files should use the same naming scheme as .rd5 instead of mimicking as SRTM tiles
• We should add documentation which also explains using the LiDAR tiles

Most of the information is from reading the source or older issues, so maybe @abrensch can confirm if I'm correct.

[abrensch]

Most of the information is from reading the source or older issues,
so maybe @abrensch can confirm if I'm correct.

Ja sure, the first part is correct and the "should" part depends on your motivation...

A good source is my 2015 Fossgis Presentation where, starting from Minute 16:34, I talked about
the usage of srtm data: https://av.tib.eu/media/17557

Some additional facts from my fading memory:

• I introduced the "bef" format as part of my "30m project" to have a good compromise between file size and decoding speed. File Size of current 90m bef dataset is 10 GB

• CGIAR SRTM Data are 6001 x 6001 Pixels so they have a common pixel line between neigbouring Tiles, which makes linear interpolation in a 4-pixel-square easy: you never have to access 2 Files to do that. There exists also srtm files with 6000 x 6000 pixels (or 3600 x 3600 for 30m/1 degree) that do not have this common line

• In "BEF", I included the possibility to have a "border" of more than one pixel in order to do interpolations accessing more than 4 pixels but still accessing only one file.

• Another confusing fact with Nasa 30m SRTM Data is that above 50 degree north/ below 50 degree south they switch to 30m x 60m pixels.

[afischerdev]

.bef files should use the same naming scheme as .rd5 instead of mimicking as SRTM tiles

Good idea, this is the main feature for me.
The current naming includes only 60N to 60S in a proper way, so e.g. a tile named srtm_39_01.bef has the neighbour tiles in North srtm_39_00, then srtm_39_-01 and so on.

Other naming
For me, SRTM is a collective term for satellite raster data may be completed with some other values.
There are serveral output formats like ascii, binary or bitmap data. And this can have different resolution. May be different internal logic.
The formats are our focus. Not the organisation they come from.

srtm as part of the name indicates what the classes are for. ElevationRaster is good and neutral in your sense but also very general.

Perhaps one converter class could handle the different formats. e.g. SrtmTileConverter
Or one class for each general format, like SrtmAsciiTileConverter, SrtmBinaryTileConverter, SrtmBitmapTileConverter

[abrensch]

In order to deploy the master branch as a pre-processor I had to comment out the query for hgt-style files in "PosUnifier"

Reason was a performance Bottle-neck from doing expensive ops on a per-node basis like File.exists and String.format

Maybe we can radically cleanup the "PosUnifier" to use only a normalized format (5x5 befs with 30m resolution and hgt-style names) and put all the magic in the converter tool that then should be able to create befs from mixed sources, putting lidar where available and falling back to cgiar srtm where not?

[zod]

Ja sure, the first part is correct and the "should" part depends on your motivation...

I think it would be sensible to cleanup some stuff while my memory is still fresh. But let's see how much time I can devote to this.

A good source is my 2015 Fossgis Presentation where, starting from Minute 16:34, I talked about the usage of srtm data: https://av.tib.eu/media/17557

Thanks, that's quite helpful.

* CGIAR SRTM Data are 6001 x 6001 Pixels so they have a common pixel line between neigbouring Tiles, which makes linear interpolation in a 4-pixel-square easy: you never have to access 2 Files to do that. There exists also srtm files with 6000 x 6000 pixels (or 3600 x 3600 for 30m/1 degree) that do not have this common line

I've downloaded SRTM 90m DEM Version 4 from https://srtm.csi.cgiar.org/download but it seems those files are 6000 x 6000 (ncols and nrows are both 6000 and each row contains 6000 values).

* In "BEF", I included the possibility to have a "border" of more than one pixel in order to do interpolations accessing more than 4 pixels but still accessing only one file.

By looking at the code this handling doesn't seem related to "BEF" which just contains coordinates and a grid. I think the border handling is in ConvertLidarTile.

srtm as part of the name indicates what the classes are for. ElevationRaster is good and neutral in your sense but also very general.

Maybe we should use DemRaster, which should be correct, regardless of it's DTM or DSM and how this data is obtained (e.g. LiDAR or Radar)

Maybe we can radically cleanup the "PosUnifier" to use only a normalized format (5x5 befs with 30m resolution and hgt-style names) and put all the magic in the converter tool that then should be able to create befs from mixed sources, putting lidar where available and falling back to cgiar srtm where not?

This would be a clean solution but would also break functionality for existing users which create rd5 files directly from "Ersi ASCII" files.

[eddy-geek]

https://mapterhorn.com/coverage/ (github) is a good source of high-precision Lidar elevation data in Europe

It was not available at the time of the last work as per #644 (comment)

However I'm not clear what is the state here, #829 seems to imply that we are still on 3'' data on the main server ? Does trying to integrate the mapterhorn pmtiles format make sense for the brouter project ?

Mapterhorn v0.0.4 distributes Terrarium-encoded terrain rgb tiles as webp images with 512 pixel width. The tiles are bundled into multiple PMTiles archives. Alternatively, single tiles can be retrieved from a zxy-endpoint.

[afischerdev]

@eddy-geek
Thanks for your hint.

However I'm not clear what is the state here, #829 seems to imply that we are still on 3'' data

Yes that is correct. It needs the server update to make use of some new functions.
The BRouter online version is running with library 1.7.0.

So please hang on a while.