Create a simple guide showing how to use this with a common FE library such as leafletjs or even QGIS

[ToddG]

WHAT
Create a simple guide showing how to use this with at least one of:

• QGIS
• LeafletJS
• MapLibre

WHY
I'd like to use this as part of a gleam app I'm writing. I'm currently using Martin, and while it's fast, it's not as configurable as I'd like. Since my app is already written for the BEAM, I'd like to continue to use BEAM based technologies rather than cobbling things together.

HOW

• create an examples folder/directory in your repo
• add a sample elixir app that uses this library
• frontend app has some demo data that it exposes
• add a sample frontend app that displays this demo data

[iStefo]

Hi @ToddG, thanks for reaching out and showing interest in this library. I understand that the current documentation isn't helpful in finding out how the (encoded) tiles can be connected to a mapping application or library.

There are two major missing building blocks that would be needed to complete an example as you suggested:

1. A tileserver that can deliver vector tiles, probably by presenting a TileJSON document with tiles pointing to a x/y/z-addressable endpoint that delivers the actual vector tiles.
2. A data backend that can be queried for a certain bounding box (and zoom/detail level) for those x/y/z-based tile requests.
3. (and ideally an actual client application, e.g. MapLibre in Livebook)

This library only fills the gap between retrieving the relevant location data and sending it to clients as Mapbox Vector Tiles (MVT). This narrow and well-defined scope makes the library quite reusable because it doesn't tie it to either a specific data source (e.g. a PostGIS DB) nor to how the resulting vector tiles should be served (including caching/CDN, store on disk etc.).

However, the unopinionatedness of this library also means that giving examples for these missing bits feels quite out-of-scope to me.

For example, we use this library with a self-written simple tileserver attached to a Phoenix application that we already use. For the backend, we use GenServer processes that fetch raw data from Postgres (but without PostGIS extensions) and then perform clustering with varying settings depending on the tile's zoom level. For development purposes, we currently use the linked MapLibre <-> Livebook integration.

That all being said: If you do want to create some resources that make the problem space more accessible to you (or help others with that), I would be happy to support you with that.

[ToddG]

iStefo, thank you!

Overview

I'm building stack right now that's composed of:

Stack
BE

• gleam/BEAM : runtime
• Postgres + Postgis extensions
• Martin : tileserver for dynamic custom data
• OpenStreetMap : for all the static background tiles

FE

• gleam/lustre
• leafletjs

Note, I'm not sold on leafletjs, it's just that I got it working quickly, and I'm still struggling to get maplibrejl working.

Tile Server

With respect to tileservers, I'm finding that Martin is not matching my needs:

• no fine grained cache control
• no dynamic configuration
• no built-in auth/auth

I'll have a bunch of data coming in with different caching characteristics. Some data will be a constant feed and will need to update user content in near real time. Some of the data is only changing every 15min or so (based on a cron job that queries external data sources). So being able to have fine grained cache control is a must, and it must be different for different data sources.

I'll also be hosting datasets that cannot be shared between user groups. I've hacked Martin in the past to provide authorized access for this purpose, but it was pretty janky. I'm starting to think I'd rather have all this logic inside my own BEAM servers anyway. This way I can auth/auth in one place, then serve out data as I need w/o having to cobble together martin + varnish + other random stuff.

So, once I saw this library, I thought...how hard would it be to write a simple tile server? I only need to serve up things like points and polygons...basically locations and perimeters...run some calculations like creating buffers around polygons, and determining if points-are-in-polygons... that sort of thing.

Design

Perhaps you could help me break this down a bit so I have a better idea of what I need to build and where this library fits in...:

• A tileserver that can deliver vector tiles, probably by presenting a TileJSON document with tiles pointing to a x/y/z-addressable endpoint that delivers the actual vector tiles.

• given a set of points, generate geometry

• apply some style rules to the geometry

• render geometry into tiles

• do the above for each zoom level

• do the above either at query time, in the background, at data ingest time, or some combination of the above

• A data backend that can be queried for a certain bounding box (and zoom/detail level) for those x/y/z-based tile requests.

I don't know much about this.

• Client application : (and ideally an actual client application, e.g. MapLibre in Livebook)

Wow, maplibre/livebook looks really cool. That might be helpful, I need to play with that a bit.

[iStefo]

Stack BE

• gleam/BEAM : runtime
• Postgres + Postgis extensions
• Martin : tileserver for dynamic custom data
• OpenStreetMap : for all the static background tiles

The OSM layers would be added to the mapping application/library as an "additional" source, next to the custom data sources.

FE

• gleam/lustre
• leafletjs

Note, I'm not sold on leafletjs, it's just that I got it working quickly, and I'm still struggling to get maplibrejl working.

Leaflet is a solid choice that is easy to start out with. For more complex applications I came to appreciate Maplibre for its extensibility while still great performance and excellent rendering quality.

Tile Server

With respect to tileservers, I'm finding that Martin is not matching my needs:

• no fine grained cache control
• no dynamic configuration
• no built-in auth/auth

I'll have a bunch of data coming in with different caching characteristics. Some data will be a constant feed and will need to update user content in near real time. Some of the data is only changing every 15min or so (based on a cron job that queries external data sources). So being able to have fine grained cache control is a must, and it must be different for different data sources.

A general thought here: I'm not sure how easy it is to actively "refresh" a vector source in Maplibre from the client. However, adding a Cache-Control: max-age=30 response header on the raw tile requests actually causes automatic refreshing.
So you could split your data into different sources depending on cache behavior and have Maplibre automatically refetch active tiles of the source according to the Cache-Control header.

Alternatively, use the lowest cache duration and handle caching of long-lived data sources manually in the backend :)

I'll also be hosting datasets that cannot be shared between user groups. I've hacked Martin in the past to provide authorized access for this purpose, but it was pretty janky. I'm starting to think I'd rather have all this logic inside my own BEAM servers anyway. This way I can auth/auth in one place, then serve out data as I need w/o having to cobble together martin + varnish + other random stuff.

Yes, we also have such a requirement. Luckily, if your tile server is just a regular Phoenix server, you can use all the authentication code that might already be in place. Thus, supporting multi-tenancy is not a real issue.

So, once I saw this library, I thought...how hard would it be to write a simple tile server? I only need to serve up things like points and polygons...basically locations and perimeters...run some calculations like creating buffers around polygons, and determining if points-are-in-polygons... that sort of thing.

Design

Perhaps you could help me break this down a bit so I have a better idea of what I need to build and where this library fits in...:

• A tileserver that can deliver vector tiles, probably by presenting a TileJSON document with tiles pointing to a x/y/z-addressable endpoint that delivers the actual vector tiles.
• given a set of points, generate geometry
• apply some style rules to the geometry
• render geometry into tiles
• do the above for each zoom level
• do the above either at query time, in the background, at data ingest time, or some combination of the above
• A data backend that can be queried for a certain bounding box (and zoom/detail level) for those x/y/z-based tile requests.

I don't know much about this.

• Client application : (and ideally an actual client application, e.g. MapLibre in Livebook)

Wow, maplibre/livebook looks really cool. That might be helpful, I need to play with that a bit.

So, this is roughly what we do:

These routes are in our Phoenix Router:

scope "/tiles" do
  get "/", TileController, :tilejson
  get "/:zoom/:x/:y", TileController, :tile
end

TileController, for serving the TileJSON:

def tilejson(conn, params) do
  # 1. Extract relevant parameters from query
  server_params = get_filter_from_params(params)

  # 2. Start a GenServer that can already prefetch relevant data
  {:ok, _server_pid} = ClusterServer.start_for_params(server_params)

  # 3. Generate a token that allows finding the same GenServer for answering tile requests
  token = TokenHelper.generate_token(conn, server_params)

  conn
  |> put_status(:ok)
  |> json(
    %{
      tilejson: "3.0.0",
      tiles: [
        # 4. Attach token to tile URL
        "#{conn.scheme}://#{conn.host}:#{conn.port}#{conn.request_path}/{z}/{x}/{y}?&token=#{token}"
      ],
      tile_size: 4096,
      vector_layers: [
        %{
          id: "clusters",
          fields: %{
            count: "Number",
            color: "String"
          }
        },
        %{
          id: "outliers",
          fields: %{
            id: "String"
          }
        }
      ]
    }
  )
end

TileController, for serving the raw tiles:

@extent 4096
@sm SphericalMercator.new(size: @extent)

alias VectorTile.{
  Feature,
  Layer,
  Tile
}

def tile(conn, %{"x" => x, "y" => y, "zoom" => zoom, "token" => token}) do
  # Tile Bounds in WGS 84
  tile_bounds = SphericalMercator.bbox(@sm, x, y, zoom)

  # Find GenServer that holds the data
  {:ok, server_token} = TokenHelper.verify_token(conn, query_param(conn, :token))
  {:ok, server_pid} = ClusterServer.start_for_params(server_token)
  
  # Query for features within tile bounds
  features = ClusterServer.query(server_pid, tile_bounds, zoom)

  # Tile with bbox in Web Mercator for feature position interpolation
  tile = Tile.new(bbox: SphericalMercator.convert(@sm, tile_bounds, "900913"), extent: @extent)

  # Create layer structs to hold the features
  luster_layer = %Layer{
    name: "clusters",
    version: 2,
    extent: @extent
  }

  outlier_layer = %Layer{
    name: "outliers",
    version: 2,
    extent: @extent
  }

  # Assign features to respective layers 
  {cluster_layer, outlier_layer} =
    Enum.reduce(
      features,
      {cluster_layer, outlier_layer},
      fn %RObject{x: x, y: y, data: data}, {cluster_layer, outlier_layer} ->
        case data do
          {:outlier, id} ->
            outlier_layer =
              Layer.add_feature(
                outlier_layer,
                Feature.point([x, y], project: &project_point(&1, tile)),
                id: id
              )

            {cluster_layer, outlier_layer}

          {:cluster, count, color} ->
            cluster_layer =
              Layer.add_feature(
                cluster_layer,
                Feature.point([x, y], project: &project_point(&1, tile)),
                count: count,
                color: color
              )

            {cluster_layer, outlier_layer}
        end
      end
    )
  
  # Add layers to tile
  tile = %{tile | layers: [cluster_layer, outlier_layer]}

  conn
  |> put_resp_content_type("application/vnd.mapbox-vector-tile")
  |> send_resp(200, Protobuf.encode_to_iodata(tile))
end

TileController, for projecting feature coordinates into tile coordinate system:

defp project_point(point, tile) do
  SphericalMercator.forward(@sm, point)
  |> VectorTile.Coordinates.interpolate(tile)
end

That's really all there is to the tileserver. We're currently thinking about an addition to this library to simplify the feature handling even more, but essentially, you're just building your VectorTile structs and then sending them down.

All your specific caching and querying would be handled by your version of ClusterServer.

[ToddG]

Thank you ! That's very helpful. I'll take a run at implementing this in gleam and get back to you with my results.