Certain selector combinations negatively affect compile time

[amandasaurus]

I have ported the openstreetmap-carto style to vector tiles, and am using tessera, which is based on tilelive, to serve the raster tiles (which are created from vectortiles). However it takes a very long time to start up (~4 minutes) and that's from carto's processing of the cartocss files. I initially reported this on the tessera issue tracker, see more background there

Using simple print statement debugging, I can see that some of the longest parts are inside the rules = inheritDefinitions(matching, env); call. For some osm-carto layers, there are thousands of items in the matching variable. (e.g. test-polyg-low-zoom has 6,005. bridges has 1,099, test-poly as 6,064 etc). It takes about 1 second (on my machine) for every 1,000 items. so some of these layers take about 5sec to process. There are lots of layers, so it's about 3½ minutes to parse the whole file.

A weird thing is that I can use the same node_modules to parse the upstream project.mml into mapnik XML in 24sec. This is only a problem when parsing a vector tiles file.

I have patched tilelive-tm{style,source} to use the latest carto (0.17.2).

You can test this yourself by checking out the openstreetmap-carto-vector-tiles project and doing make tessera (which will serve the tiles after a while).

[nebulon42]

I'm undecided if this is a bug or not, but this would need improvement. Thanks for the investigation so far. openstreetmap-carto has slow layers also in the Mapnik XML case but not that slow. I know not enough about the vector workflow to say what could be different there. Needs more investigation.

[amandasaurus]

Some more "print statement debugging" tells me that for the upstream project, the layers have much less items in the matching vartiable. text-poly-low-zoom only has 1,016 layers, bridges has 951, text-poly has 1.054. This is probably why the non-vector-tile version is much faster (30sec vs 3½min).

I know openstreetmap-carto is complicated and slow, but it's the difference which is the bug for me. If it took about the same time, that would be one thing.

To me, this is a bug, since it negatively affects what should be a valid work flow.

[nebulon42]

Ok, let's call it a bug then.

[nebulon42]

Doesn't run for me. make tessera failed. I have singled out the command below.

$ ./node_modules/.bin/tessera -c tessera-serve-vector-tiles.json
/dir/openstreetmap-carto-vector-tiles/node_modules/tilelive-cache/index.js:232
    uri.query = uri.query || {};
                   ^

TypeError: Cannot read property 'query' of undefined
    at Object.<anonymous> (/dir/openstreetmap-carto-vector-tiles/node_modules/tilelive-cache/index.js:232:20)
    at Object.load (/dir/openstreetmap-carto-vector-tiles/node_modules/locking-cache/index.js:85:17)
    at /dir/openstreetmap-carto-vector-tiles/node_modules/tessera/server.js:100:16
    at Array.forEach (native)
    at module.exports (/dir/openstreetmap-carto-vector-tiles/node_modules/tessera/server.js:88:25)
    at Object.<anonymous> (/dir/openstreetmap-carto-vector-tiles/node_modules/tessera/bin/tessera.js:82:30)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)

Have you seen that before?

[amandasaurus]

I've seen that. You can work around it by running the tmsource on port 8081 in one tessera process/terminatl window, and the tmstyle in another window. I'll post full details on Monday

[amandasaurus]

I have fixed the my vector tiles project, so make tessera will work now. Please pull from the current master.

[nebulon42]

Some random notes not necessarily related to the behaviour, primarily for myself:

• Is there a way to easily change datasource parameters (db, user, password) and not having to change them in osm-carto.tm2source/data.yml?
• tilelive-tmsource/tilelive-tmstyle uses an outdated version of carto (0.14.x) as also noted above
• the slow rendering happens in tilelive-tmstyle, not in tilelive-tmsource
• not any really siginificant differences in terms of Stylesheet object as input for the renderer when comparing vanilla osm-carto and osm-carto-vector-tiles
• Kosmtik seems to have the same problem, since I'm much more familiar with Kosmtik I will concentrate on this.
• I suspect it might have to do something with the layers being referenced by their class attribute in the style files.
• in the matching variable there are an identical number of references to amenity-points.mss for the vector and non-vector case
• in the same variable the number of "elements": [ is almost 6x higher for the vector case, the overall number of lines deviates about the factor 6-7
• classIndex is not correctly set for the vector case, because the layer classes are not correctly set in the vector case
• adding the respective class to project.yml e.g. - {id: text-poly-low-zoom, class: text-low-zoom} did not change the behaviour, but now layer classes are correctly set

[nebulon42]

I have turned on benchmarking mode for carto, which you can achieve if you call new carto.Renderer({ benchmark: true }, {}).render(opts) in tilelive-tmsource. It reveals that the following layers are quite slow:

processing layer: text-poly-low-zoom
Inheritance time: 43857ms
...
processing layer: text-poly
Inheritance time: 54167ms
...
processing layer: text-point
Inheritance time: 53358ms

These three layers alone take nearly 3 minutes on my machine.

For comparison compiling osm-carto with the benchmark option on the command line gives:

processing layer: text-poly-low-zoom
Inheritance time: 1411ms
...
processing layer: text-poly
Inheritance time: 2062ms
...
processing layer: text-point
Inheritance time: 2044ms