consider implementing validation on gtfs feeds

[carocad]

There are a couple of tools already that can do validation of GTFS feeds. I have checked some of them but they all seems very focused on their own way of doing things and might not match properly with the in-memory representation of kamal.

The most known implementation of GTFS validation is: https://github.com/google/transitfeed/wiki/FeedValidator

After a careful check I found out that the validation is still pretty straightforward and we might be able to implement it in kamal without much work.

[derhuerst]

There are a couple of tools already that can do validation of GTFS feeds. I have checked some of them but they all seems very focused on their own way of doing things and might not match properly with the in-memory representation of kamal.

Just FYI: I'm very interested in writing a GTFS validator (yet another) that is a) truly FS-independent and b) streaming. But it's quite a lot of work to validate GTFS properly and not just built another half-baked validator.

Also, I'm fluent in JS and not very familiar with Clojure.

[carocad]

a GTFS validator (yet another) that is a) truly FS-independent

this relatively straightforward. At least in Java with a ZipInpuStream you can stream the file wherever it is (local/remote). Even in Js you should be able to use a plain Buffer/Stream to do the same.

b) streaming

although you can stream a zip file, I am pretty sure that it is not possible to stream the validation. This is due to GTFS data being deeply linked to each other. For example, you cannot validate stop_times without trips, nor routes nor agencies. That means that you need to (at least for some time) keep the complete gtfs feed in memory.

From that point onwards you might return lazy sequences of validations over chunks of input. In kamal we are using Datascript, which also comes with a Js interface. You might want to take a look at it, together with the gtfs.edn repo which we use to follow the schema.

Hope it helps

[derhuerst]

this relatively straightforward. At least in Java with a ZipInpuStream you can stream the file wherever it is (local/remote). Even in Js you should be able to use a plain Buffer/Stream to do the same.

Yup, did this in gtfs-utils. It wants a function (filename) => readStream.

b) streaming

although you can stream a zip file, I am pretty sure that it is not possible to stream the validation. This is due to GTFS data being deeply linked to each other. For example, you cannot validate stop_times without trips, nor routes nor agencies. That means that you need to (at least for some time) keep the complete gtfs feed in memory.

Although you will have to keep some items in memory, not the complete GTFS feed, but just a reduced form. This is what gtfs-utils does. Still you can reach far lower memory limits than having the whole feed in memory.

Using a more memory-efficient language would also be an option obviously. 😁

From that point onwards you might return lazy sequences of validations over chunks of input.

This should only work if the files are sorted in the right way, correct? How would you make sure that your chunk of a.txt matches the referenced data in b.txt?

[carocad]

Although you will have to keep some items in memory, not the complete GTFS feed, but just a reduced form. This is what gtfs-utils does. Still you can reach far lower memory limits than having the whole feed in memory.

Certainly, though I am pretty sure this also limits the amount of validations that you can do in the data as you would always have to keep in your head the current state of the validator and figure out which information are you able to use at that point, which in my opinion limits the usability of such a tool.

This should only work if the files are sorted in the right way, correct?

I think I didnt expressed myself properly here. I was talking about returning a stream of validation errors back. In Clojure, lazy sequences are performed on chunks of 32 elements, which means that after those are validated, you can release its memory and validate the rest, which can help a lot in keeping memory usage under control.

[derhuerst]

I think I didnt expressed myself properly here. I was talking about returning a stream of validation errors back. In Clojure, lazy sequences are performed on chunks of 32 elements, which means that after those are validated, you can release its memory and validate the rest, which can help a lot in keeping memory usage under control.

Yes, but by far the largest part would not come from keeping the invalid rows, but from keeping all data necessary to validate each row.

[carocad]

Absolutely.

Could you share some insights into your experiences with other validators and what you expect from this one in terms of memory and time?

Like expected maximum memory consumption, duration. Maybe do you have a specific feed in mind to test a prototype against?

[derhuerst]

Could you share some insights into your experiences with other validators and what you expect from this one in terms of memory and time?

I don't have much experience with them TBH. I wrote gtfs-utils for analysis purposes, not for validation. Nevertheless, the memory consumption should be similar.

I know of GTFS feeds being a couple of hundred MBs (unpacked). This may increase to the range of a GB because there will likely be more country-wide GTFS feeds soon.

[carocad]

This may increase to the range of a GB because there will likely be more country-wide GTFS feeds soon.

mmm that will definitely become quite a challengue. I will think more about it. So far from my end, it is not easy to come up with a solution that can work properly mainly because my requirement is for it to work online as well. Since kamal lets you submit transactions to modify the underlying GTFS feed, it would be best if this validation could also check that the new gtfs feed is valid. Otherwise it would force the user to always rely on zip feeds which can get pretty tricky once they reach a certain size, just like you mentioned

[derhuerst]

Not sure what exactly you want to support online, but

1. I don't think ZIP GTFS feeds are a good idea: Any reasonable server should nowadays served individual files gzipped; On disk, having a ZIP instead of files is very unpractical.
2. Routing engines directly importing GTFS seems like a bad idea. All routing engines I know have routing built in in some way, and IMHO it doesn't belong there; It represents unnecessary complexity, nr of files, etc. GTFS -> routing-compatible format should be a preprocessing step.
3. Given 2, the online API for modifying the graph should be much more practical/use-case-focused than GTFS, which feels like a DB dump. In fact GTFS-RT is meant for that and doesn't use the relational-DB-like format of GTFS.

[carocad]

This is getting a bit out of topic. However I think the discussion is interesing in general

Any reasonable server should nowadays served individual files gzipped

Do you have a reference for this? afaik, the current best practice is to deliver a single zip file

Routing engines directly importing GTFS seems like a bad idea

This is entirely subjetive and an implementation detail at best. For reference, kamal is implemented in such a way that the abstraction behind GTFS feeds is minimal. In most cases the gtfs entities map directly to our internal api. For example: in GTFS trips.txt trip_heasign -> trip/headsign in Datascript.

In fact GTFS-RT is meant for that and doesn't use the relational-DB-like format of GTFS

this is a misunderstanding. I agree that real-time updates should come through something like GTFS-realtime and not handled here. My point before was about the permanent updates to a GTFS feeds; like changing the name of a stop. If a server (like kamal) already has all the information about a feed and (potentially) knows how to validate it, why not directly use it? I think that moving that logic to yet another service that has to recreate all that logic is wasteful