Feature request: Downsampling of the NMEA data

[it-jonas]

It would be great if there were a possibility to downsample the NMEA data like the AIS Dispatcher does.

[stripydog]

I've taken a look at what AIS dispatcher is doing. If I understand correctly there's no distinction being made in the downsampling between message types so with this downsampling you might never propagate static data for a vessel with a class b transmitter. I'm guessing though that maybe marinetraffic etc. maintain some static data associated with each mmsi so this is not such a big deal. Is that the case?

This is certainly doable but my concern is that it's a little off-mission as it involves decoding all the AIS and making decisions based on data content so might be better done externally. Let me think about this a bit more though as I understand the utility for remote bandwidth constrained sites pumping data to AIS sites.

[it-jonas]

Hello thanks for your fast reply!
I'm not a specialist about NMEA and AIS data. But I run kplex on 3 AIS relay stations and I'd like to reduce the data running through our sat connection. Right now we have like 250-300 messages per minute running through our service on each station. And yes we also provide this data for example for marinetraffic.
I understood the down sampling service which the ais-dispatcher provides the way that duplicated messages in a certain time frame will be transmitted only once. So if you define a time frame of 300s than a static message would be transmitted only once every 300s.
Decoding and filtering the messages to transmit the position etc. of the same vessel also just every XYs would be great, but more like a nice to have feature I think.
Thanks a lot for your nice work!

[stripydog]

I do understand the utility of this, especially if you're transmitting over satellite. Not all AIS messages contain the same info and notably "static" data about a vessel (e.g. name, length etc.) are transmitted with a much lower frequency than position reports. Both will be tagged with the same MMSI though. Taken at face value, if AIS dispatcher just allows one message with a given MMSI every N seconds, it could filter out almost all the static data reports. That may not be significant if marinetraffic (or wherever) holds those data (e.g. from previous station reports). Also, that may not actually be what AIS dispatcher is doing. It could be that rather than actually decoding the messages and filtering on MMSI it's scanning for duplicates in the first 7 characters of the payload field of VDM messages without actually decoding the data which would be the quick and effective way of doing things and would pass different message types from the same MMSI within one time window. Note that the messages AIS dispatcher is filtering out won't be duplicates, but whereas position updates every 2-10 seconds are useful for collision avoidance, they're not really necessary for vessel tracking via marinetraffic.

Anyway...this type of filtering (rate limiting based on a substring match in the payload) doesn't easily fit into the current filtering framework so I was thinking again about a "plugin" mechanism which I've been considering for years and have an idea how I might approach it. In the meantime I've also started to write some AIS decoding software which is proving educational. An intermediate solution might be something like..instead of current invocation of kplex, remove the AIS interface from your main kplex.conf and rather than invoking kplex the normal way do something like:
kplex -f- serial:filename=/dev/ttyUSB0,baud=38400 file:direction=out | myfilter | kplex file:direction=in
, ie one instance of kplex to read AIS, then pipe it to another program ("myfilter") and pipe that to stdin of the kplex which sends to marinetraffic. The simplest version of "myfilter" could be a script which passes any VDM sentence with a fragment count (second field) > 1 as it's really position reports we want to cut down on, then for others looks at the first 7 characters of the payload in field 6 and passes the sentence if and only if no sentence with those same first 7 characters in field 6 has been seen in the past N seconds (easy in any scripting language with a dict/hash table type). I have a pretty busy week but I can knock up an example if you don't feel like having a go (this is my reference for AIS, but using the above strategy no decoding is required: https://gpsd.gitlab.io/gpsd/AIVDM.html ). The AIS decoding stuff I'm looking at should also work but might be a little overkill.

Longer term I'm going to re-vist the plugin idea, but it may be a couple of months before that comes to fruition

[stripydog]

I'm thinking something kinda like this....excuse any non-idiomatic or even downright wrong code here: I don't claim any python proficiency but it's probably the easiest choice for most people:

import sys
import time
import argparse

lastpass = {}

parser = argparse.ArgumentParser(description='Downsample AIS input')
parser.add_argument('-f', type=int, default=300, metavar='s',
                   help='Downsample frequency (seconds)')
args = parser.parse_args()

for line in sys.stdin:
    fields = line.split(",")
    if len(fields) == 7 and fields[1] == "1":
        now = time.time()
        msgid = (fields[5])[:7]
        if msgid in lastpass and ( now - lastpass[msgid] ) < args.f:
            continue
        lastpass[msgid] = now
    sys.stdout.write(line)
    sys.stdout.flush()

[it-jonas]

Hello,
thanks a lot for all your afford. I also already tested piping the Input into a filtering script. But thought it would be awesome to have this integrated into the main application. I'll develop within the next weeks a solution for our specific problem.
A plugin possibility would be great a great solution!

[stripydog]

fwiw had a reply from AISHub to say that their downsampling only affects position reports so filtering based on the first 7 characters of the armoured AIS info as above should replicated this behaviour (if my calculations are correct which they may not be)

[stripydog]

Update: Today I pushed the "filter" branch to GitHub, not because it's ready for public consumption but because I'd done too much work on it to leave it uncommitted on my laptop. It's probably full of precarious race conditions and lacks suitable messages but does basically work and with the minor modification to the above python program of providing with a default downsample frequency (because the new "xofilter=" and "xifilter=" options currently only take a single argument so I can pass command line parameters) I was achieving the required downsampling.

I'm working on a set of AIS decoding libraries and a utility to use them but for downsampling, as previously suggested, a simple python script will do.

I'll smooth out some of the rough edges in the coming week and hopefully ask you to test it soon.

[stripydog]

The xfilter branch still has some tweaks I could make but is ready for testing with details of how to use external filters in the README but basically the deal is...
use "xofilter=" (for output filters) or "xifilter=" (for input filters) and specify the command you want to run all your output (or input) through on an interface. If the command takes arguments, put the whole command line in quotes, e.g.
xofilter="/tmp/myprog -f 1800"
Take care to make any filter program line buffered or flush stdout after every write. Filter programs should read from stdin, write to stdout and make their own arrangements for stderr. They shouldn't block SIGTERM.
If a filter program is adding or changing sentences (rather than just passing/dropping them) make sure one of the following is true:

• You are ensuring the NMEA checksum is valid within your program
• Your downstream app doesn't care about checksums and you don't have checksumming enabled for the interface with the xfilter
• You're producing sentences without checksums and stripping off checksums from modified sentences and have the "checksum=add" or "checksum=addonly" options on the interface to add missing checksums (this is a new feature not in 1.4)

[stripydog]

Have you had a chance to give this a try? I have in development a program which will be able to decode AIS and filter by message type and rate limit by mmsi. It would be able to be used as a kplex external filter. What other functionality would you be interested in seeing?

[julled]

Hey @stripydog, thanks for all your work on klpex! I really enjoy using the tool!

Did you continue working on this? I have also the same usecase of only letting one position per AIS through per each boat every X seconds.

Was this merged? Or is there a complete example how to do this? Or should the script in #36 (comment) work?