Deal with Fluksosensors that have reset to 0

[JrtPec]

We have a sensor that has reset its counter to 0 some years ago, resulting in a large negative gas consumption. This screws up the prediction model, while this could be easily fixed imo.

[icarus75]

This might occur after a clean firmware flash on an FLM02 or a repair of an FLM03.

[saroele]

So while it should only occur rarely, it screws up any analysis. What are our options? 1. check the tmpo data after obtaining it from tmpo-py for negative differences? All data is supposed to be monotonically increasing, so this is easy to check. But can it be that minor negative differences occur more frequently as a result of sensor tolerance or measurement errors? 2. add a check-and-fix in tmpo-py? 3. go even more upstream and fix it in the fluksometer? 4. ?

…

On Fri, Mar 24, 2017 at 9:27 AM, Bart Van Der Meerssche < ***@***.***> wrote: This might occur after a clean firmware flash on an FLM02 or a repair of an FLM03. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/opengridcc/opengrid/issues/194#issuecomment-288961613>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAiVBqpaj9_GgGGwK6wRBPdIrRj9lVljks5ro35xgaJpZM4MfTss> .

[JrtPec]

1. Seems easy to implement:

ts = ts.diff()
ts = ts[ts > 0]
ts = ts.cumsum()

This will basically ignore the resets, and start again from 0. Two downsides: I suspect this will be enormously slow, and if you are caching the counter values (or saving them, like EnergieID) it will create more problems, you'll have to start saving the offset from which to start counting or something.

I don't think we will have to deal with minor negative differences, as I suspect that counters are limited by software to only count up (unless they get reset of course).

I really hope we can find a fix in tmpo-py or more upstream... @icarus75 any ideas?

[icarus75]

1/ Down-counting can occur in an FLM02 hooked up to a P1 port. When net energy is being injected into the grid, the counter value will decrease. The FLM03 works with separate P+ and P- registers so that even in a P1 case, the FLM03's counters will only increase.
2/ A valid check will thus combine detection of a large delta with a near-zero new counter value.
3/ The primary cause of counter 0-ing is a complete re-flashing of an FLM or battery removal of a Kube. In both cases the counter registers are zeroed. There's no way for the FLM or FLK to recover from this amnesia.
4/ tmpo-py currently serves as a utility lib to reconstruct the original time series, be it counter or gauge, encoded in tmpo format. We could decide to include a 0-counter value check into tmpo-py when reconstructing a tmpo counter time series. If yes, then I would suggest to explicitly request this via a named parameter.
5/ Since tmpo-py will only decode blocks that are part of the requested time interval, you will get a different output if the 0-event is part of this time interval or not.

[JrtPec]

So if I have understood correctly yesterday: all counters should always be counting UP, except for P1-port counters on an FLM02.

@icarus75 Would it be possible to get this information from the TMPO header: wether it is a counter or not, is it an FLM02, is it a P1?

I would propose to add a flag to tmpo-py's series method that can fix the offset. But users have to beware that absolute counter values may change when they use this flag.

[icarus75]

Correct.

Here's a typical tmpo block. "h" stands for header, "t" for time and "v" for value.

{"h":{"cfg":{"id":"f8c4b8e8628096c0057dc8810480a83c","class":"kube","kid":1,"unit":"V","type":"battery","rid":7,"data_type":"gauge"},"tail":[1490788557,2.9],"vsn":1,"head":[1490788407,2.9]},"t":[0,75,75],"v":[0,0,0]}

A sensor is linked to a P1 port when its "class" equals "cosem".
A sensor is a counter when "data_type" equals "counter".