how much should history be scaled?

[brianjmurrell]

Hi.

Great project!

I'm trying to feed your PitDroid from my own temperature controller project. I have it all pretty much working. Your app can fetch data from my controller. The only part I don't yet have working is history scaling.

How many points of data does PitDroid expect from /luci/lm/hist at maximum? Hopefully that question makes sense.

[brianjmurrell]

So, just looking at the code a bit here:

if ((mSamples.size() == 0 || mSamples.size() > 500 || timeSinceLastUpdate > kMaxUpdateDelta)
				&& (currentTime - mLastHistoryTime) > kMinHistoryUpdateTime)

which basically says:

if we have no samples at all yet, or
   we have more than 500 samples, or
   we have not updated in 5 seconds or more and
   our last full history update was at least 5 seconds ago:
    get a full history

It seems that you expect a history fetch to never provide more than 500 samples, otherwise every update cycle fetches a full history rather than just grabbing the current status. Am I understanding that correctly?

[colings]

I'm not sure, but I believe Heatermeter returns 200 samples max. If it doesn't have a ton of samples yet it'll return up to that many at a high resolution, then as time goes on they'll get further and further apart. I think the max it returns is a 24 hour span. The Heatermeter source is open, so you could take a look at that and see what he's doing. In general you'll not be hitting the 500 sample limit very often since every time you lock your phone it'll end up pulling the history again when you unlock. It mostly matters if you've got it plugged into a charger just sitting open continuously.

…

On Jan 29, 2018 7:17 AM, "Brian J. Murrell" ***@***.***> wrote: So, just looking at the code a bit here: if ((mSamples.size() == 0 || mSamples.size() > 500 || timeSinceLastUpdate > kMaxUpdateDelta) && (currentTime - mLastHistoryTime) > kMinHistoryUpdateTime) which basically says: if we have no samples at all yet, or we have more than 500 samples, or we have not updated in 5 seconds or more and our last full history update was at least 5 seconds ago: get a full history It seems that you expect a history fetch to never provide more than 500 samples, otherwise every update cycle fetches a full history rather than just grabbing the current status. Am I understanding that correctly? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAhK1LVJ0MqY5n33PRjss_jc9opdvLPzks5tPcTWgaJpZM4Rwf-4> .

[brianjmurrell]

Wow. 200 samples seems pretty low. At per-second sampling that means losing precision in just over 3 minutes. I can surely understand the lack of need for high precision over many hours, but to lose it so quickly...

In any case, I've limited my controller to spitting out 500 entries max (and I think 250 at minimum) as that's the limit your code wants before it starts requesting full histories (and as such any return of history > 500 entries would just mean a pull from history again instead of switching back to sampling the current status every second).

I'm currently just working on downsampling my stored data once it gets to the point of not being able to average down to the 250-500 entry range within the 5 second timeout your code wants (which is a perfectly reasonable timeout). Something along the lines of reducing the source data to 10% of it's size every time it grows by 10x what it was when it was downsampled the last time. I.e. a bucket of data at 1s samples and once that reaches 5000 entries (the limit of how much data I can reduce to a max of 500 entries in less than 5 seconds by computation), downsample it to 10s samples and once that reaches 5000 again, downsample it to 100s samples, etc.

I know that this is what RRD does but the platform I'm on doesn't have an RRD implementation (that I know of) so I have to hack up a poor man's version of it.

As an aside, it would be interesting to make the history size (mSamples.size() > 500) configurable in the app to see how it affects performance and visuals when pulling from a source that will feed more than 200 entries before dropping precision.

[brianjmurrell]

So given that heatermeter only gives you up 200 points of data from history, and your app continues to pull data from hmstatus every second until it hits 500 total samples (200 at whatever resolution is necessary to get the entire history of the cook into 200 data points and 300 more high resolution data points), the graph only ever contains between 200 and 500 points of data, is that correct? If so, that seems to be pretty few for an average phone display that could typically be 1000 or more pixels wide, even in portrait orientation.

[colings]

For the data it's visualizing it's actually plenty of points in my experience. You're really only looking at an overall trend in the graph, and even spikes like lid open temp drops have enough resolution IMO. Also, the more points you plot the worse your performance gets when interacting with the graph (zooming, scrolling, etc). You're welcome to experiment though.

…

On Tue, Jan 30, 2018 at 9:10 AM, Brian J. Murrell ***@***.***> wrote: So given that heatermeter only gives you up 200 points of data from history, and your app continues to pull data from hmstatus every second until it hits 500 total samples (200 at whatever resolution is necessary to get the entire history of the cook into 200 data points and 300 more high resolution data points), the graph only ever contains between 200 and 500 points of data, is that correct? If so, that seems to be pretty few for an average phone display that could typically be 1000 or more pixels wide, even in portrait orientation. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAhK1HB4y0iaJjpMa4WPJLgif81D-z8Vks5tPzDKgaJpZM4Rwf-4> .