Simplify rtstats tracking of NEXRAD2 topologies

[akrherz]

Here's a report for the number of 'data points' in the database for the past hour, grouped by feedtype

 select 
(select feedtype from ldm_feedtypes where id = p.feedtype_id) as feedtype, count(*) 
from ldm_rtstats r JOIN ldm_feedtype_paths p on (r.feedtype_path_id = p.id) 
where r.entry_added > '2016-10-07 12:17+00' GROUP by feedtype ORDER by count DESC;
  feedtype  | count  
------------+--------
 NEXRAD2    | 633109
 IDS|DDPLUS |  43039
 HDS        |  25728
 EXP        |  21411
 NEXRAD3    |  21125
 NGRID      |  12801
 NOTHER     |  12642
 NIMAGE     |   8160
 LIGHTNING  |   6935
 FNEXRAD    |   4070
 CONDUIT    |   3143
 FSL2       |   2531
 UNIWISC    |   2397
 NNEXRAD    |   1868
 WMO        |   1252
 FSL4       |   1171
 WSI        |    901
 FSL5       |    651
 FSL3       |    543
 PCWS       |    352
 SPARE      |    242
 NGRAPH     |    176
 GPSSRC     |    112
 NLDN       |    108
 NTEXT      |     63
 GPS        |     29
 PPS        |     15
(27 rows)

So NEXRAD2 dominates the number of entries! This is because each RADAR site shows up as an LDM product originator and so the ldm_feedtype_paths explodes for each host feeding data. For instance, my NEXRAD2 only host has 15,523 entries for this one hour period. This creates unmitigated plotting disasters, like currently here. My initial thought is to aggregate these results up for downstream hosts of the top tier CRAFT sites. So while the top tier sites would retain these detailed stats, the downstream hosts would only see the top tier CRAFT site as the product origin. Perhaps this is not a big enough issue to fuss with? The reason I am currently down this path is that one of my JSON webservices is currently emitting 18 MB response for a test host.

[akrherz]

Reviewing, I think the aggregation is doing what I hoped. The inbound stats are not touched, but the computed / aggregated hourly are. Will leave open for now

[akrherz]

Steve Emerson expressed concern to me that they don't want the level II feed paths aggregated. I'll need to review / think some more about this.

[mzuranski]

I think I found the problem. As you are forming a feedtype path you are using get_ldm_feedtype_path_id to aggregate the stats. Problem is, given the nature of NEXRAD2 origins those paths wouldn't exist; they would have to have been created by some other non-NEXRAD2 report, which of course would never happen. The result is only those origins that are also relays would have their stats aggregated, all others would have that id be returned as null and the path would not be added.

I'm currently testing a potential fix by replacing get_ldm_feedtype_path_id with get_or_set_ldm_feedtype_path_id and while I can say that's having some effect, I need to let it run longer before I can see how much I trust it. Will keep you posted.