Need to build in sanity checks and tests

[chrislkeller]

• Sanity checks to ensure updates are being captured accurately and identify any ridiculous escalation in the numbers
  • For example, going from 500,000 registrants to 2.5 million in a day
  • Tests to identify data from duplicate dates or duplicate counties
  • Tests to add up data in subcategories to ensure figures equal a total

[chrislkeller]

Some ideas here from Brent Jones...

I built a slackbot for COVID data (using the NYT data), cases and deaths. I wanted to look at all the counties in our area and aggregate them. I recognized individual counties might throw up outliers but the aggregate might still be useful. So I decided to produce an update every time the NYT data updated (with new numbers in our area), but just flag suspect data and let the reporters decide if it seemed too noisy to use.

So for new cases and deaths, for each county (and also for the aggregate), I’m flagging negative values. Then I’m also calculating the standard deviation for the last month’s worth of data and checking to see if the value falls outside of the mean ± 2 st.dev, flagging if it does. And finally I’m checking for zero values, and flagging that if 0 is outside the mean - 1 standard deviation.

If the data for a county or the region trips one of these flags, I say what the value was as well as what I expected it to be, so folks can see how far off it is. For example: “Value for new cases was 220, higher than I expected. Based on the prior 30 days of data, I expected this number to be between 12 and 218.”