GitHub Statistics

[writetoritu]

We need to compute the following for a given data cycle.

number of edits to conventions doc
number of edits to ETL code by each site
number of issues opened in data model repo
number of issues opened in each site’s repo

where a data cycle is defined in the following way:

• Data cycle 1 = January – Feb 2015
• cycle 2 = march - april 2015
• and so on (as specified here https://github.com/PEDSnet/Data-Quality/blob/master/SecondaryReports/DCC_Data_Cycles.csv)

Would it be possible to write a tool for it?

[bruth]

Yes. Out of curiosity how will this information be used? Does it need to be presented to anyone other than you and @utidjianl? Do you have an expectation of a user interface to for viewing these stats?

[penningtonj]

These are important stats to quantify the operational burden of data model changes. What're the current options for getting these data out?

[writetoritu]

@bruth thanks - We don't need these stats urgently.
We wanted to see the changes in GitHub docs and GH issues in each cycle and would like to view them along with the number of data quality issues per cycle. This will demonstrate the network's progress over various cycles and also highlight if there is any correlation between the use of GitHub for data discussions and the number of data quality issues.

[bruth]

number of edits to conventions doc

The ETL conventions service has all the information needed to compute this. An enhancement needs to be made to push changes detected by the service to a log or database so they can be aggregated over time.

number of edits to ETL code by each site

This can be determined using git data similar to the ETL conventions service. Each site's repository appears to be organized a bit differently. Are there select directories or files that should be considered in determining a code edit? We could match filenames based on extension, e.g. *.sql or *.scala.

The output of this would be a set of changes per site each of which will need to be aggregated over time.

number of issues opened in data model repo

This is virtually a straight API to GH with a bit of filtering. We may need to evaluate the issues for non-data model changes, but that will come later. Again, output is a stream aggregated over time.

number of issues opened in each site’s repo

Same as above.

[writetoritu]

We can keep all abovementioned measures simple.
e.g. number of edits to ETL code by each site ~ any commit to that site (no matter which directory or which file)