To review and resolve differences between dev and prod outputs, DE and GR need to be able to see overviews and record-level details of the differences.
In general the process for resolving diffs could be something like:
- DE describes diffs
- DE and GR review diffs
- DE resolves diffs (either by flagging them as "acceptable", flagging them as "pending source data fix", or by fixing the dev code)
Sharing/updating the latest state of the differences shouldn't be an entirely manual process. GR already uses the DE QA app and it seems like a great place to facilitate "QA" of the dev CSCL outputs.
current state
We use a CSCL ETL - Issue tracking excel file and Data Discrepancy Tracking folder in SharePoint to describe and track diffs. But docs like these are hard to keep up to date since the state of the diffs are changing when we change code, source data, and prod data.
The scripts to generate diffs in products/cscl/poc_validation/ are only either run during builds or locally ad-hoc.
There's no mechanism for DE to flag diffs as "acceptable" (meaning prod is wrong or the diffs aren't consequential to downstream usage). This makes all diffs appear unresolved, in-need of investigation, or causes by dev code issues.
desired state
DE should be able to:
- generate diffs without re-running the entire build
DE and GR should be able to:
- easily review all diffs between dev and prod outputs
- document explanations for diffs (either by field or by row)
- flag diffs as "acceptable" or "pending source data fix" as a valid way to resolving diffs (either by field or by row)
The QA app should have:
- overviews of diffs
- for each file: total dev and prod row counts, diffs by row, diffs by field
- views of relevant rows in dev and prod files/tables
- mechanism to explain and flag diffs in a way that preserves that documentation
To review and resolve differences between dev and prod outputs, DE and GR need to be able to see overviews and record-level details of the differences.
In general the process for resolving diffs could be something like:
Sharing/updating the latest state of the differences shouldn't be an entirely manual process. GR already uses the DE QA app and it seems like a great place to facilitate "QA" of the dev CSCL outputs.
current state
We use a CSCL ETL - Issue tracking excel file and Data Discrepancy Tracking folder in SharePoint to describe and track diffs. But docs like these are hard to keep up to date since the state of the diffs are changing when we change code, source data, and prod data.
The scripts to generate diffs in products/cscl/poc_validation/ are only either run during builds or locally ad-hoc.
There's no mechanism for DE to flag diffs as "acceptable" (meaning prod is wrong or the diffs aren't consequential to downstream usage). This makes all diffs appear unresolved, in-need of investigation, or causes by dev code issues.
desired state
DE should be able to:
DE and GR should be able to:
The QA app should have: