Data uploads to private repo but policyengine-uk downloads from public repo

[MaxGhenis]

Problem

The CI workflow uploads data to policyengine/policyengine-uk-data-private (Hugging Face), but policyengine-uk downloads from the public repo policyengine/policyengine-uk-data.

This means that when new data is built and uploaded via CI (e.g., PR #220 adding salary sacrifice imputation), it goes to the private repo but isn't accessible to users of policyengine-uk.

Details

Upload script (upload_completed_datasets.py:16-18):

upload_data_files(
    files=dataset_files,
    hf_repo_name="policyengine/policyengine-uk-data-private",  # <-- PRIVATE
    ...
)

Download source in policyengine-uk (policyengine-uk/simulation.py:109):

self.build_from_url(
    "hf://policyengine/policyengine-uk-data/enhanced_frs_2023_24.h5"  # <-- PUBLIC (no -private suffix)
)

Impact

• PR Add salary sacrifice imputation to dataset pipeline #220 (salary sacrifice imputation) was merged and CI ran successfully
• Data was built and uploaded to the private repo (100% upload completed in logs)
• But Microsimulation() downloads from the public repo
• Users don't get the updated data

Verification

After clearing the HF cache and re-downloading, the data is unchanged:

• SS contributors: 1.22m
• Total SS: £6.91bn

This confirms the public repo wasn't updated.

Suggested Fix

Either:

1. Change upload_completed_datasets.py to upload to policyengine/policyengine-uk-data (the public repo that policyengine-uk downloads from)
2. Or change policyengine-uk to download from policyengine/policyengine-uk-data-private (would require HF token authentication)

[MaxGhenis]

This issue was based on a misunderstanding. Upon further investigation:

1. Upload DID succeed: The CI workflow completed successfully, uploading 271MB (100%) to HuggingFace
2. Data is synced: Both policyengine-uk-data-private and policyengine-uk-data repos have identical files (same SHA)
3. Salary sacrifice imputation worked: The new pension_contributions_via_salary_sacrifice column exists in the dataset with meaningful values:
   • Total SS contributions: £6.79bn
   • 1.20m contributors (1.8% participation rate)
   • Mean contribution: £4,727

The initial confusion arose because:

• The policyengine-uk simulation package doesn't have a formula for pension_contributions_via_salary_sacrifice yet
• But the raw data IS present in the HDF5 file and can be accessed directly

Closing as resolved - the upload infrastructure is working correctly.

[MaxGhenis]

Correction to closing comment

Further investigation shows the data pipeline is working correctly:

What actually happened with PR #220:

Metric	v1.21.0 (before)	v1.22.0 (after)	Change
Total SS	£3.52bn	£6.79bn	+93%
Contributors	0.58m	1.20m	+107%
Revenue impact	£0.59bn	£1.11bn	+88%

Key findings:

1. Both HF repos (-private and public) have identical data (same SHA: 6280f14f...)
2. The upload to private is intentional due to UK Data Service restrictions
3. There appears to be an automated sync from private → public
4. PR Add salary sacrifice imputation to dataset pipeline #220's salary sacrifice imputation DID take effect and IS reflected in the published data

The issue was filed based on stale cached data. After clearing the HF cache, the new v1.22.0 data with the imputation is accessible.

[MaxGhenis]

Reopening — sync stopped working

The Nov 2025 investigation found both repos had identical SHAs, but further investigation shows there is no automated sync mechanism — no HuggingFace webhooks, no GitHub Actions workflow, no mirroring script. The repos likely had identical content from a manual sync or coincidental timing.

As of Feb 2026, the public repo (policyengine/policyengine-uk-data) was last updated ~Jan 27, while the private repo has had successful uploads on Feb 16-20. This means columns added since then (like highest_education from b1a567b) are missing from the public dataset, causing KeyError: 'highest_education' in policyengine-uk's student loan uprating code.

Root cause: upload_completed_datasets.py only uploads to -private. There was never a durable public upload.

Fix: PR #280 adds a second upload_files_to_hf() call for the public repo, making the push workflow self-sufficient.