Backfill config gen, and as-of date changes #286

natemcintosh · 2025-05-20T19:10:08Z

Nate's Summary

This aims to do issues:

It became clear that we needed to pass the job ids created during config generation to the job creation script. The options I could think of were:

Have config generation spit out the job ids, parse them with bash, and then loop over job.py for each job id.
Put a modified copy of job.py in the same script as the backfill config generation.

Option 2 felt much safer. Have tested this in Batch, and things run!

Example model output

You can see output from this in the nssp-rt-testing storage container, in the job ids starting with your-backfill-name....

Copilot's Summary

This pull request introduces a new feature for generating and running backfill configurations, along with various supporting changes. The most significant updates include adding a new script for backfill configuration, updating the Makefile to include a command for running this script, and refining documentation and parameter handling in related files.

New Feature: Backfill Configuration and Execution

New script for backfill configuration: Added azure/backfill_generate_and_run.py to generate and upload configuration files for the EpiNow2 pipeline, and optionally execute tasks in Azure Batch. This includes detailed parameter handling, validation, and Azure integration.
Makefile update: Added a backfill-config-and-run command to the Makefile to streamline running the new backfill script. This provides an easy way to configure and execute backfill tasks.
Documentation update: Added an entry in NEWS.md to announce the new commands for generating backfill configurations.

Improvements to Parameter Handling

Refinement of as_of_date parameter: Updated the as_of_date parameter description in R/config.R for better clarity and alignment with its usage.
generate_configs.py adjustment: Modified the as_of_date calculation to use the report date directly, ensuring consistency and simplifying logic. [1] [2]

github-actions · 2025-05-20T19:11:59Z

Thank you for your contribution @micahwiesner67 🚀! Your pkgdown-site is ready for download 👉 here 👈!
_{(The artifact expires on 2025-06-03T17:14:08Z. You can re-generate it by re-running the workflow here.)}

codecov · 2025-05-20T19:12:28Z

Codecov Report

All modified and coverable lines are covered by tests ✅

📢 Thoughts on this report? Let us know!

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

athowes · 2025-05-21T09:40:37Z

Makefile

@@ -42,6 +42,21 @@ rerun-config:
 		--job-id=$(JOB) \
 		--report-date-str=$(REPORT_DATE)

+backfill-config:


How possible is it to expose arguments here for them to be input via the target backfill-config?

Perhaps this is already working using Typer?

Yes, passing in different arguments in a nice way is something I haven't entirely figured out yet. My best thought so far is to have the user run the help message to read about the arguments, and then alter them in this command to their needs.

So the user can pass the arguments in, it's just that they don't have help on figuring out what the arguments are (unless they read the help message)?

If so, interesting problem. Wonder if there is a better solution.

azure/generate_backfill_configs.py

Co-authored-by: Adam Howes <adamthowes@gmail.com>

I decided to do this as it became clear that we needed to pass the job ids created during config generation to the job creation script. Either that happens with bash, or like this with python, which feels much safer. Have tested this in Batch, and things run!

…tes'

zsusswein

Is it worth putting the duplicated code here in a function and importing it? There seems to be quite a bit of logic here shared with existing stuff and it could be annoying to maintain.

natemcintosh · 2025-05-22T15:28:31Z

Is it worth putting the duplicated code here in a function and importing it? There seems to be quite a bit of logic here shared with existing stuff and it could be annoying to maintain.

I was waiting for someone to say this 😆. At what point do we make the azure/ folder its own little python package? Part of the problem with putting the common functionality is that it requires various azure packages. Right now, everything in the azure/ folder is a uv script, each of which creates its own environment. If we wanted to import functionality that uses azure packages, we'd probably have to create a small package.

kgostic

Approving so as not to hold you up, but I have some comments.

Most importantly, could you put your head together with @PatrickTCorbett to talk about how people are supposed to set up a run, and add some documentation?

azure/backfill_generate_and_run.py

kgostic · 2025-05-22T17:01:22Z

Makefile

+	uv run --env-file .env azure/backfill_generate_and_run.py \
+		--state=all \
+		--disease="COVID-19,Influenza" \
+		--str-report-dates="2025-04-30,2025-05-07,2025-05-14" \


I like this as an example, but I don't think this is going to be the full range of default dates. And I don't think that people can pass these args easily into the makefile without them being defined as makefile vars?

If, (e.g.) we want to do a backfill run for every weds in the 2024-2025 season, or change any of these other options, what's the recommended workflow? Modify the makefile locally and then call make backfill-config-and-run?

Whatever is recommended, I think it would be good to add an example call (e.g. to the SOP.md)?

Do we have an official start and end date for the 2024-2025 season. I will add an example showing how to easily generate those dates.

Has this been resolved? Otherwise, looks good to me

Not yet. Still need to add that little bit more documentation, and then I think it'll be ready to go.

Co-authored-by: Katie Gostic <uep6@cdc.gov>

for more information, see https://pre-commit.ci

zsusswein · 2025-05-23T15:12:31Z

At what point do we make the azure/ folder its own little python package?

Definitely out of scope for this PR, but what about sticking it in the STF package you're using for pool setup? Probably way more trouble than it's worth because it's a different format than what they do...........

I do think this is maybe worth doing though. This feel worth hashing out in an issue?

natemcintosh added 6 commits May 16, 2025 13:16

use report date as the as of date

9e108ce

after review, don't need datetime

58a6230

better docstring explanation

fbb8e88

start of backfill gen script

4d0d89b

start of thinking about possible inptu options

35a674f

backfill config gen script complete

0cb4c2f

natemcintosh temporarily deployed to production May 20, 2025 19:12 — with GitHub Actions Inactive

natemcintosh temporarily deployed to production May 20, 2025 19:13 — with GitHub Actions Inactive

natemcintosh added 3 commits May 20, 2025 17:37

backfill config gen now working

7188cfa

news update

a590b47

Merge branch 'main' into nam-config-gen-updates

5ab60e8

natemcintosh temporarily deployed to production May 20, 2025 21:42 — with GitHub Actions Inactive

natemcintosh temporarily deployed to production May 20, 2025 21:45 — with GitHub Actions Inactive

athowes reviewed May 21, 2025

View reviewed changes

Update azure/generate_backfill_configs.py

5c7274b

Co-authored-by: Adam Howes <adamthowes@gmail.com>

natemcintosh temporarily deployed to production May 21, 2025 13:20 — with GitHub Actions Inactive

natemcintosh added 3 commits May 21, 2025 15:38

start of job generation in this script

645d6de

Merge branch 'main' into nam-config-gen-updates

dbc531b

natemcintosh marked this pull request as ready for review May 21, 2025 21:14

natemcintosh temporarily deployed to production May 21, 2025 21:16 — with GitHub Actions Inactive

natemcintosh temporarily deployed to production May 21, 2025 21:17 — with GitHub Actions Inactive

natemcintosh added 2 commits May 21, 2025 17:18

docs got out of phase somehow

668d76b

Merge remote-tracking branch 'refs/remotes/origin/nam-config-gen-upda…

05cf989

…tes'

natemcintosh temporarily deployed to production May 21, 2025 21:21 — with GitHub Actions Inactive

natemcintosh temporarily deployed to production May 21, 2025 21:22 — with GitHub Actions Inactive

natemcintosh requested a review from kgostic May 21, 2025 21:25

natemcintosh requested review from zsusswein and micahwiesner67 May 21, 2025 21:25

zsusswein reviewed May 22, 2025

View reviewed changes

kgostic approved these changes May 22, 2025

View reviewed changes

natemcintosh and others added 3 commits May 22, 2025 14:30

Apply suggestions from code review

aee4872

Co-authored-by: Katie Gostic <uep6@cdc.gov>

Merge branch 'main' into nam-config-gen-updates

467225f

[pre-commit.ci] auto fixes from pre-commit.com hooks

4fee95b

for more information, see https://pre-commit.ci

pre-commit-ci bot temporarily deployed to production May 22, 2025 18:58 Inactive

pre-commit-ci bot temporarily deployed to production May 22, 2025 18:59 Inactive

natemcintosh added 2 commits May 23, 2025 08:25

more info on state abbr and string splitting

bc54f8a

use mutli-line strings

1ec19b7

natemcintosh temporarily deployed to production May 23, 2025 12:27 — with GitHub Actions Inactive

natemcintosh temporarily deployed to production May 23, 2025 12:28 — with GitHub Actions Inactive

micahwiesner67 approved these changes May 27, 2025

View reviewed changes

Merge branch 'main' into nam-config-gen-updates

0262b8c

micahwiesner67 temporarily deployed to production May 27, 2025 17:14 — with GitHub Actions Inactive

micahwiesner67 temporarily deployed to production May 27, 2025 17:15 — with GitHub Actions Inactive

Backfill config gen, and as-of date changes #286

Are you sure you want to change the base?

Backfill config gen, and as-of date changes #286

Uh oh!

Conversation

natemcintosh commented May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Nate's Summary

Example model output

Copilot's Summary

New Feature: Backfill Configuration and Execution

Improvements to Parameter Handling

Uh oh!

github-actions bot commented May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented May 20, 2025

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zsusswein left a comment

Choose a reason for hiding this comment

Uh oh!

natemcintosh commented May 22, 2025

Uh oh!

kgostic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zsusswein commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

natemcintosh commented May 20, 2025 •

edited

Loading

github-actions bot commented May 20, 2025 •

edited

Loading

zsusswein commented May 23, 2025 •

edited

Loading