fix(utils.py): add dependency handling wrapper

[MattWellie]

Useful where inter-job dependencies need to be set, but its possible that any job in the chain may not exist. Fairly lengthy example provided in the docstring. This is a shorthand for the use case:

"my task may have produced a job. If it did, I need to set up a depends_on relationship with a collection of prior jobs... it's possible they also didn't create a job."

There are >100 instances of depends_on relationships being set in code all over our organisation, and many/most of these instances require some floppy logic, as we have to adapt to either side of the relationship being None/empty list: https://github.com/search?q=org%3Apopulationgenomics%20%22depends_on(%22&type=code

When incorporated into a chain of jobs, we're continually given multiple decisions:

• if the current task didn't create a Job, do nothing
• if the current task did create a Job, but there are no prior jobs, set no dependencies but append to a list (for future jobs to dependency-set against)
• if the current task created a job, and there were previous jobs, set a depends_on relationship, and append to a list

This method aims to cleanly cater to all these cases, skipping or dependency setting as required.

A boolean flag determines whether the final action is to extend the 'tail' (previous job list) or not. A tail cannot be extended if it is a single Job, or a None value.

---

I've added a range of demonstrative test cases, and a chunk of docstring. As a further demonstration/utilisation I've also implemented this in stage.py. This code block now changes from:

for output_job in outputs.jobs:
    if output_job:
        for input_job in inputs.get_jobs(target):
            assert input_job, f'Input dependency job for stage: {self}, target: {target}'
            output_job.depends_on(input_job)

to

dependency_handler(target=outputs.jobs, tail=inputs.get_jobs(target), append_or_extend=False)

This provides a good example of how floppy this loop was previously, vs. how tersely it can now be written.

---

This workflow change has been deployed and confirmed working with https://batch.hail.populationgenomics.org.au/batches/1124257

This PR is loosely paired with the stacked PR chain populationgenomics/test_workflows_shared#19, populationgenomics/test_workflows_shared#20, populationgenomics/test_workflows_shared#21

[cpg-software-ci-bot]

📊 SonarQube Summary

Metric	This PR	Main Branch
✅ Coverage	77.1%	76.9%
💨 Code Smells	47	47
🐞 Bugs	0	0
🔐 Vulnerabilities	0	0
🚨 Security Hotspots	1	0
📝 New Issues	0	0
🌟 Quality Gate	✅ OK	✅ OK

🔗 View Main Branch Report
🔗 View PR Report

[github-actions]

🐳 Docker Image Built

A new Docker image has been built for this PR:

Image: australia-southeast1-docker.pkg.dev/cpg-common/images-dev/cpg_flow:cd87c5f4971ab1b82c55eb555de5bd9060c070f8

Pull command:

docker pull australia-southeast1-docker.pkg.dev/cpg-common/images-dev/cpg_flow:cd87c5f4971ab1b82c55eb555de5bd9060c070f8

🔗 View in Google Cloud Console

---

This comment was automatically generated by the Docker workflow.

[MattWellie]

The commit immediately prior to this comment splits the logic into two methods

1. set dependencies
2. decide whether to append jobs to the tail

I don't know if this is better as two separate methods, but Sonar was complaining that there were too many conditional switches in the logic. This split satisfies Sonar, and results in extra test cases focused only on that functionality.

[rameshka]

Hey @MattWellie, the new changes look great—thanks for introducing the reusable utility method. I’ve left a few comments for you to review. Also, appreciate the detailed PR description and the additional notes on the logic; they make it easy to follow.

[MattWellie]

Testing the latest state of this branch here https://batch.hail.populationgenomics.org.au/batches/1124653/jobs/1

and here https://batch.hail.populationgenomics.org.au/batches/1124660