Triggering processing workflow after new photo upload method

[FeezyHendrix]

What type of PR is this? (check all applicable)

• 🍕 Feature
• 🐛 Bug Fix
• 📝 Documentation
• 🧑‍💻 Refactor
• ✅ Test
• 🤖 Build or CI
• ❓ Other (please specify)

Related Issue

Implements the batch processing step (Step 4) of the drone image processing workflow.

Describe this PR

This PR adds the final processing step to the drone image workflow, enabling batch processing of classified and reviewed images.

Key Features:

Batch Processing (Step 4)

• Summary view showing tasks and image counts ready for ODM processing
• Background job to move images from staging (user-uploads/) to task folders ({task_id}/images/)
• Triggers ODM processing jobs for each task with assigned images
• Real-time status updates during processing

Backend Changes:

• move_batch_images_to_tasks() in ImageClassifier - copies images from staging to task folders in S3
• get_batch_processing_summary() - returns task-grouped summary for the UI
• New API endpoints:
  • GET /api/projects/{id}/batch/{batch_id}/processing-summary/ - Get processing summary
  • POST /api/projects/{id}/batch/{batch_id}/process/ - Start batch processing
• process_batch_images ARQ background task that moves images and triggers ODM jobs

Frontend Changes:

• ImageProcessing.tsx component with task list and processing controls
• Service functions: getProcessingSummary(), startBatchProcessing()
• Processing status display with job tracking

Screenshots

Review Guide

1. Navigate to Step 4 (Processing) in the drone image workflow
2. Verify the task list shows correct image counts per task
3. Click "Start Processing" to trigger the batch job
4. Check backend logs to confirm images are moved to task folders and ODM jobs are queued

API Endpoints to test:

• GET /api/projects/{id}/batch/{batch_id}/processing-summary/
• POST /api/projects/{id}/batch/{batch_id}/process/

Checklist before requesting a review

• 📖 Read the HOT Code of Conduct: https://docs.hotosm.org/code-of-conduct
• 👷‍♀️ Create small PRs. In most cases, this will be possible.
• ✅ Provide tests for your changes.
• 📝 Use descriptive commit messages.
• 📗 Update any related documentation and include any relevant screenshots.

[spwoodcock]

This is a good improvement I think!
I actually quite like that the user can trigger the processing of all the task areas they flew in one go.

Comments / user story:

• Create a project in DroneTM.
• Divide the area into tasks.
• Multiple operators go out and assign themselves tasks for collecting imagery.
• Users go to the field, fly perhaps 4-12 task areas each, then return to base.
• In the evening, they upload their task area imagery (4-12 tasks worth of imagery in one go if that want, via the new uploader approach).
• They click on a task area
• [where the workflow diverges]
  • Existing worflow:
    • The user was uploading imagery for a specific task area.
    • This made it obvious to them when to mark the task as 'complete'.
    • Once marked 'complete', they started processing the intermediate products for the task area - START PROCESSING.
  • New workflow
    • The user uploads all of their photos for all tasks.
    • Now they need to them click on each task individually and mark it 'complete'.
    • [bonus] It would be great if we could do an automated check here to determine the task area coverage from the imagery provided. If the user uploaded imagery that covers the entire area, then can we mark it as complete automatically? Perhaps we already do this to determine if the task is ready to process - I forgot - please clarify 🙏
    • [question 1]: if processing fails, or the pipeline misses a task to process, can the user still trigger the task-level processing on the task page? Using the previous functionality?
    • [question 2]: do we still utilise the task statuses here? Once all imagery is uploaded, the task status should change to 'complete. This could be automated as mentioned above, but as part of the new workflow this step appears to be skipped? Ideally we only trigger processing for task areas that are marked as status='complete'.

[question 3]: this triggers the processing of the intermediate products for task areas. These 'fast orthophotos' are simply used for visual inspection of the task area, to assess the imagery looks good enough (coverage, quality, etc). Once all of the task areas are processed, how do we go about triggering the full 'final orthomosaic' processing for the project?

p.s.

How come every PR has pre-commit issues & merge conflicts? 😆

Nothing should have changed on dev since your last work, so I think you may accidentally be pulling in commits from other PRs or something?

[spwoodcock]

Can we have some tests on https://dev.dronetm.org of this workflow too? 🙏

[spwoodcock]

Also, for users that expect imagery to be uploaded on the task page, we should probably add a prompt to say 'To upload imagery, please go to the main [project page](link)'

[spwoodcock]

TODO (copied from Slack):

• Ensure visual inspection for each tasks in the review, by showing the images on the map.
• Ensure percentage of coverage is shown and flagged to the user, when the image coverage is less than 100%
• Remove the upload image area on the Task Details page (and redirect the user to the Project Details page via a helpful prompt).
• Ensure the task status is not updated to IMAGE_UPLOADED when a set of imagery is uploaded, but instead from the user manually verifying all the imagery is there with 'Mark Fully Flown' button.
• Fix the 500 error on dev server for project creation (I think it was introduced by latest changed).
• On image upload, it shouldn't be possible to add duplicates as new images (I think if images are already classified then we don't compare the hash of the newly uploaded images against them? If the user uploaded imagery, they are classified, but not enough coverage for processing, then the user will upload more images - these shouldn't clash / duplicate).
• Ensure code is cleaned up & that all replaced endpoints are removed (to avoid confusion).

[FeezyHendrix]

Sam, The above comments has been looked into.

[spwoodcock]

Thanks! Looking forward to seeing the changes 😁

The branch has merge conflicts though & pre-commit issues not solved.

I'll wait until they are solved before I review 👍

[spwoodcock]

Installing pre-commit hooks are useful, as they help catch simple issues like this 😄

F821 Undefined name `log`
   --> src/backend/app/tasks/task_logic.py:592:13
    |
590 |             #
591 |             # Keeping this case to avoid breaking existing API calls, but it now does nothing.
592 |             log.warning(
    |             ^^^
593 |                 f"Deprecated IMAGE_UPLOAD event received for project {project_id}, "
594 |                 f"task {task_id}. This event no longer updates task status."
    |

Found 1 error.

[spwoodcock]

Also

error: Failed to parse `uv.lock`
  Caused by: Dependency `opencv-python-headless` has missing `source` field but has more than one matching package

was the uv lock updated using uv, or using another tool?

[spwoodcock]

I fixed the uv lock file issues and pre-commit issue 👍

Could you possibly provide a video of the latest functionality in action before we sign off? 🙏

[FeezyHendrix]

@spwoodcock Thanks! I would provide the video to you by tomorrow.