Deploy Data Upload Manager (DUM) to production

[ramesh-maddegoda]

Are you sure this is not a new requirement or bug?

Yes

Task Type

Sub-task

💡 Description

Are you sure this is not a new requirement or bug?

Yes

Task Type

Sub-task

💡 Description

Description

Deploy the latest version of the Data Upload Manager (DUM) to MCP Production.

This deployment incorporates the following key updates:

• Server-side deduplication to the DUM ingestion path (checks both staging and archive buckets for existing files and skips duplicates)
• Update terraform to include required tags
• Allow DUM client to use MD5 calculations by setting usedforsecurity=False to hashlib.md5()
• Clearer, color-coded error handling on the client.

Tasks

• Deploy DUM server-side components using Terraform (ECS / Lambda / infrastructure updates)
• Release the latest DUM client package to PyPI
• Verify single-file upload flow from client → API → S3
• Test duplicate file uploads
• Validate S3 metadata tagging and synchronization with the database
• Verify error handling for failed uploads, authentication issues, and invalid requests

Acceptance Criteria

• DUM Server is deployed and reachable via DUM client
• DUM Client is published on PyPI and compatible with the latest features
• Duplicate uploads are handled gracefully without system failures
• S3 metadata remains consistent and correctly synced with the database
• CloudWatch logs are generated and visible for all DUM components

[anilnatha]

Status: Encountered issues with terraform, the remote states are not in sync the way we expected. MCP Test saw 10 changes, but in Prod 39 changes were observed. Only test was deployed and terraform state was fixed. Some of the AWS resources may have been modified manually or with another terraform config. For the moment, DUM in Prod is not deployed as it requires more discussion.