DMP-4312: Dets Clean up automated Task#3216
Conversation
Ben-Edwards-cgi
requested review from
cakeben,
karen-hedges and
npv0
and removed request for
a team
# Conflicts: # src/integrationTest/java/uk/gov/hmcts/darts/testutils/stubs/DartsDatabaseStub.java # src/main/java/uk/gov/hmcts/darts/task/api/AutomatedTaskName.java
hmcts-jenkins-cnp
bot
added
ns:darts-modernisation
prd:darts
rel:darts-api-pr-3216
labels
| } | ||
|
|
||
| List<List<CleanUpDetsDataProcessorImpl.CleanUpDetsProcedureResponse>> batchesToDeleteBlobStoreRecordFor = | ||
| ListUtils.partition(eodIdsToCleanUp, config.getChunkSize() / config.getThreads()); |
There was a problem hiding this comment.
This calculation "outconfig.getChunkSize() / config.getThreads()" should not be passed as a parameter to ensure there are no issues with the value returned
| RAISE NOTICE '% Duplicate count = %', c_msg_prefix, v_dup_count; | ||
|
|
||
| IF v_dup_count > 0 THEN | ||
| RAISE EXCEPTION 'Validation failed: Duplicate records [%] found.', v_dup_count; |
There was a problem hiding this comment.
This will stop the whole process, should this stop the whole process or skip the duplicates?
| log.info("Processing clean up of DETS data with chunk size: {}", chunkSize); | ||
|
|
||
| List<CleanUpDetsDataProcessorImpl.CleanUpDetsProcedureResponse> eodIdsToCleanUp = | ||
| cleanUpDetsDataBatchProcessor.callDetsCleanUpStoredProcedure(chunkSize, minimumStoredAge); |
There was a problem hiding this comment.
If the callDetsCleanUpStoredProcedure throws an exception say because there are duplicate eods, then the whole job finishes. Should there be a try catch inside the while loop so the next chunk can be processed?
|
|
||
| for (CleanUpDetsProcedureResponse response : eodsCleanedUp) { | ||
| try { | ||
| log.debug("Processing clean up response for EOD ID: {}, Location: {}", response.getOsrUuid(), response.getDetsLocation()); |
There was a problem hiding this comment.
We have the functionality for feature flag logging, can you please change these debug messages to be feature flag controlled so we can turn them on and off when required. I will find an example of how this is currently done
There was a problem hiding this comment.
migrations need to be updated as we have had various other flyway migration merged
There was a problem hiding this comment.
migrations need to be updated as we have had various other flyway migration merged
| @@ -0,0 +1,10 @@ | |||
| INSERT INTO darts.automated_task (aut_id, task_name, task_description, cron_expression, cron_editable, batch_size, | |||
| created_ts, created_by, last_modified_ts, last_modified_by, task_enabled) | |||
| VALUES (36, 'CleanUpDetsData', 'Cleans up Dets files that have successfully been stored in ARM', '0 24 0-6,19-23 ? * *', true, 100_000, | |||
There was a problem hiding this comment.
this should not use a fixed value 36 for id but should use nextval('aut_seq') as we have added more automated tasks
2 participants
JIRA link (if applicable)
https://tools.hmcts.net/jira/browse/DMP-4312
Change description
Created the DETs Clean Up Job
Does this PR introduce a breaking change? (check one with "x")