add handling for multiple read/write permissions issues

[mdeshotel]

This PR attempts to resolve a permissions issue with the cache files for historical simulations. For writing cache files the following is implemented:

1. Read data from s3.
2. Write to a local netcdf file with unique hash name.
3. Once the data is fully written to the hash name, copy it to its cache name so that future iterations can access it.

For read conflicts (if they exists), the following logic is applied:

1. Try to read the local netcdf file if it exists.
2. If an error is received, produce warning, wait 1 second, and try again.
3. Do step above for 10 seconds
4. If 10 seconds/attempts are tried and the file is still locked, raise an error.

Additions

Removals

Changes

Testing

1. Run suite in RTE was ran.

Screenshots

Notes

Todos

Checklist

• PR has an informative and human-readable title
• Changes are limited to a single goal (no scope creep)
• Code can be automatically merged (no conflicts)
• Code follows project standards (link if applicable)
• Passes all existing automated tests
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future todos are captured in comments
• Visually tested in supported browsers and devices (see checklist below 👇)
• Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)
• Reviewers requested with the Reviewers tool ➡️

Testing checklist

Target Environment support

• Linux

[mxkpp]

Most comments apply throughout.

Generally also, please write a shared function that can be used multiple times.

[mxkpp]

See 2 comments

[mxkpp]

We may be ready to merge after resolving merge conflicts.

Changes look reasonable.

I am about to run tests for AORC + NWM CONUS, and for NWM oCONUS.

Edit: tests for CONUS and oCONUS ran well