Verify native library integrity during extraction/load

[samfreund]

Description

We've hit a problem where the CombinedRuntimeLoader extracts native files, but gets interrupted in the middle. This is bad, cause all CombinedRuntimeLoader used to check a file was its existence. This change uses the hash of the file to verify that it's correct. This will be checked at runtime, everytime, if the file is extant. If this check fails, we will delete the extant file and attempt to reextract. We also check a newly extracted file, if that hash does not match we error.

Note that this is reliant on PhotonVision/wpilib-tool-plugin#8 and should follow #2367

Meta

Merge checklist:

• Pull Request title is short, imperative summary of proposed changes
• The description documents the what and why, including events that led to this PR
• If this PR changes behavior or adds a feature, user documentation is updated
• If this PR touches photon-serde, all messages have been regenerated and hashes have not changed unexpectedly
• If this PR touches configuration, this is backwards compatible with all settings going back to the previous seasons's last release (seasons end after champs ends)
• If this PR touches pipeline settings or anything related to data exchange, the frontend typing is updated
• If this PR addresses a bug, a regression test for it is added

[samfreund]

Testing locally seems to be succesful. I can corrupt a file, and it'll be replaced when files are re-extracted.

[mcm001]

time find ~/.wpilib/ | xargs md5sum takes 0.06 seconds at least. Extraction and verification currently takes 0.6 seconds after they've already been extracted to disk once. I can also see that we re-hash .wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libphotontargetingJNI.so 4 times (??!?!?), so let's figure out why and make sure it is only extracted once.

One run from my laptop: (units in milliseconds)

Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libphotontargetingJNI.so took 20
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libwpinetjni.so took 1
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libntcore.so took 5
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libopencv_calib3d.so.4.10 took 6
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libwpiHaljni.so took 1
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libphotontargeting.so took 8
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libwpiutiljni.so took 1
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libopencv_stitching.so.4.10 took 1
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libopencv_java4100.so took 3
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libopencv_video.so.4.10 took 0
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libwpiutil.so took 7
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libopencv_photo.so.4.10 took 1
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libopencv_imgproc.so.4.10 took 14
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libcscorejni.so took 0
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libopencv_ml.so.4.10 took 1
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libwpiHal.so took 2
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libmrcal_jni.so took 7
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libapriltagjni.so took 0
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libwpimathjni.so took 2
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libwpinet.so took 5
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libopencv_objdetect.so.4.10 took 2
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libopencv_core.so.4.10 took 8
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libcscore.so took 1
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libopencv_features2d.so.4.10 took 2
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libwpimath.so took 4
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libopencv_flann.so.4.10 took 1
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libopencv_highgui.so.4.10 took 0
Verify /home/matt/.wpilib/nativecache/linux/x86-64/29afd81adb712c5998955e0a8f8c0d0f/libapriltag.so took 1

[samfreund]

On an rpi3b, time find | xargs md5sum results in:

real	0m0.468s
user	0m0.421s
sys	0m0.045s

This was run on an extraction of the latest jar's native libraries.

[mcm001]

The bug here is that every call to CombinedRuntimeLoader::loadLibraries calls extractLibraries. We should keep track of this and skip if we've already done it

[mcm001]

Looks like on my Ubuntu laptop, main loads native libraries in 0.18 seconds, and this branch is still sitting at 0.28 seconds. Half of that time is from extraction and hashing, and half of that time is from setup (before the actual loop over libraries). We can't speed that first bit up on a cold JVM startup really, so the question is if the extra 100ms (on this machine) we spend could be made any faster

[mcm001]

No more juice left to squeeze on Linux x64 at least