[VL] finalizeS3FileSystem is never called

[JunhyungSong]

Backend

VL (Velox)

Bug description

/// Teardown the AWS SDK C++.
/// Velox users need to manually invoke this before exiting an application.
/// This is because Velox uses a static object to hold the S3 FileSystem
/// instance. AWS C++ SDK library also uses static global objects in its code.
/// The order of static object destruction is not determined by the C++
/// standard.
/// This could lead to a segmentation fault during the program exit.
/// Ref https://github.com/aws/aws-sdk-cpp/issues/1550#issuecomment-1412601061
void finalizeS3FileSystem();

This is written in Velox (RegisterS3FileSystem.h). But, this is never called by Gluten. Also, shouldn't this be called when there are too many S3FileSystem created in the cache? Looks like there is no cache eviction mechanism for S3FileSystem which can be technically memory & resource leak since S3FileSystem contains S3Client.

Gluten version

No response

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

[FelixYBW]

The clean up should be done in velox after one file is scaned. There is a filehandlercache but we set it off in Gluten.

@rui-mo doesn't velox do this?