-
Notifications
You must be signed in to change notification settings - Fork 2
Closed
Labels
bugSomething isn't workingSomething isn't workingenhancementNew feature or requestNew feature or request
Description
GenTS currently opens file handles per-task (per time series file), which possibly adds enough latency/overhead to cause significant slowdowns/hangs on parallel filesystems at scale.
One possible solution is to cache file handles per Dask worker so additional reads of the same history file don't require creating a redundant file handle. This would likely need to include some sort of blocking that prevents the total number of file handles from blowing up system limits.
This introduces additional complexity, but if the performance boosts are significant it could be a worthwhile enhancement.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't workingenhancementNew feature or requestNew feature or request