Parallelize fstatat for faster directory listing #26
Conversation
- Refactor `nativeFillBuffer` to use a 3-phase approach: Serial Gather -> Parallel Stat -> Serial Sort. - Use `pthread` to parallelize `fstatat` calls, significantly reducing latency for large directories. - Add robust error handling for thread creation and fix a potential memory leak in the output loop. - Preserve existing "Zero Garbage" philosophy by using efficient memory management. Co-authored-by: mewmix <42463809+mewmix@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
…ow, larger local buffer; add timing logs for list phases
… base_index/prefix) perf(search): reuse per-worker getdents64 buffer; lower min threads to 2 to reduce I/O thrash perf(sort): add NEON ASCII strcasecmp behind GLAIVE_EXPERIMENTAL_FASTSORT flag chore: keep native timing logs for list phases
1 participant
This change optimizes the directory listing performance in
glaive_core.cby parallelizing thefstatatcalls. Previously,nativeFillBufferperformedgetdents64andfstatatserially for each file, which caused significant delays in large directories due to I/O latency. The new implementation gathers all filenames first, then uses a thread pool to performfstatatin parallel, and finally sorts and outputs the results. This ensures 'instant feedback' and better responsiveness. Robustness checks forpthread_createand memory leak fixes were also applied.PR created automatically by Jules for task 8531284504720419051 started by @mewmix