Skip to content

[🍒6.2][llvm][cas] Prevent corruption on ENOSPC on sparse filesystems #10921

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: swift/release/6.2
Choose a base branch
from
Open

[🍒6.2][llvm][cas] Prevent corruption on ENOSPC on sparse filesystems #10921

wants to merge 1 commit into from

Conversation

benlangmuir
Copy link

  • Explanation: On at least some platform/filesystem combos, if we run out of disk space when writing to mmap'd files, the error is not detected until page flush, which is too late to prevent corrupt data being written to the CAS. To prevent that we will now preallocate disk space (fcntl F_PREALLOCATE on Darwin, posix_fallocate on Linux). This can't handle all possible I/O errors, but it fixes the common one without significantly impacting performance. Note: this requires us to bump the cas format version since we are storing an extra mandatory field.
  • Scope: Affects caching builds.
  • Issues: rdar://152273395
  • Original PRs: [llvm][cas] Prevent corruption on ENOSPC on sparse filesystems #10870
  • Reviewers: @cachemeifyoucan
  • Risk: Low. No impact to non-caching builds. The preallocation APIs should never cause us to lose data. The worst case would be if we discovered a situation where they introduce a spurious error that causes caching builds to fail unnecessarily.
  • Testing: Existing tests passed. Manually tested the out-of-space conditions on apfs and tmpfs, including both allocation within a mapped file going over the disk limit and a new standalone file going over the disk limit, to ensure the ENOSPC is propagated without writing any corrupt data.

@benlangmuir
[llvm][cas] Prevent corruption on ENOSPC on sparse filesystems
fb8f519 
If a platform and filesystem do not detect disk out of space errors during page
fault but instead defer it to page flush, we can corrupt the CAS data without
receiving an error until it is too late. Fix that on platforms that
support preallocating disk space by ensuring all CAS allocations are
allocated to disk before writing. For standalone files, we ensure the
entire file is allocated up front. For bump pointer allocated files
(index, datapool, cache), we allocate in chunks of 1 MB to amortize the
cost, and store the current disk-allocated size into the database file
next to the bump pointer so that every thread and process can check
their allocations.

The overhead from these additional checks is <1% in a store-heavy stress test
on Darwin/APFS, and effectively zero overhead on a non-sparse filesystem
Darwin/tmpfs.

rdar://152273395
(cherry picked from commit cca6e01)
@benlangmuir benlangmuir requested a review from a team as a code owner June 27, 2025 20:52
@benlangmuir
Copy link
Author

@swift-ci please test llvm

@benlangmuir
Copy link
Author

@swift-ci please test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@akyrtzi akyrtzi akyrtzi approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@benlangmuir @akyrtzi