feat: add --no-push flag for local-only writes (overlay semantics)#91
Closed
feat: add --no-push flag for local-only writes (overlay semantics)#91
Conversation
Allow writes to be stored locally in the cache/staging directory without ever uploading to remote storage. Reads check local files first, then fall back to remote content (overlay semantics). This is a simpler alternative to a full overlayfs-style upperdir implementation: instead of adding a parallel write path, we reuse the existing advanced_writes staging infrastructure and simply skip creating the FlushManager, so dirty files stay local indefinitely. Useful for caching expensive-to-produce artifacts (e.g. compilation caches) on top of a shared read-only remote bucket.
Print the exact staging directory path so the user knows where their local writes are stored on disk.
POSIX Compliance (pjdfstest) |
Benchmark Results |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--no-pushCLI flag: allows local writes without ever uploading to remoteadvanced_writesstaging infrastructure (no new write path)How it works
--no-pushsetsread_only = falseandadvanced_writes = true, but skips creating theFlushManager. The result:--cache-dir/staging/)FlushManagerto enqueue to)This achieves overlay semantics in ~30 lines instead of ~200 (compared to dacorvo's overlay-write-dir which implements a parallel
--upperdirwrite path).Use case
Mounting a shared remote bucket (e.g. AWS Neuron compilation cache) and writing local artifacts on top without pushing back. See Slack thread.
Notes
--cache-dir/staging/. The path is logged at mount time.