feat: add --no-push flag for local-only writes (overlay semantics)#90
Draft
feat: add --no-push flag for local-only writes (overlay semantics)#90
Conversation
Allow writes to be stored locally in the cache/staging directory without ever uploading to remote storage. Reads check local files first, then fall back to remote content (overlay semantics). This is a simpler alternative to a full overlayfs-style upperdir implementation: instead of adding a parallel write path, we reuse the existing advanced_writes staging infrastructure and simply skip creating the FlushManager, so dirty files stay local indefinitely. Useful for caching expensive-to-produce artifacts (e.g. compilation caches) on top of a shared read-only remote bucket.
POSIX Compliance (pjdfstest) |
Benchmark Results |
Print the exact staging directory path so the user knows where their local writes are stored on disk.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--no-pushCLI flag that allows local writes without ever uploading to remote storageadvanced_writesstaging infrastructure instead of implementing a parallel write pathContext
Alternative to dacorvo/hf-mount#overlay-write-dir which implements full overlayfs semantics with a dedicated
--upperdir. That approach adds ~200 lines touching many VFS methods. This approach achieves the same overlay behavior in ~30 lines by reusing the existing staging/dirty-tracking mechanism and simply not creating theFlushManager.How it works
--no-pushforcesadvanced_writes = trueandread_only = falseFlushManageris not created, so dirty files are never uploaded--cache-dir)Use case
Mounting a shared remote bucket (e.g. AWS Neuron compilation cache) and writing local artifacts on top without pushing back. See Slack thread for context.
Limitation
Local writes live in the staging directory under
--cache-dir. For persistence across sessions, the user should pass a stable--cache-dirpath. The staging dir uses inode numbers as filenames, so persistence across different mount sessions of the same source requires the same inode assignment order (which is deterministic for the same remote state).