-
-
Notifications
You must be signed in to change notification settings - Fork 34.5k
Virtual File System for Node.js #61478
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
Review requested:
|
|
The
notable-change
Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section. |
|
Nice! This is a great addition. Since it's such a large PR, this will take me some time to review. Will try to tackle it over the next week. |
| */ | ||
| existsSync(path) { | ||
| // Prepend prefix to path for VFS lookup | ||
| const fullPath = this.#prefix + (StringPrototypeStartsWith(path, '/') ? path : '/' + path); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we use path.join?
| /** | ||
| * Gets the underlying VirtualFileSystem instance. | ||
| * @returns {VirtualFileSystem} | ||
| */ | ||
| get vfs() { | ||
| return this.#vfs; | ||
| } | ||
|
|
||
| /** | ||
| * Gets the mount prefix for the mock file system. | ||
| * @returns {string} | ||
| */ | ||
| get prefix() { | ||
| return this.#prefix; | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do these need to be getters? Why can't we expose the actual values.
If a user overwrites them, they can if they wish?
| validateObject(files, 'options.files'); | ||
| } | ||
|
|
||
| const { VirtualFileSystem } = require('internal/vfs/virtual_fs'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't we import this at the top level / lazy load it at the top level?
| ArrayPrototypePush(this.#mocks, { | ||
| __proto__: null, | ||
| ctx, | ||
| restore: restoreFS, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| restore: restoreFS, | |
| restore: ctx.restore, |
nit
lib/internal/vfs/entries.js
Outdated
| * @param {object} [options] Optional configuration | ||
| */ | ||
| addFile(name, content, options) { | ||
| const path = this._directory.path + '/' + name; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we use path.join?
| let entry = current.getEntry(segment); | ||
| if (!entry) { | ||
| // Auto-create parent directory | ||
| const dirPath = '/' + segments.slice(0, i + 1).join('/'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's use path.join
| let entry = current.getEntry(segment); | ||
| if (!entry) { | ||
| // Auto-create parent directory | ||
| const parentPath = '/' + segments.slice(0, i + 1).join('/'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
path.join?
| } | ||
| } | ||
| callback(null, content); | ||
| }).catch((err) => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| }).catch((err) => { | |
| }, (err) => { |
lib/internal/vfs/virtual_fs.js
Outdated
| const bytesToRead = Math.min(length, available); | ||
| content.copy(buffer, offset, readPos, readPos + bytesToRead); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Primordials?
| } | ||
|
|
||
| callback(null, bytesToRead, buffer); | ||
| }).catch((err) => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| }).catch((err) => { | |
| }, (err) => { |
|
Left an initial review, but like @Ethan-Arrowood said, it'll take time for a more in depth look |
|
It's nice to see some momentum in this area, though from a first glance it seems the design has largely overlooked the feedback from real world use cases collected 4 years ago: https://github.com/nodejs/single-executable/blob/main/docs/virtual-file-system-requirements.md - I think it's worth checking that the API satisfies the constraints that users of this feature have provided, to not waste the work that have been done by prior contributors to gather them, or having to reinvent it later (possibly in a breaking manner) to satisfy these requirements from real world use cases. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #61478 +/- ##
==========================================
- Coverage 89.80% 89.79% -0.02%
==========================================
Files 672 681 +9
Lines 203907 207293 +3386
Branches 39203 39692 +489
==========================================
+ Hits 183121 186139 +3018
- Misses 13113 13475 +362
- Partials 7673 7679 +6
🚀 New features to boost your workflow:
|
|
And why not something like OPFS aka whatwg/fs? const rootHandle = await navigator.storage.getDirectory()
await rootHandle.getFileHandle('config.json', { create: true })
fs.mount('/app', rootHandle) // to make it work with fs
fs.readFileSync('/app/config.json')OR const rootHandle = await navigator.storage.getDirectory()
await rootHandle.getFileHandle('config.json', { create: true })
fs.readFileSync('sandbox:/config.json')fs.createVirtual seems like something like a competing specification |
5e317de to
977cc3d
Compare
I generally prefer not to interleave with WHATWG specs as much as possible for core functionality (e.g., SEA). In my experience, they tend to perform poorly on our codebase and remove a few degrees of flexibility. (I also don't find much fun in working on them, and I'm way less interested in contributing to that.) On an implementation side, the core functionality of this feature will be identical (technically, it's missing writes that OPFS supports), as we would need to impact all our internal fs methods anyway. If this lands, we can certainly iterate on a WHATWG-compatible API for this, but I would not add this to this PR. |
|
Small prior art: https://github.com/juliangruber/subfs |
8d711c1 to
73c18cd
Compare
|
I also worked on this a bit on the side recently: Qard@73b8fc6 That is very much in chaotic ideation stage with a bunch of LLM assistance to try some different ideas, but the broader concept I was aiming for was to have a module.exports = new VirtualFileSystem(new LocalProvider())I intended for it to be extensible for a bunch of different interesting scenarios, so there's also an S3 provider and a zip file provider there, mainly just to validate that the model can be applied to other varieties of storage systems effectively. Keep in mind, like I said, the current state is very much just ideation in a branch I pushed up just now to share, but I think there are concepts for extensibility in there that we could consider to enable a whole ecosystem of flexible storage providers. 🙂 Personally, I would hope for something which could provide both read and write access through an abstraction with swappable backends of some variety, this way we could pass around these virtualized file systems like objects and let an ecosystem grow around accepting any generalized virtual file system for its storage backing. I think it'd be very nice for a lot of use cases like file uploads or archive management to be able to just treat them like any other readable and writable file system. |
just a bit off topic... but this reminds me of why i created this feature request: Would not lie, it would be cool if NodeJS also provided some type of static example that would only work in NodeJS (based on how it works internally) const size = 26
const blobPart = BlobFrom({
size,
stream (start, end) {
// can either be sync or async (that resolves to a ReadableStream)
// return new Response('abcdefghijklmnopqrstuvwxyz'.slice(start, end)).body
// return new Blob(['abcdefghijklmnopqrstuvwxyz'.slice(start, end)]).stream()
return fetch('https://httpbin.dev/range/' + size, {
headers: {
range: `bytes=${start}-${end - 1}`
}
}).then(r => r.body)
}
})
blobPart.text().then(text => {
console.log('a-z', text)
})
blobPart.slice(-3).text().then(text => {
console.log('x-z', text)
})
const a = blobPart.slice(0, 6)
a.text().then(text => {
console.log('a-f', text)
})
const b = a.slice(2, 4)
b.text().then(text => {
console.log('c-d', text)
})An actual working PoC (I would not rely on this unless it became officially supported by nodejs core - this is a hack) const blob = new Blob()
const symbols = Object.getOwnPropertySymbols(blob)
const blobSymbol = symbols.map(s => [s.description, s])
const symbolMap = Object.fromEntries(blobSymbol)
const {
kHandle,
kLength,
} = symbolMap
function BlobFrom ({ size, stream }) {
const blob = new Blob()
if (size === 0) return blob
blob[kLength] = size
blob[kHandle] = {
span: [0, size],
getReader () {
const [start, end] = this.span
if (start === end) {
return { pull: cb => cb(0) }
}
let reader
return {
async pull (cb) {
reader ??= (await stream(start, end)).getReader()
const {done, value} = await reader.read()
cb(done ^ 1, value)
}
}
},
slice (start, end) {
const [baseStart] = this.span
return {
span: [baseStart + start, baseStart + end],
getReader: this.getReader,
slice: this.slice,
}
}
}
return blob
}currently problematic to do: also need to handle properly clone, serialize & deserialize, if this where to be sent of to another worker - then i would transfer a MessageChannel where the worker thread asks main frame to hand back a transferable ReadableStream when it needs to read something. but there are probably better ways to handle this internally in core with piping data directly to and from different destinations without having to touch the js runtime? - if only getReader could return the reader directly instead of needing to read from the ReadableStream using js? |
Redesign VFS with a Provider-based architecture: - Add VirtualProvider base class defining the provider interface - Add MemoryProvider for in-memory file storage with full read/write - Add SEAProvider for read-only access to SEA assets - Add VirtualFileSystem class as thin wrapper delegating to providers - Add VirtualFileHandle and MemoryFileHandle for file operations - Register node:vfs as a builtin module (scheme-only access) - Add comprehensive API documentation in doc/api/vfs.md The provider interface supports: - Essential primitives: open, stat, readdir, mkdir, rmdir, unlink, rename - Derived methods: readFile, writeFile, exists, copyFile, etc. - Capability flags: readonly, supportsSymlinks - Dynamic content providers (functions called on each read) - Lazy directory population
Remove backward compatibility methods (addFile, addDirectory, addSymlink, has, remove) from VirtualFileSystem. Users should now use the standard fs-like API (writeFileSync, mkdirSync, symlinkSync, existsSync, unlinkSync). For dynamic content and lazy directories, use the provider methods: - provider.setContentProvider(path, fn) for dynamic file content - provider.setPopulateCallback(path, fn) for lazy directory population Also adds: - MemoryProvider.setReadOnly() to make provider immutable after setup - Fix router.js to handle root mount point (/) correctly
- Remove setContentProvider() and setPopulateCallback() methods - Remove getContent() backward compat wrapper from MemoryProvider - Remove section separator comments from memory.js and sea.js - Remove related tests for dynamic content and lazy directories - Update documentation to remove the removed methods
Ethan-Arrowood
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some initial comments on docs/api/fs.md only. I'm going to focus on reviewing docs first and then move on to actual implementations.
| * A synchronous function `() => string|Buffer` for dynamic content | ||
| * An async function `async () => string|Buffer` for async dynamic content | ||
|
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lets add a little more explanation to "dynamic content". I think the code example below already does a better job by including the tidbit "(evaluated on each read)". It'd be good to call that here in the description.
I think its particularly important to call this out for the async one since unknowing users could potentially be making expensive network requests every time that data is read.
Like
const data = await fetchData();
vfs.addFile('/data.json', JSON.stringify(data));Is quite different from:
vfs.addFile('/data.json', async () => {
const data = await fetchData();
return JSON.stringify(data);
});| * `populate` {Function} Optional callback to dynamically populate the directory. | ||
|
|
||
| Adds a virtual directory. If `populate` is provided, it receives a scoped VFS | ||
| for adding files and subdirectories within this directory. The callback is | ||
| invoked lazily on first access. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can populate be async? Can the user safely do async things within it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Allowing populate to be a sync or async iterable would be helpful here as well.
|
|
||
| // Create a VFS without module loading hooks (fs operations only) | ||
| const fsOnlyVfs = fs.createVirtual({ moduleHooks: false }); | ||
| ``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing mjs code block.
Seems like most of the examples in here are missing the mjs counterpart. I'm not going to add comments for all of them.
|
|
||
| vfs.unmount(); | ||
|
|
||
| fs.existsSync('/vfs/test.txt'); // false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this just unmount the /vfs path, or the entire virtual filesystem?
Like what would fs.existsSync('/test.txt'); return now?
And do remounts work? Like if I vfs.mount('/vfs'); again, would fs.existsSync('/vfs/test.txt'); return true again?
| * Returns: {boolean} | ||
|
|
||
| Returns `true` if the VFS contains a file or directory at the given path. | ||
|
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing example.
| * `path` {string} The path to remove. | ||
| * Returns: {boolean} `true` if the entry was removed, `false` if not found. | ||
|
|
||
| Removes a file or directory from the VFS. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would this differ at all from rm, rmSync, unlink, etc. ?
|
|
||
| Returns `true` if virtual working directory support is enabled for this VFS | ||
| instance. This is determined by the `virtualCwd` option passed to | ||
| `fs.createVirtual()`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can this property be set by the user to enable virtualCwd later? Or should we include in the description this is meant to be read only?
| > **Note:** VFS hooks are not automatically shared with worker threads. Each | ||
| > worker thread has its own `process` object and must set up its own VFS | ||
| > instance if virtual cwd support is needed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is interesting to me. Isn't process.chdir() not available in worker threads? So I wouldn't expect the VFS to change that 🤔
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For that we'd need to move the implementation into C++ and arrange for shared access, which gets complicated very quickly. I think omitting that for now is ideal but we might want to make sure we don't rule it out for later.
| (`writeFile`, `appendFile`, etc.) are not supported. | ||
| * **No file watching**: `fs.watch()` and `fs.watchFile()` do not work with | ||
| virtual files. | ||
| * **No real file descriptor**: Virtual file descriptors (10000+) are managed |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the 10000+ part?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Haven't checked but I assume open returns file descriptors (ie integers) higher than 10k (arbitrary number) to denote that they aren't true ones and avoid clashing with actual system fds. That's how we implemented it in the fslib.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Although thinking about it 10k feels a little low for apps that open/close a lot. In fslib we simply reserve the upper byte of an i32, and it seemed to work well (note: you can't use the upper bit; unix requires fds to be signed).
| virtual files. | ||
| * **No real file descriptor**: Virtual file descriptors (10000+) are managed | ||
| separately from real file descriptors. | ||
|
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How do fs.Stats work?
|
Hold off from |
bnb
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some suggestions, plus I'd like to request that we also include ESM examples for every cjs example we're adding in doc/api/fs.md.
I'd be happy to commit to the PR directly with ESM examples if you'd like.
| const vfs = fs.createVirtual(); | ||
|
|
||
| // Add files to the VFS | ||
| vfs.addFile('/config.json', JSON.stringify({ debug: true })); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
might be wroth passing something more generic here, to minimize possible confusion about what's being passed.
| vfs.addFile('/config.json', JSON.stringify({ debug: true })); | |
| vfs.addFile('/config.json', JSON.stringify({ foo: true })); |
|
|
||
| // Now files are accessible via standard fs APIs | ||
| const config = JSON.parse(fs.readFileSync('/app/config.json', 'utf8')); | ||
| console.log(config.debug); // true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Paired with my previous suggestion:
| console.log(config.debug); // true | |
| console.log(config.foo); // true |
| const vfs = fs.createVirtual(); | ||
|
|
||
| // Add files to the VFS | ||
| vfs.addFile('/config.json', JSON.stringify({ debug: true })); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Paired with previous suggestions:
| vfs.addFile('/config.json', JSON.stringify({ debug: true })); | |
| vfs.addFile('/config.json', JSON.stringify({ foo: true })); |
|
|
||
| // Now files are accessible via standard fs APIs | ||
| const config = JSON.parse(fs.readFileSync('/app/config.json', 'utf8')); | ||
| console.log(config.debug); // true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Paired with previous suggestions:
| console.log(config.debug); // true | |
| console.log(config.foo); // true |
| const vfs = fs.createVirtual(); | ||
|
|
||
| // Static content | ||
| vfs.addFile('/config.json', '{"debug": true}'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Paired with previous suggestions:
| vfs.addFile('/config.json', '{"debug": true}'); | |
| vfs.addFile('/config.json', '{"foo": true}'); |
| const content = fs.readFileSync('/virtual/module.js', 'utf8'); | ||
| const mod = require('/virtual/module.js'); | ||
| ``` | ||
|
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we give an example of what will happen if you mount a second time? For example:
const fs = require('node:fs');
const vfs = fs.createVirtual();
vfs.addFile('/module.js', 'module.exports = "hello"');
vfs.mount('/virtual');
// Now accessible at /virtual/module.js
const content = fs.readFileSync('/virtual/module.js', 'utf8');
const mod = require('/virtual/module.js');
vfs.mount('/different');
// what happens if we use mod? what happens if we use load `/different/module.js`?| #### `vfs.unmount()` | ||
|
|
||
| <!-- YAML | ||
| added: REPLACEME | ||
| --> | ||
|
|
||
| Unmounts the VFS, removing it from the file system namespace. After unmounting, | ||
| the virtual files are no longer accessible through standard `fs` operations. | ||
|
|
||
| ```cjs | ||
| const fs = require('node:fs'); | ||
|
|
||
| const vfs = fs.createVirtual(); | ||
| vfs.addFile('/test.txt', 'content'); | ||
| vfs.mount('/vfs'); | ||
|
|
||
| fs.existsSync('/vfs/test.txt'); // true | ||
|
|
||
| vfs.unmount(); | ||
|
|
||
| fs.existsSync('/vfs/test.txt'); // false | ||
| ``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO makes more sense to have this directly after mount?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Our doc conventions really prefer that these are listed in alphabetical order
| > **Note:** VFS hooks are not automatically shared with worker threads. Each | ||
| > worker thread has its own `process` object and must set up its own VFS | ||
| > instance if virtual cwd support is needed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"Automatically" here implies that VFS hooks can be shared with worker threads. Is that the case? If so, can we mention it here (and point to where users can learn more?)
- Fix template string in documentation examples - Remove unused imports (isVirtualFd, VFS_FD_BASE, kEntries, isWrite) - Add createENOTDIR to top-level imports, remove duplicate require - Use DateNow from primordials instead of Date.now() - Add MathMin, Boolean to primordials at top of sea.js - Add parentheses to arrow function parameter
SEA assets are automatically mounted at /sea when running as a Single Executable Application. No user action is required.
VFS Provider Architecture UpdateThis PR has been updated with a significant refactoring to introduce a Provider-based architecture. Here's a summary of the changes: Key Changes
Public APIconst vfs = require('node:vfs');
// Create a VFS with default MemoryProvider
const myVfs = vfs.create();
// Or with explicit provider
const myVfs = vfs.create(new vfs.MemoryProvider());
// Standard fs operations
myVfs.writeFileSync('/file.txt', 'content');
myVfs.mkdirSync('/dir');
// Mount to make accessible via fs module
myVfs.mount('/virtual');
// For SEA - assets automatically available at /sea
// No code needed - just use fs.readFileSync('/sea/asset.txt')Exported Classes
cc @Qard |
- Use getLazy utility for lazy loading VirtualFileSystem and SEAProvider - Import getLazy from internal/util instead of manual ??= pattern
- Remove references to non-existent setContentProvider() and setPopulateCallback() methods - Add mjs examples to vfs.create(), promises API, and fs integration - Clarify unmount/remount behavior - VFS can be remounted after unmount - Document virtualCwd limitations (construction-time only, no workers) - Add Implementation details section explaining Stats objects and FD range - Fix "Note that" lint warning Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Addressed Review Comments (Ethan-Arrowood)I've addressed the documentation review comments:
Note: Some original comments referenced methods/features from an earlier design iteration that were removed in the Provider architecture refactoring (e.g., |
Virtual cwd works for virtual paths in Workers, but process.chdir() to real filesystem paths will throw ERR_WORKER_UNSUPPORTED_OPERATION. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
RealFSProvider wraps a real filesystem directory, allowing it to be mounted at a different VFS path. This is useful for: - Mounting a directory at a different path - Enabling virtualCwd support in Worker threads - Creating sandboxed views of real directories The provider prevents path traversal attacks by ensuring resolved paths stay within the configured root directory. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add VirtualFileSystem, VirtualProvider, MemoryProvider, SEAProvider, and RealFSProvider to the type-parser for documentation generation.
| const kPath = Symbol('kPath'); | ||
|
|
||
| // FD range: 10000+ to avoid conflicts with real fds | ||
| const VFS_FD_BASE = 10_000; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alternatively here... it might be interesting to use negative fds for vfs nodes. Not sure if practically possible but it would avoid the potential for collision.
| --> | ||
|
|
||
| * `path` {string} The virtual path for the file. | ||
| * `content` {string|Buffer|Function} The file content, or a function that |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Suggest not limiting this to Buffer... also support Uint8Array explicitly.
For "dynamic content", why not also accept sync/async iterables here so that the content can be provided by a stream or generator.
We might also want to support Blob and File here.
|
|
||
| Adds a virtual file. The `content` can be: | ||
|
|
||
| * A `string` or `Buffer` for static content |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When a string is provided, is there a default encoding?
| * `prefix` {string} The path prefix where the VFS will be mounted. | ||
|
|
||
| Mounts the VFS at a specific path prefix. All paths in the VFS become accessible | ||
| under this prefix. Only one mount point can be active at a time. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this last sentence mean this will throw if it is called twice?
Having vfs.mount(...) return a disposable object that will unmount on disposal would be nice.
const myMount = vfs.mount(prefix);
myMount.unmount();
// or
{
using myMount = vfs.mount(prefix);
// ...
}There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alternatively, having vfs itself support Symbol.dispose would be nice (if it is already and I haven't read far enough thru then ignore this ;-) ...)
|
|
||
| const vfs = fs.createVirtual(); | ||
| vfs.addFile('/module.js', 'module.exports = "hello"'); | ||
| vfs.mount('/virtual'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be good for the docs here to explain a bit about what happens if there's already an existing real file system path /virtual. Is it shadowed? Does mount error? etc
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If it automatically shadows, then I can see a potential security hole in this. For instance, if I can mount a path that shadows any real file system path, then a module could obfuscate mounting over a critical system path and trick users into reading malicious data or writing data to something that intercepts the data and forwards it somewhere. Not a critical concern given our security model but something that would be worth documenting
|
|
||
| Enables overlay mode, where the VFS is checked first for all file system | ||
| operations. If a path exists in the VFS, it is used; otherwise, the operation | ||
| falls through to the real file system (if `fallthrough` is enabled). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would add warnings to the docs here about how this could be abused to secretly shadow critical file system paths.
| * `path` {string} The path to check. | ||
| * Returns: {boolean} | ||
|
|
||
| Returns `true` if the VFS contains a file or directory at the given path. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should document the behavior if "overlay" mode is enabled.
| added: REPLACEME | ||
| --> | ||
|
|
||
| * `path` {string} The path to remove. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should document the behavior when "overlay" mode is enabled.
| const mod = require('/virtual/module.js'); | ||
| ``` | ||
|
|
||
| #### `vfs.overlay()` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I'd actually prefer this to be an option passed to createVirtual
| When the VFS is unmounted, `process.chdir()` and `process.cwd()` are restored | ||
| to their original implementations. | ||
|
|
||
| > **Note:** VFS hooks are not automatically shared with worker threads. Each |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| > **Note:** VFS hooks are not automatically shared with worker threads. Each | |
| > VFS hooks are not automatically shared with worker threads. Each |
| console.log(myVfs.provider.readonly); // true | ||
| ``` | ||
|
|
||
| ### `vfs.mount(prefix)` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This feels like it's duplicating the docs in fs.md a bit too much
|
|
||
| * {boolean} | ||
|
|
||
| Returns `true` if the provider supports symbolic links. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Symbolic link support needs more explanation. For instance, in "overlay" mode... would it be possible for a real file system symbolic link to target a vfs symbolic link and vis versa? I assume yes but that gets tricky especially when worker threads are involved... sometimes the link works other times it doesn't, etc.
A first-class virtual file system module (
node:vfs) with a provider-based architecture that integrates with Node.js's fs module and module loader.Key Features
Provider Architecture - Extensible design with pluggable providers:
MemoryProvider- In-memory file system with full read/write supportSEAProvider- Read-only access to Single Executable Application assetsVirtualProvider- Base class for creating custom providersStandard fs API - Uses familiar
writeFileSync,readFileSync,mkdirSyncinstead of custom methodsMount Mode - VFS mounts at a specific path prefix (e.g.,
/virtual), clear separation from real filesystemModule Loading -
require()andimportwork seamlessly from virtual filesSEA Integration - Assets automatically mounted at
/seawhen running as a Single Executable ApplicationFull fs Support - readFile, stat, readdir, exists, streams, promises, glob, symlinks
Example
SEA Usage
When running as a Single Executable Application, bundled assets are automatically available:
Public API
Disclaimer: I've used a significant amount of Claude Code tokens to create this PR. I've reviewed all changes myself.