vfs: file system abstraction by jnohlgard · Pull Request #5616 · RIOT-OS/RIOT

jnohlgard · 2016-07-08T20:36:26Z

This PR provides a basic virtual file system (VFS) layer for RIOT. The intention is to provide a much simpler layer than what was proposed in #2474 and #1265.

The overall design goals are:

Provide implementations for all newlib "file" syscalls
Keep it simple, do not add every possible file operation from Linux VFS.
Easy to map existing file system implementations for resource constrained systems onto the VFS layer API
Avoid keeping a central enum of all file system drivers that has to be kept up to date with external packages etc.
Use POSIX <errno.h> numbers as negative return codes for errors, avoid the global errno variable.
Only absolute paths to files (no per-process working directory)
No dynamic memory allocation

The file systems for resource constrained systems that I have been looking at (so far) are Contiki CFS and SPIFFS.
I have also been looking at Linux kernel headers and books/articles on writing Linux device drivers for more ideas for the design.

The API should be easy to understand for users who are familiar with the POSIX file functions (open, close, read, write, fstat, lseek).

The VFS layer keeps track of mounted file systems and open files, the vfs_open function searches the array of mounted file systems and dispatches the call to the file system instance with the longest matching mount point prefix.
Subsequent calls to vfs_read, vfs_write, etc will do a look up in the table of open files and dispatch the call to the correct file system driver for handling.

vfs_mount takes a string containing the mount point, a file system driver specification (struct file_system), and an opaque pointer that only the FS driver knows how to use, which can be used to keep driver parameters in order to allow dynamic handling of multiple devices.

Also included is a simple file system implementation of a static const file system called ConstFS. This was created as a tool to assist in unit testing of the VFS layer, and as an exercise in implementing file system drivers, and to provide an example for developers who wish to write their own file system driver to hook into the VFS layer.

TODO:

Not implemented:

Simultaneous access to the same file.

How to try it

Right now the only place to run the code is through a newly added embunit unit test batch:

make -C tests/unittests tests-vfs term BOARD=native

(also successfully tested on BOARD=mulle)

Open questions

Does it make sense to add an "inode" layer to separate the file name handling from the file opening?
Should mode be given as mode_t or int? (newlib uses int, POSIX uses mode_t which is unsigned long on 32 bit platforms)
~~Should mounted file systems be changed into a linked list? (originally was statically allocated array with fixed maximum mount point length)~~ - implemented

kaspar030 · 2016-07-09T08:43:49Z

Nice! Thank you for tackling this. Looks very nice already!

Some thoughts that came up while skimming the diff:

did you consider extending structs by embedding instead of using pointers, (like netdev2 does)?
any reason you're using "struct xxx..." within structs, instead of the typedefs?

jnohlgard · 2016-07-09T09:22:00Z

@kaspar030

did you consider extending structs by embedding instead of using pointers, (like netdev2 does)?

My thoughts were that some of these structs should be constant and static, e.g. the file system driver, and should not consume valuable RAM, while other parts are variable. Right now I think all of it is kept in RAM, but the goal is to keep the static parts in flash instead and using pointers from the RAM parts to the flash parts.
-- updated: file operations and file system operations are const and kept in flash

any reason you're using "struct xxx..." within structs, instead of the typedefs?

No particular reason, I started using them in one place to get a pointer to the same type of struct, and then it just kind of kept on. I'll see if it still compiles if I modify it to use the typedefs instead.
-- updated: modified to use structname_t instead of struct structname

Kijewski · 2016-07-09T22:56:33Z

sys/fs/constfs/constfs.c

+            off += filp->pos;
+            break;
+        case SEEK_END:
+            off += fp->size;


jnohlgard · 2016-07-11T10:12:43Z

updated todo-list with some more ideas.

vincent-d · 2016-07-11T10:21:51Z

This looks very nice.

I was working on the same topic, but my work is much less advanced than yours. I'm trying to get SPIFFS working and we worked on a low level flash interface as well. I'll push some work soon and will try to rebase my SPIFFS port on top of your vfs.

Thanks for that work

jnohlgard · 2016-07-11T10:28:47Z

@vincent-d Nice! I have had plans to add SPIFFS but I did not feel that I knew where to begin, so I started with this instead, it's good to see that others are working towards the same goal.
I am currently under way to add a device file system, DevFS, similar to /dev on Linux, in order to be able to access low level devices from the shell, such as NVRAM. Maybe we can fit the generic flash driver onto that as well? (see TODO in top post)

jnohlgard · 2016-07-11T11:49:28Z

squashed, WIP commit list was getting out of hand.

kaspar030 · 2016-07-12T08:07:58Z

How do we deal with directories?

jnohlgard · 2016-07-12T08:12:00Z

Updated with more work on DevFS, including:

DevFS auto_init mounts /dev
Shell command vfs to access the VFS mounted file systems from the shell
Newlib fixes
basic fcntl implementation
VFS file operations implementation for /drivers/nvram to access the contents of NVRAM devices via VFS.
Configured NVRAM VFS on DevFS for Mulle (/dev/mulle-fram).
improved unit tests

jnohlgard · 2016-07-12T08:13:44Z

@kaspar030 not sure. CFS and SPIFFS both implement specific opendir, closedir, readdir. Maybe those functions should be added as part of vfs_file_system_ops_t, what is your opinion?

vincent-d · 2016-07-12T11:29:29Z

sys/vfs/vfs.c

+        return -ENOENT;
+    }
+    /* Increment open files counter for this mount */
+    atomic_inc(&_vfs_mounts[md].open_files);


This should not be done here but only in vfs_open, otherwise the unlink also increments the counter and umount fails afterwards.

Actually, this is on purpose as a kind of semaphore to avoid races between umount and open/unlink. There were a few missing decrements of open_files in vfs_unlink though that I have fixed now.

jnohlgard · 2016-07-12T17:30:57Z

I'll do an update tomorrow morning with opendir, readdir, closedir support for devfs and constfs (and the VFS support code of course)

jnohlgard · 2016-07-13T06:59:58Z

Updated with the directory interface. No unit tests yet or ConstFS, DevFS support for directories.

miri64 · 2016-07-13T07:34:32Z

I know FAT is not the most ideal FS for our purposes with all its layers of backwards compatibility, but we also should consider porting the FatFS library as a package using this interface. After all: FAT is still the file system type most SD cards are shipped with. I saw this library used on systems that are way more constraint than the ones RIOT targets (the CCC's r0ket badge e.g.), so space shouldn't be an issue.

jnohlgard · 2017-03-06T18:54:11Z

Rebased on top of #6587. Squashed and ready for review.

Edit: Also, removed the MIPS change and added an #ifdef for the MIPS board which uses newlib without specifying. See #6639 for the proper fix, but I don't want that PR to block this, this has been waiting for too long now..

This line gave a -Wunused-value in gcc-4.6 before the stdatomic.h header for msp430 was modified as a workaround.

Enable adding more C++ specifics inside the same ifdef block (e.g. sys/include/vfs.h)

…unctions and constants

The VFS layer provides file system abstractions to allow using a unified interface to access files from mounted file systems.

miri64 · 2017-03-08T13:37:38Z

Does this need a re-ACK?

jnohlgard · 2017-03-08T14:24:36Z

@miri64 Needs at least someone to approve the workaround for the mips board in the beginning of fcntl.h and statvfs.h

jnohlgard · 2017-03-09T05:57:11Z

@vincent-d Does your ACK hold?

vincent-d · 2017-03-09T08:27:02Z

Yes! ACK

miri64 · 2017-03-09T08:41:35Z

qemu error on Jarvis seems unrelated and Murdock (1) is still the imperative CI, so merge at will.

jnohlgard · 2017-03-09T08:44:00Z

Yay! :D

miri64 · 2017-03-09T08:45:36Z

Congratz \o/

vincent-d · 2017-03-09T09:00:53Z

Great!!

kaspar030 · 2017-03-09T09:06:25Z

Congrats and a lot of thanks to @gebart! Good job!

emmanuelsearch · 2017-03-09T09:09:08Z

great!!

kYc0o · 2017-03-09T14:16:43Z

Super! Thanks @gebart !

jnohlgard mentioned this pull request Jul 8, 2016

fs: Filesystem module (WIP) #2474

Closed

6 tasks

kaspar030 self-assigned this Jul 9, 2016

Kijewski reviewed Jul 9, 2016
View reviewed changes

jnohlgard force-pushed the pr/vfs branch from 4924520 to c5aae55 Compare July 11, 2016 11:46

jnohlgard force-pushed the pr/vfs branch 2 times, most recently from 518d49b to 7549a62 Compare July 12, 2016 08:05

jnohlgard added this to the Release 2016.10 milestone Jul 12, 2016

vincent-d reviewed Jul 12, 2016
View reviewed changes

vincent-d mentioned this pull request Jul 12, 2016

pkg: add SPIFFS support #5625

Merged

jnohlgard force-pushed the pr/vfs branch from ddf2afd to 0a3c29a Compare July 13, 2016 06:57

jnohlgard force-pushed the pr/vfs branch from d207cf1 to 1bfc799 Compare March 6, 2017 18:52

Joakim Nohlgård and others added 16 commits March 7, 2017 10:21

msp430: Discard order argument when using __sync_xxx for atomics

33566a2

tests/emb6: Revert workaround for false positive

0f74b5b

This line gave a -Wunused-value in gcc-4.6 before the stdatomic.h header for msp430 was modified as a workaround.

tools/externc: Remove #endif in extern C pattern match

5638359

Enable adding more C++ specifics inside the same ifdef block (e.g. sys/include/vfs.h)

sys/posix: Add fcntl.h

03d9f0f

sys/posix: Add sys/statvfs.h

89ba41c

cpu/atmega_common: Improve POSIX headers

738876a

atmega_common: Update unistd.h with prototypes of all POSIX defined f…

9ec001b

…unctions and constants

msp430: add libc missing symbols and defines

3485ab4

msp430: Add all expected POSIX unistd.h declarations

5e27bf4

sys/vfs: A virtual file system (VFS) layer for RIOT

dcc3732

The VFS layer provides file system abstractions to allow using a unified interface to access files from mounted file systems.

newlib: Use vfs for file I/O syscalls

a1faeb9

uart_stdio: Bind uart_stdio to vfs fds for stdin/out/err

fdd129d

devfs: Dynamic file system for device nodes

0371769

cpu: native: add vfs wrappers

26b99a5

drivers/nvram: Add vfs compatible functions

e021b3b

boards/mulle: Add FRAM to DevFS

8f81121

neiljay mentioned this pull request Mar 10, 2017

cpu: Add PIC32MZ support #6066

Merged

Conversation

jnohlgard commented Jul 8, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to try it

Open questions

Uh oh!

kaspar030 commented Jul 9, 2016

Uh oh!

jnohlgard commented Jul 9, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Kijewski Jul 9, 2016

Choose a reason for hiding this comment

Uh oh!

jnohlgard Jul 10, 2016

Choose a reason for hiding this comment

Uh oh!

jnohlgard commented Jul 11, 2016

Uh oh!

vincent-d commented Jul 11, 2016

Uh oh!

jnohlgard commented Jul 11, 2016

Uh oh!

jnohlgard commented Jul 11, 2016

Uh oh!

kaspar030 commented Jul 12, 2016

Uh oh!

jnohlgard commented Jul 12, 2016

Uh oh!

jnohlgard commented Jul 12, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vincent-d Jul 12, 2016

Choose a reason for hiding this comment

Uh oh!

jnohlgard Jul 12, 2016

Choose a reason for hiding this comment

Uh oh!

jnohlgard commented Jul 12, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jnohlgard commented Jul 13, 2016

Uh oh!

miri64 commented Jul 13, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jnohlgard commented Mar 6, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

miri64 commented Mar 8, 2017

Uh oh!

jnohlgard commented Mar 8, 2017

Uh oh!

jnohlgard commented Mar 9, 2017

Uh oh!

vincent-d commented Mar 9, 2017

Uh oh!

miri64 commented Mar 9, 2017

Uh oh!

jnohlgard commented Mar 9, 2017

Uh oh!

miri64 commented Mar 9, 2017

Uh oh!

vincent-d commented Mar 9, 2017

Uh oh!

kaspar030 commented Mar 9, 2017

Uh oh!

emmanuelsearch commented Mar 9, 2017

Uh oh!

kYc0o commented Mar 9, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

jnohlgard commented Jul 8, 2016 •

edited

Loading

jnohlgard commented Jul 9, 2016 •

edited

Loading

jnohlgard commented Jul 12, 2016 •

edited

Loading

jnohlgard commented Jul 12, 2016 •

edited

Loading

miri64 commented Jul 13, 2016 •

edited

Loading

jnohlgard commented Mar 6, 2017 •

edited

Loading