Skip to content

package: add generic FatFs#6072

Merged
BytesGalore merged 2 commits intoRIOT-OS:masterfrom
MichelRottleuthner:fatfs_testing
May 23, 2017
Merged

package: add generic FatFs#6072
BytesGalore merged 2 commits intoRIOT-OS:masterfrom
MichelRottleuthner:fatfs_testing

Conversation

@MichelRottleuthner
Copy link
Contributor

@MichelRottleuthner MichelRottleuthner commented Nov 7, 2016

This PR adds a package for a generic FAT file system. As it's a RIOT package it mainly contains a makefile that can be used to fetch the code from the original authors page.
Beyond this the PR also includes two FatFs abstraction layer implementations. One for the sdcard_spi driver and one that wraps around file-io systemcalls that makes it fairly easy to use it with FAT-images under native (see README for details).
Also included is a simple interactive test application. Currently with the test app you can mount your volume, list, read, create and write to files.
One thing I'd like to discuss about this thing: Since the FatFs package is generally neither bound to a specific media (like sdcard in this case) nor is the sdcard bound to be only used with FatFs - what would be the best and "RIOT-like" way to provide the abstraction layer (diskio.c)?. For now I've included it to this PR to make testing easier but I think it should be moved somewhere else.

Waiting for 6031 so that building with sdcard_spi_diskio is possible.

@MichelRottleuthner MichelRottleuthner changed the title Fatfs testing package: add generic FatFs Nov 7, 2016
@OlegHahm
Copy link
Member

OlegHahm commented Nov 8, 2016

There was FAT support in former RIOT versions, but we decided to kick it out at some point. IIRC the main reasons were:
1.) Unclear license situation. It is not clear if all code in the ChaN implementation is correctly licensed.
2.) Not designed for simple flash storage without internal wear leveling.
3.) Horrible design of FAT FS itself.

Have you considered to use another FS that is more suited for MCUs and flash storage? Maybe you can take a look at the on-going discussions with respect to file systems: https://github.com/RIOT-OS/RIOT/search?q=fs&type=Issues

@OlegHahm OlegHahm added Type: new feature The issue requests / The PR implemements a new feature for RIOT Discussion: RFC The issue/PR is used as a discussion starting point about the item of the issue/PR Area: pkg Area: External package ports labels Nov 8, 2016
@MichelRottleuthner
Copy link
Contributor Author

MichelRottleuthner commented Nov 8, 2016

1.) Unclear license situation.

Hmm can you give me a pointer to where that was discussed? The only related info I could find about that in the RIOT context is within #2474 where @x3ro stats "custom license compatible to BSD"

2.) Not designed for simple flash storage without internal wear leveling.

True, but I'd agree to comments by @miri64 like this one or by @gebart : "FAT would be useful for SD card storage, which provide internal wear levelling, but it would probably wear out simpler flash chips too quickly to be useful on those" (found here).
Apart from that, FatFs could be still a usefull addition especially for simple read-only tasks (config files with the ability to modify on any system and simply put back the sd-card to the MCU). Also if you just want to log data as in "continuously write data to a huge sd-card without deleting files for a long time" (my use case for now) wear-leveling might be not a real problem to you.

3.) Horrible design of FAT FS itself.

Yep! Nevertheless some guys ( @alignan @smlng I'm pointing at you :P) were eager to try it out.

@OlegHahm
Copy link
Member

OlegHahm commented Nov 8, 2016

There's some part of the discussion here: #1265 (comment)

Nope, wrong pointer. It might be that this discussion was in pre-RIOT, pre-Github days. I will see if I can find something. @kaspar030, do you remember?

@OlegHahm
Copy link
Member

OlegHahm commented Nov 8, 2016

lso if you just want to log data as in "continuously write data to a huge sd-card without deleting files for a long time" (my use case for now) wear-leveling might be not a real problem to you.

True, but in this case I don't see the need for any file system.

However, don't get me wrong: I'm not in general opposed to this PR.

Personally, I'm still not convinced that a file system is really needed on these systems, but that's another discussion. But if we want to support file systems, we should have a clear understanding of what and how we want to integrate it into RIOT.

@OlegHahm
Copy link
Member

OlegHahm commented Nov 8, 2016

The only license issue I could find was related to long file names and a MS patent, but that doesn't seem to be a problem: http://elm-chan.org/fsw/ff/en/appnote.html#lfn

Maybe I remember wrong, but I will still try to see if anyone else remembers.

@alignan
Copy link
Contributor

alignan commented Nov 8, 2016

Personally, I'm still not convinced that a file system is really needed on these systems

The point of using a micro-SD instead of a flash memory is to allow the information to be retrieved and read externally, without having to use a specific test/feature such as dumping the memory content over USB, etc (which would be huge). Also, in cases external configuration information has to be written to the microSD cards, it would be easier just to clone a card.

I'm not particularly fixed to FatFS as the solution, but I do see a need for a file system for the Micro-SD

@OlegHahm
Copy link
Member

OlegHahm commented Nov 8, 2016

The point of using a micro-SD instead of a flash memory is to allow the information to be retrieved and read externally, without having to use a specific test/feature such as dumping the memory content over USB, etc (which would be huge). Also, in cases external configuration information has to be written to the microSD cards, it would be easier just to clone a card.

I see use cases for a memory card, but for simple configuration data or log information, one doesn't need a filesystem. Cloning a card would be done with something like dd anyway which doesn't care about a filesystem or not.

But let's not discuss this here. 😉

@miri64
Copy link
Member

miri64 commented Nov 8, 2016

Cloning a card would be done with something like dd anyway which doesn't care about a filesystem or not.

What a user friendly way to get a bitmap to your device ;-)

@MichelRottleuthner
Copy link
Contributor Author

True, but in this case I don't see the need for any file system.

Sure. I simply did it for ease of use and the fact that i can manipulate files if needed - with the implication that I may need to think a bit more about how to use the FS.

Personally, I'm still not convinced that a file system is really needed on these systems, but that's another discussion.

I agree. Its not "really" needed as in "one could always implement/port a specific fs if needed". But also it could be of great help for someone that just wants to store/load data in an interoperable way and wants to get started quickly without reinventing the weel on how to store the data on the bare media.

...we should have a clear understanding of what and how we want to integrate it into RIOT

Agree. And I'm totally open minded to any kind of suggestions to tackle this :)

I'm not particularly fixed to FatFS as the solution, but I do see a need for a file system for the Micro-SD

ACK! Would also like to see a better fs without all the shortcomings. For me the key thing is: FatFs is actually usable in it's current state and its dead simple (from the user POV). FAT has it's place and may still be there for longer than we'd all prefer so why not provide compatibility...

@kaspar030
Copy link
Contributor

Nope, wrong pointer. It might be that this discussion was in pre-RIOT, pre-Github days. I will see if I can find something. @kaspar030, do you remember?

AFAIR the guy doing FireKernel's FAT support didn't use chan or (had to) heavily modify it, so we kicked it out as unmaintainable.

I'd like to see FAT for RIOT, just because it's the common denominator all OS can read. Wear leveling shouldn't be a problem with SD cards as they deal with that internally.

@OlegHahm
Copy link
Member

OlegHahm commented Nov 8, 2016

AFAIR the guy doing FireKernel's FAT support didn't use chan or (had to) heavily modify it, so we kicked it out as unmaintainable.

I just checked: the latest version of FireKernel used ChaN FatFs R0.08a. I haven't run a diff versus the upstream version.

I'd like to see FAT for RIOT, just because it's the common denominator all OS can read.

It is so sad that in 2016 we still have to use FAT as common FS, but yeah, you're kind of right. However, it's interesting how drastically you've changed your mind: #1265 (comment) 😉

@kaspar030
Copy link
Contributor

It is so sad that in 2016 we still have to use FAT as common FS, but yeah, you're kind of right. However, it's interesting how drastically you've changed your mind: #1265 (comment)

Yup. Back then I remembered the horrors Michael went through to get FAT working. Now it seems as ChaN's FatFS is actually usable. ;)

@kaspar030
Copy link
Contributor

@MichelRottleuthner Have you considered basing this on #5616?

@MichelRottleuthner
Copy link
Contributor Author

Yes I think that would make sence but I haven't looked into the details so I'm not exactly sure what is needed to make this work with vfs.

@MichelRottleuthner
Copy link
Contributor Author

@cgundogan @smlng you can now also use FatFs on native see README.md for details.

@cgundogan
Copy link
Member

@MichelRottleuthner 👍 I will look into the native tests later today. Do you mind splitting the sdcard_spi based code parts into another PR, which depends on this PR? This would make it easier to review the FatFS+native relevant code.

@MichelRottleuthner
Copy link
Contributor Author

splitting the sdcard_spi based code parts into another PR

As expressed in the opening comment I'd like to do that anyway. The question that remains open for discussion is: how (i.e. where would be the best place to put this code)?
If done consequently the gluecode for native would also have to be kicked out of this PR, which wouldn't really help with testing for now I think.
Currently I think an additional module (fatfs-diskio or whatever) that is also included in this PR would be best to ease testing and keep it clean at the same time. What do the others think?

*
* @}
*/
#define ENABLE_DEBUG (1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please define all ENABLE_DEBUGs as (0)

{
if(idx < sizeof(volume_files) / sizeof(dummy_volume_t)){
return &volume_files[idx];
}else{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

else statements should go to a new line


static dummy_volume_t volume_files[] = {
{
.image_path = "../../riot_fatfs_disk.img",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be parameterized somehow

{
dummy_volume_t *volume = get_volume_file(pdrv);

if(volume != NULL && volume->opened){
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

put parens around separate conditions

no longer used (needed at _USE_TRIM == 1) */
#define CTRL_TRIM 4

#define RTC_YEAR_OFFSET 1900
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

many of these macros are defined multiple times. can you extract them into a common h file?

}

#ifdef FATFS_RTC_AVAILABLE
DWORD get_fattime (void){
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems to be the same function as above. would make sense to put all common functions / macros into a common file

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

* @return 0 if disk is initialized
* @return STA_NOINIT if disk id exists, but disk isn't initialized
*/
DSTATUS disk_status (BYTE pdrv)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same?

@smlng
Copy link
Member

smlng commented Jan 12, 2017

@PeterKietzmann I think @MichelRottleuthner should whitelist supported boards with SPI as defined in pkg test Makefile

@smlng
Copy link
Member

smlng commented Jan 12, 2017

plus dependence on #6031

@PeterKietzmann PeterKietzmann added this to the Release 2017.01 milestone Jan 14, 2017
@PeterKietzmann PeterKietzmann modified the milestones: Release 2017.01, Release 2017.04 Jan 26, 2017
@miri64
Copy link
Member

miri64 commented Mar 9, 2017

@MichelRottleuthner please rebase.

@miri64 miri64 removed the State: waiting for other PR State: The PR requires another PR to be merged first label Mar 9, 2017
@miri64
Copy link
Member

miri64 commented Mar 9, 2017

VFS (#5616) was merged, so if you want to you can add a wrapper for that, but given that this PR was already ACK'd, I would say we do this as a follow-up ;-)

@miri64
Copy link
Member

miri64 commented Mar 22, 2017

Was approved, but needs rebase and some fixing ;-)

@@ -0,0 +1,2 @@
MODULE = fatfs_diskio
include $(RIOTBASE)/Makefile.base
Copy link
Member

@miri64 miri64 Mar 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this module is fatfs specific: why is it in sys and not pkg/fatfs/? Please move it (sorry for ringing in so late, but just noticed).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping @miri64

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pong.

@kaspar030 kaspar030 modified the milestone: Release 2017.04 Apr 21, 2017
@kaspar030 kaspar030 added the CI: needs squashing Commits in this PR need to be squashed; If set, CI systems will mark this PR as unmergable label May 8, 2017
@MichelRottleuthner
Copy link
Contributor Author

squashed.

@BytesGalore BytesGalore removed the CI: needs squashing Commits in this PR need to be squashed; If set, CI systems will mark this PR as unmergable label May 12, 2017
…of authors webserver, added Makefile.dep for fatfs, cleanup code for using rtc
ifneq (,$(filter tcp,$(USEMODULE)))
DIRS += net/transport_layer/tcp
endif

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep - seems like I couldn't resist removing the asymmetry in this file when moving the diskio stuff^^

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you removing the patch or shall I merge?

Copy link
Member

@miri64 miri64 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor nitpick, but I'm not blocking this PR because of this.

@miri64
Copy link
Member

miri64 commented May 22, 2017

Press merge at will.

@BytesGalore
Copy link
Member

beside the unrelated newline strips, looks good to me -> ACK'n'go

@BytesGalore BytesGalore merged commit 2ef5cff into RIOT-OS:master May 23, 2017
@jnohlgard jnohlgard added the Area: fs Area: File systems label Feb 19, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area: fs Area: File systems Area: pkg Area: External package ports CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR Discussion: RFC The issue/PR is used as a discussion starting point about the item of the issue/PR Type: new feature The issue requests / The PR implemements a new feature for RIOT

Projects

None yet

Development

Successfully merging this pull request may close these issues.