ESPFlash: An Arduino Library for Storing Data in the ESP Filesystem

27/12/2022 UPDATE: I have finally managed to fix a few annoying bugs with this library and release a new version. Hopefully it continues to get some use out there!

SPIFFS (or SPI Flash File System) is very cool. It is an Open Source Library intended for SPI NOR flash devices on embedded targets. Embedded targets like the ESP8266 and ESP32, which depending on the model can have upward of 3 megabytes of NOR Flash storage available. The cool thing about SPIFFS is that it makes using this flash storage more intuitive with a simple filesystem type interface where you do not have to think about the various intricacies involved.

This being said, it only reduces the complexity to a certain extent. Anyone who has spent some time using the library would have inevitably lost time with simple problems such as keeping filenames less than 32 characters in length, casting data to confine to the 8 bit nature of saving data to NOR Flash memory, and developing different SPIFFS handling functions for different kinds of arrayed data that you want to save.

It seems like a classic use case where generic programming with templates could reduce the complexity of SPIFFS usage by removing the need to constantly cast data types and provide the size of said data to store it in SPIFFS. After making this observation I was surprised to learn that no Arduino library existed that simplified SPIFFS usage, and thus the idea for ESPFlash was born.

The ESPFlash Library

ESPFlash is an abstraction layer that simplifies the storing of vectorised data in the filesystem on the ESP8266 and ESP32. It was created to make SPIFFS usage simple and easier to understand. It can be found on GitHub here. It can also be found using Arduino’s Library Manager, and available when searched for when using the Arduino IDE. Simple examples also exist to help get people started.

ESPFlash Specification

After some consideration I came up with simple specification that ESPFlash should fulfill. The specification requirements include:

  • Simple template based interface to store and retrieve generic vectorised data in flash memory using SPIFFS.
  • Automatically start SPIFFS if it has not already been started.
  • Automatically truncate filenames that are over 32 characters in length.
  • Keep running count of the number of “elements” stored in a file.
  • Ability to overwrite elements.
  • Ability to append elements.
  • Ability to get single elements.
  • Ability to get a multiple number of elements.
  • Ability to clear elements.
  • Ability to add elements that are stored in flash with PROGMEM.

In addition to this, I wanted to develop two extra modules to further simplify to common use cases. The purpose of these modules are the following:

  • Simple SPIFFS based integer counter
  • Simple SPIFFS based string storage

How ESPFlash Simplifies SPIFFS Usage

Here are a couple of examples of how ESPFlash simplifies SPIFFS usage. It does not show the full functionality of ESPFlash. For the full API, you can checkout the GitHub repository.

Creating a file with a filename of “/intExample” and storing a single integer in the file with error checking.

ESPFlash

int testData = 10;
ESPFlash<int> intExample("/intExample")
bool success = intExample.set(10);

SPIFFS

int testData = 10;
bool success = false; 
uint32_t bytesWritten = 0;
File file;
SPIFFS.begin();
file = SPIFFS.open("/intExample", "w");
if(file)
{
  int* testPointer = &testData;
  bytesWritten = file.write((uint8_t*)testPointer, sizeof(testData));
  if(bytesWritten == sizeof(int))
  {
    success = true;
  }
}
file.close();

Creating a file with a filename of “/charArrayExample” and store 10 chars in the file with error checking.

ESPFlash

char testData[10] = {'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j'};
ESPFlash<char> charExample("/charArrayExample");
bool success = charExample.setElements(testData, sizeof(testData));

SPIFFS

char testData[10] = {'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j'};
bool success = false; 
uint32_t bytesWritten = 0;
File file;
uint32_t elementsSizeInBytes = 0;
SPIFFS.begin();  
file = SPIFFS.open("/charArrayExample", "w");
if(file)
{
  elementsSizeInBytes = sizeof(char)*sizeof(testData);
  bytesWritten = file.write((uint8_t*)testData, elementsSizeInBytes);
  if(bytesWritten == elementsSizeInBytes)
  {
    success = true;
  }
}
file.close();

Open a file with a filename of “/lengthExample” and get the number of elements stored in the file with error checking.

ESPFlash

ESPFlash<double> lengthExample("/lengthExample");
uint32_t numberOfElements = lengthExample.length();

SPIFFS

File file;
uint32_t sizeInBytes = 0;
uint32_t sizeInElements = 0;
SPIFFS.begin();  
file = SPIFFS.open("/lengthExample", "r");
if(file)
{      
  sizeInBytes = file.size();
  sizeInElements = sizeInBytes/sizeof(double);
}
file.close();

Open a file with a filename of “/floatArrayExample” and get the last 5 float elements stored in the file with error checking.

ESPFlash

float testGet[5];
ESPFlash<float> floatExample("/floatArrayExample");
bool success = floatExample.getBackElements(testGet, sizeof(testGet));

SPIFFS

float testGet[5];
File file;
uint32_t fileSizeInBytes = 0;
uint32_t fileSizeInElements = 0;
uint32_t firstElementIndex = 0;
uint32_t bytesRead = 0;
uint32_t getArraySizeInElements = sizeof(testGet);
uint32_t getArraySizeInBytes = sizeof(float)*sizeof(testGet);
bool success = false;
SPIFFS.begin();  
file = SPIFFS.open("/floatArrayExample", "r");
if(file)
{
  fileSizeInBytes = file.size();
  fileSizeInElements = fileSizeInBytes/sizeof(float);
  if(getArraySizeInElements <= fileSizeInElements)
  {
    firstElementIndex = fileSizeInBytes  - getArraySizeInBytes;
    file.seek(firstElementIndex, SeekSet);
    bytesRead = file.read((uint8_t*)testGet, getArraySizeInBytes);
    file.close();
    if(bytesRead == getArraySizeInBytes)
    {
      success = true;
    }
  }
}
file.close();

Truncate a filename in excess of 32 characters so it can be used with SPIFFS.

ESPFlash

/* Filenames are automatically truncated if in excess of 32 characters */
/* The file extension is preserved */
ESPFlash<float> floatExample("/thisFilenameIsLargerThan32Characters.txt");

SPIFFS

const char* filename = "/thisFilenameIsLargerThan32Characters.txt";
char filenameBuffer[32];
if(strlen(fileName) < 32)
{
  strcpy(filenameBuffer, fileName);
}
else
{
  strncpy(this->fileName, fileName, 27);
  char* pch = strrchr(fileName, '.');
  strcpy(filenameBuffer+27, pch);
}

Conclusion

ESPFlash is far simpler than using the SPIFFS library directly. It also reduces code footprint, and makes SPIFFS error detection easy. It is hoped that this library will be useful to someone down the track!

12 thoughts on “ESPFlash: An Arduino Library for Storing Data in the ESP Filesystem

  1. Krishan Kumar

    Hi,
    I am using the Arduino nano 33 BLE Sense Board. I have configured all successfully and got the results for temperature and humidity on serial monitor.
    Now I want to show this on web browser and using the approach of making a web server to show the temperature and humidity data. I have ethernet shield2 but it is not compatible with my board, I can switch to use esp8266 chip.

    Can you please suggest on it to make the web server with Arduino Nano 33 BLE Sense Board.

    Any reply will be appreciated!.

    Regards
    Krishan Kumar

    Reply
    1. dale-blog Post author

      Hi Krishan,

      You should be able to implement a web server on the Nano 33 BLE using any Arduino ethernet/wifi shield with the SPI port and associated arduino libraries.

      I have never tried it, so won’t comment further.

  2. Vladimir Skorepa

    Hi Dale,
    looks like useful way how to store data in ESP. ;-) But I encounter difficulties, trying to explain:
    running on ESP32 Dev Module (library version 1.0.4) with Arduino 1.8.13
    Building example https://github.com/DaleGia/ESPFlash/blob/master/examples/millisArray/millisArray.ino

    When building the example, it cant build with this output:
    In constructor ‘ESPFlash::ESPFlash()’:
    error: ‘SPIFFS’ was not declared in this scope
    SPIFFS.begin();

    There are many error messages with the same message: SPIFFS library isnt included.

    When I include the SPIFFS to example (before ESPFlash), then it can be build, but the ESP32 resets in loop with this output:

    Rebooting…
    ets Jun 8 2016 00:22:57

    rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
    configsip: 0, SPIWP:0xee
    clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
    mode:DIO, clock div:1
    load:0x3fff0018,len:4
    load:0x3fff001c,len:1216
    ho 0 tail 12 room 4
    load:0x40078000,len:9720
    ho 0 tail 12 room 4
    load:0x40080400,len:6352
    entry 0x400806b8
    /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/queue.c:621 (xQueueTakeMutexRecursive)- assert failed!
    abort() was called at PC 0x4008805f on core 0

    Backtrace: 0x4008b560:0x3ffe38f0 0x4008b78d:0x3ffe3910 0x4008805f:0x3ffe3930 0x400ef92c:0x3ffe3950 0x40086713:0x3ffe3970 0x40086159:0x3ffe3990 0x400863d3:0x3ffe39e0 0x400ef565:0x3ffe3a10 0x400ef726:0x3ffe3a40 0x400ef777:0x3ffe3a60 0x400d65b2:0x3ffe3a80 0x400d720c:0x3ffe3ac0 0x400d1189:0x3ffe3b80 0x400d1119:0x3ffe3bb0 0x400d4e13:0x3ffe3bd0 0x400818b5:0x3ffe3bf0 0x40081a91:0x3ffe3c20 0x4007906f:0x3ffe3c40 0x400790d5:0x3ffe3c70 0x400790e0:0x3ffe3ca0 0x400792a9:0x3ffe3cc0 0x400806ea:0x3ffe3df0 0x40007c31:0x3ffe3eb0 0x4000073d:0x3ffe3f20

    Why I have to manually include the SPIFFS library? Why it dont work then?

    Cheers, Vladimir.

    Reply
  3. Vladimir Skorepa

    Hi Dale,
    I have an update: the “powerCycleCounter.ino” works with manually included SPIFFS library and deleted the line with ESP.reset() (dont know this command).

    Cheers, Vladimir.

    Reply
    1. dale-blog Post author

      Thanks for reporting this Valdimir. It is a known issue (already reported on github) about the lack of spiffs include. I will solve this and update the library as soon as I can test the solution.

      – Dale

  4. James Costello

    Vladimir,
    Hi, I` trying to use your library to store SSID`s (as per your example) but i`m getting an error on compiling. Testing with a simpified versions i get an error.
    CODE:
    ——
    #include
    #include “ESPFlashString.h”
    #define DEFAULT_SSID “ESPFlashString Test”
    #define SSID_FILEPATH “/ssid”
    void setup()
    {
    ESPFlashString ssid(SSID_FILEPATH, DEFAULT_SSID);
    Serial.begin(115200);
    Serial.println(ssid.get());
    }
    —-
    ERROR:
    core\core.a(main.cpp.o):(.literal._Z8loopTaskPv+0x4): undefined reference to `loop()’

    core\core.a(main.cpp.o): In function `loopTask(void*)’:

    C:\Users\james\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\cores\esp32/main.cpp:17: undefined reference to `loop()’
    collect2.exe: error: ld returned 1 exit status
    exit status 1
    Error compiling for board ESP32 Dev Module.

    Reply
    1. dale-blog Post author

      Hi James,

      This error seems to be unrelated to the example, and possibly to do with your Arduino installation/setup.

      I am unsure how to solve the compilation issues.

      – Dale

  5. Ralph McCleery

    Hi Dale,
    Just experimenting with your library for use in a current project.

    I tried to make a simple sketch to create one of the variables I want to save which is an incrementing array index.

    The sketch works fine but I know its not correct because of what happened before I put the counter reset in place. I’m guessing I need to do a test to see if the file exists on the first loop and assume the keyword to use is “getFileName”? and then maybe “setFileName” when it doesn’t exist?

    Unfortunately the syntax needed eludes me?
    Any help would be greatly appreciated.
    Regards
    Ralph

    BTW I know it’s been mentioned above but to make your library work with all your examples as they are written you have to change “#include ” to “#include ” in the “ESPFlash.h” file of your library

    Reply
    1. Ralph McCleery

      those includes were meant to be FS.h to SPIFFS.h but the system stripped them out?? Sorry

    2. Ralph McCleery

      Self learning has it’s challenges but I’ve figured out what was causing the initial incorrect INT value and thought it might be worth posting to help others in the same situation.

      I discovered that if I used “esptool.py” and did an “erase_flash” followed by a simple sketch to format the “SPIFFS” partition and then load my espFLASH test sketch it returned the correct values on first boot loop. However if I just do a “SPIFFS format” to start over the first boot int value is incorrect.

      So to only need to do a SPIFFS format to start over I found the following seems to give a correct and repeatable result.

      ESPFlash espIndexVal(“/averageIndexVal”);
      if(espIndexVal.getFileName()){
      averageIndex = espIndexVal.get();
      if(averageIndex > 9){
      averageIndex = 0;
      }
      }else{
      espIndexVal.clear();
      averageIndex = 0;
      }

      Regards
      Ralph

  6. Jan

    Hi Dale!
    I just came across your espFlash-library … finding just the thing I was looking for. Great!! And thanks for sharing!
    At first try I could not compile your examples, but an include of “SPIFFS.h” just at the top of each did the job.
    In the “powerCycleCounter”-example I had to change “ESP.reset()” to “ESP.restart()” to get it work.
    Those issues have been mentioned already (a year ago or so) and so I am wondering if you are still working on your library (for example the method “setElementAt()” is still not implemented) or if you closed the project?
    I would appreciate a short answer.
    Regards,
    Jan

    Reply
    1. dale-blog Post author

      Hey Jan,

      I haven’t worked on it in a while, But I will see if I can look at this this week and fix some of the basic issues. Otherwise feel free to submit a pull request for anything and I will take a look as soon as I get a chance.

      Thanks
      Dale

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.