Monthly Archives: April 2020

ESPFlash: An Arduino Library for Storing Data in the ESP Filesystem

SPIFFS (or SPI Flash File System) is very cool. It is an Open Source Library intended for SPI NOR flash devices on embedded targets. Embedded targets like the ESP8266 and ESP32, which depending on the model can have upward of 3 megabytes of NOR Flash storage available. The cool thing about SPIFFS is that it makes using this flash storage more intuitive with a simple filesystem type interface where you do not have to think about the various intricacies involved.

This being said, it only reduces the complexity to a certain extent. Anyone who has spent some time using the library would have inevitably lost time with simple problems such as keeping filenames less than 32 characters in length, casting data to confine to the 8 bit nature of saving data to NOR Flash memory, and developing different SPIFFS handling functions for different kinds of arrayed data that you want to save.

It seems like a classic use case where generic programming with templates could reduce the complexity of SPIFFS usage by removing the need to constantly cast data types and provide the size of said data to store it in SPIFFS. After making this observation I was surprised to learn that no Arduino library existed that simplified SPIFFS usage, and thus the idea for ESPFlash was born.

The ESPFlash Library

ESPFlash is an abstraction layer that simplifies the storing of vectorised data in the filesystem on the ESP8266 and ESP32. It was created to make SPIFFS usage simple and easier to understand. It can be found on GitHub here. It can also be found using Arduino’s Library Manager, and available when searched for when using the Arduino IDE. Simple examples also exist to help get people started.

ESPFlash Specification

After some consideration I came up with simple specification that ESPFlash should fulfill. The specification requirements include:

  • Simple template based interface to store and retrieve generic vectorised data in flash memory using SPIFFS.
  • Automatically start SPIFFS if it has not already been started.
  • Automatically truncate filenames that are over 32 characters in length.
  • Keep running count of the number of “elements” stored in a file.
  • Ability to overwrite elements.
  • Ability to append elements.
  • Ability to get single elements.
  • Ability to get a multiple number of elements.
  • Ability to clear elements.
  • Ability to add elements that are stored in flash with PROGMEM.

In addition to this, I wanted to develop two extra modules to further simplify to common use cases. The purpose of these modules are the following:

  • Simple SPIFFS based integer counter
  • Simple SPIFFS based string storage

How ESPFlash Simplifies SPIFFS Usage

Here are a couple of examples of how ESPFlash simplifies SPIFFS usage. It does not show the full functionality of ESPFlash. For the full API, you can checkout the GitHub repository.

Creating a file with a filename of “/intExample” and storing a single integer in the file with error checking.

ESPFlash

int testData = 10;
ESPFlash<int> intExample("/intExample")
bool success = intExample.set(10);

SPIFFS

int testData = 10;
bool success = false; 
uint32_t bytesWritten = 0;
File file;
SPIFFS.begin();
file = SPIFFS.open("/intExample", "w");
if(file)
{
  int* testPointer = &testData;
  bytesWritten = file.write((uint8_t*)testPointer, sizeof(testData));
  if(bytesWritten == sizeof(int))
  {
    success = true;
  }
}
file.close();

Creating a file with a filename of “/charArrayExample” and store 10 chars in the file with error checking.

ESPFlash

char testData[10] = {'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j'};
ESPFlash<char> charExample("/charArrayExample");
bool success = charExample.setElements(testData, sizeof(testData));

SPIFFS

char testData[10] = {'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j'};
bool success = false; 
uint32_t bytesWritten = 0;
File file;
uint32_t elementsSizeInBytes = 0;
SPIFFS.begin();  
file = SPIFFS.open("/charArrayExample", "w");
if(file)
{
  elementsSizeInBytes = sizeof(char)*sizeof(testData);
  bytesWritten = file.write((uint8_t*)testData, elementsSizeInBytes);
  if(bytesWritten == elementsSizeInBytes)
  {
    success = true;
  }
}
file.close();

Open a file with a filename of “/lengthExample” and get the number of elements stored in the file with error checking.

ESPFlash

ESPFlash<double> lengthExample("/lengthExample");
uint32_t numberOfElements = lengthExample.length();

SPIFFS

File file;
uint32_t sizeInBytes = 0;
uint32_t sizeInElements = 0;
SPIFFS.begin();  
file = SPIFFS.open("/lengthExample", "r");
if(file)
{      
  sizeInBytes = file.size();
  sizeInElements = sizeInBytes/sizeof(double);
}
file.close();

Open a file with a filename of “/floatArrayExample” and get the last 5 float elements stored in the file with error checking.

ESPFlash

float testGet[5];
ESPFlash<float> floatExample("/floatArrayExample");
bool success = floatExample.getBackElements(testGet, sizeof(testGet));

SPIFFS

float testGet[5];
File file;
uint32_t fileSizeInBytes = 0;
uint32_t fileSizeInElements = 0;
uint32_t firstElementIndex = 0;
uint32_t bytesRead = 0;
uint32_t getArraySizeInElements = sizeof(testGet);
uint32_t getArraySizeInBytes = sizeof(float)*sizeof(testGet);
bool success = false;
SPIFFS.begin();  
file = SPIFFS.open("/floatArrayExample", "r");
if(file)
{
  fileSizeInBytes = file.size();
  fileSizeInElements = fileSizeInBytes/sizeof(float);
  if(getArraySizeInElements <= fileSizeInElements)
  {
    firstElementIndex = fileSizeInBytes  - getArraySizeInBytes;
    file.seek(firstElementIndex, SeekSet);
    bytesRead = file.read((uint8_t*)testGet, getArraySizeInBytes);
    file.close();
    if(bytesRead == getArraySizeInBytes)
    {
      success = true;
    }
  }
}
file.close();

Truncate a filename in excess of 32 characters so it can be used with SPIFFS.

ESPFlash

/* Filenames are automatically truncated if in excess of 32 characters */
/* The file extension is preserved */
ESPFlash<float> floatExample("/thisFilenameIsLargerThan32Characters.txt");

SPIFFS

const char* filename = "/thisFilenameIsLargerThan32Characters.txt";
char filenameBuffer[32];
if(strlen(fileName) < 32)
{
  strcpy(filenameBuffer, fileName);
}
else
{
  strncpy(this->fileName, fileName, 27);
  char* pch = strrchr(fileName, '.');
  strcpy(filenameBuffer+27, pch);
}

Conclusion

ESPFlash is far simpler than using the SPIFFS library directly. It also reduces code footprint, and makes SPIFFS error detection easy. It is hoped that this library will be useful to someone down the track!

ESPStringTemplate: An Arduino Library for Building Web Pages

I recently began a project where I needed to build some dead simple, HTML only static web pages during runtime with the ESP8266 to represent data stored on the filesystem. Depending on the number of files located on the filesystem various buttons, forms, and text needed to be generated. In addition I needed some templating capability to replace certain “tokens” located in the web page with other substrings. Projects with similar functionality exist usually number the name of FSBrowser.

Naturally with the popularity of the ESP8266 and Arduino, some libraries that do a pretty good job of this already exist. They are:

  • The ESPAsyncWebServer contains a simple templating engine in addition to it’s web server capability. It works by “extracting placeholder names from response text and passing it to user provided function which should return actual value to be used instead of placeholder”.
  • PageBuilder is a “HTML assembly aid” that provides a way to combine “PageElements” to a build a web page which can be stored in SPIFFS. It provides templating capability by allowing “tokens” to be tied to template handling functions. It can also be bound to a web server.
  • ESPTemplateProcessor is a templating library that reads a file from SPIFFS, processes the read file with a template handling function, then sends the result to a bound web server.
  • EspHtmlTemplateProcessor is based off ESPTemplateProcessor, although it has a few extra features.

While I certainly could have managed by using one of these libraries, I soon found the implementation resulting from their usage to be difficult to read and not very flexible. I found myself having to develop upwards of ten templating handler functions to replace tokens in a very small amount of generated HTML code. I also found the libraries that bound to web servers to be restrictive in the web server based libraries that could be used.

So with these observations I made the decision to make a dead simple string templating library that would fill the niche of my application.

ESPStringTemplate Requirements

The requirements for the ESPStringTemplate library are:

  • Ability to build web pages using string “elements”.
  • Web pages should be stored in statically allocated buffers, where elements can be added to them.
  • String elements should be able to be stored in both RAM and program flash memory (PROGMEM).
  • Templating ability on string elements using simple string literals (instead of handler functions).
  • Multiple token templating ability on string elements (without the use of handler functions).

The ESPStringTemplate Library

The ESPStringTemplate Library can be found on GitHub here. It is also a part of Arduino’s Library Manager, and can be found using the Arduino IDE when it is searched for. The GitHub README gives a good outline of the usage of the library. Simple examples also exist to help get people started. I hope this library can be useful for someone else down the track!

Just as a note, there is actually nothing stopping the use of ESPStringTemplate on any microcontroller!

ESPStringTemplate Examples

These examples are taken straight from the ESPStringTemplate GitHub page, but are also included here.

Simple example – Using string literals to build a page in a statically allocated buffer.

static char buffer[200];
ESPStringTemplate webpage(buffer, sizeof(buffer));
webpage.add("Hi!<br><br>");
webpage.add("This is an example of the ESPStringTemplate Library.<br><br>");
webpage.add("These strings are stored in RAM, and are added to the provided statically allocated buffer.<br><br>");

Using string literals stored in flash (program memory) to build a web page.

static char buffer[200];
ESPStringTemplate webpage(buffer, sizeof(buffer));
webpage.add_P(PSTR("This is an example of the ESPStringTemplate Library.<br><br>");
webpage.add_P(PSTR("This string is stored in flash using PROGMEM, and are added to the provided statically allocated buffer."));

Using reusable HTML elements with a simple token replacement.

static const char _PAGEHEADER[] PROGMEM = "<html><body>";
static const char _CONTENT[]    PROGMEM = "%CONTENT%";
static const char _PAGEFOOTER[] PROGMEM = "</body></html>";

static char buffer[200];
ESPStringTemplate webpage(buffer, sizeof(buffer));
webpage.add_P(_PAGEHEADER);
webpage.add_P(_CONTENT, "%CONTENT%", "TEST CONTENT");
webpage.add_P(_PAGEFOOTER);

Using reusable HTML elements with a multiple token replacement.

static const char _PAGEHEADER[] PROGMEM = "<html><body>";
static const char _CONTENT[]   PROGMEM = "%CONTENTA% and %CONTENTB% and %CONTENTC%";
static const char _PAGEFOOTER[] PROGMEM = "</body></html>";

static char buffer[200];
ESPStringTemplate webpage(buffer, sizeof(buffer));
TokenStringPair pair[3];
webpage.add_P(_PAGEHEADER);
pair[0].setPair("%CONTENTA%", "Replacing this token");
pair[1].setPair("%CONTENTB%", "this token");
pair[2].setPair("%CONTENTC%", "this last token as well...");
webpage.add_P(_CONTENT, pair, 3);
webpage.add_P(_PAGEFOOTER);

Using ESPAsyncWebServer to serve created webpage

static char buffer[200];
WiFi.softAP("ESPStringTemplate Example");

AsyncWebServer server(80);
server.on("/", HTTP_GET, [](AsyncWebServerRequest *request)
{
  ESPStringTemplate webpage(buffer, sizeof(buffer));  
  webpage.add("Hi!");
  /* Send the webpage from SPIFFS where it is stored. */
  request->send(200, "text/html", buffer);
}
server.begin();