Tag Archives: ESP8266

ESPCaptiveImagePortal: A Captive Portal That Displays Images

I hate the internet. I hate it not only because it shows us the decay of society in real-time, but also because I do not really understand it. And who can blame me? Technologically it is one giant bowl of spaghetti, and as an idiosyncratic wannabe iconoclast it is even more difficult for me to comprehend on a social level. So naturally when COVID19 hit Europe like a tonne of bricks and the internet immediately imploded I channeled this hatred hard. Now hold your horses, I am not dishing out this pseudo philosophical claptrap in a failed attempt to be topical. This has a point. Just let me get to it…

A friends birthday was coming up, and I came up with a concept for a gift that attempts to satirise not only the way people use the internet to dish out their unqualified opinion, but also the shallow way in which people deliver that opinion.

The Concept

The concept was to have a device create a WiFi access point called “CORONA VIRUS CURE INFORMATION”, and once a user connects to the access point a captive portal would automatically serve a web page that only displays a GIF telling the user to “go home”. Something a little like this, featuring yours truly.

Obviously this kind of concept could be refined to be a little more usable to the general population who are not as bitter as myself. Maybe a “CAT OF THE DAY” web portal that displays pictures of the household cats? Yes. That sounds much more palatable. Or how about including a configuration page where the user could name the access point whatever they like, and an upload page so custom images could be displayed by the captive portal? I can see it now, looking for my local public WiFi Hotspot and being inundated with “5G KILLS BABIES”, “BILL GATES KILLS BABIES”, and “LOCKDOWN KILLS BABIES”. Yes. This is it. This is what I wanted.

And so the concept for the ESPCaptiveImagePortal was born.

The Specifications

After some musing I came up with the following requirements for ESPCaptiveImagePortal:

  • Implement an WiFi access point with a web server.
  • Implement a captive portal that redirects most traffic to a web page that displays a simple image. This page should be able to display different images each time it is loaded if more than one image is available. It should also be able to display a wide range of image formats, including animated images.
  • Implement a configuration page allowing the user to upload images, delete images, and change the access point SSID. This page should have a hard coded username and password for basic authentication.
  • Implement using the ESP8266.

The ESPCaptiveImagePortal Project

The ESPCaptiveImagePortal project can be found on GitHub here. In addition to the ESP8266 arduino core it requires a few Arduino libraries to compile. This includes ESPAsyncTCP, ESPAsyncWebServer, DNSServer, ESPStringTemplate, and ESPFlash. Most of these libraries will be able to be installed using the Arduino IDE through Library Manager, but if you are having trouble, they can also be installed by downloading the code off github directly and installing them manually.

A Working Example

Here is an example of how the ESPCaptiveImagePortal works using the CAT OF THE DAY example I mentioned a little earlier.

CAT OF THE DAY example

Configuration

Obviously we don’t want just anyone who has access to the access point to be able to upload images, so ESPCaptiveImagePortal relies on three forms of security:

  1. The connected user knows the ESP8266 IP address.
  2. The connected user knows the configuration page URI (/supersecretpage by default)
  3. The connected user knows the hardcoded username and password.

It’s important to note the username and password is implemented using basic HTTP authentication which is incredibly weak, and allows anyone sniffing the network to grab the credentials in plain text. Obviously from a security point of view this is not ideal, but I considered it sufficient for such a project.

To access the configuration page, you have to know the IP address of the ESP8266, which is hardcoded to 192.168.4.1 by default. The URI for the configuration page is also hardcoded to “/supersecretpage” by default. So if those defaults are used, it can be accessed by using any browser with “192.168.4.1/supersecretpage” in the address bar.

The authentication that allows access ends up looking a bit like this. Default values are hardcoded in the project that can be changed very easily.

The below image shows the configuration page. It displays the total number of WiFi connections that have been made to the device as well as the following configuration options:

  • Ability to delete uploaded images
  • Ability to upload new images
  • Ability to change WiFi SSID

The below animation shows a sped up example from start to finish of the ESPCaptiveImagePortal configuration, including getting to the configuration page, uploading an image, changing the SSID, and observing the result all through a browser on a smartphone.

A Note On Image Upload Size and Speed

One last final note is that image upload speed is slow due to the use of internal flash memory (SPIFFS) on the ESP8266. The available space on the filesystem in which images are stored is also very constrained, although this will differ depending on the ESP8266 module that is used. It is highly recommended that the image size does not exceed 500Kb in size. If the image is a non-animated image, an image size of ~100Kb is recommended, which results in a negligible upload and download time while still rendering useful results.

Conclusion

It’s hoped that someone will get a kick out of this project. I had a lot of fun developing it, and learnt a lot about the ESP8266 on the way. I will follow up this post with a technical post that will discuss how I achieved all of the specified features, some thoughts on the ESP8266, and what I would do differently in the future, so watch this space!

ESPFlash: An Arduino Library for Storing Data in the ESP Filesystem

SPIFFS (or SPI Flash File System) is very cool. It is an Open Source Library intended for SPI NOR flash devices on embedded targets. Embedded targets like the ESP8266 and ESP32, which depending on the model can have upward of 3 megabytes of NOR Flash storage available. The cool thing about SPIFFS is that it makes using this flash storage more intuitive with a simple filesystem type interface where you do not have to think about the various intricacies involved.

This being said, it only reduces the complexity to a certain extent. Anyone who has spent some time using the library would have inevitably lost time with simple problems such as keeping filenames less than 32 characters in length, casting data to confine to the 8 bit nature of saving data to NOR Flash memory, and developing different SPIFFS handling functions for different kinds of arrayed data that you want to save.

It seems like a classic use case where generic programming with templates could reduce the complexity of SPIFFS usage by removing the need to constantly cast data types and provide the size of said data to store it in SPIFFS. After making this observation I was surprised to learn that no Arduino library existed that simplified SPIFFS usage, and thus the idea for ESPFlash was born.

The ESPFlash Library

ESPFlash is an abstraction layer that simplifies the storing of vectorised data in the filesystem on the ESP8266 and ESP32. It was created to make SPIFFS usage simple and easier to understand. It can be found on GitHub here. It can also be found using Arduino’s Library Manager, and available when searched for when using the Arduino IDE. Simple examples also exist to help get people started.

ESPFlash Specification

After some consideration I came up with simple specification that ESPFlash should fulfill. The specification requirements include:

  • Simple template based interface to store and retrieve generic vectorised data in flash memory using SPIFFS.
  • Automatically start SPIFFS if it has not already been started.
  • Automatically truncate filenames that are over 32 characters in length.
  • Keep running count of the number of “elements” stored in a file.
  • Ability to overwrite elements.
  • Ability to append elements.
  • Ability to get single elements.
  • Ability to get a multiple number of elements.
  • Ability to clear elements.
  • Ability to add elements that are stored in flash with PROGMEM.

In addition to this, I wanted to develop two extra modules to further simplify to common use cases. The purpose of these modules are the following:

  • Simple SPIFFS based integer counter
  • Simple SPIFFS based string storage

How ESPFlash Simplifies SPIFFS Usage

Here are a couple of examples of how ESPFlash simplifies SPIFFS usage. It does not show the full functionality of ESPFlash. For the full API, you can checkout the GitHub repository.

Creating a file with a filename of “/intExample” and storing a single integer in the file with error checking.

ESPFlash

int testData = 10;
ESPFlash<int> intExample("/intExample")
bool success = intExample.set(10);

SPIFFS

int testData = 10;
bool success = false; 
uint32_t bytesWritten = 0;
File file;
SPIFFS.begin();
file = SPIFFS.open("/intExample", "w");
if(file)
{
  int* testPointer = &testData;
  bytesWritten = file.write((uint8_t*)testPointer, sizeof(testData));
  if(bytesWritten == sizeof(int))
  {
    success = true;
  }
}
file.close();

Creating a file with a filename of “/charArrayExample” and store 10 chars in the file with error checking.

ESPFlash

char testData[10] = {'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j'};
ESPFlash<char> charExample("/charArrayExample");
bool success = charExample.setElements(testData, sizeof(testData));

SPIFFS

char testData[10] = {'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j'};
bool success = false; 
uint32_t bytesWritten = 0;
File file;
uint32_t elementsSizeInBytes = 0;
SPIFFS.begin();  
file = SPIFFS.open("/charArrayExample", "w");
if(file)
{
  elementsSizeInBytes = sizeof(char)*sizeof(testData);
  bytesWritten = file.write((uint8_t*)testData, elementsSizeInBytes);
  if(bytesWritten == elementsSizeInBytes)
  {
    success = true;
  }
}
file.close();

Open a file with a filename of “/lengthExample” and get the number of elements stored in the file with error checking.

ESPFlash

ESPFlash<double> lengthExample("/lengthExample");
uint32_t numberOfElements = lengthExample.length();

SPIFFS

File file;
uint32_t sizeInBytes = 0;
uint32_t sizeInElements = 0;
SPIFFS.begin();  
file = SPIFFS.open("/lengthExample", "r");
if(file)
{      
  sizeInBytes = file.size();
  sizeInElements = sizeInBytes/sizeof(double);
}
file.close();

Open a file with a filename of “/floatArrayExample” and get the last 5 float elements stored in the file with error checking.

ESPFlash

float testGet[5];
ESPFlash<float> floatExample("/floatArrayExample");
bool success = floatExample.getBackElements(testGet, sizeof(testGet));

SPIFFS

float testGet[5];
File file;
uint32_t fileSizeInBytes = 0;
uint32_t fileSizeInElements = 0;
uint32_t firstElementIndex = 0;
uint32_t bytesRead = 0;
uint32_t getArraySizeInElements = sizeof(testGet);
uint32_t getArraySizeInBytes = sizeof(float)*sizeof(testGet);
bool success = false;
SPIFFS.begin();  
file = SPIFFS.open("/floatArrayExample", "r");
if(file)
{
  fileSizeInBytes = file.size();
  fileSizeInElements = fileSizeInBytes/sizeof(float);
  if(getArraySizeInElements <= fileSizeInElements)
  {
    firstElementIndex = fileSizeInBytes  - getArraySizeInBytes;
    file.seek(firstElementIndex, SeekSet);
    bytesRead = file.read((uint8_t*)testGet, getArraySizeInBytes);
    file.close();
    if(bytesRead == getArraySizeInBytes)
    {
      success = true;
    }
  }
}
file.close();

Truncate a filename in excess of 32 characters so it can be used with SPIFFS.

ESPFlash

/* Filenames are automatically truncated if in excess of 32 characters */
/* The file extension is preserved */
ESPFlash<float> floatExample("/thisFilenameIsLargerThan32Characters.txt");

SPIFFS

const char* filename = "/thisFilenameIsLargerThan32Characters.txt";
char filenameBuffer[32];
if(strlen(fileName) < 32)
{
  strcpy(filenameBuffer, fileName);
}
else
{
  strncpy(this->fileName, fileName, 27);
  char* pch = strrchr(fileName, '.');
  strcpy(filenameBuffer+27, pch);
}

Conclusion

ESPFlash is far simpler than using the SPIFFS library directly. It also reduces code footprint, and makes SPIFFS error detection easy. It is hoped that this library will be useful to someone down the track!

ESPStringTemplate: An Arduino Library for Building Web Pages

I recently began a project where I needed to build some dead simple, HTML only static web pages during runtime with the ESP8266 to represent data stored on the filesystem. Depending on the number of files located on the filesystem various buttons, forms, and text needed to be generated. In addition I needed some templating capability to replace certain “tokens” located in the web page with other substrings. Projects with similar functionality exist usually number the name of FSBrowser.

Naturally with the popularity of the ESP8266 and Arduino, some libraries that do a pretty good job of this already exist. They are:

  • The ESPAsyncWebServer contains a simple templating engine in addition to it’s web server capability. It works by “extracting placeholder names from response text and passing it to user provided function which should return actual value to be used instead of placeholder”.
  • PageBuilder is a “HTML assembly aid” that provides a way to combine “PageElements” to a build a web page which can be stored in SPIFFS. It provides templating capability by allowing “tokens” to be tied to template handling functions. It can also be bound to a web server.
  • ESPTemplateProcessor is a templating library that reads a file from SPIFFS, processes the read file with a template handling function, then sends the result to a bound web server.
  • EspHtmlTemplateProcessor is based off ESPTemplateProcessor, although it has a few extra features.

While I certainly could have managed by using one of these libraries, I soon found the implementation resulting from their usage to be difficult to read and not very flexible. I found myself having to develop upwards of ten templating handler functions to replace tokens in a very small amount of generated HTML code. I also found the libraries that bound to web servers to be restrictive in the web server based libraries that could be used.

So with these observations I made the decision to make a dead simple string templating library that would fill the niche of my application.

ESPStringTemplate Requirements

The requirements for the ESPStringTemplate library are:

  • Ability to build web pages using string “elements”.
  • Web pages should be stored in statically allocated buffers, where elements can be added to them.
  • String elements should be able to be stored in both RAM and program flash memory (PROGMEM).
  • Templating ability on string elements using simple string literals (instead of handler functions).
  • Multiple token templating ability on string elements (without the use of handler functions).

The ESPStringTemplate Library

The ESPStringTemplate Library can be found on GitHub here. It is also a part of Arduino’s Library Manager, and can be found using the Arduino IDE when it is searched for. The GitHub README gives a good outline of the usage of the library. Simple examples also exist to help get people started. I hope this library can be useful for someone else down the track!

Just as a note, there is actually nothing stopping the use of ESPStringTemplate on any microcontroller!

ESPStringTemplate Examples

These examples are taken straight from the ESPStringTemplate GitHub page, but are also included here.

Simple example – Using string literals to build a page in a statically allocated buffer.

static char buffer[200];
ESPStringTemplate webpage(buffer, sizeof(buffer));
webpage.add("Hi!<br><br>");
webpage.add("This is an example of the ESPStringTemplate Library.<br><br>");
webpage.add("These strings are stored in RAM, and are added to the provided statically allocated buffer.<br><br>");

Using string literals stored in flash (program memory) to build a web page.

static char buffer[200];
ESPStringTemplate webpage(buffer, sizeof(buffer));
webpage.add_P(PSTR("This is an example of the ESPStringTemplate Library.<br><br>");
webpage.add_P(PSTR("This string is stored in flash using PROGMEM, and are added to the provided statically allocated buffer."));

Using reusable HTML elements with a simple token replacement.

static const char _PAGEHEADER[] PROGMEM = "<html><body>";
static const char _CONTENT[]    PROGMEM = "%CONTENT%";
static const char _PAGEFOOTER[] PROGMEM = "</body></html>";

static char buffer[200];
ESPStringTemplate webpage(buffer, sizeof(buffer));
webpage.add_P(_PAGEHEADER);
webpage.add_P(_CONTENT, "%CONTENT%", "TEST CONTENT");
webpage.add_P(_PAGEFOOTER);

Using reusable HTML elements with a multiple token replacement.

static const char _PAGEHEADER[] PROGMEM = "<html><body>";
static const char _CONTENT[]   PROGMEM = "%CONTENTA% and %CONTENTB% and %CONTENTC%";
static const char _PAGEFOOTER[] PROGMEM = "</body></html>";

static char buffer[200];
ESPStringTemplate webpage(buffer, sizeof(buffer));
TokenStringPair pair[3];
webpage.add_P(_PAGEHEADER);
pair[0].setPair("%CONTENTA%", "Replacing this token");
pair[1].setPair("%CONTENTB%", "this token");
pair[2].setPair("%CONTENTC%", "this last token as well...");
webpage.add_P(_CONTENT, pair, 3);
webpage.add_P(_PAGEFOOTER);

Using ESPAsyncWebServer to serve created webpage

static char buffer[200];
WiFi.softAP("ESPStringTemplate Example");

AsyncWebServer server(80);
server.on("/", HTTP_GET, [](AsyncWebServerRequest *request)
{
  ESPStringTemplate webpage(buffer, sizeof(buffer));  
  webpage.add("Hi!");
  /* Send the webpage from SPIFFS where it is stored. */
  request->send(200, "text/html", buffer);
}
server.begin();