RemoteSensorParameterCalculator: A Simple Web Application For Calculating Remote Sensor Parameters

1 Reply

For a while now I have had to calculate various camera sensor related parameters for remote sensing applications. Usually this is done using excel. But I have recently been getting some joy out of making super simple single page web applications that solve simple problems. With a small amount of code, you can create applications that will work on basically any device in the world. As someone who spends their days cursing over porting code for embedded devices and different operating systems, I can really see the beauty in this. So with this in mind, RemoteSensorParameterCalculator was born. The project is open source and can be found here. Feel free to use it where ever, just give me attribution.

Jetson AGX Xavier Developer Kit: Tips and Tricks to Save You Time and Sanity

2 Replies

How cool are the Nvidia Jetson series of “Advanced AI Embedded Systems”! They pack all kinds of potential for machine vision applications, the features of which I won’t go into here. Mainly because I cannot even understand most of them. But hey, AI sounds pretty cool. For me, Tensor Cores conjure up images of Tony Starks nuclear power source. While CUDA cores sound really cute. ARM must be because they cost an arm and a leg, amirite?

But do you know what does not sound cool and cute? Releasing a software update that renders the devices GPIO that is available on the units 40 pin expansion header useless. Who uses GPIO anyway. It is not like this computer is marketed as an embedded system or anything. Why test it before a software release? It is not like Nvidia is a large company with massive resources retailing a small computer system for more than $2000AUD which if you ask me, IS the price of an arm AND A LEG!

Yes I am a little bitter…

Anyway I am writing this because I lost three days of my life trying to get the GPIO on my Jetson AGX Xavier Developer Kit working only to find out from a forum post that “There is a known issue in jp5.0 GPIO” and that it is not currently possible to use Linux sysfs and Jetson.GPIO. Supposedly you can get it working by manipulating the GPIO register addresses directly, but good luck with that. The Xavier SoC Technical Reference Manual is horrendous, and as far as I can tell it is missing information that would allow the development of a bare metal Hardware Abstraction Layer using mmap (some people have had success doing this on the Nvidia Jetson Nano). Hopefully by the time you are reading this the next release of JetPack is out and this issue no longer exists. But currently the issue does exist.

Here are some tips I have compiled that might save you some time getting the thing up and running:

  • Do not use JetPack 5 or 5.0.1. Give JetPack 4.6.1 a go. GPIO functionality does not yet work with JetPack 5, and most likely other peripherals do not work as well. I sense that there is a lot wrong with JetPack 5 on the Jetson AGX Xavier and it should simply not be used yet.
  • You can only setup and flash the Jetson AGX Xavier with an Ubuntu/Red Hat/CentOS operating system as that is all that is supported by Nvidia SDK Manager. Plan accordingly.
  • If you try to use a virtual machine (like me) to run SDK Manager, use VMWare. Virtual Box seems to not work when flashing the Jetsons. Possibly due to the way the USB device dropouts are handled by Virtual Box. I didn’t give Docker a go, it is probably the better option.
  • Make sure you use an Ubuntu 18 LTS operating system to install SDK Manager flash the Jetson AGX Xavier because because a newer Ubuntu LTS cannot install older JetPack versions on the Jetson AGX Xavier for some reason (SDK Manager greys out the options). Disregard this if you plan on using JetPack 5.
  • Make sure your Ubuntu operating system has around 70GB of disk space (or it can be expanded in the case of a virtual machine). SDK Manager needs a lot of storage space to do it’s thing. I had to recreate my virtual machine twice due to this oversight.
  • It can be a pain sometimes to detect the Xavier with Nvidia SDK manager when using VMWare. I had success by plugging into the USBC port on the Jetson and using USB2.0 compatibility in VMWare.

The Arduino Nano 33 BLE Sense Sensor Library You Have Been Waiting For

15 Replies

Quick Links

UPDATE 21/03/2023

Nano33BLESensor has a new release with version 1.1.0 now available. It solves some issues related to MbedOS and dependency version issues.

The Arduino Nano 33 BLE Sense was released in July 2019 and was a real step up for Arduino! Using the Arm Cortex-M4F based nRF52840, the Nano 33 BLE’s arrival made the days of Atmel based 8-bit microcontrollers seem numbered. With the proliferation of dirt cheap Arduino clones this was probably the result of Arduino seeing the writing on the wall that keeping it simple was probably not going to bring in the dough anymore. In the end the sheer amounts of fairly useless Arduino projects seem a bit dull these days, and with more complex offerings such as the Nano 33 series, a new era of seriously cool projects may be on the horizon.

Previously in The Hacky Super Loop Arduino Nano 33 BLE Sense Example You Have Been Waiting For I attempted to address the lack of meaningful working examples for this board. The interfaces for each sensor are not always ideal, and no concrete examples are widely available that enable uses to utilise Mbed OS to gather sensor values.

The ability to use Mbed OS with Arduino is a real step up for Arduino. Finally, a reasonably performant and relatively deterministic way of arranging our more complex Arduino projects! In attempt to simplify the collection of sensor data and at the same time utilise the power of Mbed OS, Nano33BLESensor was born.

The Nano33BLESensor Library

Nano33BLESensor leverages Mbed OS to automatically place sensor measurements in a ring buffer that can be integrated into programs in a simple manner. This means Nano33BLESensor takes care of placing measurements in to the buffer “in the background”, and your program can retrieve them from the buffer at a later time when your program has time. It can be found on GitHub here. It can also be found using Arduino’s Library Manager, and available when searched for when using the Arduino IDE. Simple examples also exist to help get people started.

Nano33BLESensor Features

  • Class implementation with common interface for the following sensor measurements
    • 3-axis Accelerometer
    • 3-axis Gyroscope
    • 3-axis Magnetic
    • RMS Microphone
    • Barometric Pressure
    • Temperature (with humidity)
    • Proximity
    • RGBC Colour
    • Gesture
  • Mbed OS usage, allowing easy integration with programs.
  • Ring buffer usage, allowing the softening of time constraints in regard to the reading sensor measurements.
  • Excellent examples for all sensors designed for BLE and Serial Plotter that help you to get started.

In the end, the result of this is:

  • Super simple initialisation of on board sensors.
  • No code required beyond initialisation for collection of sensor data.
  • Super simple usage of sensor data.
  • Common interface among different sensors.
  • Using Mbed OS effectively makes the reading of sensor measurements happen “in the background”, and keeps it out of the main program loop.

Nano33BLESensor Examples

The Nano33BLESensor Library comes with a series of powerful examples that include sensor measurement data output using both Serial and Bluetooth. It is possible to view this data at the same time using Arduino’s Serial Plotter and a Bluetooth packet analysis tool such as Bluetooth LE Explorer. Some of the examples available on the GitHub repository are as follows:

Below are the results of a few of those examples.

Accelerometer Example

Gyroscope Example

Entire IMU Sensor Example

In order to view the data in a visually acceptable way, Bluetooth LE Explorer must be configured in a specific way. The below animation shows this setup.

How Nano33BLESensor Simplifies Sensor Usage

Here are a couple of examples of how Nano33BLESensor simplifies the collection of sensor data with the Nano 33 BLE Sense. For the full API, you can checkout the GitHub repository.

Initialising the on board accelerometer, reading data, and printing it.

Arduino_LSM9DS1

#include <Arduino_LSM9DS1.h>

float accelerometerX, accelerometerY, accelerometerZ;
void setup()
{
  IMU.begin()
}

void loop()
{
  if(IMU.accelerationAvailable())
  {
    IMU.readAcceleration(accelerometerX, accelerometerY, accelerometerZ);
    Serial.printf("%f,%f,%f\r\n", accelerometerX, accelerometerY, accelerometerZ);
  }
}

Nano33BLESensor

#include "Nano33BLEAccelerometer.h"

Nano33BLEAccelerometerData accelerometerData;
void setup()
{
  Accelerometer.begin()
}

void loop()
{ 
  if(Accelerometer.pop(accelerometerData))
  {
     Serial.printf("%f,%f,%f\r\n", accelerometerData.x, accelerometerData.y, accelerometerData.z);
  }
}

Initialising the on board colour sensor, reading data, and printing it.

Arduino_APDS9960 

#include <Arduino_APDS9960.h>

int colourR, colourG, colourB, colourC;
void setup()
{
  APDS.setGestureSensitivity(50);
  APDS.begin();
  APDS.setLEDBoost(0);
}
void loop()
{
  if (APDS.colorAvailable())
  {
    APDS.readColor(colourR, colourG, colourB, colourC);
    Serial.printf("%d,%d,%d,%d\r\n", colourR, colourG, colourB, colourC);
  }
  delay(1000);
}

Nano33BLESensor

#include "Nano33BLEColour.h"

Nano33BLEColourData colourData;
void setup()
{
  Colour.begin()
}

void loop()
{ 
  if(Colour.pop(colourData))
  {
    Serial.printf( "%d,%d,%d,%d\r\n", colourData.r, colourData.g, colourData.b, colourData.c);
  }
}

Initialising the on board temperature sensor, reading data, and printing it

Arduino_HTS221

#include <Arduino_HTS221.h>

float temperature, humidity;
void setup()
{
  HTS.begin();
}
void loop()
{
  temperature = HTS.readTemperature();
  humidity = HTS.readHumidity();
  Serial.printf("%f, %f\r\n", temperature, humidity);
  delay(1000);
}

Arduino33BLESensor

#include "Nano33BLETemperature.h"

Nano33BLETemperatureData temperatureData;
void setup()
{
  Temperature.begin()
}

void loop()
{ 
  if(Temperature.pop(temperatureData))
  {
    Serial.printf("%f,%f\r\n", temperatureData.temperatureCelsius, temperatureData.humidity);
  }
}

Conclusion

Nano33BLESensor enables some interesting possibilities with the Nano 33 BLE. It simplifies the usage of onboard sensors by using a common interface, and leverages Mbed OS to allow the softening of time constraints in regard to the reading sensor measurements. It is hoped that this library will be useful to someone down the track!

The Hacky Super Loop Arduino Nano 33 BLE Sense Example You Have Been Waiting For

1 Reply

Update: For even better examples of the non-hacky variety, check out the Nano33BLESensor Library in The Arduino Nano 33 BLE Sense Sensor Library You Have Been Waiting For.

Quick Links

The Arduino Nano 33 BLE Sense was released in July 2019 and was a real step up for Arduino! Using the Arm Cortex-M4F based nRF52840, the Nano 33 BLE’s arrival made the days of Atmel based 8-bit microcontrollers seem numbered. With the proliferation of dirt cheap Arduino clones this was probably the result of Arduino seeing the writing on the wall that keeping it simple was probably not going to bring in the dough anymore. In the end the sheer amounts of fairly useless Arduino projects seem a bit dull these days, and with more complex offerings such as the Nano 33 series, a new era of seriously cool projects may be on the horizon.

A common complaint about the Nano 33 BLE Sense is the lack of working examples available for the board. Only a simple PDM Serial Plotter example exists when you install the board package, and this seems to have caused some confusion among users who were expecting an example to exist that uses all of the available on board sensors. In fact examples do exist for all the on board sensors, however they are available only on the individual sensor library repositories.

Below is a table including each sensor, it’s component name, a brief description, and a link to the example code.

SensorNameDescriptionExample Code Link
IMU LSM9DS1 3 acceleration channels, 3 angular rate channels, 3 magnetic field channels Arduino_LSM9DS1
Microphone MP34DT05PDM omnidirectional digital microphonePDM
IR APDS9960  Gesture, light, proximityArduino_APDS9960
Barometric Pressure LPS22HB  Barometric pressureArduino_LPS22HB
Temperature/Humidity HTS221  Temperature/Humidity Arduino_HTS221

Let’s Be Honest About What We Really Wanted

We wanted a super loop that would read all the values from the sensors and print them on the serial monitor/plotter. We wanted to be able to buy this board, fire it up, and to be able to look cool to all our friends immediately. We didn’t want to have to think. In fact we didn’t want to have to do anything. It did not matter that the actual values would probably be useless, we just wanted it to be there.

Perhaps the reason this super loop example we want is so terrible is because of the fact that the Nano 33 BLE Sense can utilise Mbed OS. Finally, a reasonably performant and relatively deterministic way of arranging our more complex Arduino projects! We can finally bid farewell to the almighty super loop!

Alas, I understand the RTOS can be a new and scary venture for some. And having just got my Nano 33 BLE sense, I decided to create a simple example program that can display all the available sensor data on serial plotter to check with the sensors are at least powered on and outputting semi-meaningful data without any of the BLE functionality.

The Example Project

The repository for nano-33-sense-serial-example can be found here. The data itself might be fairly useless for most applications, as it is mostly raw digital values pulled straight from the sensors which need to be converted to useful metrics before being meaningful. It plots the following raw data:

  • x/y/z acceleration data
  • x/y/z gyroscope data
  • x/y/z magnetic data
  • RMS microphone data
  • left/right/up/down gesture data
  • r/g/b light data
  • proximity data
  • Barometric pressure data
  • Temperature/Humidity data

Some of the difficulty in displaying this data in a coherent way is the different sampling frequencies in which sensor data is obtained. The table below shows these sampling frequencies.

Data TypeSampling Frequency
x/y/z acceleration data 109Hz
x/y/z gyroscope data 109Hz
x/y/z magnetic data 20Hz
16mS RMS microphone data 62.5Hz
left/right/up/down gesture data
Undefined
r/g/b light data
Undefined
proximity data
Undefined
Barometric pressure data
On request (1Hz in example)
Temperature/Humidity data On request (1Hz in example)

Because of this, all data is plotted with a frequency of 20Hz in an attempt to display the data in a way where the values will somewhat correlate to each other. There are some simple macro based configuration options available to allow some control over what data is actually outputted.

What the Output Looks Like

All Sensors

IMU Sensor

Gesture/Light/Proximity Sensor

Barometric Pressure Sensor

RMS Microphone Output

Temperature/Humidity Sensor

Conclusion

The nano-33-sense-serial-example is a great way to visualise all of the onboard sensors on the Nano 33 BLE sense. Since writing this example, I have largely replicated this project using Mbed OS in an effort to create a simple, low power alternative that can be used as the base for many projects that use the Nano 33 BLE sense. You can read about it in The Arduino Nano 33 BLE Sense Sensor Library You Have Been Waiting For.

ESPCaptiveImagePortal: A Captive Portal That Displays Images

2 Replies

I hate the internet. I hate it not only because it shows us the decay of society in real-time, but also because I do not really understand it. And who can blame me? Technologically it is one giant bowl of spaghetti, and as an idiosyncratic wannabe iconoclast it is even more difficult for me to comprehend on a social level. So naturally when COVID19 hit Europe like a tonne of bricks and the internet immediately imploded I channeled this hatred hard. Now hold your horses, I am not dishing out this pseudo philosophical claptrap in a failed attempt to be topical. This has a point. Just let me get to it…

A friends birthday was coming up, and I came up with a concept for a gift that attempts to satirise not only the way people use the internet to dish out their unqualified opinion, but also the shallow way in which people deliver that opinion.

The Concept

The concept was to have a device create a WiFi access point called “CORONA VIRUS CURE INFORMATION”, and once a user connects to the access point a captive portal would automatically serve a web page that only displays a GIF telling the user to “go home”. Something a little like this, featuring yours truly.

Obviously this kind of concept could be refined to be a little more usable to the general population who are not as bitter as myself. Maybe a “CAT OF THE DAY” web portal that displays pictures of the household cats? Yes. That sounds much more palatable. Or how about including a configuration page where the user could name the access point whatever they like, and an upload page so custom images could be displayed by the captive portal? I can see it now, looking for my local public WiFi Hotspot and being inundated with “5G KILLS BABIES”, “BILL GATES KILLS BABIES”, and “LOCKDOWN KILLS BABIES”. Yes. This is it. This is what I wanted.

And so the concept for the ESPCaptiveImagePortal was born.

The Specifications

After some musing I came up with the following requirements for ESPCaptiveImagePortal:

  • Implement an WiFi access point with a web server.
  • Implement a captive portal that redirects most traffic to a web page that displays a simple image. This page should be able to display different images each time it is loaded if more than one image is available. It should also be able to display a wide range of image formats, including animated images.
  • Implement a configuration page allowing the user to upload images, delete images, and change the access point SSID. This page should have a hard coded username and password for basic authentication.
  • Implement using the ESP8266.

The ESPCaptiveImagePortal Project

The ESPCaptiveImagePortal project can be found on GitHub here. In addition to the ESP8266 arduino core it requires a few Arduino libraries to compile. This includes ESPAsyncTCP, ESPAsyncWebServer, DNSServer, ESPStringTemplate, and ESPFlash. Most of these libraries will be able to be installed using the Arduino IDE through Library Manager, but if you are having trouble, they can also be installed by downloading the code off github directly and installing them manually.

A Working Example

Here is an example of how the ESPCaptiveImagePortal works using the CAT OF THE DAY example I mentioned a little earlier.

CAT OF THE DAY example

Configuration

Obviously we don’t want just anyone who has access to the access point to be able to upload images, so ESPCaptiveImagePortal relies on three forms of security:

  1. The connected user knows the ESP8266 IP address.
  2. The connected user knows the configuration page URI (/supersecretpage by default)
  3. The connected user knows the hardcoded username and password.

It’s important to note the username and password is implemented using basic HTTP authentication which is incredibly weak, and allows anyone sniffing the network to grab the credentials in plain text. Obviously from a security point of view this is not ideal, but I considered it sufficient for such a project.

To access the configuration page, you have to know the IP address of the ESP8266, which is hardcoded to 192.168.4.1 by default. The URI for the configuration page is also hardcoded to “/supersecretpage” by default. So if those defaults are used, it can be accessed by using any browser with “192.168.4.1/supersecretpage” in the address bar.

The authentication that allows access ends up looking a bit like this. Default values are hardcoded in the project that can be changed very easily.

The below image shows the configuration page. It displays the total number of WiFi connections that have been made to the device as well as the following configuration options:

  • Ability to delete uploaded images
  • Ability to upload new images
  • Ability to change WiFi SSID

The below animation shows a sped up example from start to finish of the ESPCaptiveImagePortal configuration, including getting to the configuration page, uploading an image, changing the SSID, and observing the result all through a browser on a smartphone.

A Note On Image Upload Size and Speed

One last final note is that image upload speed is slow due to the use of internal flash memory (SPIFFS) on the ESP8266. The available space on the filesystem in which images are stored is also very constrained, although this will differ depending on the ESP8266 module that is used. It is highly recommended that the image size does not exceed 500Kb in size. If the image is a non-animated image, an image size of ~100Kb is recommended, which results in a negligible upload and download time while still rendering useful results.

Conclusion

It’s hoped that someone will get a kick out of this project. I had a lot of fun developing it, and learnt a lot about the ESP8266 on the way. I will follow up this post with a technical post that will discuss how I achieved all of the specified features, some thoughts on the ESP8266, and what I would do differently in the future, so watch this space!

ESPFlash: An Arduino Library for Storing Data in the ESP Filesystem

12 Replies

27/12/2022 UPDATE: I have finally managed to fix a few annoying bugs with this library and release a new version. Hopefully it continues to get some use out there!

SPIFFS (or SPI Flash File System) is very cool. It is an Open Source Library intended for SPI NOR flash devices on embedded targets. Embedded targets like the ESP8266 and ESP32, which depending on the model can have upward of 3 megabytes of NOR Flash storage available. The cool thing about SPIFFS is that it makes using this flash storage more intuitive with a simple filesystem type interface where you do not have to think about the various intricacies involved.

This being said, it only reduces the complexity to a certain extent. Anyone who has spent some time using the library would have inevitably lost time with simple problems such as keeping filenames less than 32 characters in length, casting data to confine to the 8 bit nature of saving data to NOR Flash memory, and developing different SPIFFS handling functions for different kinds of arrayed data that you want to save.

It seems like a classic use case where generic programming with templates could reduce the complexity of SPIFFS usage by removing the need to constantly cast data types and provide the size of said data to store it in SPIFFS. After making this observation I was surprised to learn that no Arduino library existed that simplified SPIFFS usage, and thus the idea for ESPFlash was born.

The ESPFlash Library

ESPFlash is an abstraction layer that simplifies the storing of vectorised data in the filesystem on the ESP8266 and ESP32. It was created to make SPIFFS usage simple and easier to understand. It can be found on GitHub here. It can also be found using Arduino’s Library Manager, and available when searched for when using the Arduino IDE. Simple examples also exist to help get people started.

ESPFlash Specification

After some consideration I came up with simple specification that ESPFlash should fulfill. The specification requirements include:

  • Simple template based interface to store and retrieve generic vectorised data in flash memory using SPIFFS.
  • Automatically start SPIFFS if it has not already been started.
  • Automatically truncate filenames that are over 32 characters in length.
  • Keep running count of the number of “elements” stored in a file.
  • Ability to overwrite elements.
  • Ability to append elements.
  • Ability to get single elements.
  • Ability to get a multiple number of elements.
  • Ability to clear elements.
  • Ability to add elements that are stored in flash with PROGMEM.

In addition to this, I wanted to develop two extra modules to further simplify to common use cases. The purpose of these modules are the following:

  • Simple SPIFFS based integer counter
  • Simple SPIFFS based string storage

How ESPFlash Simplifies SPIFFS Usage

Here are a couple of examples of how ESPFlash simplifies SPIFFS usage. It does not show the full functionality of ESPFlash. For the full API, you can checkout the GitHub repository.

Creating a file with a filename of “/intExample” and storing a single integer in the file with error checking.

ESPFlash

int testData = 10;
ESPFlash<int> intExample("/intExample")
bool success = intExample.set(10);

SPIFFS

int testData = 10;
bool success = false; 
uint32_t bytesWritten = 0;
File file;
SPIFFS.begin();
file = SPIFFS.open("/intExample", "w");
if(file)
{
  int* testPointer = &testData;
  bytesWritten = file.write((uint8_t*)testPointer, sizeof(testData));
  if(bytesWritten == sizeof(int))
  {
    success = true;
  }
}
file.close();

Creating a file with a filename of “/charArrayExample” and store 10 chars in the file with error checking.

ESPFlash

char testData[10] = {'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j'};
ESPFlash<char> charExample("/charArrayExample");
bool success = charExample.setElements(testData, sizeof(testData));

SPIFFS

char testData[10] = {'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j'};
bool success = false; 
uint32_t bytesWritten = 0;
File file;
uint32_t elementsSizeInBytes = 0;
SPIFFS.begin();  
file = SPIFFS.open("/charArrayExample", "w");
if(file)
{
  elementsSizeInBytes = sizeof(char)*sizeof(testData);
  bytesWritten = file.write((uint8_t*)testData, elementsSizeInBytes);
  if(bytesWritten == elementsSizeInBytes)
  {
    success = true;
  }
}
file.close();

Open a file with a filename of “/lengthExample” and get the number of elements stored in the file with error checking.

ESPFlash

ESPFlash<double> lengthExample("/lengthExample");
uint32_t numberOfElements = lengthExample.length();

SPIFFS

File file;
uint32_t sizeInBytes = 0;
uint32_t sizeInElements = 0;
SPIFFS.begin();  
file = SPIFFS.open("/lengthExample", "r");
if(file)
{      
  sizeInBytes = file.size();
  sizeInElements = sizeInBytes/sizeof(double);
}
file.close();

Open a file with a filename of “/floatArrayExample” and get the last 5 float elements stored in the file with error checking.

ESPFlash

float testGet[5];
ESPFlash<float> floatExample("/floatArrayExample");
bool success = floatExample.getBackElements(testGet, sizeof(testGet));

SPIFFS

float testGet[5];
File file;
uint32_t fileSizeInBytes = 0;
uint32_t fileSizeInElements = 0;
uint32_t firstElementIndex = 0;
uint32_t bytesRead = 0;
uint32_t getArraySizeInElements = sizeof(testGet);
uint32_t getArraySizeInBytes = sizeof(float)*sizeof(testGet);
bool success = false;
SPIFFS.begin();  
file = SPIFFS.open("/floatArrayExample", "r");
if(file)
{
  fileSizeInBytes = file.size();
  fileSizeInElements = fileSizeInBytes/sizeof(float);
  if(getArraySizeInElements <= fileSizeInElements)
  {
    firstElementIndex = fileSizeInBytes  - getArraySizeInBytes;
    file.seek(firstElementIndex, SeekSet);
    bytesRead = file.read((uint8_t*)testGet, getArraySizeInBytes);
    file.close();
    if(bytesRead == getArraySizeInBytes)
    {
      success = true;
    }
  }
}
file.close();

Truncate a filename in excess of 32 characters so it can be used with SPIFFS.

ESPFlash

/* Filenames are automatically truncated if in excess of 32 characters */
/* The file extension is preserved */
ESPFlash<float> floatExample("/thisFilenameIsLargerThan32Characters.txt");

SPIFFS

const char* filename = "/thisFilenameIsLargerThan32Characters.txt";
char filenameBuffer[32];
if(strlen(fileName) < 32)
{
  strcpy(filenameBuffer, fileName);
}
else
{
  strncpy(this->fileName, fileName, 27);
  char* pch = strrchr(fileName, '.');
  strcpy(filenameBuffer+27, pch);
}

Conclusion

ESPFlash is far simpler than using the SPIFFS library directly. It also reduces code footprint, and makes SPIFFS error detection easy. It is hoped that this library will be useful to someone down the track!

ESPStringTemplate: An Arduino Library for Building Web Pages

Leave a reply

I recently began a project where I needed to build some dead simple, HTML only static web pages during runtime with the ESP8266 to represent data stored on the filesystem. Depending on the number of files located on the filesystem various buttons, forms, and text needed to be generated. In addition I needed some templating capability to replace certain “tokens” located in the web page with other substrings. Projects with similar functionality exist usually number the name of FSBrowser.

Naturally with the popularity of the ESP8266 and Arduino, some libraries that do a pretty good job of this already exist. They are:

  • The ESPAsyncWebServer contains a simple templating engine in addition to it’s web server capability. It works by “extracting placeholder names from response text and passing it to user provided function which should return actual value to be used instead of placeholder”.
  • PageBuilder is a “HTML assembly aid” that provides a way to combine “PageElements” to a build a web page which can be stored in SPIFFS. It provides templating capability by allowing “tokens” to be tied to template handling functions. It can also be bound to a web server.
  • ESPTemplateProcessor is a templating library that reads a file from SPIFFS, processes the read file with a template handling function, then sends the result to a bound web server.
  • EspHtmlTemplateProcessor is based off ESPTemplateProcessor, although it has a few extra features.

While I certainly could have managed by using one of these libraries, I soon found the implementation resulting from their usage to be difficult to read and not very flexible. I found myself having to develop upwards of ten templating handler functions to replace tokens in a very small amount of generated HTML code. I also found the libraries that bound to web servers to be restrictive in the web server based libraries that could be used.

So with these observations I made the decision to make a dead simple string templating library that would fill the niche of my application.

ESPStringTemplate Requirements

The requirements for the ESPStringTemplate library are:

  • Ability to build web pages using string “elements”.
  • Web pages should be stored in statically allocated buffers, where elements can be added to them.
  • String elements should be able to be stored in both RAM and program flash memory (PROGMEM).
  • Templating ability on string elements using simple string literals (instead of handler functions).
  • Multiple token templating ability on string elements (without the use of handler functions).

The ESPStringTemplate Library

The ESPStringTemplate Library can be found on GitHub here. It is also a part of Arduino’s Library Manager, and can be found using the Arduino IDE when it is searched for. The GitHub README gives a good outline of the usage of the library. Simple examples also exist to help get people started. I hope this library can be useful for someone else down the track!

Just as a note, there is actually nothing stopping the use of ESPStringTemplate on any microcontroller!

ESPStringTemplate Examples

These examples are taken straight from the ESPStringTemplate GitHub page, but are also included here.

Simple example – Using string literals to build a page in a statically allocated buffer. 

static char buffer[200];
ESPStringTemplate webpage(buffer, sizeof(buffer));
webpage.add("Hi!<br><br>");
webpage.add("This is an example of the ESPStringTemplate Library.<br><br>");
webpage.add("These strings are stored in RAM, and are added to the provided statically allocated buffer.<br><br>");

Using string literals stored in flash (program memory) to build a web page.

static char buffer[200];
ESPStringTemplate webpage(buffer, sizeof(buffer));
webpage.add_P(PSTR("This is an example of the ESPStringTemplate Library.<br><br>");
webpage.add_P(PSTR("This string is stored in flash using PROGMEM, and are added to the provided statically allocated buffer."));

Using reusable HTML elements with a simple token replacement.

static const char _PAGEHEADER[] PROGMEM = "<html><body>";
static const char _CONTENT[]    PROGMEM = "%CONTENT%";
static const char _PAGEFOOTER[] PROGMEM = "</body></html>";

static char buffer[200];
ESPStringTemplate webpage(buffer, sizeof(buffer));
webpage.add_P(_PAGEHEADER);
webpage.add_P(_CONTENT, "%CONTENT%", "TEST CONTENT");
webpage.add_P(_PAGEFOOTER);

Using reusable HTML elements with a multiple token replacement.

static const char _PAGEHEADER[] PROGMEM = "<html><body>";
static const char _CONTENT[]   PROGMEM = "%CONTENTA% and %CONTENTB% and %CONTENTC%";
static const char _PAGEFOOTER[] PROGMEM = "</body></html>";

static char buffer[200];
ESPStringTemplate webpage(buffer, sizeof(buffer));
TokenStringPair pair[3];
webpage.add_P(_PAGEHEADER);
pair[0].setPair("%CONTENTA%", "Replacing this token");
pair[1].setPair("%CONTENTB%", "this token");
pair[2].setPair("%CONTENTC%", "this last token as well...");
webpage.add_P(_CONTENT, pair, 3);
webpage.add_P(_PAGEFOOTER);

Using ESPAsyncWebServer to serve created webpage 

static char buffer[200];
WiFi.softAP("ESPStringTemplate Example");

AsyncWebServer server(80);
server.on("/", HTTP_GET, [](AsyncWebServerRequest *request)
{
  ESPStringTemplate webpage(buffer, sizeof(buffer));  
  webpage.add("Hi!");
  /* Send the webpage from SPIFFS where it is stored. */
  request->send(200, "text/html", buffer);
}
server.begin();

Moving DaleGi in a Slightly New Direction

Leave a reply

How time flies!

It has been almost two years now since dalegi.com said “hello world” to the world, and I have had a nice time writing what I have managed to write on the blog! In this time there have almost been 3000 visitors to the site, and that has been just about enough for me to have the motivation to occasionally post about different things. Without a doubt the most popular posts have been about Bosnian Bureaucratic Wonders, however there is also a lot of interest in my post about the PMSA003 pollution sensor, as well as the Parametric Digital Transducer Loudspeaker. It has been a good creative outlet for me as someone who likes to write, but does not get much of a chance to write anything but code these days!

I have noticed lately I have struggled to find the time and motivation to write which might be due to the style I have tried to keep on the blog. This was one of a resource of information, and a portfolio for myself. I have also noticed a lack of motivation to start new personal projects outside of my day job, which really is a shame!

So with this in mind, I have decided to change things up a bit. From here on out I will try to write more in the style of articles belonging to a series. The series might be for a specific project, or maybe a topic in general. I find that this helps to motivate me not only finish projects, but also write more. I am also going to loosen up a bit with the article content and provide content more in the vein of a thought process, discussion, with the occasional bad joke.

So with this being said, here is to moving in slightly different directions!

Setup Guide for SEGGER J-Link and SystemView with STM32 Nucleo Boards

Leave a reply

SEGGER SystemView is a very cool graphical tool for embedded systems that enables the visualisation of run-time behaviour in an application by the recording on configurable events. These events can be recorded using J-Link, IP, or UART communication. A huge plus for using SystemView is that it is totally free to use, provided that in one event recording session you do not exceed one million recorded events. This limit is actually very easy to exceed in larger applications, but still renders the free version of the software very useful.

I recently setup SystemView to work with a STM32 Nucleo based development board to debug the behaviour of FreeRTOS in a visual manner. In this setup I was also using System Workbench for STM32 with SEGGER J-Link debugging. This particular setup was pretty neat because it provided multiple methods for debugging the system, and was entirely free. The STM32 Nucleo and Discovery boards are very well suited for this use case as they contain an on-board ST-LINK programmer/debugger, and SEGGER provide a free way to convert the ST-LINK module to a J-Link module.

The entire setup was a little protracted so I kept notes on everything I did to get everything work. It is these notes that I will share in this post. It is not meant to be an exhaustive guide and will not provide any kind of in depth information about how SystemView works, but is provided here in the hope that it may save someone else time somewhere down the track.

It is assumed that you already have System Workbench for STM32 installed. Beyond that, I completed each tasks in the following order:

  1. Converting ST-LINK into J-Link
  2. GNU MCU Eclipse install
  3. SEGGER J-Link Install
  4. J-Link debug configuration setup in System Workbench
  5. SystemView Install
  6. Adding SystemView Module to System Workbench project

Converting ST-LINK into J-Link

Instructions are here.

  • Install ST-LINK USB drivers (should already be done): Download
  • Install J-Link software package V5.12b or later : Download
  • Install SEGGER STLinkReflash utility: Download
  • Start the STLinkReflash utility (STLinkReflash.exe)
    • Agree to the license terms (enter “A”)
    • Connect ST-LINK on-board to PC 
    • Select “Upgrade to J-Link” (Enter “1”)
    • Wait for operation to complete

GNU MCU Eclipse Install

Open System Workbench for STM32 (Eclipse) and do the following:

  • Open Eclipse Marketplace
  • Search for GNU MCU Eclipse and install
  • Confirm all features
  • Agree to the license terms
  • Wait for operation to complete
  • Restart Eclipse

SEGGER J-Link Install

Instructions are here.

  • Install J-Link: Download
  • Test install with connected converted J-Link board by running JLinkGDBServerCL with CMD. On my system this is located at “C:\Program Files (x86)\SEGGER\JLink\JLinkGDBServerCL.exe”. 
    • Accept the terms of use as they pop up.
    • Select the correct target device. For me it is STM32F429ZI. This will differ depending on the development board you are using.

J-Link Debug Configuration Setup in System Workbench

Instructions are here.

  • Define the path in Eclipse for J-Link folder. Window→Preferences→MCU→Global SEGGER J-Link Path
    • Create new J-Link Debug Configuration (Run→Debug Configurations→GDB SEGGER J-Link Debugging)
    • On Debugger tab add Device name (see Supported Device names for precise names). For my project it is STM32F429ZI. This will differ depending on the development board you are using.
    • On Startup tab add CPU freq and SWO freq (must be a sub-multiple of 6Mhz). On my board CPU freq is 180Mhz
    • On Common tab, change the Save as option to Shared file.
    • Add {cross_prefix} and {cross_suffix} build variables (Window→Preferences→C/C++/Build Variables)
      • cross_prefix=”arm-none-eabi-
      • cross_suffix=””
      • These may already exist, if they do not, do not added them again.

SystemView Install

  • Download installer: Download
  • Follow the prompts and install

Adding SystemView Module to System Workbench project

  • Add the SystemView module to the project (it is architecture independent). It can be found in C:\Program Files (x86)\SEGGER\SystemView_V252d on my system. It includes the following files:
    • SEGGER
      • SEGGER.h
      • SEGGER_RTT.h
      • SEGGER_SYSVIEW.h
      • SEGGER_SYSVIEW_ConfDefaults.h
      • SEGGER_SYSVIEW_Int.h
      • SEGGER_SYSVIEW.c
      • SEGGER_RTT.c
    • Config
      • Global.h
      • SEGGER_RTT_Conf.h
      • SEGGER_SYSVIEW_Conf.h
  • Inside main() function (or other startup function) add the SEGGER_SYSVIEW_Conf() initialisation function. 

Conclusion

If all is well, everything is ready to start debugging with J-Link, SystemView and System Workbench for STM32.

Sarajevo

How to Live and Work in Bosnia and Herzegovina

3 Replies

So you want to live and work in Bosnia and Herzegovina? Great! You just spent your the summer in Sarajevo, you became a regular at Kino Bosna, gained 5 kilograms from eating Krompiršua and Ćevapi every meal, and developed an ironic attachment to turbo folk from long nights at Sloga. “Sarajevo is the new Berlin” you declare! You decide you will teach English, sell bracelets on Ferhadija, and push handmade cigarettes in Stup to make ends meet in order to fulfill your bohemian lifestyle. Armed with your new life plan, you only need to find a way to become legal. Well, buckle up. Because I do not have to be your mother to tell you “Okay honey, let me know when you run out of money and want me to buy your flight home”.

But I digress.

Now that you know your place, I will briefly delve into why this is probably not going to happen. But if you do persist, this post hopes to explain some of the options available that will allow you to live in Bosnia for an extended period of time and the process of obtaining a Work Permit in the event that you would like to work here. I will also share some interesting statistics on the demographics of those with Work Permits and some personal insights as someone who managed to successfully navigate the process for becoming a foreigner who lives and works in BiH legally.

Due to the complexity of living and working in BiH as a foreigner, this post will be more concise than my two previous posts on Bosnian Bureaucratic Wonders, How to Get a White Card in Bosnia and Herzegovina, and How to Get a Volunteer Visa in Bosnia and Herzegovina. It is also designed to give a general overview on the process of obtaining a Work Permit in BiH and provide a variety of links and resources to set you in the right direction.

How Do I legally Live in Bosnia and Herzegovina?

The BiH Service for Foreigner’s Affairs (SPS) website provides a good outline on the different types of visas and temporary residence permits available to foreigners in English. The BiH Ministry of Security (MSB) also provide basic information on the entry and stay of aliens in BiH on their website.

For nationals included in the Visa-free regime, it is possible to reside in BiH for 90 days out of every 180 days. You cannot legally volunteer or work while in BiH as a part of the Visa-free regime. To legally reside in BiH longer than the Visa-Free regime allows for, it is necessary to obtain either a temporary residence permit, or a long stay visa (visa D).

Temporary Residence Permit

A temporary residence permit is usually valid for a period no longer than 12 months, and allows an individual to stay in BiH for the validity period. A brief outline regurgitated from the SPS website on the grounds for granting temporary residence is as follows:

  • Marriage to a citizen of Bosnia and Herzegovina.
  • Extramarital community (Common Law Marriage) with a citizen Bosnia and Herzegovina
  • Family reunification
  • Education (including volunteering)
  • Employment as specified in an issued work permit
  • Work without a work permit
  • Treatment or rehabilitation
  • Stay in a nursing home
  • Stay for humanitarian reasons
  • Stay based on an international treaty to which Bosnia and Herzegovina is a party
  • Stay out of other legitimate reasons
  • Ownership of real estate in Bosnia and Herzegovina

The SPS website provides information regarding the required documents for an application in all of the above circumstances.

Long Stay Visa (Visa D)

A Long Stay Visa, or “Visa D”, is usually valid for a period of 180 days starting from the date of first entry into BiH. Information about the Long Stay Visa can be found on the BiH MSB website.  In general, the Long Stay Visa is issued to individuals who can prove that there is a valid purpose for an individual to stay in BiH a period of more than 90 days within a 6 month period.  The application for this kind of visa is usually based on a “letter of invitation” from a Bosnian citizen or organisation.

How Do I Legally Work in Bosnia and Herzegovina?

To legally work in BiH, usually you must a obtain a Work Permit.  Work Permits can be valid for a maximum of 12 months after being issued, and must be renewed after this period. They are usually obtained on the basis of receiving a job offer from a company registered in BiH.

In some exceptional cases, individuals can obtain a Temporary Residence Grounded on Work Without a Work Permit. This allows an individual to work in BiH without a Work Permit. One of these exceptional cases is being the founder of a company in BiH. You can find out if you qualify for this type of temporary residence by visiting the BiH SPS website here or reading Article 50 through 71 of the Rulebook on Entry and Stay of Aliens.

Who Has Work Permits in Bosnia and Herzegovina?

A comprehensive migration profile report was released by the Ministry of Security (MSB)  for 2016/2017 period detailing various demographic data on issued Work Permits in BiH.

The demographic data pulled from this report displays an odd situation in regard to who typically obtains Work Permits in BiH in regard to industry, age, and qualification level. Around 2500 Work Permits were issued in 2017. The majority of Work Permits in BiH are issued to citizens of Serbia, Turkey, Croatia, and China. The overwhelming majority of issued Work Permits are issued to men. Roughly half of those who receive work permits have a university level qualification. Most people who receive work permits are over 60 years old, and are commonly issued for those who work in the Wholesale/Retail, Vehicle Repair, Processing, Real Estate, Education, and Construction industries.

How Do I Obtain a Work Permit and Temporary Residence on the basis of a Job Offer?

The Foreign Investment Promotional Agency (FIPA) of BiH provide a nice guide on how to obtain Work Permits in BiH. Information on the required documentation for the Temporary Residence Grounded on Work With a Work Permit can be found on the PB-4.1 document listed on the SPS website. A brief overview of the process can also be found in the Frequently Asked Questions section of the SPS website in BHS. The following steps are outlined in these sources:

  1. Receive a job offer from a BiH company.
  2.  Work Permit application is then submitted by the future employer on behalf of the employee to the relevant government employment agency. The employee will be required to provide certified identity documents as well as a nostrification of their university degree if applicable, while the employer will have to provide registration documents as well as proof they do not have any outstanding tax obligations. The employee should be located outside of BiH while the Work Permit application is processing.
  3. Once the Work Permit is obtained, the employee may enter BiH either using their 90 day visa-waiver period (or other applicable short-stay visas) and apply for Temporary Residence Grounded on Work With a Work Permit. The employee may reside in BiH while the Temporary Residence Grounded on Work With a Work Permit application is processing.
  4. Once the Temporary Residence Grounded on Work With a Work Permit application has been processed, the employee may start working.

This entire process typically takes 3 months or more. It is recommended that if the employer does not take care of the process, the employee should hire a local lawyer. In my experience, the whole process costs around 400KM in government administrative fees, and 600KM in lawyer fees when applying in the Kanton Sarajevo.

Conclusion

It is hoped that this post has cleared a few things up and saved time for those thinking about living and working in BiH as a foreigner. While it may have dashed the hopes of those wishing to stave off their early life crisis by becoming professional bracelet and cigarette pushers, it is possible to navigate the visa process and do things legally in BiH when you put your mind to it. Navigating the Bosnian bureaucracy as a foreigner in practice can be stressful and frustrating, and it is recommended to hire a local lawyer when trying to apply for the various visas and permits by yourself.

Photo by @edina.sp

Useful Links