ESP32: Parsing URL variables

Introduction

In this tutorial we are going to learn how to parse variables from an URL using the PathVariableHandlers library, the ESP32 and the Arduino core.

You can install this library from the Arduino IDE libraries manager, like shown in figure 1 below.

This library allows to easily define a template URL with variables and then use it to handle actual URLs and extract the variables from them.

The tests shown below were performed on a ESP32-E FireBeetle board from DFRobot.

A basic example

To get started, we will need the two following includes:

• TokenIterator.h: exposes the TokenIterator class, which we will be using in our code to represent the template URL and the actual URL.
• UrlTokenBindings.h: exposes the UrlTokenBindings class, which will allow us to extract the URL path variables values.

#include <TokenIterator.h>
#include <UrlTokenBindings.h>

After this we will jump to the Arduino Setup. As usual, we will start by opening a serial connection, so we can output the results of our program.

Serial.begin(115200);

Then we are going to define a variable that will hold our template URL. This means that this string will contain the URL variable definitions in the following format:

/path/:variable

As can be seen, if we want to declare a path variable, we add a colon before its name.

Although this library is able to handle multiple URL variables, we are going to keep our example simple and declare a URL template that has only a variable. In our case, we will simulate a URL for a sensor resource, which is identified by an ID, being this ID the actual variable for which we want to get the value.

char templateUrl[] = "sensor/:sensorId";

After this, we will define a variable holding an URL that follows the pattern of our template URL. We will assign the value “1” for our sensor identifier.

char url[] = "sensor/1";

Now that we have both our URLs, we are going to create an instance of the class TokenIterator to represent each one of them. The constructor of this class receives the following parameters:

• URL or template URL, as a string.
• Length of the URL. We can use the strlen function to get the length.
• Separator character for each segment of the URL, as a char. Since we are handling URLs, each segment is separated by the “/” character.

TokenIterator templateUrlIterator(templateUrl, strlen(templateUrl), '/');
TokenIterator urlIterator(url, strlen(url), '/');

Now that we have our TokenIterator objects, we need to create an object of class UrlTokenBindings, which will allow to match the URL parameter with the actual URL and extract the values of the variables.

As input of the constructor, it receives the TokenIterator of the template URL and the TokenIterator of the actual URL. Please take in consideration that this should be the actual order and, at the time of writing, the library example had this call wrong and doesn’t work (opened issue here).

UrlTokenBindings bindings(templateUrlIterator, urlIterator);

Now that we have the UrlTokenBindings, we simply need to call the hasBinding method, passing as input the name of the variable we want to check (in this case, without the colon). This method will return a Boolean indicating if the variable exists (true) or not (false).

In case it returns true, we can get the actual value with a call to the get method, also passing as input the name of the variable. This method will return the value as a string. In our case, we will be printing the value directly to the serial port, but if we wanted to interpret it as an integer, we would need to perform the conversion.

if (bindings.hasBinding("sensorId")) { 
    Serial.println(bindings.get("sensorId"));
}

The complete code is shown below.

#include <TokenIterator.h>
#include <UrlTokenBindings.h>

void setup() {
  Serial.begin(115200);

  char templateUrl[] = "sensor/:sensorId";
  char url[] = "sensor/1";

  TokenIterator templateUrlIterator(templateUrl, strlen(templateUrl), '/');
  TokenIterator urlIterator(url, strlen(url), '/');

  UrlTokenBindings bindings(templateUrlIterator, urlIterator);

  if (bindings.hasBinding("sensorId")) { 
    Serial.println(bindings.get("sensorId"));
  }
}

void loop() {}

To test the code, compile it and upload it to your ESP32. After the procedure finishes, open the IDE serial monitor. You should obtain a result similar to figure 2. As can be seen, we obtained the value “1”, which was precisely the sensor ID we defined in our code.

A webserver URL variables example

In this section we will analyze a more practical example, where we will parse the URL of a request performed to a HTTP web server running on the ESP32. We will be using the Async HTTP web server library, which is covered in detail in this previous post (including the installation). As such, we will mostly focus on the URL handling part.

To get started, we will do all library includes. Besides the two includes for the URL handling we already covered in the previous section, we will also need the WiFi.h and the ESPAsyncWebServer.h, so we can setup the HTTP server.

#include <WiFi.h>
#include <ESPAsyncWebServer.h>
#include <TokenIterator.h>
#include <UrlTokenBindings.h>

We will also need to define two variables holding the WiFi network credentials, and to create an object of class AsyncWebServer, which is needed to setup the HTTP server.

const char* ssid = "yourNetworkName";
const char* password =  "yourNetworkPass";

AsyncWebServer server(80);

Moving on to the Arduino setup funcion, we will take care of opening a serial connection and connecting the ESP32 to the WiFi network, using the credentials we have declared above. Once the WiFi connection is established, we will print the IP address assigned to the ESP32 on the network, since it is needed to reach the server.

Serial.begin(115200);
 
WiFi.begin(ssid, password);
 
while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.println("Connecting to WiFi..");
}
 
Serial.println(WiFi.localIP());

We will now register a route in our web server, which will be listening to HTTP GET requests. This route will follow the same pattern we covered in the previous section: a sensor resource that is identified by its ID.

When declaring a route using the async HTTP web server library, one option for supporting path variables is using a regex rule. Nonetheless, this requires defining a build flag, like shown here. We are not going to follow this approach.

Instead, we are going to add a “*” to the endpoint, after “/sensor/”, which will match all the requests using a URL starting with “/sensor/”. Note however that this won’t automatically bind any route parameter and everything starting with “/sensor/” will be matched with this route, even if there are more forward slashes in the request URL. As such, this approach is only feasible for simple use cases.

As such, our route will be:

/sensor/*

As a recap from the HTTP server introductory tutorial, we register a route with a call to the on method on our AsyncWebServer object. It receives the following inputs:

• The route. We will use the value we mentioned before.
• The HTTP method the route will be listening for. We will pass the value HTTP_GET.
• The route handling function. We will use the C++ lambda syntax to define our handling function.

server.on("/sensor/*", HTTP_GET, [](AsyncWebServerRequest *request){
     // Route handling function
}

Moving on to the implementation of the handling function, we will start by defining a variable that contains the URL template, in the format we have covered in the previous section.

char templatePath[] = "/sensor/:sensorId";

Then we will access the actual URL value that the request was made to. Recall that we have used the “*”, which means that different URLs will match this route.

We can access this value by calling the url method on the AsyncWebServerRequest object to which we receive a pointer as input of the route handling function.

Since this method returns a String object, we need to convert it to a char array, so we can later create a TokenIterator object. We can do that with a call to the toCharArray method. This method receives as input a char buffer, where the string will be written, and the length of the buffer. I’ll be using a buffer with a size of 30, which is more than enough for our test scenario.

char urlBuffer[30];
request->url().toCharArray(urlBuffer, 30);

After this, we will create our TokenIterator objects, one representing the URL template and the other representing the actual request URL. We will also create the UrlTokenBindings that will allow us to extract the route variable.

int urlLength = request->url().length();

TokenIterator templateIterator(templatePath, strlen(templatePath), '/');
TokenIterator pathIterator(urlBuffer, urlLength, '/');
UrlTokenBindings bindings(templateIterator, pathIterator);

To finish our route handling function implementation, we will return back to the client the response. The body of the response will be the variable value extracted from the route.

request->send(200, "text/plain", bindings.get("sensorId"));

The complete route handling function can be seen below.

server.on("/sensor/*", HTTP_GET, [](AsyncWebServerRequest *request){

    char templatePath[] = "/sensor/:sensorId";

    char urlBuffer[30];
    request->url().toCharArray(urlBuffer, 30);

    int urlLength = request->url().length();

    TokenIterator templateIterator(templatePath, strlen(templatePath), '/');
    TokenIterator pathIterator(urlBuffer, urlLength, '/');
    UrlTokenBindings bindings(templateIterator, pathIterator);
  
    request->send(200, "text/plain", bindings.get("sensorId"));
});

To finalize our setup function, we will start our server with a call to the begin method.

server.begin();

The whole setup function can be seen in the snippet below.

void setup(){
  Serial.begin(115200);
 
  WiFi.begin(ssid, password);
 
  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.println("Connecting to WiFi..");
  }
 
  Serial.println(WiFi.localIP());
 
  server.on("/sensor/*", HTTP_GET, [](AsyncWebServerRequest *request){

    char templatePath[] = "/sensor/:sensorId";

    char urlBuffer[30];
    request->url().toCharArray(urlBuffer, 30);

    int urlLength = request->url().length();

    TokenIterator templateIterator(templatePath, strlen(templatePath), '/');
    TokenIterator pathIterator(urlBuffer, urlLength, '/');
    UrlTokenBindings bindings(templateIterator, pathIterator);
  
    request->send(200, "text/plain", bindings.get("sensorId"));
  });
 
  server.begin();
}

The complete code is shown below.

#include <WiFi.h>
#include <ESPAsyncWebServer.h>
#include <TokenIterator.h>
#include <UrlTokenBindings.h>

const char* ssid = "yourNetworkName";
const char* password =  "yourNetworkPass";

AsyncWebServer server(80);
 
void setup(){
  Serial.begin(115200);
 
  WiFi.begin(ssid, password);
 
  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.println("Connecting to WiFi..");
  }
 
  Serial.println(WiFi.localIP());
 
  server.on("/sensor/*", HTTP_GET, [](AsyncWebServerRequest *request){

    char templatePath[] = "/sensor/:sensorId";

    char urlBuffer[30];
    request->url().toCharArray(urlBuffer, 30);

    int urlLength = request->url().length();

    TokenIterator templateIterator(templatePath, strlen(templatePath), '/');
    TokenIterator pathIterator(urlBuffer, urlLength, '/');
    UrlTokenBindings bindings(templateIterator, pathIterator);
  
    request->send(200, "text/plain", bindings.get("sensorId"));
  });
 
  server.begin();
}
 
void loop(){}

To test the code, once again compile it and upload it to your device, using the Arduino IDE. When the precedure finishes, open the IDE serial monitor and copy the IP address that gets printed. Then, open a web browser of your choice and paste the following in the address bar, changing #yourEspIp# by the IP you have just copied:

http://#yourEspIp#/sensor/10

Note that I’ve used the value 10 as sensor Id, for illustration purposes. You can use other values if you want. You should get a result similar to figure 3. As can be seen, the value 10 was extracted from the URL and returned back to us.

Processing .csv lines

One interesting thing about the PathVariableHandlers that can go unnoticed is the fact that it can actually be used to parse other type of content. As we mentioned in the first code section, when creating the TokenIterator object, we can actually specify the separator between the segments of the “path”.

If we think about a .csv file, for example, we can see it as a collection of tokens separated by commas. As such, we can declare a template representing the header of a .csv file, as shown below (it is an arbitrary example with a name, age and gender columns):

:name,:age,:gender

Then we can use it to iterate through each line of the csv. A complete example is shown below for the example header.

#include <TokenIterator.h>
#include <UrlTokenBindings.h>

void setup() {
  Serial.begin(115200);

  char templateCsv[] = ":name,:age,:gender";
  char lineOfCsv[] = "john,10,f";

  TokenIterator templateCsvIterator(templateCsv, strlen(templateCsv), ',');
  TokenIterator lineOfCsvIterator(lineOfCsv, strlen(lineOfCsv), ',');
  UrlTokenBindings bindings(templateCsvIterator, lineOfCsvIterator);

  Serial.println(bindings.get("name"));
  Serial.println(bindings.get("age"));
  Serial.println(bindings.get("gender"));
  
}

void loop() {}

Upon running the code, you should get a result similar to figure 4.