Introduction
In this tutorial we will learn how to calculate the difference between two JSON objects using the nlohmann/json library on the ESP32. This operation is called JSON diff.
For installation instructions and an introduction to the nlohmann/json lib, please check this post.
The operation we are covering below will allow to compare two JSON objects and represent the difference between them as a set of operations that describe each difference. These diffs are represented using the JSON patch notation, which can be used to represent a set of operations that need to be applied to a source JSON object to be transformed in a target JSON object. You can consult all the JSON patch supported operations here.
Although it is interesting to know about the existence of the JSON Patch functionality, our focus for this tutorial is to learn how to obtain the differences between the two objects, regardless of how the result will be used.
Note that the code we are covering here extends beyond the scope of the ESP32 programming. All the JSON related functionality can easily be used in a generic C++ program.
The version of the nlohmann/json library used on this tutorial was 3.10.2. The Arduino core version used was 2.0.0 and the Arduino IDE version was 1.8.15, running on Windows 8.1.
The tests shown below were performed on a ESP32-E FireBeetle board from DFRobot.
A basic JSON diff example
We will start our code with the json.hpp include, which will expose to us the JSON handling functionalities.
#include <json.hpp>
In the Arduino setup, we will start by opening a serial connection, to later print the results of our test.
Serial.begin(115200);
Then we will define two strings, each one containing a JSON object that could represent some properties of a person. There will be differences between the two objects: the first one will contain a property called name and the second will contain a property called age instead. Both of them will contain a property called nationality.
char str1[] = R"(
{
"name": "John",
"nationality": "portuguese"
}
)";
char str2[] = R"(
{
"age": 10,
"nationality": "portuguese"
}
)";
After that we are going to parse both JSON strings to json objects. We can do it simply by calling the parse static method, passing as input the string we want to parse. As output, this method will return the parsed json.
nlohmann::json obj1 = nlohmann::json::parse(str1);
nlohmann::json obj2 = nlohmann::json::parse(str2);
To perform the difference operation between the two json objects, we can call the diff static method. As first input we pass the json that is the source of the comparison, and as second the json that is the target.
As output, this method will return a json array that corresponds to the set of patch operations needed to convert the first json in the second [1]. In other words, we obtain the differences between the source and the target in a representation that we may use to automatically transform one object in the other. If that’s not our objective, we can simply interpret the result to understand what changed between the two objects.
nlohmann::json difference = nlohmann::json::diff(obj1, obj2);
We will finish the Arduino setup by printing the diff object to the serial port. We start by serializing the json object to a string, with some indentation for better readability. Recall from here that we can perform the serialization with a call to the dump method, passing as input an integer with the indent level.
std::string serializedObject = difference.dump(3);
Note that the result of the serialization is a std::string, which we cannot directly print using the print and println methods of the Serial object. Consequently, we need to first call the c_str method on our string before passing it to the printing function.
std::string serializedObject = difference.dump(3);
Serial.println(serializedObject.c_str());
Since we don’t have any computation to perform on the Arduino main loop, we will simply leave it empty.
void loop() {}
The whole code is shown below
#include <json.hpp>
void setup() {
Serial.begin(115200);
char str1[] = R"(
{
"name": "John",
"nationality": "portuguese"
}
)";
char str2[] = R"(
{
"age": 10,
"nationality": "portuguese"
}
)";
nlohmann::json obj1 = nlohmann::json::parse(str1);
nlohmann::json obj2 = nlohmann::json::parse(str2);
nlohmann::json difference = nlohmann::json::diff(obj1, obj2);
std::string serializedObject = difference.dump(3);
Serial.println(serializedObject.c_str());
}
void loop() {}
Upon running the code on your ESP32, you should obtain a result similar to figure 1. As can be seen, we have obtained a JSON array that reflects the differences between the source and target json objects.
If we analyze the array in detail, we can see that there are two objects, each one representing a different operation. The operation is identified for both of them on the “op” property.
The first object is a “remove” operation. If we check the “path” property, we can easily infer that it is referring to the “name” property, which existed in the first json object, but not in the second.
The second object corresponds to an “add” operation. Once again, the “path” clarifies that a property called “age” was added. In this particular operation, we also have the additional information that “age” was added with a “value” of 10.
Since the “nationality” property was not changed, it doesn’t appear in the diff result.
As can be seen, the resulting description of the diff operation is very easy to interpret and eventually parse if we need to. Still, it is represented in such a way that can be used to execute JSON patch operations, opening even more possibilities of usage.
Diff on updated properties
Now that we covered the basics and the API to perform JSON diffs over objects, we will look into the specific behavior for different cases. As such, the code will be exactly the same in the following sections, except for the JSON objects that we are going to diff.
On this section we are going to focus our attention on properties that exist both in the source and target JSON objects, but change value.
As such, our first object will contain two properties: an integer called “prop1” and a string called “prop2“.
char str1[] = R"(
{
"prop1": 1,
"prop2": "a"
}
)";
In the second object, we will have exactly the same property names. Nonetheless, the value of “prop1” will change from 1 to 2. The value of “prop2” will change from “a” to 10, so we cover an example where the type of the property actually changes.
char str2[] = R"(
{
"prop1": 2,
"prop2": 10
}
)";
The whole code is shown below.
#include <json.hpp>
void setup() {
Serial.begin(115200);
char str1[] = R"(
{
"prop1": 1,
"prop2": "a"
}
)";
char str2[] = R"(
{
"prop1": 2,
"prop2": 10
}
)";
nlohmann::json obj1 = nlohmann::json::parse(str1);
nlohmann::json obj2 = nlohmann::json::parse(str2);
nlohmann::json difference = nlohmann::json::diff(obj1, obj2);
std::string serializedObject = difference.dump(3);
Serial.println(serializedObject.c_str());
}
void loop() {}
You can analyze the result of the operation in figure 2 below.
In this case, we can easily check that the operation is called “replace“. Naturally, in both cases, the “path” refers to the changed property. The “value” basically indicates what is the new value assigned to the property. As can be seen, it is irrelevant if the type of the property changed or not, as we will only get the final value that was assigned to it.
Diff on JSON array properties
In this section we are going to check the result when performing diffs on JSON arrays. Once again, we will focus on the JSON objects we want to diff, as the remaining code will be equal.
So, the first object will contain a property called arr, which is an array of integers that will contain the numbers 1, 2, 3 and 4.
char str1[] = R"(
{
"arr": [1,2,3,4]
}
)";
The second object will also have the arr property, but with some differences: the second element (originally with the value 2) will be replaced by the value 5. Additionally, we will remove the fourth element from the array (originally with the value 4).
Besides that, we will add a new property called newArr, which is an array of strings. Our aim here is to later check how the value of this new property is represented in the resulting JSON diff.
char str2[] = R"(
{
"arr": [1,5,3],
"newArr": ["a", "b", "c"]
}
)";
The complete code is shown below.
#include <json.hpp>
void setup() {
Serial.begin(115200);
char str1[] = R"(
{
"arr": [1,2,3,4]
}
)";
char str2[] = R"(
{
"arr": [1,5,3],
"newArr": ["a", "b", "c"]
}
)";
nlohmann::json obj1 = nlohmann::json::parse(str1);
nlohmann::json obj2 = nlohmann::json::parse(str2);
nlohmann::json difference = nlohmann::json::diff(obj1, obj2);
std::string serializedObject = difference.dump(3);
Serial.println(serializedObject.c_str());
}
void loop() {}
The result from running the previous code is shown below in figure 3.
The first object corresponds to the “replace” operation. When evaluating the “path” property, we can quickly understand that the operation was executed on the arr property, on the position 1 (note that the indexes are zero-based, so this is actually the second element of the array). As such, we can quickly conclude that, for arrays, the positions that are changed are specified as part of the path. Naturally, the “value” property contains the new value.
The second object represents a “remove” operation, once again indicating the index of the removed array element on the “path“. As expected from the first example, “remove” operations don’t indicate a “value” because there is no new value.
Finally we can check the “add” operation. Note that the “value” contains the whole newly added array of strings.
Diff on nested JSON properties
We will conclude our analysis by performing the diff on nested properties. For this test, we will use a data structure representing a possible person entity. One of the properties of this JSON object is called “address” and it contains two nested properties that we will change.
char str1[] = R"(
{
"name": "John",
"age": 10,
"address": {
"street": "St. Street",
"code": "1234-12"
}
}
)";
char str2[] = R"(
{
"name": "John",
"age": 10,
"address": {
"street": "New Avenue",
"code": "5785-214"
}
}
)";
The complete code is available on the snippet below.
#include <json.hpp>
void setup() {
Serial.begin(115200);
char str1[] = R"(
{
"name": "John",
"age": 10,
"address": {
"street": "St. Street",
"code": "1234-12"
}
}
)";
char str2[] = R"(
{
"name": "John",
"age": 10,
"address": {
"street": "New Avenue",
"code": "5785-214"
}
}
)";
nlohmann::json obj1 = nlohmann::json::parse(str1);
nlohmann::json obj2 = nlohmann::json::parse(str2);
nlohmann::json difference = nlohmann::json::diff(obj1, obj2);
std::string serializedObject = difference.dump(3);
Serial.println(serializedObject.c_str());
}
void loop() {}
The expected result is shown below on figure 4.
Naturally, the operation is still “replace“, since we only updated the values. In this case, since the properties are nested, the “path” has an additional level to clearly identify the location of the property. This follows a similar pattern to the one that is used to represent indexes on an array.
Suggested ESP32 Readings
- Deserializing JSON objects from the file system
- Serialize JSON object to file
- Getting started with the Nlohmann/json json library
- Preserve order of keys on json properties insertion
Additional resources
References
[1] https://json.nlohmann.me/api/basic_json/diff/