Python: Parsing JSON

The objective of this post is to explain how to parse the response of a HTTP request which is in JSON format.

Introduction

The objective of this post is to explain how to parse the response of a HTTP GET request which is in JSON format.

We will use the Requests library to perform the request and to parse the response.

 

The code

We will start by making a GET request to the jsonplaceholder website, which has a fake online REST API for testing and prototyping [1].

So, this part of the code will be the same of the previous tutorial where we performed a GET request to this website. We start by importing the Requests library and then we make the request using the get method of this module. We finally print the result using the text property of the Response object returned by the get method, which we stored in a variable.

import requests
response = requests.get(‘http://jsonplaceholder.typicode.com/users’)
print response.text

Important: In order for WordPress to not escape the single quote in the end of URLs, I used a special character single quote. When copying and running the code, we need to change the single quote at the end of the URL to a regular one (just delete this and put a regular one). Otherwise, an error will occur and the code will not execute.

We will also print the return code of this request. To do so, we just need to access the status_code parameter of the Response object.

To make things more readable, we will first do a print with just a space to insert a new line and then we will concatenate a string saying “Status code: ” with the actual status code.

In Python we can concatenate two strings with the “+” operator [2]. Nevertheless, we cannot directly concatenate a string and an integer. Since the status_code parameter is an integer, we need to use the str function to convert it to a string. The code is shown bellow.

print ” “
print “Status code: ” + str(response.status_code)

Just to confirm that we are really receiving a response in JSON format, we will also check the headers of the response. To do so, we just need to access the headers parameter of the Response object. This object is of type dictionary [3].

You can read more about Python dictionaries here but, for the purposes of this post, you just need to know that a dictionary is like an array but it can be indexed with keys (of type string) instead of numbers (like a normal array).

To check the type of the response, we will need to access the content-type field of the header, which describes the data contained in the body of the response [4].

We will again concatenate a string with the actual value of the header, for the sake of readability. In this case, since we will be concatenating 2 strings, no conversion needs to be done and we can just use the “+” operator.

print ” “
print “Content type: ” + response.headers[‘content-type’]

To get the JSON content parsed in an easy to use object, we need to call the json method of the Response object.

If the decoding of the JSON fails, an exception will be raised [5]. You can read more about Python exceptions and how to handle them here. In this case, just to keep the code simple, we will not handle the exception. But please keep in mind that handling exceptions is the best coding practice.

data = response.json()

In this case, we are retrieving a list of users from the fake REST API. As we will see when printing the response, an array of 10 user objects will be returned in the JSON response. You can check bellow, in figure 1, the structure of the object representing each user.

Figure 1 – JSON structure of a user, returned in the HTTP GET request.

So, our data object will be a Python list, with an entry for each user object. You can read more on Python lists here.

To confirm that we are working with a list, we can just print the type of the data object, as shown below.

print ” ”
print “Parsed list of users: ” + str(type(data))

To access an element of a list in Python, we just need to specify its index (indexes start at 0). To check the size of our list, we just use the len function and pass as input the list.

print len(data)

As seen in the structure of a user in JSON, we will now have, in each user object, key-value pairs representing the characteristics of our users, in the format “key”: “value”. This maps perfectly to the previously mentioned Python dictionaries.

So, in each element of our list of users, we will have a dictionary object representing the parsed content that describes that user.

To confirm that, we can just print the type of the first element of our list.

print ” “
print “First element of list: ” + str(type(data[0]))

Now, to print an element of our user object from JSON, we just access the corresponding index of the Python list and specify the key of the attribute. The key will be the same present in the JSON object.

We will print the id and the name of the 10th user. Since the list starts at index zero, it will be the element at index 9.

print ” “
print data[9][‘id’]
print data[9][‘name’]

If we go back to the structure of the user object. shown in figure 1, we had a key-value pair where the key was “id”, and we had another key-value pair where the key was “name”. So, this maps correctly.

In our user JSON object, we also have some complex elements, such as the Address. In this case, the Address is also an object, nested inside the user object. So, the Address object also has more key-value pairs inside its structure.

This is very easy to map to our Python parsed object. In Python, a dictionary is like an array of key-values, where the value can be another object. This object can be another dictionary. So, if the address is a complex object which has multiple key-values in its structure, it is another dictionary in Python.

So, if we access data[9][‘address’], we are accessing to the 10th element of a list, which has a dictionary representing a JSON object of a user, and then we are accessing to the value of the “address” key of that dictionary, which is another dictionary representing a JSON object of an address, which is nested inside the user.

You can confirm the previous information with the code bellow.

print “”
print “Adress Type: ” + str(type(data[9][‘address’]))
print data[9][‘address’][‘street’]
print “Street type: ” + str(type(data[9][‘address’][‘street’]))

In this case, the address will be of type dictionary because it has other key-value elements under its hierarchy, but the street will be of type unicode (which corresponds to a String in Unicode format).

 

Final result

You can check the result of the execution of the code bellow, in figure 2.

Figure 2 – Output of the JSON parsing Python script.

We can see the last element of the JSON response printed. The others were printed before and are not shown here.

Then we have the HTTP status code, which is 200. This code corresponds to the “OK” code.

Then we have the content-type of the response which, as expected, is of type JSON.

Then, the type of our JSON parsed object is list, which his coherent with what we were expecting. The size of this list is 10, because the JSON response had 10 user objects.

When accessing to an element of this list, we have a dictionary type, also as we were expecting.

Then, we have the values of 2 elements of the user object returned. In this case, we have the ID and the name of the 10th user, that we printed in our code.

Finally, we have the type of the object representing the address, which is also a Python dictionary, and we have the type and values of the street, which is an element of type key-value in that address.

References

[1] http://jsonplaceholder.typicode.com/

[2] http://www.pythonforbeginners.com/concatenation/string-concatenation-and-formatting-in-python

[3] http://docs.python-requests.org/en/master/api/#requests.Response

[4] https://www.w3.org/Protocols/rfc1341/4_Content-Type.html

[5] http://docs.python-requests.org/en/master/user/quickstart/#json-response-content

Related Posts

• Python: HTTP GET requests
• Python: HTTP POST Requests
• Python: An introduction

Technical details

• Python version:2.7.8
• Requests library: 2.11.1