Python dlib: face landmarks detection

Contents hide

1 Introduction

2 Detection with the 68 landmarks model

3 Drawing the landmarks as points

4 Distinguish different parts of the face

5 Detection with the 5 landmarks model

6 Suggested dlib tutorials

7 References

Introduction

In this tutorial we are going to learn how to use dlib and Python to detect face landmarks in an image.

These landmarks are points on the face such as the corners of the mouth, along the eyebrows, on the eyes, and so on [1], depending on the model used. We are going to test two models: a 68 face landmarks model and a 5 face landmarks model. Both of them are already made available by dlib and we need to download them to run our programs below:

The 5 landmarks model identifies the corners of the eyes and bottom of the nose [2]. The 68 landmarks model identifies the following parts of the face [3]:

Jaw (from ear to ear)
Nose
Left eyebrow
Right eyebrow
Left eye
Right eye
Mouth

You can check here the indexes of the points for each part of the face detected with the 68 landmarks model. The indexes listed in the image on the link will be the same used by dlib, which means we can easily identify which points belong to each part of the face.

In order to detect the face landmarks, we previously need to detect the face in the image. The procedure to do it with dlib was already covered here.

This tutorial was based on the Python example provided by dlib, which I encourage you to try.

The code was tested using dlib version 19.22.0 and Python version 3.7.2, on Windows 8.1.

Detection with the 68 landmarks model

We will start by importing the dlib module.

import dlib

After that we will load an image from the file system, where we will search for faces and face landmarks. We do this with a call to the load_rgb_image function, passing as input the path to the image, as a string. As output, this function will return a ndarray representing the image.

img = dlib.load_rgb_image("C:/Users/N/Desktop/Test.jpg")

Now that we loaded our image, we will create an object of class image_window, which will allow us to display our image and the facial landmarks detected. As input of the constructor we will pass our image and the name we want to assign to the window.

win = dlib.image_window(img, "Image")

Then we will detect faces in the image using the approach we have already covered on this previous tutorial. We start by calling the get_frontal_face_detector function, to obtain an instance of the default dlib face detector. Then we will call our detector, passing as input the image. As output we obtain an array of objects of class rectangle (one per each face detected).

Now that we have the face detections, we can draw them on top of the image by calling the add_overlay method on the image_window object, passing as input the array we have just obtained.

detector = dlib.get_frontal_face_detector()
faces = detector(img)
win.add_overlay(faces)

After this we will create an object of class shape_predictor, which will allow us to detect the face landmarks. As input of the constructor we need to pass the path to the trained shape predictor model file we have previously downloaded.

predictor = dlib.shape_predictor("C:/Users/N/Desktop/shape_predictor_68_face_landmarks.dat")

As can be seen here, the instances of the shape_predictor class are callable. So, to perform the prediction, we simply need to call our object, passing as first input our image and as second a dlib rectangle representing the bounding box of the face where the landmark prediction should be done.

The result of the prediction will be an object of class full_object_detection, which represents the location of an object in an image along with the positions of each of its constituent parts. In our particular use case, it represents the detected landmarks in the face. Note that the parts that compose the object detection can be outside the face bounding box we passed to the detector, if appropriate [4].

The full_object_detection object we have obtained can be directly passed to the add_overlay method of our window (it is an overloaded function, as can be seen here), to draw the landmarks.

To leave our code ready for multiple faces in the same image, we will first iterate through the faces array we have obtained before and do the prediction and drawing each one.

for face in faces:
    landmarks = predictor(img, face)
    win.add_overlay(landmarks)

To finalize, we will wait for the window to be closed to finish our program.

win.wait_until_closed()

The complete code can be seen below.

import dlib

img = dlib.load_rgb_image("C:/Users/N/Desktop/Test.jpg")
win = dlib.image_window(img, "Image")

detector = dlib.get_frontal_face_detector()
faces = detector(img)
win.add_overlay(faces)

predictor = dlib.shape_predictor("C:/Users/N/Desktop/shape_predictor_68_face_landmarks.dat")

for face in faces:
    landmarks = predictor(img, face)
    win.add_overlay(landmarks)

win.wait_until_closed()

To test the code, simply run it in a tool of your choice. I’ll be using PyCharm, a Python IDE. You should get an output similar to figure 1.

As can be seen, the face was correctly detected and a rectangle was drawn to delimit it. Additionally, the face landmarks mentioned in the introductory section for the 68 points model were drawn and connected by lines.

**Figure** 1 – Face landmarks detection using the 68 points model.

Drawing the landmarks as points

Like we saw in the previous section, if we use the full_object_detection directly to draw the landmarks, the points will be connected by lines. In this section we will show an alternative way of representing the detected landmarks by drawing them as points in the image.

Most of the code will be similar to the previous section, except for the iteration over the faces. We will focus our attention in that part of the code.

So, we will still iterate through each of the faces found and call our predictor to obtain the landmarks of each face.

for face in faces:
    landmarks = predictor(img, face)
    # remaining code

After that, we will print the number of parts that compose our full_detection_object instance. This can be done with a call to the num_parts attribute. When testing the code we expect this line to print the value 68.

print(landmarks.num_parts)

Then, we will access each individual part by calling the parts method on the full_detection_object instance. This method takes no arguments and returns an array of objects of class Point.

We will iterate through each Point and draw it over the image with a call to the add_overlay_circle method on our window object. As first input we pass an object of class Point (representing the center of the circle) and as second we pass the circle radius. In essence, we will be representing each landmark point as a small circle.

for part in landmarks.parts():
     win.add_overlay_circle(part, 2)

The full loop is shown below.

for face in faces:
    landmarks = predictor(img, face)

    print(landmarks.num_parts)

    for part in landmarks.parts():
        win.add_overlay_circle(part, 2)

The complete code is shown below.

import dlib

img = dlib.load_rgb_image("C:/Users/N/Desktop/Test.jpg")
win = dlib.image_window(img, "Image")

detector = dlib.get_frontal_face_detector()
faces = detector(img)
win.add_overlay(faces)

predictor = dlib.shape_predictor("C:/Users/N/Desktop/shape_predictor_68_face_landmarks.dat")

for face in faces:
    landmarks = predictor(img, face)

    print(landmarks.num_parts)

    for part in landmarks.parts():
        win.add_overlay_circle(part, 2)

win.wait_until_closed()

Once again, simply run the code in a tool of your choice. You should get an output similar to figure 2. As can be seen, this time the landmarks are represented as small unconnected circles. If you check the command line, the value 68 should have also been printed, as expected.

**Figure 2** – Representing the facial landmarks as small circles.

Distinguish different parts of the face

In this section we will use different colors for the points of each different part of the face detected by the model. Most of the code will be similar to the previous section, except that we are going to divide the 68 parts of the full_detection_object in sub-arrays, each one representing a part of the face.

To draw each point inside each sub-array we will still call the add_overlay_circle method on the window object, passing a third argument with the color (when not specified, like in the previous section, it defaults to full red).

To specify a color, we need to pass a rgb_pixel object as this third argument of the add_overlay_circle method. The constructor of the class receives the R, the G and the B, as integers between 0 and 255.

The whole code is shown below.

import dlib

img = dlib.load_rgb_image("C:/Users/N/Desktop/Test.jpg")
win = dlib.image_window(img, "Image")

detector = dlib.get_frontal_face_detector()
faces = detector(img)
win.add_overlay(faces)

predictor = dlib.shape_predictor("C:/Users/N/Desktop/shape_predictor_68_face_landmarks.dat")

for face in faces:
    landmarks = predictor(img, face)

    parts = landmarks.parts()

    # jaw, ear to ear
    for part in parts[0:17]:
        win.add_overlay_circle(part, 2, dlib.rgb_pixel(255,0,0))

    # left eyebrow
    for part in parts[17:22]:
        win.add_overlay_circle(part, 2, dlib.rgb_pixel(0, 255, 0))

    # right eyebrow
    for part in parts[22:27]:
        win.add_overlay_circle(part, 2, dlib.rgb_pixel(0, 0, 255))

    # line on top of nose
    for part in parts[27:31]:
        win.add_overlay_circle(part, 2, dlib.rgb_pixel(0, 0, 0))

    # bottom part of the nose
    for part in parts[31:36]:
        win.add_overlay_circle(part, 2, dlib.rgb_pixel(255, 0, 255))

    # left eye
    for part in parts[36:42]:
        win.add_overlay_circle(part, 2, dlib.rgb_pixel(255, 255, 0))

    # right eye
    for part in parts[42:48]:
        win.add_overlay_circle(part, 2, dlib.rgb_pixel(0, 255, 255))

    # outer part of the lips
    for part in parts[48:60]:
        win.add_overlay_circle(part, 2, dlib.rgb_pixel(100, 100, 100))

    # inner part of the lips
    for part in parts[60:68]:
        print(part)
        win.add_overlay_circle(part, 2, dlib.rgb_pixel(255, 255, 255))

win.wait_until_closed()

The expected result is shown in figure 3. As can be seen, each different part of the face that we can identify is represented in a different color. Note that we can go as far as distinguish between the inner part and the outer part of the lips. In this particular face pose, some of the points of the inner part of the lips overlap, which is why we see only 5 white points instead of the 8 predicted by the model (this was not so noticeable in figure 2 because we used the same color for all the points).

**Figure 3** – Facial landmarks represented with different colors.

Detection with the 5 landmarks model

To finalize this tutorial, we will try to identify the face landmarks with the 5 landmarks detector. We will follow the approach of drawing a point per each landmark.

Naturally, when creating the shape_detector object, we need to pass the path to the corresponding model.

predictor = dlib.shape_predictor("C:/Users/N/Desktop/shape_predictor_5_face_landmarks.dat")

The complete code is shown below. Note that we have set a slightly bigger radius for the points, since the number is now lower.

import dlib

img = dlib.load_rgb_image("C:/Users/N/Desktop/Test.jpg")
win = dlib.image_window(img, "Image")

detector = dlib.get_frontal_face_detector()
faces = detector(img)
win.add_overlay(faces)

predictor = dlib.shape_predictor("C:/Users/N/Desktop/shape_predictor_5_face_landmarks.dat")

for face in faces:
    landmarks = predictor(img, face)

    for part in landmarks.parts():
        win.add_overlay_circle(part, 3)

win.wait_until_closed()

Figure 4 shows the expected result. As can be seen, this time we obtain the 5 points only.