Real Time Film-Lead Face Identify


Since I love Friends of six so much, I decide to make a demo for identifying their faces in the video. BTW, the demo is naive, you can make more effort on this for a better result. And real time means on a good GPU rather than a bad PC, since two CNN take a while.



The pipeline of my work is easy understood.
First, I collect 10 images for the six lead. Rachel Monica Phoebe Ross Joey Chandler.
Then, I use MTCNN to detect face, and crop these faces to save them. One of my reference, it uses opencv built-in function to finish this, it’s fast but less effective in tough condition.
After that, VGG_face2 is used to get features for each people, and save it to .pkl format.
Finally, a run demo is made to combine all above. For simplicity, I use Euclidean Distance here. You may change it to other metric way like cosine distance.

I will show you some details.

MTCNN for face detection

I have shown three ways for face detection in my previous article.

def corp_faces(images, image_path, save_path, pnet, rnet, onet):
    Use MTCNN crop faces
    :param images: a list, all images for this 'people'
    :param save_path: where to save crop face
    count = 0
    for image_name in images:
        print('processing ' + image_path + '/' + image_name)
        frame = cv2.imread(image_path + '/' + image_name)
        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

        if gray.ndim == 2:
            img = to_rgb(gray)

        bounding_boxes, _ = detect_face.detect_face(img, minsize, pnet, rnet, onet, threshold, factor)

        nrof_faces = bounding_boxes.shape[0]  # number of faces
        print('Total faces:{}'.format(nrof_faces))

        # crop and save
        for face_position in bounding_boxes:
            face_position = face_position.astype(int)
            crop = frame[face_position[1]:face_position[3], face_position[0]:face_position[2], :]

            cv2.imwrite(save_path + '/' + (str(count) + '.jpg'), crop)
            count += 1

Before using it, please clone my github. Or you can download the file in directory model_checkpoint only, and modify my code.

Use vgg_face2 get face features

VGG_face2 is a higher version of vgg face. You can choose base net of vgg resnet. Gratefully, rcmalli have made a keras version for vgg face. Vgg face will generate a 2048 dimentional vector.

pip install keras_vggface

Here is an official demo.

import numpy as np
from keras.preprocessing import image
from keras_vggface.vggface import VGGFace
from keras_vggface import utils

model = VGGFace() # default : VGG16 , you can use model='resnet50' or 

img = image.load_img('../image/ajb.jpg', target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = utils.preprocess_input(x, version=1) # or version=2
preds = model.predict(x)
print('Predicted:', utils.decode_predictions(preds))


After done tow steps above, you almost get there. The thing lefted is reading video and doing the pipeline above. I really suggest you try other metric ways.

def detect_face(self, video_path):
    cap = cv2.VideoCapture(video_path)

    # save video
    # Define the codec and create VideoWriter object
    out = cv2.VideoWriter('output.avi', cv2.VideoWriter_fourcc(*"XVID"), 20.0, (720, 400))

        ret, frame =
        if ret:
            gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

            if gray.ndim == 2:
                img = to_rgb(gray)

            # crop face
            bounding_boxes, _ = detect_face.detect_face(img, minsize, self.pnet, self.rnet, self.onet, threshold, factor)
            face_imgs = np.empty((len(bounding_boxes), self.face_size, self.face_size, 3))

            if bounding_boxes.shape[0]:
                for i, face_position in enumerate(bounding_boxes):
                    face_position = face_position.astype(int)
                    crop = frame[face_position[1]:face_position[3], face_position[0]:face_position[2], :]
                    if crop.shape[0] and crop.shape[1]:
                        crop_resize = cv2.resize(crop, (self.face_size, self.face_size), interpolation=cv2.INTER_AREA)
                        face_imgs[i, :, :, :] = crop_resize
                        cv2.rectangle(frame, (face_position[0], face_position[1]),
                                        (face_position[2], face_position[3]), (0, 255, 0), 2)

            # vgg_face2 features
            if len(face_imgs) > 0:
                features_faces = self.vgg_model.predict(face_imgs)
                predicted_names = [self.identify_face(features_face) for features_face in features_faces]

            # draw results
            if bounding_boxes.shape[0]:
                for index, face in enumerate(bounding_boxes):
                    label = "{}".format(predicted_names[index])
                    save_frame = draw_label(frame, (int(face[0]), int(face[1])), label)
                    out.write(save_frame)  # save
                out.write(frame)  # save

            #cv2.imshow('Friends', frame)
            #if cv2.waitKey(5) == 27:  # ESC key press

    # When everything is done, release the capture

How to use my repo

>>>git clone
1. collect peoples image you wanna identify
2. put it in `images` directory, subdirectory named by peoples' name
3. put video in root directory
4. change video name in `` to keep consistent
>>> python
>>> python
>>> python

Details refer to my repo. If you have better results, let me know.


That’s all, hope this help you. :)


Three ways for face detection

Face detection is one of the most popurlay field in computer vision. Recently I make some demos for face detection. Three ways has been test, python-opencv face++ API MTCNNWhat’s face detectionFace detection is not a hard way these days with deep...…

Face Detection继续阅读