Semantic segmentation#


The ground truth exposed in this modality is the parent body part of each pixel visible in the corresponding visual spectrum image.

This modality consists of the following files:

Relevant file



Camera folder


Datapoint folder


This file contains a version of the visual spectrum image that has been converted into a semantic segmentation map.

Image1 Image2

A semantic segmentation map of a human head (left) and the corresponding visual spectrum image (right)

In a semantic segmentation map, the color value of each pixel has been replaced with a color value that indicates which class it belongs to. The color key is found in semantic_segmentation_metadata.json.

Using this ground truth, you can train your model to recognize individual parts of the face.

To process a semantic segmentation map, we recommend using code along the following lines:

import cv2

def load_segmentation(path):
    segmentation = cv2.cvtColor(cv2.imread(segmentation_path, cv2.IMREAD_UNCHANGED), cv2.COLOR_BGR2RGB)
    return segmentation

path = "semantic_segmentation.png"
segmentation = load_segmentation(path)

We strongly recommend against using PIL or plt.imread to load the semantic segmentation file, because the uint16 color will be reduced to uint8 and the color key will no longer be accurate.

See for more about how to use semantic segmentation to isolate objects in the datapoint.


This file contains a lookup table of the RGB values that were assigned to each semantic object in our database. All datasets use the same lookup table, though each individual instance of this file is filtered to list only items present in the scene. For example, if none of the actors in the scene have a beard, the “beard” item will not appear in the file.

The lookup table is constantly being added to and adjusted as we introduce more features into our platform, so we recommend using whichever version accompanied the datapoint you are working with.

At the top of this file is a single field called version, a String that provides version tracking for this file. Whenever you access this file in a datapoint, make sure to check that the version matches what you expect it to be; otherwise its format and fields may not be recognized.

Click here to view an example of a complete semantic_segmentation_metadata.json file.