Eye keypoints#
Overview#
The ground truth exposed in this modality is a set of eye keypoints using a standard developed by Datagen.
This modality consists of the following file:
Relevant file |
Location |
---|---|
actor_metadata.json |
Datapoint folder |
This modality consists of the following files:
Relevant file |
Location |
---|---|
eyes_key_points.json |
key_points folder |
all_key_points.json |
key_points folder |
actor_metadata.json#
This file contains information on the subject’s eyes, in two parts:
The 3D and 2D coordinates of a set of eye landmarks defined by Datagen.
Information about the direction of gaze, including normalized vectors and the target of the gaze in 3D coordinates.
The file has the following format:
"apex_of_cornea_point": {
// 3D and 2D coordinates of the apex of the cornea, for each eye.
},
"center_of_rotation_point": {
// 3D and 2D coordinates of the eyeball's center of rotation, for each eye.
},
"iris_circle": {
// 3D and 2D coordinates of 12 keypoints that define the circle of the iris, for each eye.
},
"center_of_iris_point": {
// 3D and 2D coordinates of the center of the iris, for each eye.
},
"pupil_circle": {
// 3D and 2D coordinates of 12 keypoints that define the circle of the pupil, for each eye.
},
"center_of_pupil_point": {
// 3D and 2D coordinates of the center of the pupil, for each eye.
},
"eye_gaze": {
// Detailed information about the direction in which the subject is looking, for each eye.
},
"eyelid_closure_intensity_level": 1
Note
There are additional fields in actor_metadata.json that are part of the Actor metadata modality.
Objects and fields:
apex_of_cornea_point: Object. This object contains the 2D (pixel) <2D-coordinates> and 3D (global) <3D-coordinates> coordinates of the apex (the highest point) of the cornea. The object has the following structure; all values are Floats:
"apex_of_cornea_point": { "2d": { "camera_1": { "right_eye": { "x": 78.11973863600198, "y": 87.67236783337898 }, "left_eye": { "x": 78.28364055998486, "y": 166.6936280299794 } } }, "3d": { "right_eye": { "x": 0.7163663506507874, "y": 0.07476498186588287, "z": 0.16999228298664093 }, "left_eye": { "x": 0.7749997973442078, "y": 0.07624759525060654, "z": 0.17110392451286316 } } },
The apex of the cornea is a crucial part of determining the optical axis vector, which is the direction that a person appears to be looking from the point of view of an external observer.
The apex of the cornea.#
center_of_rotation_point: Object: This object contains the 2D (pixel) <2D-coordinates> and 3D (global) <3D-coordinates> coordinates of the center of the eyeball, around which the eyeball rotates. The object has the following structure; all values are Floats:
"center_of_rotation_point": { "2d": { "camera_1": { "right_eye": { "x": 78.37613239589773, "y": 89.64786393539038 }, "left_eye": { "x": 78.7087573615332, "y": 167.01729179913585 } }, }, "3d": { "right_eye": { "x": 0.716782808303833, "y": 0.08541443198919296, "z": 0.16397099196910858 }, "left_eye": { "x": 0.7742490172386169, "y": 0.08667740225791931, "z": 0.16510838270187378 } } },
The center of rotation is a crucial part of determining the visual axis vector, which is the direction that a person is actually looking.
The eye’s center of rotation.#
iris_circle: Object. This object contains the coordinates of a series of points that define the iris. The coordinates are provided in both 3D (global) <3D-coordinates> and 2D (pixel) <2D-coordinates> form. The object has the following overall structure:
"iris_circle": { "2d": { "camera_1": { "right_eye": [ //List of 12 points in 2D format ], "left_eye": [ //List of 12 points in 2D format ] } }, "3d": { "right_eye": [ //List of 12 points in 3D format ], "left_eye": [ //List of 12 points in 3D format ] } },
Each list of 2D points contains twelve objects with the following structure, each containing two Floats. This gives you not only which pixel contains the keypoint but also precisely where the keypoint is inside the pixel:
{ "x": 70.0365604652201, "y": 88.08070188104806 },
Each list of 3D points contains twelve objects with the following structure, each of which contains three Floats:
{ "x": 0.7164850234985352, "y": 0.08002109080553055, "z": 0.17416279017925262 },
Twelve keypoints that define the boundary of the iris, and another point at its center.#
center_of_iris_point: Object. This object contains the 2D (pixel) <2D-coordinates> and 3D (global) <3D-coordinates> coordinates of the center of the iris. The object has the following structure; all values are Floats:
"center_of_iris_point": { "2d": { "camera_1": { "right_eye": { "x": 78.17674222131468, "y": 88.09278511201991 }, "left_eye": { "x": 78.37546983864796, "y": 166.7629219787955 } } }, "3d": { "right_eye": { "x": 0.7165213823318481, "y": 0.07704445719718933, "z": 0.16874763369560242 }, "left_eye": { "x": 0.7748080492019653, "y": 0.07848701626062393, "z": 0.16979405283927917 } } },
pupil_circle: Object. This object contains the coordinates of a series of points that define the pupil. The coordinates are provided in both 3D (global) <3D-coordinates> and 2D (pixel) <2D-coordinates> form. The object has the following overall structure:
"pupil_circle": { "2d": { "camera_1": { "right_eye": [ //List of 12 points in 2D format ], "left_eye": [ //List of 12 points in 2D format ] } }, "3d": { "right_eye": [ //List of 12 points in 3D format ], "left_eye": [ //List of 12 points in 3D format ] } },
Each list of 2D points contains twelve objects with the following structure, each containing two Floats. This gives you not only which pixel contains the keypoint but also precisely where the keypoint is inside the pixel:
{ "x": 108.69194143549674, "y": 108.4702370774488 },
Each list of 3D points contains twelve objects with the following structure, each of which contains three Floats:
{ "x": 0.7164993286132812, "y": 0.07862702757120132, "z": 0.17070025205612183 },
Twelve keypoints that define the boundary of the pupil, and another point at its center.#
center_of_pupil_point: Object. This object contains the 2D (pixel) <2D-coordinates> and 3D (global) <3D-coordinates> coordinates of the center of the pupil. The object has the following structure; all values are Floats:
"center_of_pupil_point": { "2d": { "camera_1": { "right_eye": { "x": 78.18429623941194, "y": 88.14901093221658 }, "left_eye": { "x": 78.38774457906169, "y": 166.77200226278353 } } }, "3d": { "right_eye": { "x": 0.7164666056632996, "y": 0.07730603218078613, "z": 0.16853395104408264 }, "left_eye": { "x": 0.7747995257377625, "y": 0.07873070985078812, "z": 0.1696217656135559 } } },
The center of the pupil is a crucial part of determining the optical axis vector, which is the direction that a person appears to be looking from the point of view of an external observer.
eye_gaze: Object. This object contains information on where the subject is looking, presented in three ways:
axis_directions: Object. This object contains the normalized optical and visual axis vectors for each eye.
The optical axis vector is the direction that the subject appears to be looking from the point of view of an external observer. It is calculated using the following formula:
\[optical~axis~direction = apex~of~cornea~point - center~of~pupil~point\]The visual axis vector is the direction that the subject is actually looking. (The direction of this vector is always very slightly offset from the direction of the optical axis vector.) The visual axis vector is calculated using the following formula:
\[visual~axis~direction = center~of~rotation~point - fovea~point\]The optical and visual axis vectors are presented using this format:
"axis_directions": { "right_eye": { "axis_directions": { "visual_axis_direction": { "x": 0.014786778155726103, "y": -0.8652962281691663, "z": 0.5010427014816077 }, "optical_axis_direction": { "x": -0.034199164818520936, "y": -0.8668075176884541, "z": 0.49746873711269146 } } }, "left_eye": { "axis_directions": { "visual_axis_direction": { "x": 0.009155659821882036, "y": -0.8657963448364028, "z": 0.5003127653389934 }, "optical_axis_direction": { "x": 0.0690889121195633, "y": -0.8566151210336242, "z": 0.5113093551253336 } } } },
target_point: Object. This object contains the target of the subject’s gaze in global 3D coordinates <3D-coordinates>. In humans, This is the point of convergence of the direction of gaze of the right and left eyes are looking at the same point, which is presented as a set of three Floats, as follows:
"target_point": { "right_eye": { "x": 0.9370604157447815, "y": -13.373147964477539, "z": 7.9055495262146 }, "left_eye": { "x": 0.9370604157447815, "y": -13.373147964477539, "z": 7.9055495262146 } },
eye_gaze_direction_type: String. This object contains a brief description of the general direction that the subject is looking. Typical values are forward, up, down, left, right, top_left, top_right, bottom_left, and bottom_right.
eyelid_closure_intensity_level: Int. This object describes how much the subject’s eyes are closed, on a scale of 1 (completely open) to 5 (completely closed).
Using this ground truth, you can train your model to recognize the shape, location, and substructures of the human eye, as well as estimate the direction of gaze.
On this platform, the data in this file is not relevant to this modality.
eyes_key_points.json#
This file is not provided in the Faces platform.
The 3D and 2D coordinates of a set of eye landmarks defined by Datagen.
The file has the following format:
{
"version": "2.0.0",
"eyes": {
"left": {
"cornea": {
"apex": {
//3D and 2D coordinates
},
"center": {
//3D and 2D coordinates
}
},
"pupil": {
"circle": {
"0": {
//3D and 2D coordinates
},
"1": {
//3D and 2D coordinates
},
"2": {
//3D and 2D coordinates
},
"3": {
//3D and 2D coordinates
},
"4": {
//3D and 2D coordinates
},
"5": {
//3D and 2D coordinates
},
"6": {
//3D and 2D coordinates
},
"7": {
//3D and 2D coordinates
},
"8": {
//3D and 2D coordinates
},
"9": {
//3D and 2D coordinates
},
"10": {
//3D and 2D coordinates
},
"11": {
//3D and 2D coordinates
}
},
"center": {
//3D and 2D coordinates
}
},
"rotation": {
"center": {
//3D and 2D coordinates
}
},
"iris": {
"circle": {
"0": {
//3D and 2D coordinates
},
"1": {
//3D and 2D coordinates
},
"2": {
//3D and 2D coordinates
},
"3": {
//3D and 2D coordinates
},
"4": {
//3D and 2D coordinates
},
"5": {
//3D and 2D coordinates
},
"6": {
//3D and 2D coordinates
},
"7": {
//3D and 2D coordinates
},
"8": {
//3D and 2D coordinates
},
"9": {
//3D and 2D coordinates
},
"10": {
//3D and 2D coordinates
},
"11": {
//3D and 2D coordinates
}
},
"center": {
//3D and 2D coordinates
}
},
"right": {
"cornea": {
"apex": {
//3D and 2D coordinates
},
"center": {
//3D and 2D coordinates
}
},
"pupil": {
"circle": {
"0": {
//3D and 2D coordinates
},
"1": {
//3D and 2D coordinates
},
"2": {
//3D and 2D coordinates
},
"3": {
//3D and 2D coordinates
},
"4": {
//3D and 2D coordinates
},
"5": {
//3D and 2D coordinates
},
"6": {
//3D and 2D coordinates
},
"7": {
//3D and 2D coordinates
},
"8": {
//3D and 2D coordinates
},
"9": {
//3D and 2D coordinates
},
"10": {
//3D and 2D coordinates
},
"11": {
//3D and 2D coordinates
},
"center": {
//3D and 2D coordinates
}
},
"rotation": {
"center": {
//3D and 2D coordinates
}
},
"iris": {
"circle": {
"0": {
//3D and 2D coordinates
},
"1": {
//3D and 2D coordinates
},
"2": {
//3D and 2D coordinates
},
"3": {
//3D and 2D coordinates
},
"4": {
//3D and 2D coordinates
},
"5": {
//3D and 2D coordinates
},
"6": {
//3D and 2D coordinates
},
"7": {
//3D and 2D coordinates
},
"8": {
//3D and 2D coordinates
},
"9": {
//3D and 2D coordinates
},
"10": {
//3D and 2D coordinates
},
"11": {
//3D and 2D coordinates
}
},
"center": {
//3D and 2D coordinates
}
}
}
}
}
Every keypoint in this file is presented in both 2D and 3D coordinates. The 2D coordinates <2D coordinates> are the location of the key point in the accompanying RGB image <Rendered image>, which gives you not only the pixel in which the keypoint appears but also the precise location inside that pixel. The 3D coordinates <3D coordinates> are the location of the key point in the global coordinate system, measured in meters.
Each keypoint is presented using the same format. All values are Floats:
"global_3d": {
"x": 6.167073726654053,
"y": -2.6398603916168213,
"z": 1.6212266683578491
},
"pixel_2d": {
"x": 108.72337095079274,
"y": 108.44930035598685
}
Objects and fields:
version: String. Version tracking for this file. Whenever you access this file in a datapoint, make sure to check that the version matches what you expect it to be; otherwise its format and fields may not be recognized.
eyes: Object. Contains data about the human eyes.
left and right: Objects. Divides the information about the eyes into data about the left eye and data about the right eye. These two objects have identical structure, as follows:
cornea: Object. Data about the location of the cornea, which is the transparent cover over the front of the eye. This object contains the following points:
apex: Object. The coordinates of the apex of the cornea, which is the cornea’s highest point.
center: Object. The coordinates of the geometric center of the cornea, which is not necessarily its highest point.
pupil: Object. Data about the location of the pupil, which is the black hole at the center of the iris. Contains the following objects:
circle: Object. Contains 12 coordinates that define the circle of the pupil, based on a standard developed by Datagen. These coordinates are numbered 0 to 11.
center: Object. The coordinates of the center of the pupil.
rotation: Object. Data about the rotation of the eye. Contains the following point:
center: Object. The coordinates of the center of rotation of the eyeball.
iris: Object. Data about the location of the iris, which is the ring around the pupil that gives the eye its color.
circle: Object. Contains 12 coordinates that define the circle of the iris, based on a standard developed by Datagen. These coordinates are numbered 0 to 11.
center: Object. The coordinates of the center of the iris.
Using this ground truth, you can train your model to recognize the shape, location, and substructures of the human eye.
all_key_points.json#
On this platform, the data in this file is not relevant to this modality.
For convenience, this file includes the same set of keypoints as eyes_key_points.json above, located at this position in the hierarchy:
{
"eye": {
"left": {
// keypoints of the left eye
}
"right": {
// keypoints of the right eye
}
}
}
For details on the structure of the “left” and “right” objects, see eyes_key_points.json above.
Note
all_key_points.json also includes additional eye keypoints that represent part of the Body keypoints modality.