Rendered image#

Overview#

The rendered image is the central modality of each datapoint you generate. All of the other modalities represent ground truths against which the images are compared.

This modality consists of the following files:

Relevant file	Location
environment.json	Camera folder
infrared_spectrum.png	Camera folder
lights_metadata.json	Datapoint folder
visible_spectrum_01.png	Camera folder

Relevant file	Location
environment.json	Camera folder
infrared_spectrum.png	Camera folder
lights_metadata.json	Scene folder
visible_spectrum_day.png, visible_spectrum_night.png, etc.	Camera folder

environment.json#

This file describes the environment and lighting that were used in the rendered image.

The file has the following format:

{
   "version": "1.0.0",
   "environments": [
      {
         "image_name": "visible_spectrum_01.png",
         "environment": "indoor",
         "time_of_day": "morning",
         "background": "hdri"
      }
   ]
}

Objects and fields:

version: String. Version tracking for this file. Whenever you access this file in a datapoint, make sure to check that the version matches what you expect it to be; otherwise its format and fields may not be recognized.
environments: List. Contains a list of the visual spectrum files in this folder and the environment that was used to generate each one. Each object in the list consists of the following:
- image_name: String. The filename of the rendered image. Most rendered images are named “visible_spectrum_01.png”, but some files are given unique names if they contain an image with unique properties. For example, an image that simulates near-infrared lighting is named “infrared_spectrum.png”.
- environment: String. The location that is simulated in the image’s lighting and/or background. Standard values are “indoor” and “outdoor”.
  Some unique lighting scenarios, such as near-infrared lighting, have the value “N/A”.
- time_of_day: String. The time of day that is simulated in the image’s lighting and/or background. Standard values are “morning”, “evening”, and “night”.
  Some unique lighting scenarios, such as near-infrared lighting, have the value “N/A”.
- background: String. Contains the type of background used in the image. Standard values are “hdri” and “transparent”.

This file provides details a cross-reference between each rendered image in the camera folder and information about the environment depicted in that image.

The file has the following format:

{
    "version": "1.0.0",
    "environments": [
        {
            "image_name": "visible_spectrum_day.png",
            "time_of_day": "day",
            "fps": 1,
            "frame_number": 4,
            "clutter": [
                {
                    "area_name": "table",
                    "level": "medium"
                },
                {
                    "area_name": "floor",
                    "level": "high"
                }
            ],
            "domain": "smart_office",
        },
        {
            "image_name": "visible_spectrum_night.png",
            "time_of_day": "night",
            "fps": 1,
            "frame_number": 4,
            "domain": "smart_office",
            "clutter": [
                {
                    "area_name": "table",
                    "level": "medium"
                },
                {
                    "area_name": "floor",
                    "level": "high"
                }
            ],
        },
    ]
}

Note

There are additional fields in actor_metadata.json that are part of the Actor metadata modality.

Objects and fields:

version: String. Version tracking for this file. Whenever you access this file in a datapoint, make sure to check that the version matches what you expect it to be; otherwise its format and fields may not be recognized.
environments: List. A list of the visual spectrum files in this folder and the environment that was used to generate each one. Each object in the list contains the following:
- image_name: String. The filename of one of the rendered images in this folder. Each visual spectrum image in the folder appears once in this list. Rendered images are named based on the lighting scenario: “visible_spectrum_day.png”, “visible_spectrum_night.png”, “infrared_spectrum”, and so on.
- time_of_day: String. The time of day that is simulated in that image’s lighting. Standard values are “day”, “evening”, and “night”. Some unique lighting scenarios, such as near-infrared lighting, have the value “N/A”.
- fps: Int. The number of frames per second in the animation. Each animation is 10 seconds long.
- frame_number: Int. The frame number represented in the image. Frame numbers are one-based.
- domain: String. The field of computer vision that this datapoints belongs to. Standard values include in_cabin for driver monitoring systems and smart_office for home office or meeting room setups.
- environment: String. The location that is simulated in the image’s lighting. Standard values are “indoor” and “outdoor”. Some unique lighting scenarios, such as near-infrared lighting, have the value “N/A”.
- clutter: List. A list of clutter locations in the scene and the amount of clutter appearing in each one. Each clutter location is labeled with area_name, and the amount of clutter is labeled with level - which can be low, medium, high, or none.

infrared_spectrum.png#

This file contains a near-infrared datapoint: an image of the scene under a near-infrared spotlight, using the settings you chose when you generated this datapoint. Each near-infrared datapoint (and associated modalities) appears in its own datapoint folder, separate from any visible spectrum datapoints you may have generated.

An example of a datapoint using a near-infrared spotlight (left) and a visual spectrum image of the same scene (right)

You defined the image resolution and the properties of the near-infrared spotlight when you generated this datapoint. The image resolution can be found under the resolution_px object in camera_metadata.json, in the Camera folder. The properties of the near-infrared spotlight can be found in lights_metadata.json, in the Datapoint folder.

lights_metadata.json#

This file defines any special lighting objects that were used in the rendered images of this datapoint. The file has this structure:

{
    "version": "1.0.0",
    "lights": {
         // One object for each special lighting object in the datapoint
    }
}

Objects and fields:

version: String. Version tracking for this file. Whenever you access this file in a datapoint, make sure to check that the version matches what you expect it to be; otherwise its format and fields may not be recognized.

Lighting object: Near-infrared lamp#

The near-infrared lamp is currently the only special lighting object available. The lamps are labeled sequentially: infra_red_lamp_0001, infra_red_lamp_0002, infra_red_lamp_0003, etc. (If this datapoint was generated on the platform rather than through the SDK, you were limited to one near-infrared lamp.)

Each infrared lamp object appears in the lights list with the following structure:

"infra_red_lamp_0001": {
    "type": "spot",
    "spectrum": "near_infrared",
    "location": {
        "x": 0.0,
        "y": -1.600000023841858,
        "z": 0.11999999731779099
    },
    "orientation": {
        "look_at_vector": {
            "x": 0.0,
            "y": 1.0,
            "z": 1.4901162614933128e-07
        },
        "up_vector": {
            "x": 0.0,
            "y": -1.4901162614933128e-07,
            "z": 1.0
        }
    }
    "falloff": 1.0,
    "beam_angle": 6.499999896287763,
    "brightness": 300.0,
}

Objects and fields:

type: String. The type of light that is described in this JSON object. For the near-infrared lamp, the value is “spot”.
spectrum: String. The spectrum of the light that is emitted. For the near-infrared lamp, the value is “near_infrared”.
location: Object. This object contains three Floats named “x”, “y”, and “z”, which give the coordinates of the near-infrared lamp in a global coordinate system measured in meters. See About our coordinate systems for details.

orientation: Object. This object contains two vectors that describe how the lamp is oriented in the 3D space. The orientation of the lamp is the same as the orientation of the camera in the scene, which you defined through the camera yaw, pitch, and roll controls when you created this datapoint.

look_at_vector: Object. This object contains the normalized x, y, and z components of the vector that defines the direction of the spotlight. The examples below assume the lamp begins at the default position of (0, -1.6, 0.12) and is oriented in the +y direction:

x: Float. The x axis runs from right to left.

../_images/pitch0yaw3.png — *A 3° rotation in the -x direction.*#

../_images/pitch0yaw15.png — *A 1.5° rotation in the -x direction.*#

../_images/pitch0yaw0.png — *Default orientation.*#

y: Float. The y axis runs from forward to back. The lamp points by default at y=+1, so any rotation in the ±x or ±z directions will lower the value of y.

z: Float. At the lamp’s default position and orientation, the z axis runs from bottom to top. By default, the x component of the lamp’s look-at vector is 0, which means it is tilted neither downward nor upward.

../_images/pitch3yaw0.png — *A 3° rotation in the +z direction.*#

../_images/pitch15yaw0.png — *A 1.5° rotation in the +z direction.*#

up_vector: Object. This object contains the normalized x, y, and z components of the vector that defines the lamp’s “up” direction, providing you with the lamp’s orientation independent of the direction of the beam.
- x: Float. The x axis runs from left to right, and has a default of 0. Lowering the value of x tilts the lamp to the left; raising it tilts the lamp to the right.
- y: Float. The y axis runs from back to front and has a default of 0. Lowering the value of y tilts the lamp so that its top is facing away from the subject; raising the value of y tilts the lamp so that its top is facing towards the subject.
- z: Float. The z axis runs from bottom to top and has a default of 1, which means the top of the lamp is pointing straight up.

falloff: Float. A measure of how blurred the edge of the spotlight is. Falloff describes how far into the spotlight you have to go, starting from the edge and moving towards the center, before you reach the point where the beam is at full strength.

A falloff of 0 means there is no distance between the full-strength part of the beam and its edge; in other words, the light is at full strength throughout the beam, with a sudden cutoff between light and dark. A falloff of 1 means that the light is at full strength only at a single point in the very center, and dissipates gradually as you approach the edge.

You defined the value of this field as a percentage, between 0% to 100%, when you created this datapoint; in this file it is converted to a decimal between 0 and 1.

Examples of falloff#
beam_angle: Float. The width of the near-infrared spotlight, measured in degrees. The spotlight is a point source, and the light coming out of it describes a cone in space; this angle is the angle at the point of that cone. You defined the value of this field when you created this datapoint.
brightness: Float. The radiant power of the near-infrared spotlight, measured in watts. Radiant power is the amount of light energy that is output by the lamp. You defined the value of this field when you created this datapoint.

Note

The wattage values listed on a commercial light bulb are not the same as the value in this field. Those values generally refer to power draw of one of two types: 1) the amount of electrical power drawn by the bulb itself, or 2) the amount of power that would be drawn by an equivalent incandescent lightbulb. The value in this field refers to light output, not power draw.

visible_spectrum_*.png#

visible_spectrum_01.png contains the visual spectrum datapoint itself. The datapoint is an image of the subject under a visible spectrum lighting condition that you selected. To look up the specific lighting condition that was used in this image datapoint, see the environment.json file above.

../_images/Visspec.jpeg — *A sample visible spectrum image*#

The images are in the standard png format. You defined the image resolution when you generated this datapoint; it can be found under the resolution_px object in camera_metadata.json in the Camera folder.