Generating a motion sequence#

Datagen’s Humans in Context (HIC) platform generates motion sequences of actors in specific environments, such as the driver’s seat of a car or the front walk of a house.

This guide demonstrates some of the basics of how to use our SDK to submit a data generation request for these sequences.

Note

If this is your first visit to this page, it may take up to 5 minutes to activate the kernel and install the necessary packages.

!pip install datagen-tech

Importing relevant modules#

Important

If you are copy/pasting this code into your Python environment, make sure to install the datagen-tech python package first.

Begin by importing modules:

import random

import datagen
from datagen import api
from datagen.api import catalog, assets
from datagen.api.catalog import attributes

Choosing a sequence#

You can choose a sequence at random from our catalog of sequences:

sequence = random.choice(catalog.sequences.get())

Or you can choose a random sequence from a specific domain, such as home security or fitness:

print(attributes.HICDomain.__members__.keys())
sequence = random.choice(catalog.sequences.get(domain=attributes.HICDomain.IN_CABIN))

Or you can choose a specific sequence using its UUID:

sequence = catalog.sequences.get("home_office-0f3b5b8c-0a93-4e48")

Sequence structure#

sequence

Each sequence in the catalog contains the following attributes:

  • id: A UUID that identifies this sequence. This UUID is of the format [domain]-[hash], such as “in_cabin-0de2a19e-68ee-49fa”

  • attributes: A description of certain aspects of the scene:

    • The domain and subdomain

    • The specific behavior being depicted in this sequence (such as trip_fall_over_object or driver_adjusting_mirror)

    • Information about the actor in the scene: ethnicity, age, and gender

  • camera: (Editable) A camera object that defines its position and orientation in the scene as well as intrinsic parameters such as field of view, resolution, and frames per second (fps)

  • background: (Editable, optional) The background of the scene, as selected from the backgrounds asset catalog, required if using a visible-spectrum camera

  • lights: (Editable, optional) Defines the parameters of near-infrared lighting in the scene, required if using a near-infrared camera

  • clutter_areas: (Editable, optional) A list of areas in the scene where you want the platform to add clutter

  • presets: Contains a list of recommended camera positions and a list of valid clutter areas

Customizing your sequence#

Customizing the camera#

The camera object contains a number of editable and customizable features:

sequence.camera

You can apply one of the preset camera positions to the camera’s extrinsic parameters (location and orientation). This helps ensure you get a good view of the scene:

# Different sequences have different presets.
# This code uses a preset that is valid for the sequence home_office-0f3b5b8c-0a93-4e48
sequence.camera.extrinsic_params = sequence.presets.cameras["monitor_front_camera"]

You can also make more individualized changes. You have already selected a preset; let us deviate from it slightly by tilting the camera downward by one degree. Let us also change the image resolution and increase the number of frames per second:

sequence.camera.extrinsic_params.rotation.pitch -= 1
sequence.camera.intrinsic_params.resolution_height = 256
sequence.camera.intrinsic_params.resolution_width = 256
sequence.camera.intrinsic_params.fps = 30

Selecting a background#

You can select a background for your scene from the backgrounds asset catalog, as follows:

sequence.background = random.choice(catalog.backgrounds.get(domain=attributes.HICDomain.SMART_OFFICE))
sequence.background

This filters the catalog to return one of the backgrounds appropriate for the Smart Office domain.

Important

We strongly recommend that you use a domain-appropriate background for your scene. We cannot guarantee the quality of the results if you use other types of backgrounds.

Adding clutter#

If you want to add clutter (irrelevant objects) to the scene, to teach your model to ignore them, you will need two values:

  • The clutter_areas preset gives you the list of valid clutter locations

  • The assets.ClutterLevel enum gives you the list of valid clutter amounts

# Different sequences have different clutter areas.
# This code uses a clutter area that is valid for the sequence home_office-0f3b5b8c-0a93-4e48
print("Valid clutter areas: ", sequence.presets.clutter_areas)
print("Valid clutter amounts: ", assets.ClutterLevel.__members__.keys())
sequence.clutter_areas["floor"] = assets.ClutterLevel.MEDIUM

Creating and extracting your data request#

Turning your data request into an uploadable file takes two steps: First you have to create a SequenceRequest object that contains the list of sequences you want to generate, and then you need to call the api.dump() function:

sequence_request = assets.SequenceRequest(sequences=[sequence])
api.dump(sequence_request)

You now have a .json file in your Python environment, ready for upload to the platform.

Because this browser-based notebook runs the code in an isolated kernel, we need to take one last step to extract the JSON file for download. (This step is not necessary when you are working in your local environment.)

DST_JSON_PATH="datagen_data_request.json"

from IPython.display import HTML
import os, base64

def make_download_link(DST_JSON_PATH):
  filename = os.path.basename(DST_JSON_PATH)
  with open(DST_JSON_PATH, "rb") as f:
    data = base64.b64encode(f.read()).decode()
    return HTML(f'<h4>Click here to download '
                f'<a href="data:application/binary;base64,{data}" '
                f'download={filename}>{filename}</a>.</h4>')

make_download_link(DST_JSON_PATH)

Take this JSON file and upload it to the Humans in Context platform using the Upload JSON button. Generation of your data will start immediately.