Camera Handling

How to interact with cameras/sensors in ReconstructMe SDK

In ReconstructMe 3D cameras are accessed via the Sensor interface. In particular one needs to create a reme_sensor_t for each 3D camera to interact with. ReconstructMe supports a wide range of modern 3D cameras such as the Microsoft Kinect (Windows/XBox) or the ASUS Xtion family. Additionally ReconstructMe provides sensors that stream from files.

Sensor State Overview

Here is an overview of the states a sensor can be in.

State Closed

Once a sensor is created (reme_sensor_create) it is closed state. Depending on the arguments provided a successful create call ensures at least that the necessary drivers are installed on the current machine.

The transit to the Open state is accomplished through a successful reme_sensor_open call. This command takes the current sensor configuration into account (reme_sensor_bind_camera_options) and tries to open the requested sensor.

State Open

Once a sensor has been successfully opened you can access and potentially modify its capture properties (reme_sensor_bind_capture_options, reme_sensor_apply_capture_options) and access supported image frames (reme_sensor_get_image, reme_sensor_prepare_image). In contrast to convential cameras each sensor in ReconstructMe has a three-dimensional position (reme_sensor_get_position, reme_sensor_set_position) with respect to the world coordinate system (see Coordinate Systems Introduction). This position can either be modified by hand or automatically determined by tracking the sensor position via matching data in the reconstruction volume in current sensor data input.

The sensor moves into the Closed state by issuing a reme_sensor_close. In case a sensor in Closed state is not need anymore it is adviced to destroy the sensor using reme_sensor_destroy.

Creating Sensors

Each sensor is created through a call to reme_sensor_create. This method accepts a driver argument that refers to a single backend identifier for communication with the sensor (i.e. OpenNI, Kinect SDK, ...), or a sensor configuration file to use. Besides single arguments this method accepts a list of drivers / configuration paths to test.

In case a list of sensors is passed the first one working will be returned. Working in this context is defined by value of require_can_open argument:

• TRUE It is ensured that the sensor returned can be opened successfully: I.e when the driver argument is openni it is ensured that the sensor can be opened using the default OpenNI configuration; when the driver argument is a path to a configuration file it is ensured that the sensor can be opened using the particular configuration stored.

• FALSE It is ensured that the necessary drivers are installed on the current machine but no test to ensure a successful opening of the sensor is performed.

Here is a quick example

  // Create any sensor
  reme_sensor_t s;
  reme_sensor_create(c, "mskinect", true, &s);

Opening Sensors

When a sensor is opened through the reme_sensor_open call it uses the current state of the camera configuration (reme_sensor_bind_camera_options). In the following example the RGB image stream is disabled and IR stream for the Microsoft Kinect is enabled. The sensors usally offer two streams in ReconstructMe:

• REME_IMAGE_DEPTH depth image stream used for reconstruction
• REME_IMAGE_AUX Auxilary image. Usually RGB but can be anything depending on the sensor configuration.

  // Access camera properties and disable rgb and enable ir stream
  reme_options_t o_cam;
  reme_options_create(c, &o_cam);
  
  reme_sensor_bind_camera_options(c, s, o_cam);
  reme_options_set(c, o_cam, "aux_stream.type", "STREAM_IR");

  // Open the sensor. This takes camera options into account.
  reme_sensor_open(c, s);

Reading and Writing Capture Properties

Opened sensors allow access to capture properties. These properties reflect information about the current state of the sensor (frame sizes, support for frame types, position of tilt motors etc.). For example to access the frame size of the auxilary image one would perform the following.

  reme_options_t o_cap;
  reme_options_create(c, &o_cap);

  // Bind the capture options. See documentation of this method for a list of properties available.
  reme_sensor_bind_capture_options(c, s, o_cap);

  // Read frame size of auxilary image (IR).
  bool supported;
  int width, height;
  
  reme_options_get_bool(c, o_cap, "frame_info.supports_aux", &supported);
  reme_options_get_int(c, o_cap, "frame_info.aux_size.width", &width);
  reme_options_get_int(c, o_cap, "frame_info.aux_size.height", &height);

  printf("Auxilary supported: %s (%i x %i)", supported ? "true" : "false", width, height);

To modify capture properties such as the tilt angle of the Kinect one would perform the following steps

  // Here we modify the position of the tilt-angle of the Microsoft Kinect for Windows camera.
  // We clear all values before indicating that we are just interesting in setting the tilt angle
  // and no other value.

  reme_options_clear(c, o_cap);
  reme_options_set_int(c, o_cap, "tilt_angle", 0);

  // Apply capture properties
  reme_sensor_apply_capture_options(c, s, o_cap);
  

Accessing and Displaying Images

Once the sensor is open you can access its images. The example below illustrates this

  reme_image_t aux;
  reme_image_create(c, &aux);

  // Grab image. Synchronizes all image streams.
  reme_sensor_grab(c, s);

  // Prepare just the auxilary image
  reme_sensor_prepare_image(c, s, REME_IMAGE_AUX);

  // Get the image
  reme_sensor_get_image(c, s, REME_IMAGE_AUX, aux);

  // Visualize image and wait for user to close viewer
  reme_viewer_t v;
  reme_viewer_create_image(c, "Auxilary Image", &v);
  reme_viewer_add_image(c, v, aux);
  reme_viewer_update(c, v);
  reme_viewer_wait(c, v);

Positioning the Sensor

Each sensor has a 3D position and orientation with respect to the world coordinate system. The page Coordinate Systems and Sensor Positioning explains this in detail.

Tracking the Sensor Position

The position of the sensor with respect to the world coordinate sytem can be tracked automatically using reme_sensor_track_position. This is the heart of the ReconstructMe engine as it automatically determines the position of the sensor while it is moved in real-time.

Since version 1.4 ReconstructMe supports two different tracking algorithms:

• local search Local search is fast and succeeds when the camera movement between two subsequent frames is small. It used to be the only search method prior to 1.4.
• global search Added in version 1.4 based on algorithms of CANDELOR (http://www.candelor.com). Global search is slower than local search but succeeds in cases where the camera movement between two subsequent frames is large.

ReconstructMe uses a combination of the above algorithms to improve camera tracking. The behaviour of ReconstructMe can be configured with reme_sensor_set_trackmode

• REME_SENSOR_TRACKMODE_AUTO This is the default. ReconstructMe uses a combination of available search algorithms. When tracking in the last frame succeeded ReconstructMe tries first to use local search. If that fails, global search followed by a local search is attempted. If that fails as well tracking for the given frame fails and ReconstructMe moves the virtual camera into the most recent recovery position (reme_sensor_set_recovery_position). Similarily, if tracking for the last frame failed, ReconstructMe attempts to use global search followed by a local search.
• REME_SENSOR_TRACKMODE_LOCAL ReconstuctMe will only use local search independent of the tracking status of the last frame. This behaviour is identical to the behaviour of ReconstructMe before 1.4
• REME_SENSOR_TRACKMODE_GLOBAL ReconstructMe will always first use global search followed by a fine alignment of local search.

Note that although CANDELOR supports truly global search, the implementation in ReconstructMe uses the the volume data as viewed by the current camera position as the model. This leads to a partial global behaviour as the overlap between the model and the current frame has to exceed a predefined threshold. This is subject to change in future versions.

Additionally one might provide tracking hints that can temporarily change the behaviour of the tracking. See reme_sensor_set_trackhint for details.

Please see the Getting Started page for usage. You might also be interested in example_reconstructmesdk_sensor_multi_independent.cpp that allows you to track multiple sensors.

Updating the Reconstruction Volume

Once the sensor position with respect to the world coordinate is known the current sensor data can be pushed into the reconstruction volume. The reconstruction volume will then care about fusioning the data to a global model. Please refer to the Getting Started page for an introduction.

Closing the Sensor

An opened sensor can be closed as follows.

  // Close sensor communication
  reme_sensor_close(c, s);

Destroying the Sensor

A closed sensor can be destroyed as follows.

  // Destroy sensor
  reme_sensor_destroy(c, &s);