Camera Handling

How to interact with cameras/sensors in ReconstructMe SDK

In ReconstructMe 3D cameras are accessed via the Sensor interface. In particular one needs to create a reme_sensor_t for each 3D camera to interact with. ReconstructMe supports a wide range of modern 3D cameras such as the Microsoft Kinect (Windows/XBox) or the ASUS Xtion family. Additionally ReconstructMe provides sensors that stream from files.

Sensor State Overview

Here is an overview of the states a sensor can be in.

State Closed

Once a sensor is created (reme_sensor_create) it is closed state. Depending on the arguments provided a successful create call ensures at least that the necessary drivers are installed on the current machine.

The transit to the Open state is accomplished through a successful reme_sensor_open call. This command takes the current sensor configuration into account (reme_sensor_bind_camera_options) and tries to open the requested sensor.

State Open

Once a sensor has been successfully opened you can access and potentially modify its capture properties (reme_sensor_bind_capture_options, reme_sensor_apply_capture_options) and access supported image frames (reme_sensor_get_image, reme_sensor_prepare_image). In contrast to convential cameras each sensor in ReconstructMe has a three-dimensional position (reme_sensor_get_position, reme_sensor_set_position) with respect to the world coordinate system (see Coordinate Systems Introduction). This position can either be modified by hand or automatically determined by tracking the sensor position via matching data in the reconstruction volume in current sensor data input.

The sensor moves into the Closed state by issuing a reme_sensor_close. In case a sensor in Closed state is not need anymore it is adviced to destroy the sensor using reme_sensor_destroy.

Creating Sensors

Each sensor is created through a call to reme_sensor_create. This method accepts a driver argument that refers to a single backend identifier for communication with the sensor (i.e. OpenNI, Kinect SDK, ...), or a sensor configuration file to use. Besides single arguments this method accepts a list of drivers / configuration paths to test.

In case a list of sensors is passed the first one working will be returned. Working in this context is defined by value of require_can_open argument:

• TRUE It is ensured that the sensor returned can be opened successfully: I.e when the driver argument is openni it is ensured that the sensor can be opened using the default OpenNI configuration; when the driver argument is a path to a configuration file it is ensured that the sensor can be opened using the particular configuration stored.
• FALSE It is ensured that the necessary drivers are installed on the current machine but no test to ensure a successful opening of the sensor is performed.

Here is a quick example

    // Create any sensor
    reme_sensor_t s;
    reme_sensor_create(c, "mskinect", true, &s);

Opening Sensors

When a sensor is opened through the reme_sensor_open call it uses the current state of the camera configuration (reme_sensor_bind_camera_options). In the following example the RGB image stream is disabled and IR stream for the Microsoft Kinect is enabled. The sensors usally offer two streams in ReconstructMe:

• REME_IMAGE_DEPTH depth image stream used for reconstruction
• REME_IMAGE_AUX Auxilary image. Usually RGB but can be anything depending on the sensor configuration.

    // Access camera properties and disable rgb and enable ir stream
    reme_options_t o_cam;
    reme_options_create(c, &o_cam);

    reme_sensor_bind_camera_options(c, s, o_cam);
    reme_options_set(c, o_cam, "aux_stream.type", "STREAM_IR");

    // Open the sensor. This takes camera options into account.
    reme_sensor_open(c, s);

Reading and Writing Capture Properties

Opened sensors allow access to capture properties. These properties reflect information about the current state of the sensor (frame sizes, support for frame types, position of tilt motors etc.). For example to access the frame size of the auxilary image one would perform the following.

    reme_options_t o_cap;
    reme_options_create(c, &o_cap);

    // Bind the capture options. See documentation of this method for a list of properties available.
    reme_sensor_bind_capture_options(c, s, o_cap);

    // Read frame size of auxilary image (IR).
    bool supported;
    int width, height;

    reme_options_get_bool(c, o_cap, "frame_info.supports_aux", &supported);
    reme_options_get_int(c, o_cap, "frame_info.aux_size.width", &width);
    reme_options_get_int(c, o_cap, "frame_info.aux_size.height", &height);

    printf("Auxilary supported: %s (%i x %i)", supported ? "true" : "false", width, height);

To modify capture properties such as the tilt angle of the Kinect one would perform the following steps

    // Here we modify the position of the tilt-angle of the Microsoft Kinect for Windows camera.
    // We clear all values before indicating that we are just interesting in setting the tilt angle
    // and no other value.

    reme_options_clear(c, o_cap);
    reme_options_set_int(c, o_cap, "tilt_angle", 0);

    // Apply capture properties
    reme_sensor_apply_capture_options(c, s, o_cap);

Accessing and Displaying Images

Once the sensor is open you can access its images. The example below illustrates this

    reme_image_t aux;
    reme_image_create(c, &aux);

    // Grab image. Synchronizes all image streams.
    reme_sensor_grab(c, s);

    // Prepare just the auxilary image
    reme_sensor_prepare_image(c, s, REME_IMAGE_AUX);

    // Get the image
    reme_sensor_get_image(c, s, REME_IMAGE_AUX, aux);

    // Visualize image and wait for user to close viewer
    reme_viewer_t v;
    reme_viewer_create_image(c, "Auxilary Image", &v);
    reme_viewer_add_image(c, v, aux);
    reme_viewer_update(c, v);
    reme_viewer_wait(c, v);

Positioning the Sensor

Each sensor has a 3D position and orientation with respect to the world coordinate system. The page Coordinate Systems and Sensor Positioning explains this in detail.

Tracking the Sensor Position

The position of the sensor with respect to the world coordinate sytem can be tracked automatically using reme_sensor_track_position. This is the heart of the ReconstructMe engine as it automatically determines the position of the sensor while it is moved in real-time.

Since version 1.4 ReconstructMe supports two different tracking algorithms:

• local search Local search is fast and succeeds when the camera movement between two subsequent frames is small. It used to be the only search method prior to 1.4.
• global search Added in version 1.4 based on algorithms of CANDELOR (http://www.candelor.com). Global search is slower than local search but succeeds in cases where the camera movement between two subsequent frames is large.

ReconstructMe uses a combination of the above algorithms to improve camera tracking. The behaviour of ReconstructMe can be configured with reme_sensor_set_trackmode

• REME_SENSOR_TRACKMODE_AUTO This is the default. ReconstructMe uses a combination of available search algorithms. When tracking in the last frame succeeded ReconstructMe tries first to use local search. If that fails, global search followed by a local search is attempted. If that fails as well tracking for the given frame fails and ReconstructMe moves the virtual camera into the most recent recovery position (reme_sensor_set_recovery_position). Similarily, if tracking for the last frame failed, ReconstructMe attempts to use global search followed by a local search.
• REME_SENSOR_TRACKMODE_LOCAL ReconstuctMe will only use local search independent of the tracking status of the last frame. This behaviour is identical to the behaviour of ReconstructMe before 1.4
• REME_SENSOR_TRACKMODE_GLOBAL ReconstructMe will always first use global search followed by a fine alignment of local search.

Note that although CANDELOR supports truly global search, the implementation in ReconstructMe uses the the volume data as viewed by the current camera position as the model. This leads to a partial global behaviour as the overlap between the model and the current frame has to exceed a predefined threshold. This is subject to change in future versions.

Additionally one might provide tracking hints that can temporarily change the behaviour of the tracking. See reme_sensor_set_trackhint for details.

Please see the Getting Started page for usage. You might also be interested in example_reconstructmesdk_sensor_multi_independent.cpp that allows you to track multiple sensors.

Updating the Reconstruction Volume

Once the sensor position with respect to the world coordinate is known the current sensor data can be pushed into the reconstruction volume. The reconstruction volume will then care about fusioning the data to a global model. Please refer to the Getting Started page for an introduction.

Closing the Sensor

An opened sensor can be closed as follows.

    // Close sensor communication
    reme_sensor_close(c, s);

Destroying the Sensor

A closed sensor can be destroyed as follows.

    // Destroy sensor
    reme_sensor_destroy(c, &s);

Compensating for Tilt Movements

Popular 3D scanners such as the Microsoft Kinect and others have a motorized base that allows enlargement of the interaction space by tilting the sensor. These sensors, however, turn of the IR projector while the tilt is in progress, essentially making ReconstructMe blind while the tilt occurs. Since ReconstructMe currently relies on small movements between frames it will often loose track after the tilt completes.

This example shows you how to compensate for such a blind tilt by compensating through a virtual repositioning of the sensor. As most sensors make the tilt relative to the gravitation vector the tilt angle difference can only be calculated accurately for sensors that stand on a fixed platform and are not moved by hand. After tilting it is currently advised to skip a couple of depth frames to compensate wrong data from a recent projector turn-on.

Below is the main scanning loop of the example. This is an ordinary scanning loop with one slight difference: skip_frame_counter causes frames to be skipped when greater zero. This counter gets decremented in each loop cycle.

    bool continue_scanning = true;

    // After each tilt we skip a couple of frames
    int skip_frame_counter = 0;
    while (continue_scanning && REME_SUCCESS(reme_sensor_grab(c, s))) {

        // Keyboard handling
        if (_kbhit()) {
            char k = _getch();
            switch (k) {

            // Key 'r' resets the volume to empty state
            case 'r': {
                reme_volume_reset(c, v);
                reme_sensor_reset(c, s);
                break;
            }

            // Key 'f' tilt +10 degrees
            case 'f': {
                perform_tilt(c, s, o, 10);
                reme_sensor_set_trackhint(c, s, REME_SENSOR_TRACKHINT_USE_GLOBAL);
                break;
            }

            // Key 'b' tilt -10 degrees
            case 'b': {
                perform_tilt(c, s, o, -10);
                reme_sensor_set_trackhint(c, s, REME_SENSOR_TRACKHINT_USE_GLOBAL);
                skip_frame_counter = 10;
                break;
            }

            case 'x': {
                continue_scanning = false;
                break;
            }

            }
        }

        // Prepare image and depth data
        reme_sensor_prepare_images(c, s);

        if (skip_frame_counter == 0) {
            if (REME_SUCCESS(reme_sensor_track_position(c, s))) {
                reme_sensor_update_volume(c, s);
            }
        } else {
            skip_frame_counter -= 1;
        }

        // Update the viewer
        reme_sensor_get_image(c, s, REME_IMAGE_DEPTH, imgs[0]);
        reme_sensor_get_image(c, s, REME_IMAGE_VOLUME, imgs[1]);
        reme_viewer_update(c, viewer);

    }

The missing piece is the method called perform_tilt that takes a context, sensor, capture options and an incremental tilt angle (relativ to the current) in degrees.

// Perform the tilt and compensate for it.
// After tilting a couple of frames should be skipped to compensate for bad data.
void perform_tilt(reme_context_t c, reme_sensor_t s, reme_options_t o, int angle) {
    // Read current angle
    reme_sensor_bind_capture_options(c, s, o);
    int prev_angle;
    reme_options_get_int(c, o, "tilt_angle", &prev_angle);

    // Apply tilt
    reme_options_clear(c, o);
    reme_options_set_int(c, o, "tilt_angle", prev_angle + angle);
    reme_sensor_apply_capture_options(c, s, o);

    // Read tilt angle again to determine to determine actual movement
    reme_sensor_bind_capture_options(c, s, o);
    int new_angle;
    reme_options_get_int(c, o, "tilt_angle", &new_angle);

    // Compensate for tilt in sensor position
    float pos[16];
    reme_sensor_get_position(c, s, pos);
    reme_transform_compensate_tilt(c, pos, new_angle - prev_angle, pos);
    reme_sensor_set_position(c, s, pos);
}

The method reme_transform_compensate_tilt assumes that the tilt axis is parallel to the sensor positive sensor x-axis. It also assumes that the origin of the tilt movement is placed at roughly (0, 30, -25) with respect to the sensor coordinate system. As such, this compensation will currently only work for Kinect for Windows sensors. While this is an inaccurate measurement and the assumpations are partly wrong, the solution works in most cases as ReconstructMe is able to compensate for the remaining error.

It could be improved in terms of standard stereo calibration that takes the known rotation angle into account.

Note that you cannot tilt arbitrary large angles as the Kinect limits the tilt-angle to +/- 27 degrees and a large tilt may force a lot of new data to be integrated which could trigger track-lost detection.