Sensor

Detailed Description

The sensor is responsible for acquiring real world data and feeding it into the reconstruction volume. See the example below for basic usage.

  // Create a new context
  reme_context_t c;
  reme_context_create(&c);

  // Compile for OpenCL device using defaults
  reme_context_compile(c);
  
  // Create a new volume
  reme_volume_t v;
  reme_volume_create(c, &v);
  
  // Create a new sensor. Tries multiple backends using default
  // sensor configurations, returns the first working one. By default 
  // each sensor works on the first volume created.
  reme_sensor_t s;
  reme_sensor_create(c, "openni;mskinect;file", true, &s);
  reme_sensor_open(c, s);

  // For debugging purposes open a viewer for tracking the reconstruction process.
  // This will create a two-sided viewer showing the video stream and the current
  // reconstruction from the sensor point of view.
  reme_viewer_t viewer;
  reme_viewer_create_image(c, "This is ReconstructMe SDK", &viewer);
  reme_viewer_add_image(c, viewer, s, REME_IMAGE_DEPTH);
  reme_viewer_add_image(c, viewer, s, REME_IMAGE_VOLUME);

  // Perform reconstruction until no more frames are left
  int time = 0;
  while (time < 200 && REME_SUCCESS(reme_sensor_grab(c, s))) {

    // Prepare image and depth data
    reme_sensor_prepare_images(c, s);

    // Try to determine updated sensor position.
    // On succes, update volume, otherwise move to a recovery position
    // and wait for the tracking to start again.
    if (REME_SUCCESS(reme_sensor_track_position(c, s))) {
      
      // Update volume with depth data from the 
      // current sensor perspective
      reme_sensor_update_volume(c, s);

    }

    // Update the viewer
    reme_viewer_update(c, viewer);
    time += 1;
  }

  // Close and destroy the sensor, it is not needed anymore
  reme_sensor_close(c, s);
  reme_sensor_destroy(c, &s);

  // Create a new surface
  reme_surface_t m;
  reme_surface_create(c, &m);
  reme_surface_generate(c, m, v);
  reme_surface_save_to_file(c, m, "test.ply");
  
  // Visualize resulting surface
  reme_viewer_t viewer_surface;
  reme_viewer_create_surface(c, m, "This is ReconstructMeSDK", &viewer_surface);
  reme_viewer_wait(c, viewer_surface);

  reme_surface_destroy(c, &m);

  // Make sure to release all memory acquired
  reme_context_destroy(&c);

Typedef Documentation

typedef int reme_sensor_t

Handle referencing a sensor object.

A reme_sensor_t represents a RGBD sensor. Associated methods allow to create, open and interact with sensors of various types.

typedef enum _reme_sensor_image_t reme_sensor_image_t

Image type enumeration.

Each sensor might provide different frame types that are all 2D images. Not all sensors support all frames, or the number of frames supported is configuration dependant.

Enumeration Type Documentation

enum _reme_sensor_image_t

Image type enumeration.

Each sensor might provide different frame types that are all 2D images. Not all sensors support all frames, or the number of frames supported is configuration dependant.

Enumerator:

REME_IMAGE_AUX	Auxiliary color image if provided by sensor. RGB 3 channels, 1 byte per channel
REME_IMAGE_DEPTH	Depth image. RGB 3 channels, 1 byte per channel
REME_IMAGE_VOLUME	Rendered image of the volume as viewed from the current sensor perspective. RGB 3 channels, 1 byte per channel.

Function Documentation

reme_error_t reme_sensor_create	(	reme_context_t	c,
		const char *	driver,
		bool	require_can_open,
		reme_sensor_t *	s
	)

Create a new sensor object.

Creates a new sensor. The driver argument specifies which sensor to open. The following sensor drivers are currently available

• openni Microsoft Kinect for XBox, Asus Xtion, Asus Xtion Pro Live
• mskinect Microsoft Kinect for Windows
• file A generic file sensor to read previously recorded data
• s3e AITs' area based stereo matching sensors.

You might specify a specific driver

"openni" 

Alternatively you can specify a list of drivers to test, in which case it will return the first sensor that works.

"openni;mskinect;file" 

Or, specify file paths of a serialized sensor messages (don't forget to escape backslashes).

"c:/drivers/my_openni.txt;c:/drivers/my_file.txt" 

The drivers will be tested in the given order.

Set require_can_open to true to ensure the created sensor can be created and opened using either the default settings, or if file paths are specifed using the settings specified in the file.

No matter how require_can_open is set the sensor is returned in closed state. Use reme_sensor_open to open it.

Parameters

c	A valid context object
driver	The name of sensor driver to instance a sensor from
require_can_open	Ensures that the returned sensor can be opened using either the default sensor options.
s	A pointer that will receive the handle of the created sensor

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

Examples:

example_reconstructmesdk_image.cpp, example_reconstructmesdk_one_minute.cpp, example_reconstructmesdk_recorder.cpp, example_reconstructmesdk_sensor_multi_independent.cpp, example_reconstructmesdk_surface.cpp, and example_reconstructmesdk_volume.cpp.

reme_error_t reme_sensor_destroy	(	reme_context_t	c,
		reme_sensor_t *	s
	)

Destroy a previously created sensor object.

Parameters

c	A pointer to a valid context object
s	A mutable pointer to a valid sensor handle to destroy

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

Examples:

example_reconstructmesdk_one_minute.cpp, and example_reconstructmesdk_surface.cpp.

reme_error_t reme_sensor_open	(	reme_context_t	c,
		reme_sensor_t	s
	)

Open a sensor.

Attempts to opens the specified sensor using the specified sensor options.

Parameters

c	A valid context object
s	A valid sensor object

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

Examples:

example_reconstructmesdk_image.cpp, example_reconstructmesdk_one_minute.cpp, example_reconstructmesdk_recorder.cpp, example_reconstructmesdk_sensor_multi_independent.cpp, example_reconstructmesdk_surface.cpp, and example_reconstructmesdk_volume.cpp.

reme_error_t reme_sensor_close	(	reme_context_t	c,
		reme_sensor_t	s
	)

Close a an open sensor.

Indicates that no more grabbing is done from sensor. Re-opening a sensor is possible.

Parameters

c	A valid context object
s	A valid sensor object

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

Examples:

example_reconstructmesdk_one_minute.cpp, and example_reconstructmesdk_surface.cpp.

reme_error_t reme_sensor_set_volume	(	reme_context_t	c,
		reme_sensor_t	s,
		reme_volume_t	v
	)

Set the working volume.

The sensor will use this volume to keep track of its position and it will also update into this volume. By default the sensor is assigned to the first volume available.

Parameters

c	A valid context object
s	A valid sensor object
v	A valid volume object

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

reme_error_t reme_sensor_bind_camera_options	(	reme_context_t	c,
		reme_sensor_t	s,
		reme_options_t	o
	)

Access the sensor specific options.

Associated Protocol Buffers Specification

Depending on the sensor chosen on create, different specs are needed, since each sensor supports a different set of parameters.

For openni sensors

import "rgbdsensor/intrinsics.proto";

package LibRGBDSensor;

// OpenNI sensor configuration
message openni_sensor_config {

  // Default driver model
  optional string driver = 1 [default = "openni"];
  // USB device id
  optional int32 usb_device_id = 2 [default = 0];
  // Camera intrinsics
  optional LibRGBDSensor.intrinsics depth_intrinsics = 3;
  // Enable IR mode
  optional bool enable_ir = 4 [default = false];
  // Enable RGB mode
  optional bool enable_rgb = 5 [default = true];
  
}

For mskinect sensors

import "rgbdsensor/intrinsics.proto";

package LibRGBDSensor;

// Kinect sensor configuration based on MSKinect driver
message mskinect_sensor_config {

  // Default driver model
  optional string driver = 1 [default = "mskinect"];
  // USB device id
  optional int32 usb_device_id = 2 [default = 0];
  // Camera intrinsics
  optional LibRGBDSensor.intrinsics depth_intrinsics = 3;
  // Enable or disable 'Near Mode'
  optional bool enable_near_mode = 4 [default = true];
  // Enable RGB mode
  optional bool enable_rgb = 5 [default = true];
}

For file sensors

import "rgbdsensor/intrinsics.proto";

package LibRGBDSensor;

// OpenNI sensor configuration
message file_sensor_config {

  message frame_size {
    optional int32 width = 1 [default = 640];
    optional int32 height = 2 [default = 480];
  };

  // Default driver model
  optional string driver = 1 [default = "file"];
  // Path to file containing depth information
  optional string depth_file = 2 [default = "reconstructme.depths.gz"];
  // Path to file containing color information
  optional string color_file = 3 [default = "reconstructme.avi"];
  // Frame size of depth file
  optional frame_size depth_size = 4;
  // Frame size of color file
  optional frame_size color_size = 5;

  // Camera depth intrinsics
  optional LibRGBDSensor.intrinsics depth_intrinsics = 6;
  
}

Common to all configurations above is the depth_intrinsics field. It is mendatory for reconstruction and has the following structure

package LibRGBDSensor;

// Camera intrinsics
message intrinsics {

  // Camera matrix
  optional double fx = 1 [default = 571.26];
  optional double fy = 2 [default = 571.26];
  optional double cx = 3 [default = 320];
  optional double cy = 4 [default = 240];
  
  // Radial distortion coefficients
  // Set to zero to disable certain calculations
  optional double k1 = 5 [default = 0];
  optional double k2 = 6 [default = 0];
  optional double k3 = 7 [default = 0];
  
  // Tangential distortion coefficients
  // Set to zero to disable certain calculations
  optional double p1 = 8 [default = 0];
  optional double p2 = 9 [default = 0];
  
}

Note

Support for radial and tangential distortion parameters needs to be enabled using the reme_context_bind_compile_options.

Changing camera options without a call to reme_sensor_open has no effect.

Parameters

c	A valid context object
s	A valid sensor object
o	A valid options binding object

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

Examples:

example_reconstructmesdk_recorder.cpp, and example_reconstructmesdk_sensor_multi_independent.cpp.

reme_error_t reme_sensor_bind_track_options	(	reme_context_t	c,
		reme_sensor_t	s,
		reme_options_t	o
	)

Access the sensor track options.

Associated Protocol Buffers Specification

package LibReconstructMe;

// Settings that drive the update volume step
message tracking_settings {
  // Maximum allowed average SDF change of voxels when
  // tracking is established
  optional float sdf_change_on_track = 1 [default = 0.08];
  // Maximum allowed average SDF change of voxels when
  // no tracking is present
  optional float sdf_change_no_track = 2 [default = 0.05];
}

Parameters

c	A valid context object
s	A valid sensor object
o	A valid options binding object.

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

reme_error_t reme_sensor_get_position	(	reme_context_t	c,
		reme_sensor_t	s,
		const float **	coordinates
	)

Get the sensor position with respect to the world coordinate frame.

Initially this is set to identity for all sensors. The position is modified when reme_sensor_track_position succeeds, or reme_sensor_set_position is called.

Memory Management Rules Exception

The returned coordinates pointer remains valid until the sensor is destroyed. The pointer is recycled internally, which means that it will point to different values each time the sensor position has changed.

Parameters

c	A valid context object
s	A valid sensor object
coordinates	A mutable pointer to constant Transform data.

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

reme_error_t reme_sensor_get_incremental_position	(	reme_context_t	c,
		reme_sensor_t	s,
		const float **	coordinates
	)

Get the incremental movement of the sensor.

When reme_sensor_track_position succeeds this position reflects the movement of the sensor between the last position of the sensor and the current position of the sensor.

Memory Management Rules Exception

The returned coordinates pointer remains valid until the sensor is destroyed. The pointer is recycled internally, which means that it will point to different values each time the sensor position has changed.

Parameters

c	A valid context object
s	A valid sensor object
coordinates	A mutable pointer to constant Transform data.

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

reme_error_t reme_sensor_set_position	(	reme_context_t	c,
		reme_sensor_t	s,
		const float *	coordinates
	)

Set the sensor position with respect to the world coordinate frame.

Initially this is set to identity for all sensors.

Parameters

c	A valid context object
s	A valid sensor object
coordinates	A pointer to constant Transform data.

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

reme_error_t reme_sensor_set_incremental_position	(	reme_context_t	c,
		reme_sensor_t	s,
		const float *	coordinates
	)

Set the incremental movement of the sensor.

Updates the world position of the sensor by the incremental position specified.

Parameters

c	A valid context object
s	A valid sensor object
coordinates	A pointer to constant Transform data.

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

reme_error_t reme_sensor_reset	(	reme_context_t	c,
		reme_sensor_t	s
	)

Resets the sensor to identity position.

Forces the sensor to loose track.

Parameters

c	A valid context object
s	A valid sensor object

Return values

REME_ERROR_SUCCESS	On tracking success or the corresponding volume is empty.
REME_ERROR_UNSPECIFIED	is returned otherwise.

Examples:

example_reconstructmesdk_volume.cpp.

reme_error_t reme_sensor_get_image_size	(	reme_context_t	c,
		reme_sensor_t	s,
		_reme_sensor_image_t	i,
		int *	width,
		int *	height
	)

Get a specific sensor image width and height.

Returns the image dimensions width and height. The sensor needs to be open for this method to complete successfully (see reme_sensor_open).

Parameters

c	A valid context object
s	A valid sensor object
i	Image type to access
width	Width of image in pixels
height	Height of image in pixels

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

reme_error_t reme_sensor_get_image	(	reme_context_t	c,
		reme_sensor_t	s,
		_reme_sensor_image_t	i,
		const void **	bytes,
		int *	width = 0,
		int *	height = 0,
		int *	num_channels = 0,
		int *	num_bytes_per_channel = 0,
		int *	row_stride = 0
	)

Get a specific sensor image.

Each sensor might provide different frame types. Not all sensors support all frames, or the number of frames supported is configuration dependant. See reme_sensor_image_t for a complete enumeration of available image types.

Memory Management Rules Exception

The returned bytes pointer remains valid until the sensor is destroyed or the dimension of the image changes. The pointer is recycled internally, which means that it will point to different values each time the sensor images are updated.

Note

REME_IMAGE_VOLUME is currently only updated when reme_sensor_track_position is called.

Parameters

c	A valid context object
s	A valid sensor object
i	Image type to access
bytes	A mutable pointer to constant pixel data.
width	Width of image in pixels
height	Height of image in pixels
num_channels	Number of channels per pixel
num_bytes_per_channel	Number of bytes per channel
row_stride	Distance between successive rows in bytes

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

Examples:

example_reconstructmesdk_image.cpp.

reme_error_t reme_sensor_get_track_time	(	reme_context_t	c,
		reme_sensor_t	s,
		int *	track_time
	)

Get the tracking time.

The tracking time corresponds to the number of frames in which tracking succeeded. In case tracking fails the counter gets negative.

Parameters

c	A valid context object
s	A valid sensor object
track_time	The tracking time

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

reme_error_t reme_sensor_grab	(	reme_context_t	c,
		reme_sensor_t	s
	)

Trigger frame grabbing.

Provides a synchronization point to trigger image generation of all image types. This method is intended to be fast.

In order to synchronize frame grabbing from multiple sensors, call this method in sequence for each sensor before calling reme_sensor_retrieve.

Parameters

c	A valid context object
s	A valid sensor object

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_FAILED_TO_GRAB	Failed to grab from sensor. This is not necessarily an error, you might re-try.
REME_ERROR_UNSPECIFIED	Is returned otherwise.

Examples:

example_reconstructmesdk_image.cpp, example_reconstructmesdk_one_minute.cpp, example_reconstructmesdk_recorder.cpp, example_reconstructmesdk_sensor_multi_independent.cpp, example_reconstructmesdk_surface.cpp, and example_reconstructmesdk_volume.cpp.

reme_error_t reme_sensor_prepare_images	(	reme_context_t	c,
		reme_sensor_t	s
	)

Retrieve image data corresponding to the previous grab command for further processing.

Updates the internal state of all images and prepares the required data structures on the computation device.

Parameters

c	A valid context object
s	A valid sensor object

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

Examples:

example_reconstructmesdk_one_minute.cpp, example_reconstructmesdk_recorder.cpp, example_reconstructmesdk_sensor_multi_independent.cpp, example_reconstructmesdk_surface.cpp, and example_reconstructmesdk_volume.cpp.

reme_error_t reme_sensor_prepare_image	(	reme_context_t	c,
		reme_sensor_t	s,
		_reme_sensor_image_t	i
	)

Retrieve specific image data for subsequent processing.

In case REME_IMAGE_AUX or REME_IMAGE_DEPTH is passed, this method will fetch the data into internal memory. In case REME_IMAGE_VOLUME is passed, the previously prepared REME_IMAGE_DEPTH will be uploaded to the computation device for subsequent processing (reme_sensor_track_position, reme_sensor_update_volume).

This method is especially useful (when compared to reme_sensor_prepare_images) when only depth and image data is required. For example when recording, there is no need for REME_IMAGE_VOLUME and it should be skipped so no time is wasted waiting for the data to be uploaded to the computation device.

Parameters

c	A valid context object
s	A valid sensor object
i	Image type to prepare

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

Examples:

example_reconstructmesdk_image.cpp, and example_reconstructmesdk_recorder.cpp.

reme_error_t reme_sensor_track_position	(	reme_context_t	c,
		reme_sensor_t	s
	)

Attempts to track the sensor position.

Tries to track the sensor movement by matching the current depth data against the perspective from the last position. Initially the sensor position is the identity position, unless otherwise specified.

Parameters

c	A valid context object
s	A valid sensor object

Return values

REME_ERROR_SUCCESS	On tracking success or the corresponding volume is empty.
REME_ERROR_TRACK_LOST	When the tracking did not succeed. In this case the sensor is repositioned into latest recovery pose.
REME_ERROR_INVALID_LICENSE	is returned when tracking is disabled for a certain amount of time because of running in a non-commercial mode.
REME_ERROR_UNSPECIFIED	is returned otherwise.

Examples:

example_reconstructmesdk_one_minute.cpp, example_reconstructmesdk_recorder.cpp, example_reconstructmesdk_sensor_multi_independent.cpp, example_reconstructmesdk_surface.cpp, and example_reconstructmesdk_volume.cpp.

reme_error_t reme_sensor_update_volume	(	reme_context_t	c,
		reme_sensor_t	s
	)

Update the bound volume using the current sensor data.

Uses the current sensor position as the perspective to update the volume.

Parameters

c	A valid context object
s	A valid sensor object

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

Examples:

example_reconstructmesdk_one_minute.cpp, example_reconstructmesdk_recorder.cpp, example_reconstructmesdk_sensor_multi_independent.cpp, example_reconstructmesdk_surface.cpp, and example_reconstructmesdk_volume.cpp.