Sensor

Acquire data from real-world sensors or from file streams. More...

Typedefs
typedef int	reme_sensor_t
	Handle referencing a sensor object.
typedef enum _reme_sensor_image_t	reme_sensor_image_t
	Image type enumeration.

Enumerations
enum	_reme_sensor_image_t {   REME_IMAGE_AUX = 0,   REME_IMAGE_DEPTH = 1,   REME_IMAGE_VOLUME = 2 }
	Image type enumeration. More...
reme_error_t	reme_sensor_create (reme_context_t c, const char *driver, bool require_can_open, reme_sensor_t *s)
	Create a new sensor object.
reme_error_t	reme_sensor_destroy (reme_context_t c, reme_sensor_t *s)
	Destroy a previously created sensor object.
reme_error_t	reme_sensor_open (reme_context_t c, reme_sensor_t s)
	Open a sensor.
reme_error_t	reme_sensor_close (reme_context_t c, reme_sensor_t s)
	Close a an open sensor.
reme_error_t	reme_sensor_set_volume (reme_context_t c, reme_sensor_t s, reme_volume_t v)
	Set the working volume.
reme_error_t	reme_sensor_bind_camera_options (reme_context_t c, reme_sensor_t s, reme_options_t o)
	Access the sensor specific options.
reme_error_t	reme_sensor_bind_track_options (reme_context_t c, reme_sensor_t s, reme_options_t o)
	Access the sensor track options.
reme_error_t	reme_sensor_get_position (reme_context_t c, reme_sensor_t s, const float **coordinates)
	Get the sensor position with respect to the world coordinate frame.
reme_error_t	reme_sensor_get_incremental_position (reme_context_t c, reme_sensor_t s, const float **coordinates)
	Get the incremental movement of the sensor.
reme_error_t	reme_sensor_set_position (reme_context_t c, reme_sensor_t s, const float *coordinates)
	Set the sensor position with respect to the world coordinate frame.
reme_error_t	reme_sensor_set_incremental_position (reme_context_t c, reme_sensor_t s, const float *coordinates)
	Set the incremental movement of the sensor.
reme_error_t	reme_sensor_reset (reme_context_t c, reme_sensor_t s)
	Resets the sensor to identity position.
reme_error_t	reme_sensor_get_image_size (reme_context_t c, reme_sensor_t s, _reme_sensor_image_t i, int *width, int *height)
	Get a specific sensor image width and height.
reme_error_t	reme_sensor_update_image (reme_context_t c, reme_sensor_t s, reme_sensor_image_t it, reme_image_t i)
	Get a specific sensor image.
reme_error_t	reme_sensor_get_track_time (reme_context_t c, reme_sensor_t s, int *track_time)
	Get the tracking time.
reme_error_t	reme_sensor_grab (reme_context_t c, reme_sensor_t s)
	Trigger frame grabbing.
reme_error_t	reme_sensor_prepare_images (reme_context_t c, reme_sensor_t s)
	Retrieve image data corresponding to the previous grab command for further processing.
reme_error_t	reme_sensor_prepare_image (reme_context_t c, reme_sensor_t s, _reme_sensor_image_t i)
	Retrieve specific image data for subsequent processing.
reme_error_t	reme_sensor_track_position (reme_context_t c, reme_sensor_t s)
	Attempts to track the sensor position.
reme_error_t	reme_sensor_update_volume (reme_context_t c, reme_sensor_t s)
	Update the bound volume using the current sensor data.

Detailed Description

Acquire data from real-world sensors or from file streams.

The sensor is responsible for acquiring real world data and feeding it into the reconstruction volume. The life-time of reme_sensor_t sensor is usually the following

• Create a new sensor (reme_sensor_create) using either the backend name or a specific sensor configuration file. Usually this method returns with success when the necessary drivers for the sensor where found. It does usually not mean that the sensor is actually there. We say usually, since you can force a test to open the sensor at this point using the require_can_open flag.
• Modify sensor parameters (reme_sensor_bind_camera_options).
• Open the sensor (reme_sensor_open).
• Work with sensor images (reme_sensor_grab, reme_sensor_prepare_images).
• Close sensor (reme_sensor_close).
• Destroy sensor (reme_sensor_destroy).

The one-minute example illustrates the basic usage.

Typedef Documentation

typedef int reme_sensor_t

Handle referencing a sensor object.

A reme_sensor_t represents a RGBD sensor. Associated methods allow to create, open and interact with sensors of various types.

typedef enum _reme_sensor_image_t reme_sensor_image_t

Image type enumeration.

Each sensor might provide different frame types that are all 2D images. Not all sensors support all frames, or the number of frames supported is configuration dependant.

Enumeration Type Documentation

enum _reme_sensor_image_t

Image type enumeration.

Each sensor might provide different frame types that are all 2D images. Not all sensors support all frames, or the number of frames supported is configuration dependant.

Enumerator:

REME_IMAGE_AUX	Auxiliary color image if provided by sensor. RGB 3 channels, 1 byte per channel
REME_IMAGE_DEPTH	Depth image. RGB 3 channels, 1 byte per channel
REME_IMAGE_VOLUME	Rendered image of the volume as viewed from the current sensor perspective. RGB 3 channels, 1 byte per channel.

Function Documentation

reme_error_t reme_sensor_create	(	reme_context_t	c,
		const char *	driver,
		bool	require_can_open,
		reme_sensor_t *	s
	)

Create a new sensor object.

Creates a new sensor. The driver argument specifies which sensor to open. The following sensor drivers are currently available

• openni Microsoft Kinect for XBox, Asus Xtion, Asus Xtion Pro Live
• mskinect Microsoft Kinect for Windows
• file A generic file sensor to read previously recorded data
• s3e AITs' area based stereo matching sensors.

You might specify a specific driver

"openni" 

Alternatively you can specify a list of drivers to test, in which case it will return the first sensor that works.

"openni;mskinect;file" 

Or, specify file containing a sensor configuration file (see reme_sensor_bind_camera_options for camera specific options and Accessing and Modifying Options for background information about options)

"c:/drivers/my_openni.txt;c:/drivers/my_file.txt" 

The drivers will be tested in the given order.

Set require_can_open to true to ensure the created sensor can be created and opened using either the default settings, or if file paths are specifed using the settings specified in the file.

No matter how require_can_open is set the sensor is returned in closed state. Use reme_sensor_open to open it.

Parameters

c	A valid context object
driver	The name of sensor driver to instance a sensor from
require_can_open	Ensures that the returned sensor can be opened using either the default sensor options.
s	A pointer that will receive the handle of the created sensor

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

Examples:

example_reconstructmesdk_calibration.cpp, example_reconstructmesdk_image.cpp, example_reconstructmesdk_one_minute.cpp, example_reconstructmesdk_recorder.cpp, example_reconstructmesdk_sensor_multi_independent.cpp, example_reconstructmesdk_surface.cpp, and example_reconstructmesdk_volume.cpp.

reme_error_t reme_sensor_destroy	(	reme_context_t	c,
		reme_sensor_t *	s
	)

Destroy a previously created sensor object.

Parameters

c	A pointer to a valid context object
s	A mutable pointer to a valid sensor handle to destroy

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

Examples:

example_reconstructmesdk_one_minute.cpp, and example_reconstructmesdk_surface.cpp.

reme_error_t reme_sensor_open	(	reme_context_t	c,
		reme_sensor_t	s
	)

Open a sensor.

Attempts to opens the specified sensor using the specified sensor options.

Parameters

c	A valid context object
s	A valid sensor object

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

Examples:

example_reconstructmesdk_calibration.cpp, example_reconstructmesdk_image.cpp, example_reconstructmesdk_one_minute.cpp, example_reconstructmesdk_recorder.cpp, example_reconstructmesdk_sensor_multi_independent.cpp, example_reconstructmesdk_surface.cpp, and example_reconstructmesdk_volume.cpp.

reme_error_t reme_sensor_close	(	reme_context_t	c,
		reme_sensor_t	s
	)

Close a an open sensor.

Indicates that no more grabbing is done from sensor. Re-opening a sensor is possible.

Parameters

c	A valid context object
s	A valid sensor object

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

Examples:

example_reconstructmesdk_one_minute.cpp, and example_reconstructmesdk_surface.cpp.

reme_error_t reme_sensor_set_volume	(	reme_context_t	c,
		reme_sensor_t	s,
		reme_volume_t	v
	)

Set the working volume.

The sensor will use this volume to keep track of its position and it will also update into this volume. By default the sensor is assigned to the first volume available.

Parameters

c	A valid context object
s	A valid sensor object
v	A valid volume object

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

reme_error_t reme_sensor_bind_camera_options	(	reme_context_t	c,
		reme_sensor_t	s,
		reme_options_t	o
	)

Access the sensor specific options.

Associated Protocol Buffers Specification

Depending on the sensor chosen on create, different specs are needed, since each sensor supports a different set of parameters.

For openni sensors

import "rgbdsensor/intrinsics.proto";

package LibRGBDSensor;

// OpenNI sensor configuration
message openni_sensor_config {

  // Default driver model
  optional string driver = 1 [default = "openni"];
  // USB device id
  optional int32 usb_device_id = 2 [default = 0];
  // Camera intrinsics
  optional LibRGBDSensor.intrinsics depth_intrinsics = 3;
  // Enable IR mode
  optional bool enable_ir = 4 [default = false];
  // Enable RGB mode
  optional bool enable_rgb = 5 [default = true];
  // Infrared raw intensity multiplier
  optional int32 ir_alpha = 6 [default = 1];  
}

For mskinect sensors

import "rgbdsensor/intrinsics.proto";

package LibRGBDSensor;

// Kinect sensor configuration based on MSKinect driver
message mskinect_sensor_config {

  // Default driver model
  optional string driver = 1 [default = "mskinect"];
  // USB device id
  optional int32 usb_device_id = 2 [default = 0];
  // Camera intrinsics
  optional LibRGBDSensor.intrinsics depth_intrinsics = 3;
  // Enable or disable 'Near Mode'
  optional bool enable_near_mode = 4 [default = true];
  // Enable RGB mode
  optional bool enable_rgb = 5 [default = true];
  // Enable IR stream mode
  optional bool enable_ir = 6 [default = false];
  // Enable IR projector
  optional bool enable_ir_projector = 7 [default = true];
  // Infrared raw intensity multiplier
  optional int32 ir_alpha = 8 [default = 1];  
}

For file sensors

import "rgbdsensor/intrinsics.proto";

package LibRGBDSensor;

// OpenNI sensor configuration
message file_sensor_config {

  message frame_size {
    optional int32 width = 1 [default = 640];
    optional int32 height = 2 [default = 480];
  };

  // Default driver model
  optional string driver = 1 [default = "file"];
  // Path to file containing depth information
  optional string depth_file = 2 [default = "reconstructme.depths.gz"];
  // Path to file containing color information
  optional string color_file = 3 [default = "reconstructme.avi"];
  // Frame size of depth file
  optional frame_size depth_size = 4;
  // Frame size of color file
  optional frame_size color_size = 5;

  // Camera depth intrinsics
  optional LibRGBDSensor.intrinsics depth_intrinsics = 6;
  
}

Common to all configurations above is the depth_intrinsics field. It is mendatory for reconstruction and has the following structure

package LibRGBDSensor;

// Camera intrinsics
message intrinsics {

  // Camera matrix
  optional double fx = 1 [default = 571.26];
  optional double fy = 2 [default = 571.26];
  optional double cx = 3 [default = 320];
  optional double cy = 4 [default = 240];
  
  // Radial distortion coefficients
  // Set to zero to disable certain calculations
  optional double k1 = 5 [default = 0];
  optional double k2 = 6 [default = 0];
  optional double k3 = 7 [default = 0];
  
  // Tangential distortion coefficients
  // Set to zero to disable certain calculations
  optional double p1 = 8 [default = 0];
  optional double p2 = 9 [default = 0];
  
}

Note

Support for radial and tangential distortion parameters needs to be enabled using the reme_context_bind_compile_options.

Changing camera options without a call to reme_sensor_open has no effect.

Parameters

c	A valid context object
s	A valid sensor object
o	A valid options binding object

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

Examples:

example_reconstructmesdk_calibration.cpp, example_reconstructmesdk_recorder.cpp, and example_reconstructmesdk_sensor_multi_independent.cpp.

reme_error_t reme_sensor_bind_track_options	(	reme_context_t	c,
		reme_sensor_t	s,
		reme_options_t	o
	)

Access the sensor track options.

Associated Protocol Buffers Specification

package LibReconstructMe;

// Settings that drive the update volume step
message tracking_settings {
  // Maximum allowed average SDF change of voxels when
  // tracking is established
  optional float sdf_change_on_track = 1 [default = 0.08];
  // Maximum allowed average SDF change of voxels when
  // no tracking is present
  optional float sdf_change_no_track = 2 [default = 0.05];
}

Parameters

c	A valid context object
s	A valid sensor object
o	A valid options binding object.

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

reme_error_t reme_sensor_get_position	(	reme_context_t	c,
		reme_sensor_t	s,
		const float **	coordinates
	)

Get the sensor position with respect to the world coordinate frame.

Initially this is set to identity for all sensors. The position is modified when reme_sensor_track_position succeeds, or reme_sensor_set_position is called.

Memory Management Rules Exception

The returned coordinates pointer remains valid until the sensor is destroyed. The pointer is recycled internally, which means that it will point to different values each time the sensor position has changed.

Parameters

c	A valid context object
s	A valid sensor object
coordinates	A mutable pointer to constant Transform data.

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

reme_error_t reme_sensor_get_incremental_position	(	reme_context_t	c,
		reme_sensor_t	s,
		const float **	coordinates
	)

Get the incremental movement of the sensor.

When reme_sensor_track_position succeeds this position reflects the movement of the sensor between the last position of the sensor and the current position of the sensor.

Memory Management Rules Exception

The returned coordinates pointer remains valid until the sensor is destroyed. The pointer is recycled internally, which means that it will point to different values each time the sensor position has changed.

Parameters

c	A valid context object
s	A valid sensor object
coordinates	A mutable pointer to constant Transform data.

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

reme_error_t reme_sensor_set_position	(	reme_context_t	c,
		reme_sensor_t	s,
		const float *	coordinates
	)

Set the sensor position with respect to the world coordinate frame.

Initially this is set to identity for all sensors.

Parameters

c	A valid context object
s	A valid sensor object
coordinates	A pointer to constant Transform data.

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

reme_error_t reme_sensor_set_incremental_position	(	reme_context_t	c,
		reme_sensor_t	s,
		const float *	coordinates
	)

Set the incremental movement of the sensor.

Updates the world position of the sensor by the incremental position specified.

Parameters

c	A valid context object
s	A valid sensor object
coordinates	A pointer to constant Transform data.

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

reme_error_t reme_sensor_reset	(	reme_context_t	c,
		reme_sensor_t	s
	)

Resets the sensor to identity position.

Forces the sensor to loose track.

Parameters

c	A valid context object
s	A valid sensor object

Return values

REME_ERROR_SUCCESS	On tracking success or the corresponding volume is empty.
REME_ERROR_UNSPECIFIED	is returned otherwise.

Examples:

example_reconstructmesdk_volume.cpp.

reme_error_t reme_sensor_get_image_size	(	reme_context_t	c,
		reme_sensor_t	s,
		_reme_sensor_image_t	i,
		int *	width,
		int *	height
	)

Get a specific sensor image width and height.

Returns the image dimensions width and height. The sensor needs to be open for this method to complete successfully (see reme_sensor_open).

Parameters

c	A valid context object
s	A valid sensor object
i	Image type to access
width	Width of image in pixels
height	Height of image in pixels

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

reme_error_t reme_sensor_update_image	(	reme_context_t	c,
		reme_sensor_t	s,
		reme_sensor_image_t	it,
		reme_image_t	i
	)

Get a specific sensor image.

Each sensor might provide different frame types. Not all sensors support all frames, or the number of frames supported is configuration dependant. See reme_sensor_image_t for a complete enumeration of available image types.

Memory Management Rules Exception

The returned image remains valid until the sensor is destroyed or the dimension of the image changes. The pointer is recycled internally, which means that it will point to different values each time the sensor images are updated.

Note

REME_IMAGE_VOLUME is currently only updated when reme_sensor_track_position is called.

Parameters

c	A valid context object
s	A valid sensor object
it	Image type to access
i	A valid image object to receive image data

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

Examples:

example_reconstructmesdk_calibration.cpp, example_reconstructmesdk_image.cpp, example_reconstructmesdk_one_minute.cpp, example_reconstructmesdk_recorder.cpp, example_reconstructmesdk_sensor_multi_independent.cpp, example_reconstructmesdk_surface.cpp, and example_reconstructmesdk_volume.cpp.

reme_error_t reme_sensor_get_track_time	(	reme_context_t	c,
		reme_sensor_t	s,
		int *	track_time
	)

Get the tracking time.

The tracking time corresponds to the number of frames in which tracking succeeded. In case tracking fails the counter gets negative.

Parameters

c	A valid context object
s	A valid sensor object
track_time	The tracking time

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

reme_error_t reme_sensor_grab	(	reme_context_t	c,
		reme_sensor_t	s
	)

Trigger frame grabbing.

Provides a synchronization point to trigger image generation of all image types. This method is intended to be fast.

In order to synchronize frame grabbing from multiple sensors, call this method in sequence for each sensor before calling reme_sensor_retrieve.

Parameters

c	A valid context object
s	A valid sensor object

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_FAILED_TO_GRAB	Failed to grab from sensor. This is not necessarily an error, you might re-try.
REME_ERROR_UNSPECIFIED	Is returned otherwise.

Examples:

example_reconstructmesdk_calibration.cpp, example_reconstructmesdk_image.cpp, example_reconstructmesdk_one_minute.cpp, example_reconstructmesdk_recorder.cpp, example_reconstructmesdk_sensor_multi_independent.cpp, example_reconstructmesdk_surface.cpp, and example_reconstructmesdk_volume.cpp.

reme_error_t reme_sensor_prepare_images	(	reme_context_t	c,
		reme_sensor_t	s
	)

Retrieve image data corresponding to the previous grab command for further processing.

Updates the internal state of all images and prepares the required data structures on the computation device.

Parameters

c	A valid context object
s	A valid sensor object

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

Examples:

example_reconstructmesdk_one_minute.cpp, example_reconstructmesdk_recorder.cpp, example_reconstructmesdk_sensor_multi_independent.cpp, example_reconstructmesdk_surface.cpp, and example_reconstructmesdk_volume.cpp.

reme_error_t reme_sensor_prepare_image	(	reme_context_t	c,
		reme_sensor_t	s,
		_reme_sensor_image_t	i
	)

Retrieve specific image data for subsequent processing.

In case REME_IMAGE_AUX or REME_IMAGE_DEPTH is passed, this method will fetch the data into internal memory. In case REME_IMAGE_VOLUME is passed, the previously prepared REME_IMAGE_DEPTH will be uploaded to the computation device for subsequent processing (reme_sensor_track_position, reme_sensor_update_volume).

This method is especially useful (when compared to reme_sensor_prepare_images) when only depth and image data is required. For example when recording, there is no need for REME_IMAGE_VOLUME and it should be skipped so no time is wasted waiting for the data to be uploaded to the computation device.

Parameters

c	A valid context object
s	A valid sensor object
i	Image type to prepare

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

Examples:

example_reconstructmesdk_calibration.cpp, example_reconstructmesdk_image.cpp, and example_reconstructmesdk_recorder.cpp.

reme_error_t reme_sensor_track_position	(	reme_context_t	c,
		reme_sensor_t	s
	)

Attempts to track the sensor position.

Tries to track the sensor movement by matching the current depth data against the perspective from the last position. Initially the sensor position is the identity position, unless otherwise specified.

Parameters

c	A valid context object
s	A valid sensor object

Return values

REME_ERROR_SUCCESS	On tracking success or the corresponding volume is empty.
REME_ERROR_TRACK_LOST	When the tracking did not succeed. In this case the sensor is repositioned into latest recovery pose.
REME_ERROR_INVALID_LICENSE	is returned when tracking is disabled for a certain amount of time because of running in a non-commercial mode.
REME_ERROR_UNSPECIFIED	is returned otherwise.

Examples:

example_reconstructmesdk_one_minute.cpp, example_reconstructmesdk_recorder.cpp, example_reconstructmesdk_sensor_multi_independent.cpp, example_reconstructmesdk_surface.cpp, and example_reconstructmesdk_volume.cpp.

reme_error_t reme_sensor_update_volume	(	reme_context_t	c,
		reme_sensor_t	s
	)

Update the bound volume using the current sensor data.

Uses the current sensor position as the perspective to update the volume.

Parameters

c	A valid context object
s	A valid sensor object

Return values

REME_ERROR_SUCCESS	On success
REME_ERROR_UNSPECIFIED	On failure

Examples:

example_reconstructmesdk_one_minute.cpp, example_reconstructmesdk_recorder.cpp, example_reconstructmesdk_sensor_multi_independent.cpp, example_reconstructmesdk_surface.cpp, and example_reconstructmesdk_volume.cpp.