ReconstructMe SDK 2.6.43-0
Real-time 3D reconstruction engine
How to interact with cameras/sensors in ReconstructMe SDK
In ReconstructMe 3D cameras are accessed via the Sensor interface. In particular one needs to create a reme_sensor_t for each 3D camera to interact with. ReconstructMe supports a wide range of modern 3D cameras such as the Microsoft Kinect (Windows/XBox) or the ASUS Xtion family. Additionally ReconstructMe provides sensors that stream from files.
Here is an overview of the states a sensor can be in.
Once a sensor is created (reme_sensor_create) it is closed state. Depending on the arguments provided a successful create call ensures at least that the necessary drivers are installed on the current machine.
The transit to the Open state is accomplished through a successful reme_sensor_open call. This command takes the current sensor configuration into account (reme_sensor_bind_camera_options) and tries to open the requested sensor.
Once a sensor has been successfully opened you can access and potentially modify its capture properties (reme_sensor_bind_capture_options, reme_sensor_apply_capture_options) and access supported image frames (reme_sensor_get_image, reme_sensor_prepare_image). In contrast to convential cameras each sensor in ReconstructMe has a three-dimensional position (reme_sensor_get_position, reme_sensor_set_position) with respect to the world coordinate system (see Coordinate Systems Introduction). This position can either be modified by hand or automatically determined by tracking the sensor position via matching data in the reconstruction volume in current sensor data input.
Each sensor is created through a call to reme_sensor_create. This method accepts a
driver argument that refers to a single backend identifier for communication with the sensor (i.e. OpenNI, Kinect SDK, ...), or a sensor configuration file to use. Besides single arguments this method accepts a list of drivers / configuration paths to test.
In case a list of sensors is passed the first one working will be returned. Working in this context is defined by value of
TRUEIt is ensured that the sensor returned can be opened successfully: I.e when the driver argument is
openniit is ensured that the sensor can be opened using the default OpenNI configuration; when the driver argument is a path to a configuration file it is ensured that the sensor can be opened using the particular configuration stored.
FALSEIt is ensured that the necessary drivers are installed on the current machine but no test to ensure a successful opening of the sensor is performed.
Here is a quick example
When a sensor is opened through the reme_sensor_open call it uses the current state of the camera configuration (reme_sensor_bind_camera_options). In the following example the RGB image stream is disabled and IR stream for the Microsoft Kinect is enabled. The sensors usally offer two streams in ReconstructMe:
Opened sensors allow access to capture properties. These properties reflect information about the current state of the sensor (frame sizes, support for frame types, position of tilt motors etc.). For example to access the frame size of the auxilary image one would perform the following.
To modify capture properties such as the tilt angle of the Kinect one would perform the following steps
Once the sensor is open you can access its images. The example below illustrates this
Each sensor has a 3D position and orientation with respect to the world coordinate system. The page Coordinate Systems and Sensor Positioning explains this in detail.
The position of the sensor with respect to the world coordinate sytem can be tracked automatically using reme_sensor_track_position. This is the heart of the ReconstructMe engine as it automatically determines the position of the sensor while it is moved in real-time.
Since version 1.4 ReconstructMe supports two different tracking algorithms:
local searchLocal search is fast and succeeds when the camera movement between two subsequent frames is small. It used to be the only search method prior to 1.4.
global searchAdded in version 1.4 based on algorithms of CANDELOR (http://www.candelor.com). Global search is slower than local search but succeeds in cases where the camera movement between two subsequent frames is large.
ReconstructMe uses a combination of the above algorithms to improve camera tracking. The behaviour of ReconstructMe can be configured with reme_sensor_set_trackmode
Note that although CANDELOR supports truly global search, the implementation in ReconstructMe uses the the volume data as viewed by the current camera position as the model. This leads to a partial global behaviour as the overlap between the model and the current frame has to exceed a predefined threshold. This is subject to change in future versions.
Additionally one might provide tracking hints that can temporarily change the behaviour of the tracking. See reme_sensor_set_trackhint for details.
Please see the Getting Started page for usage. You might also be interested in example_reconstructmesdk_sensor_multi_independent.cpp that allows you to track multiple sensors.
Once the sensor position with respect to the world coordinate is known the current sensor data can be pushed into the reconstruction volume. The reconstruction volume will then care about fusioning the data to a global model. Please refer to the Getting Started page for an introduction.
An opened sensor can be closed as follows.
A closed sensor can be destroyed as follows.
Popular 3D scanners such as the Microsoft Kinect and others have a motorized base that allows enlargement of the interaction space by tilting the sensor. These sensors, however, turn of the IR projector while the tilt is in progress, essentially making ReconstructMe blind while the tilt occurs. Since ReconstructMe currently relies on small movements between frames it will often loose track after the tilt completes.
This example shows you how to compensate for such a blind tilt by compensating through a virtual repositioning of the sensor. As most sensors make the tilt relative to the gravitation vector the tilt angle difference can only be calculated accurately for sensors that stand on a fixed platform and are not moved by hand. After tilting it is currently advised to skip a couple of depth frames to compensate wrong data from a recent projector turn-on.
Below is the main scanning loop of the example. This is an ordinary scanning loop with one slight difference:
skip_frame_counter causes frames to be skipped when greater zero. This counter gets decremented in each loop cycle.
The missing piece is the method called
perform_tilt that takes a context, sensor, capture options and an incremental tilt angle (relativ to the current) in degrees.
The method reme_transform_compensate_tilt assumes that the tilt axis is parallel to the sensor positive sensor x-axis. It also assumes that the origin of the tilt movement is placed at roughly (0, 30, -25) with respect to the sensor coordinate system. As such, this compensation will currently only work for Kinect for Windows sensors. While this is an inaccurate measurement and the assumpations are partly wrong, the solution works in most cases as ReconstructMe is able to compensate for the remaining error.
It could be improved in terms of standard stereo calibration that takes the known rotation angle into account.
Note that you cannot tilt arbitrary large angles as the Kinect limits the tilt-angle to +/- 27 degrees and a large tilt may force a lot of new data to be integrated which could trigger track-lost detection.