Coordinate Frames

Coordinate systems involved in ReconstructMe

Since ReconstructMe operates in three dimensional space, the involved coordinate systems form three dimensional spaces. ReconstructMe uses right-handed coordinate systems which are represented as 4 by 4 homogenous matrices. The Transform explains the memory layout details and how to work with those matrices.

Sensor Coordinate System

The first coordinate system you should be aware of is the sensor coordinate system. This coordinate system is attached to the sensor in the following way

Sensor Coordinate Sytem

The main purpose of the sensor coordinate system is specify how a sensor views the environment and produces its data. That is, when holding the sensor horizontally and looking through the sensor into the room, the positive x-axis extends to the right, the positive y-axis points towards the floor and positive z-axis extends into the room.

World Coordinate System

In the world coordinate system the reconstruction and mesh extraction is performed. The reconstruction volume is defined as an axis aligned bounding box with respect to the world coordinate system. For example, the following settings

volume {
  minimum_corner {
    x: -500
    y: -500
    z: -500
  }
  maximum_corner {
    x: 500
    y: 500
    z: 500
  }
}

define a reconstruction volume that extends 500 units in each direction around the origin of the world coordinate system.

Since we don't have any absolute reference in the space, the absolute location of world coordinate system is initially undefined or arbitrary. Once the first sensor data is fed into the reconstruction volume (reme_sensor_update_volume), the position and orientation of the world coordinate system 'freezes' until the volume is resetted to empty state (reme_volume_reset).

The absolute position of the world coordinate system in real-space is thus defined

• in the exact moment the first data is fed into the reconstruction volume (reme_sensor_update_volume)
• by the absolute sensor position in the real space and
• by the initial virtual sensor position (reme_sensor_get_position, reme_sensor_set_position) with respect to the world coordinate system.

By default the initial virtual sensor position is the identity matrix, meaning that the sensor frame coincides with the world frame. You can use reme_sensor_set_position (see ManualSensorWorldPositioning) and reme_sensor_set_prescan_position (see AutomaticSensorWorldPositioning) to modify this relationship. Since version 1.6 the standard configuration files position the reconstruction volume around the origin of the world coordinate system, meaning that by default the sensor starts in the middle of the volume.

The illustration below should help you grasp that concept. In virtual space the position of the sensor has been set to be behind the world coordinate system. Thus, the world coordinate system is floating infront of the sensor in real-space. When the reconstruction starts, the world coordinate system freezes at the current position and the sensor starts to reposition itself with respect to world coordinate system via tracking (reme_sensor_track_position).

Word Coordinate System Placement in Real-Space

The world coordinate system is important to you as the reconstruction volume is defined with respect to it and surfaces (Surface) will be generated with respect to it.

Automatic Sensor to World Positioning

The position of the sensor with respect to the world when reconstruction starts can be automatically set by calling reme_sensor_set_prescan_position. This method takes as input argument a context, a sensor and a choice for the positioning algorithm. Currently, ReconstructMe supports the following choices

Placing the sensor infront of the volume

By submitting REME_SENSOR_POSITION_INFRONT the sensor will be automatically placed 'infront' of the world coordinate system. 'Infront' is herby defined as: Assume without loss of generality that the sensor held horizontally pointing towards the target. Then the position is chosen so that z-axis of the world ordinate is the natural up-direction, the sensor looks towards the positive y-axis of the world coordinate system, the sensor is located at the center of the front face of the reconstruction volume and is moved back (negative y-axis) by 300 units.

Automatic sensor positioning using ::REME_SENSOR_POSITION_INFRONT

Placing the sensor in the center of the volume

By submitting REME_SENSOR_POSITION_CENTER the sensor will be automatically placed in the center of the volume. The orientation of the sensor will not be adapted, therefore both, the sensor and world coordinate system, will be oriented equally. Note that the volume does not need to span centric around the world coordinate system as the image below illustrates.

Automatic sensor positioning using ::REME_SENSOR_POSITION_CENTER

Placing the volume with respect to the floor

While the previous methods of positioning the sensor did not involve any input data, placing the volume on the floor requires knowledge about where the floor is located in space. By passing REME_SENSOR_POSITION_FLOOR the positioning algorithm attempts to find the floor in the current sensor depth image. If that fails REME_ERROR_NO_FLOOR is returned and positioning failed.

The algorithm works best when a large portion of the image is covered by floor data (i.e planar data) and the sensor is held without roll (i.e no rotation around the sensor's z-axis). Once the floor is detected the orientation of the world coordinate system is fixed:

• the z-axis is normal to to the floor plane and points towards the natural ceiling
• the x-axis is aligned with the sensor's x-axis
• the y-axis is formed by the cross product of the former two axes.

The origin of the world coordinate system equals the point of intersection between the ray through the center pixel and the estimated floor plane. The following image illustrates this.

Automatic sensor positioning using ::REME_SENSOR_POSITION_FLOOR

The following snippet shows how to detect the floor before reconstruction starts. Note that some error checking is omitted for readability.

reme_sensor_grab(c, s);
reme_sensor_prepare_images(c, s);
while (!REME_SUCCESS(reme_sensor_set_prescan_position(c, s, REME_SENSOR_POSITION_FLOOR))) {
    reme_sensor_grab(c, s);
    reme_sensor_prepare_images(c, s);  
}

Manual Sensor to World Positioning

3D printers, in contrast to 2D printers, print three dimensional objects instead of flat sheets of paper. There are different techniques on how these devices work, but one common point is that there is a 3D printer coordinate system in which the surface model has to be placed when you intend to print it.

A lot of printers define a platform, i.e a planar region, that defines the floor of the print. The illustration below shows such a platform with the printer coordinate system in the center of the platform (x-axis:red, y-axis:green, z-axis:blue).

Printer Platform

Ideally, the resulting surface of our reconstruction already sits on this platform and needs only to be scaled to fit the printer space. This example shows you how to set up the world coordinate and the sensor position to accomplish this.

As mentioned in section CoordinateSystemsWorld, the world coordinate system is arbitrary until the reconstruction starts. So, as we are free to choose, we assume that our world coordinate system coincides with the printer coordinate system (nothing todo in the code at this point).

Next, we need to define the reconstruction volume with respect to the world coordinate system. The reconstruction volume is defined as an axis-aligned box with respect to the world coordinate system.

For this example we let our reconstruction volume to extend from the platform towards positive z-axis about 1 meter in length and extend on the other axes from -500 to 500 millimeter. This gives us a reconstruction volume similar to the one illustrated below (note that in reality the printer platform is not as big as shown below).

Reconstruction Volume

Axis-aligned boxes, as our reconstruction volume, are often represented by a minimim and maximum corner point. Both points are marked in the illustration below.

Reconstruction Volume Corner Points

The corner points of the reconstruction volume and can be modified via the fields volume_min and volume_max that are part of the reconstruction options (reme_context_bind_reconstruction_options). Here's a snippet that does that (see Accessing and Modifying Options for more details on how to modify/access options)

    // Create a new context
    reme_context_t c;
    reme_context_create(&c);

    // Create options to modify the position of the volume
    reme_options_t o;
    reme_options_create(c, &o);
    reme_context_bind_reconstruction_options(c, o);

    reme_options_set_int(c, o, "volume.minimum_corner.x", -500);
    reme_options_set_int(c, o, "volume.minimum_corner.y", -500);
    reme_options_set_int(c, o, "volume.minimum_corner.z", 0);

    reme_options_set_int(c, o, "volume.maximum_corner.x", 500);
    reme_options_set_int(c, o, "volume.maximum_corner.y", 500);
    reme_options_set_int(c, o, "volume.maximum_corner.z", 1000);

    reme_context_tune_reconstruction_options(c, REME_TUNE_PROFILE_MID_QUALITY);

    // Compile for OpenCL device using defaults
    reme_context_compile(c);

    // Create a new volume
    reme_volume_t v;
    reme_volume_create(c, &v);

As stated in the section CoordinateSystemsWorld, the initial sensor position coincides with the world coordinate system. For this tutorial we will set the initial sensor position to be to the left of platform, looking towards the platform from the outside of the reconstruction volume. The following illustrations shows this

Initial Sensor Position with Respect to the World Coordinate System

This requires a Transform which can be set up by hands or one can use the utility method reme_transform_look_at to simplify the process. This method takes the new origin of the camera, a reference point the camera looks at and a new up-vector. From these values the method establishes a sensor coordinate system with respect to the world coordinate system.

    // Create a new sensor.
    reme_sensor_t s;
    reme_sensor_create(c, "openni;mskinect;file", true, &s);
    reme_sensor_open(c, s);

    // Specify eye, ref and up with respect to world
    const float eye[3] = {0.f, -1000.f,  500.f};
    const float ref[3] = {0.f,     0.f,  500.f};
    const float up[3]  = {0.f,     0.f,    1.f};

    // Create sensor coordinate system with respect to world
    float mat[16];
    reme_transform_look_at(c, eye, ref, up, mat);

    // Set initial sensor position
    reme_sensor_set_position(c, s, mat);
    // We set the same position first recovery position
    reme_sensor_set_recovery_position(c, s, mat);

The matrix mat in the above example equals the matrix mat2 in the following handwritten example

    //                    x     y     z    origin
    float mat2[4][4] = {{1.f,  0.f,  0.f,     0.f},
        {0.f,  0.f,  1.f, -1000.f},
        {0.f, -1.f,  0.f,   500.f},
        {0.f,  0.f,  0.f,     1.f}
    };

    reme_sensor_set_position(c, s, &mat2[0][0]);
    reme_sensor_set_recovery_position(c, s, &mat2[0][0]);

The rest of the example equals standard scanning code as shown in the one-minute example. Remember, that the world coordinate system is fixed when the reconstruction starts. To achieve optimal results with the above settings, hold your sensor horizontally and about 1 meter away from the object to scan. Then the center of resulting mesh will coincide with the printer platform and z-axis will point towards the ceiling.

Marker Coordinate System

The canonical coordinate system of the marker is shown in the image below. When printed and laid on floor, positive z-axis points towards the ceiling, positive x-axis to the right and positive y-axis to the top of the marker. The origin of the coordinate system is in the center of the marker.

Fiducial Marker Coordinate System