Advanced Code Samples and Topics

Various advanced topics

Compensating for Tilt Movements

Popular 3D scanners such as the Microsoft Kinect and others have a motorized base that allows enlargement of the interaction space by tilting the sensor. These sensors, however, turn of the IR projector while the tilt is in progress, essentially making ReconstructMe blind while the tilt occurs. Since ReconstructMe currently relies on small movements between frames it will often loose track after the tilt completes.

This example shows you how to compensate for such a blind tilt by compensating through a virtual repositioning of the sensor. As most sensors make the tilt relative to the gravitation vector the tilt angle difference can only be calculated accurately for sensors that stand on a fixed platform and are not moved by hand. After tilting it is currently advised to skip a couple of depth frames to compensate wrong data from a recent projector turn-on.

Below is the main scanning loop of the example. This is an ordinary scanning loop with one slight difference: skip_frame_counter causes frames to be skipped when greater zero. This counter gets decremented in each loop cycle.

  bool continue_scanning = true;

  // After each tilt we skip a couple of frames
  int skip_frame_counter = 0;
  while (continue_scanning && REME_SUCCESS(reme_sensor_grab(c, s))) {

    // Keyboard handling
    if (_kbhit()) { 
      char k = _getch();
      switch(k) {

        // Key 'r' resets the volume to empty state
        case 'r': {
          reme_volume_reset(c, v);
          reme_sensor_reset(c, s);
          break;
                  }

        // Key 'f' tilt +10 degrees
        case 'f': {
          perform_tilt(c, s, o, 10);
          reme_sensor_set_trackhint(c, s, REME_SENSOR_TRACKHINT_USE_GLOBAL);
          break;
                  }

        // Key 'b' tilt -10 degrees
        case 'b': {
          perform_tilt(c, s, o, -10);
          reme_sensor_set_trackhint(c, s, REME_SENSOR_TRACKHINT_USE_GLOBAL);
          skip_frame_counter = 10;
          break;
                  }

        case 'x': {
          continue_scanning = false;
          break;
                  }

      }
    }

    // Prepare image and depth data
    reme_sensor_prepare_images(c, s);

    if (skip_frame_counter == 0) {
      if (REME_SUCCESS(reme_sensor_track_position(c, s))) {
        reme_sensor_update_volume(c, s);
      }
    } else {
      skip_frame_counter -= 1;
    }

    // Update the viewer
    reme_sensor_get_image(c, s, REME_IMAGE_DEPTH, imgs[0]);
    reme_sensor_get_image(c, s, REME_IMAGE_VOLUME, imgs[1]);
    reme_viewer_update(c, viewer);

  }

The missing piece is the method called perform_tilt that takes a context, sensor, capture options and an incremental tilt angle (relativ to the current) in degrees.

// Perform the tilt and compensate for it. 
// After tilting a couple of frames should be skipped to compensate for bad data.
void perform_tilt(reme_context_t c, reme_sensor_t s, reme_options_t o, int angle)
{
  // Read current angle
  reme_sensor_bind_capture_options(c, s, o);
  int prev_angle;
  reme_options_get_int(c, o, "tilt_angle", &prev_angle);
          
  // Apply tilt
  reme_options_clear(c, o);
  reme_options_set_int(c, o, "tilt_angle", prev_angle + angle);
  reme_sensor_apply_capture_options(c, s, o);

  // Read tilt angle again to determine to determine actual movement
  reme_sensor_bind_capture_options(c, s, o);
  int new_angle;
  reme_options_get_int(c, o, "tilt_angle", &new_angle);

  // Compensate for tilt in sensor position
  float pos[16];
  reme_sensor_get_position(c, s, pos);
  reme_transform_compensate_tilt(c, pos, new_angle - prev_angle, pos);
  reme_sensor_set_position(c, s, pos);
}

The method reme_transform_compensate_tilt assumes that the tilt axis is parallel to the sensor positive sensor x-axis. It also assumes that the origin of the tilt movement is placed at roughly (0, 30, -25) with respect to the sensor coordinate system. As such, this compensation will currently only work for Kinect for Windows sensors. While this is an inaccurate measurement and the assumpations are partly wrong, the solution works in most cases as ReconstructMe is able to compensate for the remaining error.

It could be improved in terms of standard stereo calibration that takes the known rotation angle into account.

Note that you cannot tilt arbitrary large angles as the Kinect limits the tilt-angle to +/- 27 degrees and a large tilt may force a lot of new data to be integrated which could trigger track-lost detection.