Color Support

How to colorize your reconstruction.

ReconstructMe allows you to create colorized triangular surface representations of your reconstruction. There are many approaches to colorize your result and the one that ReconstructMe currently supports is colorization of surface vertices. That is, each vertex receives a distinct color that is generated from all RGB frames.

The following video shows the result created by the following sections.

Colorization Prerequisites

In order to digitize with color support, you need to have a sensor or file stream that supports colors. Since the camera that captures the depth stream is physically offsetted by the camera that provides the RGB stream, you need to register the depth and color stream. Most supported sensors provide this ability directly in their SDK and you only need to enable it via the sensor options as follows

    reme_sensor_bind_camera_options(c, s, o);

    // Ensure AUX stream is aligned with the DEPTH stream
    reme_options_set_bool(c, o, "enable_align_viewpoints", true);

    reme_sensor_open(c, s);

For file streams this property is not available. However, file streams recorded with sensors that have the alignment property turned on will equally work well.

Next, you need to instruct ReconstructMe to compile with color support enabled. This is done by

    reme_context_bind_reconstruction_options(c, o);

    // Make sure that we enable color support in the program settings.
    reme_options_set_bool(c, o, "data_integration.use_colors", true);

    // Use the tuning to derive a couple of related settings automatically.
    reme_context_tune_reconstruction_options(c, REME_TUNE_PROFILE_MID_QUALITY);

    // Compile for OpenCL device using defaults
    reme_context_compile(c);

Note

Enabling color support will take twice as much GPU memory. The memory required for a volume of 256x256x256 with color support enabled can be calculated as 256(res-x) x 256(res-y) x 256(res-z) x 4(bytes per voxel) x 2(account for color support) = 128 Mbytes.

Finally, we'd like to render the volume image using colors, so we get a nice preview of the result

    reme_sensor_bind_render_options(c, s, o);

    // Use color in rendering volume image
    reme_options_set(c, o, "shade_mode", "SHADE_COLORS");

    // Apply
    reme_sensor_apply_render_options(c, s, o);

That's all there is to setup ReconstructMe with color support enabled.

Surface Vertex Colorization

Once you have completed the setup you perform reconstruction as usual. See the one minute example for reference. In order to generate a mesh with vertex colors you proceed as follows

    // Create a new surface
    reme_surface_t m;
    reme_surface_create(c, &m);
    reme_surface_generate(c, m, v);

    // Colorize surface vertices.
    reme_surface_colorize_vertices(c, m, v);

    // Inpaint vertices with invalid colors.
    reme_surface_inpaint_vertices(c, m);

    // Remesh the surface to generate an isotropic tessellation
    reme_surface_bind_remesh_options(c, m, o);

    // The following properties define the allowed edge length range
    // The tesselation will adhere to these values when splitting/collapsing
    // edges according to local color and geometry change.
    reme_options_set_real(c, o, "minimum_edge_length", 3);
    reme_options_set_real(c, o, "maximum_edge_length", 20);

    // Perform remeshing
    reme_surface_remesh(c, m);

    // Remeshing as decimation might change vertex positions,
    // we should update the color information
    reme_surface_colorize_vertices(c, m, v);
    reme_surface_inpaint_vertices(c, m);

The function reme_surface_colorize_vertices will colorize all vertices that lie within a valid region of the color volume. At volume borders and unseen regions no color information can be extracted. Vertices inside those areas will receive an invalid vertex color with is currently mapped to pure green (subject to change in later versions).

In order to paint vertices with invalid colors you can use reme_surface_inpaint_vertices. This algorithm will attempt to find a suitable color for each vertex with invalid color by looking at neighboring vertex colors. In case a connected component is isolated and every vertex in that component has an invalid vertex color, the algorithm cannot determine a suitable color.

You might want to invoke any of the surface manipulating functions before colorizing/inpainting the surface. E.g colorizing a Poisson reconstructed surface is also supported. Holes closed by the algorithm would then be automatically colorized (actually inpainted).

Finally save the colored surface.

    // Save to file including colored vertices.
    reme_surface_save_to_file(c, m, "test.ply");

Note

Not all file formats support vertex colors (i.e PLY does, but STL does not). In case it is supported, the vertex colors are automatically saved.

That's it! The entire example can be found in example_reconstructmesdk_colorize.cpp