App Note: Using Helios2+ with the Point Cloud Library for Dimensioning

Helios2 Time-of-flight cameras from LUCID Vision Labs produce depth measurements of a scene as point clouds, which are data consisting of points in 3D space relative to the camera. The open-source Point Cloud Library (PCL) is a powerful way to process point clouds produced by these cameras.

This App Note demonstrates the processing and display of point clouds for a dimensioning application that uses the Helios2+ camera. The code examples described below use PCL to extract surfaces from the point cloud and overlay useful information such as dimensions, depth, and bounding boxes.

The code examples are optimized for the Helios2+ camera. Although this code can be used by any camera in the Helios2 series, the parameters would have to be changed accordingly.

The Point Cloud Library (PCL) is an open-source library for point cloud processing. Its features include a simple interactive visualization of a point cloud, estimating surface normals to analyze smoothness, filtering / downsampling a point cloud, and more advanced algorithms including registration (to align point clouds) and segmentation (to break a point cloud into more recognizable segments / surfaces).

Note: The application described below is only one possible use case of PCL with our Time-of-Flight cameras. Our current examples use several features offered by PCL, and depending on your application and setup, many other features can be used in a similar way to how we demonstrate in this App Note. Examples can include using Helios2 Narrow cameras to analyze and process targets at a much closer range, or using several of our Helios2+ cameras to analyze and stitch multiple point clouds together.

• Processor: i9-13th generation Intel processor
• RAM: 32 GB
• OS: Windows 11 Pro

Download the Arena SDK from the LUCID website, and install it selecting the Developer option when prompted.

To install the PCL library, it is recommended to first install a package manager. PCL lists several package managers here: https://pointclouds.org/downloads/.  This application was created in Windows, and we used vcpkg. To use vcpkg to install PCL, follow the steps below:

1. Navigate to the github repo for vcpkg.
2. Git clone this repo onto your PC or download as a ZIP folder. Note: ensure the file path this folder is Git cloned to (or extracted to) does not have any spaces in it – this can affect the PCL installation later down the line.
3. Navigate to vcpkg-master folder, and execute this through the terminal:

   .\bootstrap-vcpkg.bat

4. Execute the following command to install PCL:

   vcpkg install pcl[vtk]

   This installs all the PCL libraries + VTK libraries (visual add-ons). Be sure to specify vtk in your installation to install the visualization capabilities offered by PCL – the default installation does not include this.

There are several third-party libraries that can be used to color point clouds. We used cppturbo, which is a header-only implementation for coloring point clouds.

This library is available here: https://github.com/tatsuya-s/cppturbo.

To link installed libraries with the Visual Studio C++ example projects, follow the steps below:

1. Navigate to Environment variables on your system.
2. Under ‘System variables’, add a new variable called ‘PCL_ROOT’ and set its value to the folder location of the vcpkg (e.g., C:\Users\<username>\Downloads\vcpkg if you git cloned the folder here, and C:\Users\<username>\Downloads\vcpkg-master if you installed as zip folder here).
3. Add another variable called ‘TURBO_ROOT’ and set its value to the folder location of cppturbo-master (e.g., C:\Users\<username>\Downloads\cppturbo-master if you installed cppturbo-master here).
4. After downloading our example zip folder, navigate to the folder Cpp_HTP_PCL\DLLs, copy its path, and add it to your PATH variable.

The Helios2+ camera is mounted to a frame to view several cardboard boxes stacked on a pallet. The camera is pointing downwards toward the center of the pallet and is placed about 1.8m above the pallet.

Schematic of the how the camera is set up in the cardboard box visualization application.

The code samples for this application progressively build on each other.

These code examples are described in more detail below.

This example uses PCL to show the native point cloud stream, as seen by the camera.

Before using any PCL features, this example first sets Arena SDK specific parameters for optimal stream quality, such as turning on StandardHDR on the Helios2+. The example also makes certain Arena SDK parameters user-adjustable, like ConfidenceThreshold and FlyingPixel on the Helios2+, so the user can adjust them during the stream using certain keys assigned in code. This code also uses cppturbo to color the point cloud based on depth.

This example is similar to what you would see if you connected a Helios2+ to ArenaView, the Arena SDK’s built-in camera control program.

Cardboard boxes viewed with the Display code example. The colors are based on depth.

The user-adjustable parameters are as follows:

The initialization step initializes the following variables:

Sequence of operations for the Display example.

This example uses PCL’s Region Growing Based Segmentation algorithm to segment surfaces on our cardboard boxes into smaller point clouds (which are referred to as segments). The example then displays this collection of identified segments, removing points not belonging to any segment, using the previous Display code example. We display PCL’s own output of this collection of segments, and so the color chosen for each segment is from PCL’s algorithm.

Cardboard boxes viewed with the Segmentation code example. The colors are generated by PCL for each segment.

The user-adjustable parameters are as follows:

The initialization step initializes the following variables:

Sequence of operations for the Segmentation example.

This example uses PCL’s computeCentroidAndOBB function to calculate an oriented bounding box for each segment identified in the previous code sample. This includes the bounding box’s centroid position, its dimensions, and the rotational matrix required for the box to fit onto this segment along its principal directions.

For this example, we iterate through all the segments (outputted from the segmentation algorithm), to determine the bounding box for each segment.

To make our example more structured, we create two structs:

• segmentInfo: Stores the segmented point cloud, and its centroid (used for processing each segment).
• bbInfo: Stores the information required to add a bounding box to the viewer.

Since we will add bounding boxes on our segments, we can easily identify each segment in the viewer. Therefore, we display the native output from the camera with all its points, colored with depth (output of the first example), and not the output with PCL’s own colors (output of the second example).

This implementation also offers the following three display settings, which can be toggled using the keyboard.

The user-adjustable parameters are as follows:

The initialization step initializes the following variables:

Sequence of operations for the Dimensioning example.

The previous example performs the segmentation, bounding box calculation, and the visualization in a single thread.

To improve the speed, this example splits that code into two threads: one thread of processing, and the other for visualization. This implementation uses only a few variables that are shared across the two threads: one to store a new point cloud taken from the Helios2+(sharedCloud), and one to store the corresponding bounding box for that cloud’s segments (sharedBBs).

This sequence is outlined below:

1. The visualization thread grabs a new point cloud and stores it in sharedCloud.
2. The processing thread segments sharedCloud and computes bounding boxes for its segments. Once computed, the bounding boxes are stored in sharedBBs.
3. The visualization thread iterates through sharedBBs and adds the appropriate details to each segment on the viewer.
4. Repeat.

The user-adjustable parameters are as follows:

The initialization step initializes the following variables:

Sequence of operations for the Multithreading example.

Keyboard inputs for the examples are shown below: