Empty pointcloud data when accelerate_gpu_with_glsl = True

[edgarcamilocamacho]

Hello

Required Info
Camera Model	D435i
Firmware Version	5.16.0.1
Operating System	Ubuntu 20.04.6 LTS
Platform	NVIDIA Jetson Orin AGX
Jetpack	5.1.1 [L4T 35.3.1]
Kernel Version (Linux Only)	5.10.104-tegra
Librealsense SDK Version	2.56.1
ROS Distro	Humble
RealSense ROS Wrapper Version	4.56.1

Issue Description

When I set accelerate_gpu_with_glsl = True (at the begining of the execution, not dinamically), I get empty data pointcloud like:

header:
  stamp:
    sec: 1726699016
    nanosec: 654941162
  frame_id: camera_depth_optical_frame
height: 1
width: 0
fields:
- name: x
  offset: 0
  datatype: 7
  count: 1
- name: y
  offset: 4
  datatype: 7
  count: 1
- name: z
  offset: 8
  datatype: 7
  count: 1
is_bigendian: false
point_step: 16
row_step: 0
data: []
is_dense: true

But with accelerate_gpu_with_glsl = False it works perfectly.

Maybe I'm setting something wrong, I attach the list of ros params:

rs-params.txt

[MartyG-RealSense]

Hi @edgarcamilocamacho In rare cases on some computers, a pointcloud will disappear when GLSL is enabled. There is no way to predict in advance which computers will be unable to use GLSL, unfortunately.

If you are using a Jetson then the usual way to accelerate a pointcloud will be to enable the librealsense SDK's CUDA support. This can be done by installing the Jetson version of the librealsense packages on the installation_jetson.md instruction page or by building from source code with the flag -DBUILD_WITH_CUDA=ON included in the CMake build instruction.

[edgarcamilocamacho]

So if I install librealsense and realsense-ros in the Jetson, both from apt packages, like:

• librealsense: https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_jetson.md
  

• realsense-ros: https://github.com/IntelRealSense/realsense-ros/blob/ros2-master/README.md
  

I should get both of them running with GPU, right?

There is a way to identify that it's actually using the GPU?

[edgarcamilocamacho]

Following that idea I this in my Jetson container (the sumarized steps from the links I sent before):

# apt-key adv --keyserver keyserver.ubuntu.com --recv-key F6E65AC044F831AC80A06380C8B3A55A6F3EFCDE || sudo apt-key # adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-key F6E65AC044F831AC80A06380C8B3A55A6F3EFCDE
# add-apt-repository "deb https://librealsense.intel.com/Debian/apt-repo $(lsb_release -cs) main" -u
# apt-get install -y librealsense2-utils librealsense2-dev ros-humble-realsense2-*

After that I get these versions installed:

# apt list --installed *realsense*
Listing... Done
librealsense2-dev/focal,now 2.55.1-0~realsense.3336 arm64 [installed]
librealsense2-gl/focal,now 2.55.1-0~realsense.3336 arm64 [installed,automatic]
librealsense2-udev-rules/focal,now 2.55.1-0~realsense.3336 arm64 [installed,automatic]
librealsense2-utils/focal,now 2.55.1-0~realsense.3336 arm64 [installed]
librealsense2/focal,now 2.55.1-0~realsense.3336 arm64 [installed,automatic]
ros-humble-librealsense2/focal,now 2.54.1-1focal.20231218.194516 arm64 [installed,automatic]
ros-humble-realsense2-camera-dbgsym/focal,now 4.54.1-1focal.20231230.021252 arm64 [installed]
ros-humble-realsense2-camera-msgs-dbgsym/focal,now 4.54.1-1focal.20231229.233803 arm64 [installed]
ros-humble-realsense2-camera-msgs/focal,now 4.54.1-1focal.20231229.233803 arm64 [installed]
ros-humble-realsense2-camera/focal,now 4.54.1-1focal.20231230.021252 arm64 [installed]
ros-humble-realsense2-description/focal,now 4.54.1-1focal.20231229.234552 arm64 [installed]

I got librealsense2=2.55.1 and realsense-ros=4.54.1, so aparently the apt packages are pointing to not compatible libraries.

I could install X.54.1 in both cases but then I won't have the accelerate_gpu_with_glsl (or that's what I understand, because that parameters is only mentioned in the documentation from version 4.55.1).

[edgarcamilocamacho]

Wait... seeing it closely, it installed librealsense2 and ros-humble-librealsense2. The second one has the correct version, so the questions would be:

• The ros-humble-realsense2-camera packages would use the libraries provided by ros-humble-librealsense2 instead of the librealsense2 one?
• You say the librealsense2 apt package is ready to use CUDA in the Jetson, is the package ros-humble-librealsense2 as well?

[MartyG-RealSense]

I am not involved in the compilation and publishing of packages, so do not have advice to provide about their contents.

As a general rule, assume that if installation is not being performed with the Jetson packages on the installation_jetson.md page or the JetsonHacks website's installLibrealsense.sh build script at the link below then the packages do not contain CUDA support. This is because packages that will be used on desktop / laptop PCs do not need the CUDA support because it only works with Nvidia GPUs, usually the ones built onto a Jetson board.

https://github.com/JetsonHacksNano/installLibrealsense

[MartyG-RealSense]

Hi @edgarcamilocamacho Do you have an update about this case that you can provide, please Thanks!

[edgarcamilocamacho]

These are the steps I'm following in my dockerfile (in the Jetson). I took the steps from the scripts you told me.

# Install librealsense
RUN cd /tmp && \
    wget https://github.com/IntelRealSense/librealsense/archive/refs/tags/v2.56.1.tar.gz && \
    tar -xzf v2.56.1.tar.gz && \
    cd librealsense-2.56.1 && \
    mkdir build && \
    cd build && \
    cmake .. \
        -DBUILD_PYTHON_BINDINGS:bool=true \
        -DPYTHON_EXECUTABLE=/usr/bin/python3 \
        -DBUILD_EXAMPLES=true \
        -DCMAKE_BUILD_TYPE=release \
        -DFORCE_LIBUVC=true \
        -DFORCE_RSUSB_BACKEND=false \
        -DBUILD_WITH_CUDA:bool=true \
        && \
    make -j 10 && \
    make install && \
    cd ../.. && \
    rm -r v2.56.1.tar.gz librealsense-2.56.1/

# Install realsense-ros
RUN source /setup_env.bash && \
    mkdir -p /robot/additional_ros_packages/src && \
    cd /robot/additional_ros_packages/src && \
    git clone -b 4.56.1  https://github.com/IntelRealSense/realsense-ros.git && \
    cd /robot/additional_ros_packages && \
    colcon build --merge-install --cmake-args '-DBUILD_ACCELERATE_GPU_WITH_GLSL=ON' && \
    rm -rf src build log

And I still have the same problem.

I attach the complete dockerfile.

In addition this is the docker-compose configuration:

  cameras:

    # common
    network_mode: host
    pid: host
    privileged: true
    runtime: nvidia
    restart: unless-stopped
    # specific
    image: gary.core.cameras.cameras:jetsonorin.4.3.beta

    environment:
      # display
      - XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR}
      - DISPLAY=:0

    env_file:
    # common
      - ./config/env/env

    volumes:
      # common
      - ./config:/robot/config
      - ./generic_persistent_data:/robot/generic_persistent_data
      - ./component_persistent_data/cameras:/robot/component_persistent_data
      # devices
      - /dev:/dev
      - /run/udev:/run/udev:ro
      # display
      - ${XDG_RUNTIME_DIR}/gdm/Xauthority:/root/.Xauthority
      - /tmp/.X11-unix:/tmp/.X11-unix:rw

rs.dockerfile.txt

[MartyG-RealSense]

My knowledge of Docker is admittedly limited, so there is not much that I can add to the previous advice, unfortunately.

I would recommend removing the -DBUILD_ACCELERATE_GPU_WITH_GLSL=ON pointcloud acceleration parameter, since the pointcloud will already be being accelerated by -DBUILD_WITH_CUDA:bool=true when this Dockerfile is used on Jetson.

Havibg said that, there have been a small number of past cases where problems disappear when not setting -DBUILD_WITH_CUDA to true. So you could try it either with -DBUILD_ACCELERATE_GPU_WITH_GLSL=ON or with DBUILD_WITH_CUDA:bool=true but not have both set to true at the same time.

[MartyG-RealSense]

Hi @edgarcamilocamacho Do you require further assistance with this case, please? Thanks!

[edgarcamilocamacho]

Hello @MartyG-RealSense , thanks for the fast response and sorry for the slow testing, we have several things on the table, but for sure next week we'll build the container again and test.

[MartyG-RealSense]

It's no problem at all. I look forward to your next report. Good luck!

[MartyG-RealSense]

Hi @edgarcamilocamacho Do you have an update about this case that you can provide, please? Thanks!

[vahapt]

Hi @MartyG-RealSense,
I've replicated the same problem on the following hardware setups.

Configuration	Setup 1	Setup 2
Camera Model	D435	D435
Firmware Version	5.16.0.1	5.16.0.1
Operating System & Version	Ubuntu v22.04 aarch64 Virtualized via VMWare under MacOS (M3 processor) [USB passthrough is enabled]	Ubuntu v22.04 aarch64
Kernel Version (Linux Only)	6.8.0-45	6.1.43
Platform	PC aarch64	Orange PI 5 - rockchip - aarch64
Librealsense SDK Version	2.55.1	2.55.1
Language	C++	C++
Segment	Robot	Robot
ROS Distro	Humble	Humble
RealSense ROS Wrapper Version	v4.55.1	v4.55.1

I've the following findings (Using Setup 1 hardware):
In short:

	ROS2 Repos	Librealsense Repos	Built from Source
librealsense2.so	Works	Does not Work	Works
librealsense2-gl.so	N/A	Does not Work	Cannot be built

• librealsense2.so distributed by ros works with aarch64 architecture but it does not have librealsense2-gl.so included, thus GPU point cloud generation is not working
• librealsense2.so and librealsense2-gl.so distributed by librealsense, does not work on aarch64 architecture (point cloud generation)
• librealsense2.so built from source works with aarch64 architecture, for some reason librealsense2-gl.so cannot be built even if -DBUILD_GLSL_EXTENSIONS=ON is defined. Thus I could not activate GPU based point cloud generation.

Long story:

Scenario1:

• I've installed ros modules via

sudo apt-get install ros-humble-realsense2-camera ros-humble-realsense2-camera-msgs ros-humble-realsense2-description

• During installation realsense libraries are installed to

  /opt/ros/humble/lib/aarch64-linux-gnu/librealsense2*

• This installation method does not have gl acceleration (both ros-wrapper and librealsense libraries), but it works as intended.
• I ran the realsense camera node with the following config (we'll use the same config across the scenarios):

ros2 run realsense2_camera realsense2_camera_node --ros-args -r __ns:=/ -r __node:=camera_d435_front -p initial_reset:=True -p accelerate_gpu_with_glsl:=False -p pointcloud.enable:=True -p rgb_camera.color_profile:=848x480x30 -p depth_module.infra_profile:=848x480x30 -p depth_module.depth_profile:=848x480x30 -p camera_name:=camera_d435_front -p enable_color:=False -p enable_depth:=True -p enable_infra1:=False -p enable_infra2:=False -p clip_distance:=2.5 --log-level info

• To make sure ros wrapper is linked to the correct libraries I've ran the following:

lsof -p $(pgrep -f /opt/ros/humble/lib/realsense2_camera/realsense2_camera_node) | grep libreal.*.so

lsof -p [processid] extracts list of libraries linked to the process
pgrep extracts the process id(s) of the executable
Results:

realsense 76448 zzzzz  mem       REG  259,2    28264  807521 /opt/ros/humble/lib/librealsense2_camera_msgs__rosidl_typesupport_introspection_cpp.so
realsense 76448 zzzzz  mem       REG  259,2    27392  807519 /opt/ros/humble/lib/librealsense2_camera_msgs__rosidl_typesupport_fastrtps_cpp.so
realsense 76448 zzzzz  mem       REG  259,2  6581504  807412 /opt/ros/humble/lib/aarch64-linux-gnu/librealsense2.so.2.55.1
realsense 76448 zzzzz  mem       REG  259,2  1151296  813087 /opt/ros/humble/lib/librealsense2_camera.so
realsense 76448 zzzzz  mem       REG  259,2    14344  807517 /opt/ros/humble/lib/librealsense2_camera_msgs__rosidl_typesupport_cpp.so

Scenario2:

• I've installed librealsense2 v2.55.1 via the following documentation:
  https://github.com/IntelRealSense/librealsense/blob/v2.55.1/doc/distribution_linux.md
  I've not uninstalled ros modules installed on Scenario1
  Note: librealsense2-dkms is not available for aarch64-Ubuntu 22.04 so I skipped the installation
  I've installed the following optional packages

sudo apt-get install librealsense2-utils librealsense2-dev librealsense2-gl-dev

Note2: I've not patched the kernel

• I've checked out realsense-ros and compiled manually with gl enabled

git clone https://github.com/IntelRealSense/realsense-ros.git
cd realsense-ros
colcon build --cmake-args -DBUILD_ACCELERATE_GPU_WITH_GLSL=1
#overlay this build on top of humble release
source $(pwd)/install/setup.bash

• I've launched the camera node with the same config on Scenario1

ros2 run realsense2_camera realsense2_camera_node --ros-args -r __ns:=/ -r __node:=camera_d435_front -p initial_reset:=True -p accelerate_gpu_with_glsl:=False -p pointcloud.enable:=True -p rgb_camera.color_profile:=848x480x30 -p depth_module.infra_profile:=848x480x30 -p depth_module.depth_profile:=848x480x30 -p camera_name:=camera_d435_front -p enable_color:=False -p enable_depth:=True -p enable_infra1:=False -p enable_infra2:=False -p clip_distance:=2.5 --log-level info

• To make sure ros wrapper is linked to the correct libraries I've ran the following:

lsof -p $(pgrep -f $(pwd)/install/realsense2_camera/lib/realsense2_camera/realsense2_camera_node) | grep libreal.*.s

Output:

realsense 80059 zzzzz  mem       REG              259,2     79448  138474 /home/zzzzz/realsense-ros/install/realsense2_camera_msgs/lib/librealsense2_camera_msgs__rosidl_typesupport_introspection_cpp.so
realsense 80059 zzzzz  mem       REG              259,2     60784  138463 /home/zzzzz/realsense-ros/install/realsense2_camera_msgs/lib/librealsense2_camera_msgs__rosidl_typesupport_fastrtps_cpp.so
realsense 80059 zzzzz  mem       REG              259,2   6581504  807412 /opt/ros/humble/lib/aarch64-linux-gnu/librealsense2.so.2.55.1
realsense 80059 zzzzz  mem       REG              259,2   1450648 3030330 /usr/lib/aarch64-linux-gnu/librealsense2-gl.so.2.55.1
realsense 80059 zzzzz  mem       REG              259,2   2318512  138826 /home/zzzzz/realsense-ros/install/realsense2_camera/lib/librealsense2_camera.so
realsense 80059 zzzzz  mem       REG              259,2     24904  138475 /home/zzzzz/realsense-ros/install/realsense2_camera_msgs/lib/librealsense2_camera_msgs__rosidl_typesupport_cpp.so

Note the following:
now -gl version is linked against the library built by librealsense,
but non-gl library is linked against the library built by ros

/opt/ros/humble/lib/aarch64-linux-gnu/librealsense2.so.2.55.1
/usr/lib/aarch64-linux-gnu/librealsense2-gl.so.2.55.1

When I run the node with accelerate_gpu_with_glsl:=True point cloud is not generated. Otherwise it is generated without problems because ros version of librealsense2 is running as expected.

Scenario 3:

• I've uninstalled ros-humble-realsense packages

sudo apt-get remove ros-humble-realsense2-camera ros-humble-realsense2-camera-msgs ros-humble-realsense2-description

At this point the following should return empty results:

ls /opt/ros/humble/lib/aarch64-linux-gnu/librealsense*
ls /opt/ros/humble/lib/librealsense*

• I've launched the camera node with the same config on Scenario1

ros2 run realsense2_camera realsense2_camera_node --ros-args -r __ns:=/ -r __node:=camera_d435_front -p initial_reset:=True -p accelerate_gpu_with_glsl:=False -p pointcloud.enable:=True -p rgb_camera.color_profile:=848x480x30 -p depth_module.infra_profile:=848x480x30 -p depth_module.depth_profile:=848x480x30 -p camera_name:=camera_d435_front -p enable_color:=False -p enable_depth:=True -p enable_infra1:=False -p enable_infra2:=False -p clip_distance:=2.5 --log-level info

IMPORTANT: Now we don't get point cloud data in all cases. (So there must be a problem with the libraries built by librealsense, ros version is working, lets deep dive in Scenario4)

• To make sure ros wrapper is linked to the correct libraries I've ran the following:

lsof -p $(pgrep -f $(pwd)/install/realsense2_camera/lib/realsense2_camera/realsense2_camera_node) | grep libreal.*.s

Output:

realsense 80962 zzzzz  mem       REG              259,2     79448  138474 /home/zzzzz/realsense-ros/install/realsense2_camera_msgs/lib/librealsense2_camera_msgs__rosidl_typesupport_introspection_cpp.so
realsense 80962 zzzzz  mem       REG              259,2     60784  138463 /home/zzzzz/realsense-ros/install/realsense2_camera_msgs/lib/librealsense2_camera_msgs__rosidl_typesupport_fastrtps_cpp.so
realsense 80962 zzzzz  mem       REG              259,2   7855232 3015185 /usr/lib/aarch64-linux-gnu/librealsense2.so.2.55.1
realsense 80962 zzzzz  mem       REG              259,2   1450648 3030330 /usr/lib/aarch64-linux-gnu/librealsense2-gl.so.2.55.1
realsense 80962 zzzzz  mem       REG              259,2   2318512  138826 /home/zzzzz/realsense-ros/install/realsense2_camera/lib/librealsense2_camera.so
realsense 80962 zzzzz  mem       REG              259,2     24904  138475 /home/zzzzz/realsense-ros/install/realsense2_camera_msgs/lib/librealsense2_camera_msgs__rosidl_typesupport_cpp.so

As expected, this time, both libraries are linked correctly and against the ones built by librealsense

/usr/lib/aarch64-linux-gnu/librealsense2.so.2.55.1
/usr/lib/aarch64-linux-gnu/librealsense2-gl.so.2.55.1

Scenario4:

• Lets uninstall librealsense and compile manually (following the instructions @ https://github.com/IntelRealSense/librealsense/blob/v2.55.1/doc/installation.md)

sudo apt-get remove librealsense2-dev librealsense2-gl-dev librealsense2 librealsense2-gl librealsense2-utils librealsense2-udev-rules
#Dependencies are already installed, skipping...
cd ~
git clone https://github.com/IntelRealSense/librealsense.git
cd librealsense
#Switch to v2.55.1 tag
git checkout v2.55.1
#Install USB udev rules
sudo ./scripts/setup_udev_rules.sh
#Skipping kernel patch we'll be using FORCE_RSUSB_BACKEND
mkdir build
cd build
cmake ../ -DBUILD_EXAMPLES=false -DBUILD_GRAPHICAL_EXAMPLES=false -DBUILD_GLSL_EXTENSIONS=ON -DFORCE_RSUSB_BACKEND=true -DCMAKE_BUILD_TYPE=Release
sudo make uninstall
make clean
make
sudo make install

• Above process unfortunately does not build librealsense2-gl.so
  At a glance, it seemed like src/gl/CMakeLists.txt is not included from anywhere, that's the point I get stuck.

• I've rebuilt realsense-ros without GL capabilities to see if point cloud can be generated on CPU

cd ~/realsense-ros
colcon clean workspace -y
colcon build
#overlay this build on top of humble release
source $(pwd)/install/setup.bash

• I've launched the camera node with the same config on Scenario1

ros2 run realsense2_camera realsense2_camera_node --ros-args -r __ns:=/ -r __node:=camera_d435_front -p initial_reset:=True -p accelerate_gpu_with_glsl:=False -p pointcloud.enable:=True -p rgb_camera.color_profile:=848x480x30 -p depth_module.infra_profile:=848x480x30 -p depth_module.depth_profile:=848x480x30 -p camera_name:=camera_d435_front -p enable_color:=False -p enable_depth:=True -p enable_infra1:=False -p enable_infra2:=False -p clip_distance:=2.5 --log-level info

Point cloud generation works, however on the ros logs I see the following warning: (ros/realsense builds do not print this warning)

WARNING  (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Success, number: 0
[INFO] [1728932468.838272790] [camera_d435_front]: RealSense Node Is Up!
WARNING (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Success, number: 0

[vahapt]

Update:
I've managed to compile librealsense2-gl.so with the following cmake options:

cmake ../ -DBUILD_EXAMPLES=true -DBUILD_GRAPHICAL_EXAMPLES=true -DBUILD_GLSL_EXTENSIONS=ON -DFORCE_RSUSB_BACKEND=true -DCMAKE_BUILD_TYPE=Release -DBUILD_TOOLS=true

We're back to square 1. Point cloud can now be generated with CPU but not with GPU.
On another note, (I'm not sure if it's directly related to this case) example app rs-gl works with both CPU and GPU and effectively reduces CPU load when GPU is enabled.