feat: Run entire Manipulator demo inside docker

.dockerignore

@@ -180,3 +180,6 @@ src/rai_interfaces
 
 # Demo assets
 demo_assets/
+
+# Docker files
+Dockerfile*

docker/Dockerfile

@@ -17,6 +17,8 @@ FROM osrf/ros:${ROS_DISTRO}-desktop-full
 
 ARG DEPENDENCIES=core_only
 ENV DEBIAN_FRONTEND=noninteractive
+# Increase timeout for Poetry requests.
+ENV POETRY_REQUESTS_TIMEOUT=300
 
 # Check whether the $DEPENDENCIES ARG has a valid value
 RUN /bin/bash -c 'if [ "${DEPENDENCIES}" != "core_only" ] && [ "${DEPENDENCIES}" != "all_groups" ]; then \
@@ -56,3 +58,20 @@ RUN /bin/bash -c '. /opt/ros/${ROS_DISTRO}/setup.bash && \
 
 # Build the workspace
 RUN /bin/bash -c '. /opt/ros/${ROS_DISTRO}/setup.bash && colcon build --symlink-install'
+
+# Install libraries required for GUI and NVIDIA support
+RUN apt install -y \
+    libglu1-mesa-dev \
+    libxcb-xinerama0 \
+    libfontconfig1-dev \
+    libnvidia-gl-470 \
+    libxcb-xkb-dev \
+    libxkbcommon-x11-dev \
+    libxkbcommon-dev \
+    libxcb-xfixes0-dev \
+    libxcb-xinput-dev \
+    libxcb-xinput0 \
+    libpcre2-16-0 \
+    libunwind-dev
+ENV NVIDIA_VISIBLE_DEVICES=all
+ENV NVIDIA_DRIVER_CAPABILITIES=all

docs/demos/manipulation.md

@@ -25,23 +25,23 @@ manipulation techniques.
 1. Follow the RAI setup instructions in the [quick setup guide](../setup/install.md#setting-up-developer-environment).
 2. Download additional dependencies:
 
-    ```shell
-    poetry install --with openset
-    vcs import < demos.repos
-    rosdep install --from-paths src/examples/rai-manipulation-demo/ros2_ws/src --ignore-src -r -y
-    ```
+   ```shell
+   poetry install --with openset
+   vcs import < demos.repos
+   rosdep install --from-paths src/examples/rai-manipulation-demo/ros2_ws/src --ignore-src -r -y
+   ```
 
 3. Download the latest binary release
 
-    ```bash
-    ./scripts/download_demo.sh manipulation
-    ```
+   ```bash
+   ./scripts/download_demo.sh manipulation
+   ```
 
 4. Build the ROS 2 workspace:
 
-    ```bash
-    colcon build --symlink-install
-    ```
+   ```bash
+   colcon build --symlink-install
+   ```
 
 #### Running the demo
 
@@ -52,95 +52,56 @@ manipulation techniques.
 
 1. Start the demo
 
-    ```shell
-    ros2 launch examples/manipulation-demo.launch.py game_launcher:=demo_assets/manipulation/RAIManipulationDemo/RAIManipulationDemo.GameLauncher
-    ```
+   ```shell
+   ros2 launch examples/manipulation-demo.launch.py game_launcher:=demo_assets/manipulation/RAIManipulationDemo/RAIManipulationDemo.GameLauncher
+   ```
 
 2. In the second terminal, run the streamlit interface:
 
-    ```shell
-    streamlit run examples/manipulation-demo-streamlit.py
-    ```
+   ```shell
+   streamlit run examples/manipulation-demo-streamlit.py
+   ```
 
-    Alternatively, you can run the simpler command-line version, which also serves as an example of
-    how to use the RAI API for you own applications:
+   Alternatively, you can run the simpler command-line version, which also serves as an example of
+   how to use the RAI API for you own applications:
 
-    ```shell
-    python examples/manipulation-demo.py
-    ```
+   ```shell
+   python examples/manipulation-demo.py
+   ```
 
 3. Interact with the robot arm using natural language commands. For example:
 
-    ```
-    Enter a prompt: Pick up the red cube and drop it on another cube
-    ```
+   ```
+   Enter a prompt: Pick up the red cube and drop it on another cube
+   ```
 
 !!! tip "Changing camera view"
 
     To change camera in the simulation use 1-7 keys on your keyboard once it's window is focused.
 
 ### Docker Setup
 
-!!! note "ROS 2 required"
-
-    The docker setup requires a working Humble or Jazzy ROS 2 installation on the host machine. Make sure that ROS 2 is sourced on the host machine and the `ROS_DOMAIN_ID` environment variable is set to the same value as in the [Docker setup](../setup/setup_docker.md#2-set-up-communications-between-docker-and-host-optional)
-
-!!! warning "ROS 2 distributions"
-
-    It is highly recommended that ROS 2 distribution on the host machine matches the ROS 2 distribution of the docker container. A distribution version mismatch may result in the demo not working correctly.
-
 #### 1. Setting up the demo
 
 1.  Set up docker as outlined in the [docker setup guide](../setup/setup_docker.md). During the setup, build the docker image with all dependencies (i.e., use the `--build-arg DEPENDENCIES=all_groups` argument)
-    and configure communication between the container and the host ([link](../setup/setup_docker.md#2-set-up-communications-between-docker-and-host-optional)).
-
-2.  On the host machine, download the latest binary release for the Robotic Arm Demo:
-
-    ```shell
-    ./scripts/download_demo.sh manipulation
-    ```
 
-3.  Run the docker container (if not already running):
+2.  Enable X11 access for the docker container:
 
     ```shell
-    docker run --net=host --ipc=host --pid=host -e ROS_DOMAIN_ID=$ROS_DOMAIN_ID -it rai:jazzy # or rai:humble
+    xhost +local:root
     ```
 
-    !!! tip "NVIDIA GPU acceleration"
-
-        If the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) is set up on your host machine, you can use the GPU within the RAI docker container for faster inference by adding the `--gpus all` option:
-
-        ```shell
-        docker run --net=host --ipc=host --pid=host -e ROS_DOMAIN_ID=$ROS_DOMAIN_ID --gpus all -it rai:jazzy # or rai:humble
-        ```
-
-        Sometimes, passing GPUs to the docker container may result in an error:
-
-        ```shell
-        docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].
-        ```
-
-        Restarting the docker service should resolve this error:
-
-        ```shell
-        sudo systemctl restart docker
-        ```
-
-4.  (Inside the container shell) Download additional ROS 2 dependencies:
+3.  Run the docker container with the following command:
 
     ```shell
-    vcs import < demos.repos
-    rosdep install --from-paths src/examples/rai-manipulation-demo/ros2_ws/src --ignore-src -r -y
+    docker run --net=host --ipc=host --pid=host -e ROS_DOMAIN_ID=$ROS_DOMAIN_ID -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix --gpus all -it rai:jazzy # or rai:humble
     ```
 
-5.  (Inside the container shell) Build the ROS 2 workspace:
+    !!! tip "NVIDIA Container Toolkit"
 
-    ```shell
-    source /opt/ros/${ROS_DISTRO}/setup.bash
-    colcon build --symlink-install
-    ```
+        In order to use the `--gpus all` flag, the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) must be installed on the host machine.
 
-6.  (Inside the docker container) By default, RAI uses OpenAI as the vendor. Thus, it is necessary
+4.  (Inside the docker container) By default, RAI uses OpenAI as the vendor. Thus, it is necessary
     to set the `$OPENAI_API_KEY` environmental variable. The command below may be utilized to set
     the variable and add it to the container's `.bashrc` file:
 
@@ -153,35 +114,15 @@ manipulation techniques.
 
         The default vendor can be changed to a different provider via the [RAI configuration tool](../setup/install.md#15-configure-rai)
 
-#### 2. Running the demo
-
-!!! note Source the setup shell
-
-    Ensure ROS 2 is sourced on the host machine and the `ROS_DOMAIN_ID` environment variable is set to the same value as in the [Docker setup](../setup/setup_docker.md#2-set-up-communications-between-docker-and-host-optional). Ensure that every command inside the docker container is run in a sourced shell using `source setup_shell.sh`.
-
-1. Launch the Robotic Arm Visualization on the host machine:
-
-    ```shell
-    ./demo_assets/manipulation/RAIManipulationDemo/RAIManipulationDemo.GameLauncher
-    ```
+5.  After this, follow the steps in the [Local Setup](#local-setup) from step 2 onwards.
 
-2. (Inside the container shell) Launch the Robotic Arm Demo script inside of the docker container:
+    !!! tip "New terminal in docker"
 
-    ```shell
-    ros2 launch examples/manipulation-demo.launch.py
-    ```
+        In order to open a new terminal in the same docker container, you can use the following command:
 
-3. (Inside the container shell) Open a new terminal for the docker container (e.g., `docker exec -it CONTAINER_ID /bin/bash`) and launch the streamlit interface:
-
-    ```shell
-    streamlit run examples/manipulation-demo-streamlit.py
-    ```
-
-    Alternatively, run the simpler command-line version:
-
-    ```shell
-    python examples/manipulation-demo.py
-    ```
+        ```shell
+        docker exec -it <container_id> bash
+        ```
 
 ## How it works
 
@@ -199,8 +140,8 @@ examples/manipulation-demo.py
 
 ## Known Limitations
 
--   `Grounding DINO` can't distinguish colors.
--   VLMs tend to struggle with spatial understanding (for example left/right concepts).
+- `Grounding DINO` can't distinguish colors.
+- VLMs tend to struggle with spatial understanding (for example left/right concepts).
 
 !!! tip "Building from source"