NVIDIA Omniverse Isaac Sim - RoboLab

This project provides a collection of features for moving robots and acquiring sensor data in various simulated scenes with NVIDIA Omniverse Isaac Sim (Version 4.2) and ROS 2 (Humble) in a docker container. Furthermore, the project seeks to create a connection between simulated and real environments.

Table of Contents

• NVIDIA Omniverse Isaac Sim - RoboLab
  • Table of Contents
  • Quick start
  • Setup
  • Usage
  • Configuration
  • RoboLab Demo

Quick start

1. NVIDIA Container Setup
2. Check out this repo:

git clone [email protected]:IRAS-HKA/RobotLab.git

3. Download assets.
4. Extract assets to RobotLab/data/:

unzip assets_hka_release.zip -d RobotLab/data/

5. Build and start docker container:

cd RobotLab
./start_docker.sh

6. Start RobotLab Demo in the container (takes some time to load everything):

./python.sh scripts/demos/robolab/robolab_demo.py

7. Download Isaac Sim WebRTC Streaming Client latest release
8. Start Isaac Sim WebRTC Streaming Client and connect to 127.0.0.1. (AppImage requires FUSE version 2 to run)

Setup

From the Omniverse Container Installation Documentation make sure that all the requirements under Container Setup are met. Ignore the Container Deployment steps.

Next, download the Isaac Sim WebRTC Streaming Client from the Latest Release section. To connect to a running Isaac Sim simulation application window later on, you will need to run the Isaac Sim WebRTC Streaming Client and enter the IP address of the machine running Isaac Sim. If there is a blank screen, e.g. when restarting the simulation application, you can reconnect by clicking Reload in the View menu or pressing F5.

Lastly clone this repo.

Usage

From this repo build and run the container. If you need a specific ROS_DOMAIN_ID you can change it with the DOMAIN_ID argument in the docker build command of the start_docker.sh file before executing it:

./start_docker.sh

In the container you can run Isaac Sim with WebRTC livestream mode:

./runheadless.webrtc.sh

Alternatively you can run your own Python scripts:

./python.sh scripts/my_script.py

For a first impression on how to use Isaac Sim and this repo for your own scripts, you can have a look at scripts/run_scene.py.

If you need other PIP packages for your scripts you can install them with

./python.sh -m pip install name_of_package_here

Any assets that might be needed should be stored in an assets folder inside the data folder. They can also be stored at a different path, but the new path might need to be specified when creating a new scene with the data_root parameter of the Scene class in scripts/isaac/scene.py.

The data folder also contains a config_files subfolder. It contains a dimensions.json, and simulation_app_config.json file. More information about these files can be found in the Configuration chapter.

Inside the docker container, the data folder can be found at /isaac-sim/project/data/.

For further information and help look at the Official Isaac Sim Documentation, Omniverse Isaac Forums or join the Omniverse Discord Channel. There are also Self-Paced Online Courses, Weekly Livestreams, and Further Omniverse-Related Events.

Configuration

There are various parameters for running the Isaac Sim Simulation Application. For this project, these parameters can be configured in the data/configfiles/simulation_app_config.json file. The parameters are then used at Scene initialization to create a SimulationApp object (see scripts/isaac/scene.py). The parameter description and default values can be read in the Simulation Application [omni.isaac.kit] Documentation.

In the data/configfiles/simulation_app_config.json file

• width (1280),
• height (720),
• window_width (1920),
• window_height (1080),
• headless (true), and
• renderer ("RaytracedLighting")

are currently specified.

When using the Scene object for starting the simulation application, the display_options parameter can be specified. If it is None, the data/configfiles/simulation_app_config.json file will be checked for the display_options parameter. If it is not defined in the file, the default return value of the get_display_options_code function in the scripts/utils.py script will be used instead. This default options code results in the world coordinate system, FPS, Device Memory, and Host Memory being shown in the viewport, and selected objects being outlined. If you do not know the code for your desired display options, you can use the same function to get it and set the Scene classes display_options parameter.

RoboLab Demo

The RoboLab Demo can be started with

./python.sh scripts/demos/robolab/robolab_demo.py

It loads the RoboLab asset, adds a static camera and a camera mounted near the end effector of the UR10e robot. It also starts a ROS 2 node for listening to the /joint_states topic and adjusting the UR10e robot's joints to any published JointState messages.