Skip to content
This repository was archived by the owner on Feb 16, 2024. It is now read-only.

Benchmark for Robot Semantics Dataset

Jump to bottom
Zone Trooper edited this page Jan 18, 2021 · 4 revisions

We record and update the performance of experiments against RS-RGBD dataset here. And we offer downloads for those pretrained models.

To evaluate the performance of your model, first download the full RS-RGBD dataset.

Annotations

The annotations provided are in files of command.txt, where individual linguistic sentences describing the saliant actions along with the entities associated with the actions are provided. In specific, for pouring setting, the following forms of annotations might be available:

  • none
  • <manipulator> move empty
  • <manipulator> grasp <object> with <liquid>
  • <manipulator> hold <object> with <liquid>
  • <manipulator> release <object> with <liquid>
  • <manipulator> pour <liquid> from <object> to <object>

Additionally, two options are available in to control your labels:

  • Default: Use the default annotations as they are.

  • V1_COMMAND: The commands will be formatted the same way when the paper was submitted. Refer to line 49 of the generate_clips.py to see what's changed compared to the default annotations.

Quantitative Results

We update the latest scores for all the models experimented in the paper here, using the updated codebase. Scores for all pretrained models are listed below:

Model Backbone Annotation BLEU1 BLEU2 BLEU3 BLEU4 METEOR ROUGE_L CIDEr
attn-seq2seq-cat-256 ResNet50 V1 0.757 0.670 0.602 0.567 0.500 0.807 4.044
attn-seq2seq-256 ResNet50 V1 0.764 0.677 0.601 0.546 0.496 0.812 4.096
seq2seq-256 ResNet50 V1 0.773 0.685 0.626 0.591 0.481 0.786 3.977
EDNet-256 ResNet50 V1 0.761 0.666 0.599 0.559 0.469 0.784 3.897

Make sure to match model against the correct branches of our repos.

If you wish to submit your pretrained models and your own scores, feel free to open an issue to let me know.

Clone this wiki locally