Slow inference time

[programath]

Hi, I am testing the latency of the model RFDETRBase at different platform. (Quadro GV100 and A800), the latency of inference time are 0.25s and 0.14s. which is much slower then the reported speed. Is that normal speed of rf-detr?

the timer is calcuated as follows:

image = Image.open("/vepfs-sf/wtp2/R-Human/images_resized/video3_frame_00068_204.00s.jpg")
time1 = time.time()
for _ in range(10):
    detections = model.predict(image, threshold=0.5)
print("Inference time:", (time.time() - time1) / 10)

[picjul]

Hi!

Times in README are calculated inferencing with TensorRT at FP16 precision. As I now, currently TensorRT is not supported as engine in model.predict() function.

Probably the times that you are measuring are based on PyTorch model in FP32 precision, so it's ok that they are higher.

Am I right?

[programath]

One more thing, how to use TensorRT to accelerate the inference of RFDETR? is there any tutorial?

[SkalskiP]

Hi @programath 👋🏻, unfortunately, we don't have a tutorial for this. Do you think it would be helpful?

[DatSplit]

Good afternoon @SkalskiP and @programath,

I've converted a custom RT-DETR-B and RT-DETR-L model to TensorRT for deployment on a Jetson Orin Nano 8GB.
I used the trtexec function in rfdetr/deploy/export.py to retrieve the command I should use to convert the ONNX model to TensorRT.

Would it be beneficial if I wrote a tutorial for this?
If so, what details must I at least include? @SkalskiP (reproducible code + explanation + environment details?)

[SkalskiP]

Hi @DatSplit! That would be amazing — really appreciate it! 🔥

Here’s what I’m thinking: we should start breaking out the documentation into separate .md files. Tutorials like this are super valuable, but I’d prefer not to put them directly into the README.md.

Could you open a PR that creates a docs/ directory at the root of the project, and add a new file called export.md where you walk through:

• Exporting to ONNX
• Converting to TensorRT
• Deploying on Jetson

Something like this for structure:

rf-detr /
├── docs /
│   └── export.md
├── README.md
...

Later, we’ll link to docs/export.md from the README.md to keep things clean and easy to navigate.

Let me know what you think!

[DatSplit]

Good evening @SkalskiP,

That sounds like a solid plan!
I'll be working on that documentation and will create a PR once I believe it is good enough.
Is there a "deadline" indication? (E.g., in one to two weeks)

[SkalskiP]

@DatSplit the sooner the better. It can be really rough. I can help you out with structuring it properly. I mostly care about the steps to follow.

[DatSplit]

Good afternoon @SkalskiP,

I have quite a busy weekend.
So, I'll most likely start working on it Sunday evening/Monday morning.
I expect the rough draft to be done Monday morning.

[mitzy-ByteMe]

Hi @DatSplit , how has the real-time performance (in FPS) been for the base and large models on your Jetson so far?

[SkalskiP]

@DatSplit that sounds perfect! Thank you! 🙏🏻

[pzoltowski]

Did anyone also have such a poor inference speed on ONNX? On my macbook m2 max I'm getting ~300ms using Base and testing test.jpg (5 inferences with some warmup).

I'm using CoreML provider:
CoreMLExecutionProvider::GetCapability, number of partitions supported by CoreML: 90 number of nodes in the graph: 913 number of nodes supported by CoreML: 737.

It's slower than on CPUExecutionProvider (~200ms). With pytorch MPS is ok (~80ms) but still like 8x slower than official yolov9-m.

[DatSplit]

Good afternoon @mirza298,

Important

The custom models mentioned below have been trained on an resolution of 1120! (e.g., model = RFDETRBase(resolution=1120)). This significantly
reduces the FPS. (Might be a factor 3-4)

On the Jetson Orin Nano (8 GB developer kit) TensorRT optimized:
FPS with RF-DETR-B is 5.5 FPS and with RF-DETR-L it is 2.4 FPS.