Roboflow + NVIDIA TRT
Speed up Roboflow models on NVIDIA GPUs and Jetson devices by turning on TensorRT in the Roboflow Inference server.
How to connect NVIDIA TRT to Roboflow
5 stepsInstall the NVIDIA Container Toolkit
On an x86 machine with an NVIDIA GPU, follow NVIDIA's Container Toolkit guide so Docker can use the GPU. Jetson devices with JetPack already have the NVIDIA container runtime.
Start the GPU or Jetson container
Run roboflow/roboflow-inference-server-gpu with `--gpus all`, or the Jetson image that matches your JetPack version (for example roboflow-inference-server-jetson-6.2.0).
Turn on TensorRT
Add `-e ONNXRUNTIME_EXECUTION_PROVIDERS="[TensorrtExecutionProvider,CUDAExecutionProvider,CPUExecutionProvider]"` to the docker run command, or the same variable in Docker Compose.
Mount a cache volume
Map a host folder to the container cache (the docs use ~/.inference/cache:/tmp) so compiled TensorRT engines persist and each model compiles only once.
Warm up each model
Send a first request to each model and expect it to take several minutes while TensorRT compiles it. Later requests use the cached engine.
What teams build with NVIDIA TRT
Get more frames per second from the same GPU
Use TensorRT to run video models faster on NVIDIA hardware you already own.
Run real-time models on Jetson
Enable TensorRT in the JetPack container to accelerate models on an edge device next to the camera.
Questions
No. Roboflow's current docs enable TensorRT in the standard GPU and Jetson containers with the ONNXRUNTIME_EXECUTION_PROVIDERS environment variable.
Prebuilt TensorRT model packages for private models require an Enterprise plan when you run Inference outside the Roboflow platform. Public models include TensorRT packages on all plans.
TensorRT compiles and optimizes each model the first time it loads, which can take 15 minutes or more. A persistent cache volume keeps the result so you pay that cost once per model.
Yes. The Jetson containers accept the same environment variable to turn on TensorRT.
Why are companies choosing Roboflow?
The end-to-end platform for vision AI: create datasets, train models and deploy to production.
Flexible deployment modes
Move from a scalable cloud API to edge hardware like NVIDIA Jetson without managing complex dependencies.
Integrated workflow builder
Combine multiple AI models, custom business logic and third-party integrations into one production-ready pipeline in a low-code editor.
Model monitoring and alerts
Dashboards track confidence, inference time and request volume for your deployed models, and email alerts tell your team when confidence suddenly drops.
