Serverless GPU Inference Cost Comparison: Roboflow, GCP, AWS, Azure

In this blog post, we will explore how different cloud providers compare in running custom vision model inference.

Custom RF-DETR Model

If you need OCR, face detection/comparison, or general-purpose COCO-20 classification and detection, the major clouds already offer plenty of managed options, such as AWS Rekognition, GCP Vision, and Azure AI Vision.

Here, we will be looking at serving a state-of-the-art architecture object detection model, RF-DETR (opens in a new tab), fine-tuned on a custom dataset and optimized for GPU inference.

We'll be looking specifically at the RF-DETR XL version, which achieves a score of 77.4 on COCO mAP@50 (opens in a new tab). With TRT compilation at FP16 with batch_size=1 it runs at about 12ms on T4 GPU.

Serverless Inference

We're also specifically looking at inference deployment modes that don't require an always-on GPU. We'll be looking at 3 different use-cases:

• Continuous inference at 1 inference per 10 seconds (360 req/h)
• Continuous inference at 1 inference per 30 seconds (120 req/h)
• Burst mode, 100 requests per 30 minutes (200 req/h)

Cost comparison

Roboflow Serverless Cloud API

Roboflow's Serverless Cloud API (opens in a new tab) excels at low-frequency and sporadic AI workloads. Because of the shared infrastructure and containers, we don't scale to zero and we don't charge you for cold start time (loading your model into the GPU).

We'll take into account that as of October 2026, RF-DETR XL is 0.25 credits / 1000 images (opens in a new tab), and 1 credit is $4 for most accounts

• Continuous 1 req/10sec, or 360/h: 0.09 credits/h, or $0.36/h
• Continuous 1 req/30sec, or 120/h: 0.03 credits/h, or $0.12/h
• Burst mode 100 req/30min, or 200/h: 0.05 credits/h, or $0.20/h

The actual inference time of RF-DETR XL is about 30ms, but can be longer with cold starts when requests are less frequent. Unlike the other providers, your cost for Roboflow Cloud API depends on the number of requests you send, not GPU time.

Note: Above 250k requests / month volume discounts kick in. While none of these scenarios would trigger the discounted rates, for higher volumes you can calculate your savings here (opens in a new tab).

Update: As of October 2026, Roboflow Serverless Cloud API uses per image pricing instead of GPU time. The comparison has been updated to reflect these changes.

GCP Cloud Run

Google recommends using Cloud Run for such AI workloads. As it only offers L4 and RTX Pro 6000, we'll be using L4 prices for calculations. Minimum resources alongside an L4 are 4 vCPU cores and 16GB RAM. Total cost per second would be:

• L4 GPU: $0.0001867 / sec
• 16 GB RAM: $0.000032 / sec
• 4 vCPU: $0.000072/sec
• Total: $1.045 / hour

While GCP advertises a 5-second startup time, GPU is only available with instance-based billing (not possible with request-based billing), which means that even if you send a single inference request that completes in 100 ms, you’d pay for up to 15 minutes of inference time, before the instance spins back down (with scale to zero).

For both inference frequencies, we'd need to have always-on GPU instance, as it wouldn't be able to scale to zero. So for both frequencies that'd be $1.05/h.

Burst 100 req/30min: Since L4 is about 3x faster than T4 we'd need about 400 ms of inference time to process 100 images. We'd need an additional 5 sec for cold start, and around 600 seconds to scale to 0. As we're processing bursts every 30min, that ends up to 1211 sec of inference compute every hour, so about $0.35/h.

AWS SageMaker

AWS recommends using AWS SageMaker's Asynchronous Inference for such AI workloads. They also offer Serverless Inference, but that only offers CPU compute, so it wouldn't work for our use case. They offer ml.g4dn.xlarge (T4 with 16 GB RAM, 4 vCPU and 16 GB RAM) at $0.736/hour.

AWS mentions that provisioning takes several minutes, and Scale-to-Zero takes several minutes.

For both inference frequencies, we'd need to have an always-on GPU instance, as it wouldn't be able to scale to zero. So for both frequencies that'd be $0.74/h.

Burst 100 req/30min: T4 would need about 1200 ms of inference time to process 100 images. We'd need an additional 180 sec for cold start, and around 300 seconds to scale to 0. As we're processing bursts every 30min, that ends up to ~962 sec of inference compute every hour, so about $0.198/h.

Azure Serverless GPU

Azure recommends using Serverless GPU in Azure Container Apps for vision AI workloads. They offer either T4 or A100 GPUs, and for a comparable machine that'd cost:

• T4 GPU: $0.00009 / sec
• 16 GB RAM: $0.000048 / sec
• 4 vCPU: $0.000096 / sec active
• Total: between $0.55 - $0.84 / hour (depending on vCPU usage)

No mentions about cold start, but there's a technical guide (opens in a new tab) needing 8-10 minutes for cold start. Let's say I'd take about 3 minutes for a cold start, and they mention 300 seconds for Scale-to-Zero in their documentation.

For both inference frequencies, we'd need to have an always-on GPU instance, as it wouldn't be able to scale to zero. As CPU would be mostly inactive, that'd be $0.55/h.

Burst 100 req/30min: T4 would need about 1200 ms of inference time to process 100 images. About 962 sec of GPU compute per hour, so about $0.14/h.

Conclusion

Choosing the right cloud provider for vision inference depends on your traffic patterns, your tolerance for cold starts, and your ability to manage Cloud compute. As our comparison shows the billing mechanics behind each provider can lead to vastly different monthly invoices, depending on your workloads.

If you want to avoid the headache of managing GPU idle times and instance spin-downs for custom models like RF-DETR, Roboflow provides a frictionless and cost-effective experience.