Roboflow Serverless vs. Self-Hosted Inference: What a 90-Second Traffic Spike Does to One GPU Server

When deploying a computer vision workflow in production, you need underlying infrastructure that can guarantee consistent and reliable performance even when the request rate spikes. In this article, we’ll compare how two options, serverless and self-hosted GPUs, perform.

If you run real-time vision inference on your own GPUs, you size the deployment for the request rate you expect. When conducting a benchmarking exercise below, one GPU server handled 5 requests a second without any trouble. When I sent it 20 per second for 90 seconds, its median latency climbed to the 20-second timeout, and by the end of the three-minute run it wasn't returning any responses, even with traffic back down to 5 requests per second. Roboflow Serverless got the exact same traffic at the exact same moment and kept responding to inference requests during the whole process.

Capacity planning is stricter for GPU inference than for general compute. When CPU load rises, autoscaling can usually add instances in time. But on-demand GPUs are often unavailable, and a new inference server can take many minutes to start, so a real-time GPU workload has to be sized right ahead of time. On the other hand, a serverless system takes that capacity planning off your hands. Below I walk through the test setup, the recorded run, and what's behind the serverless side.

You can watch a recording of the load test here (opens in a new tab).

A request that makes the GPU work

When benchmarking vision infrastructure, running a toy model on a small image wouldn't tell you much about load. So I conducted a test using something heavier: counting vehicles in 1280×1280 aerial frames of a parking lot.

Run the vehicle detector once on the whole frame and it finds two. The model takes a 512×512 input, so the frame gets squeezed down and most cars end up a few pixels wide. This results in poor vehicle detection model performance.

The Workflow gets around that with SAHI (opens in a new tab). It cuts the frame into 16 overlapping 427×427 tiles, runs the detector on every tile, merges the boxes where tiles overlap and counts what's left. On this frame that's 102 vehicles, even exceeding the 100 a person labeled.

That's a much better answer, and a much heavier request. The client still makes one API call to the Workflow, but behind it sit 16 model inferences plus the stitching. That's the kind of work I wanted to throw at both sides.

Same Workflow, same traffic, one URL apart

One side was Roboflow’s hosted inference product: serverless.roboflow.com. The other was a single cloud VM with an NVIDIA L40S (the same GPU model serverless runs on) running our open-source inference server (opens in a new tab), version 1.7.4 with default settings. Both used the same Locust (opens in a new tab) file with the same load test configuration, and the only thing I changed between them was the host URL. Locust ran on a Google Cloud VM rather than my laptop, which keeps local Wi-Fi out of the numbers.

Each simulated user acts like a camera. It sends a frame every 4 seconds and doesn't wait for the last answer before sending the next one, because a real camera doesn't slow down when your server does.

The run starts with a minute at 20 users, which works out to 5 requests a second. Then comes a 90-second burst to 80 users, or 20 requests a second. At 16 model calls per request, that's 320 model calls a second.

After the burst, it drops back to 20 users for 30 seconds. Any answer slower than 20 seconds counts as a failure, because by then the client has given up.

What the recorded run looked like

For the first minute the two lines sit right on top of each other. Both sides handle 5 requests a second at a median of about a second. If you'd sized one server for this traffic, you'd feel good about it.

Then the burst hits. The self-hosted server's successful requests flatten out around 13 to 15 a second while 20 a second keep arriving. Its median latency starts climbing and keeps climbing, in a fairly straight line, for the next minute.

To me that ramp looks like a backlog building up, with every new frame waiting behind the ones already in line. With 20 requests arriving and about 14 finishing each second, the line would grow by about 6 requests every second. I didn't measure the queue directly, so treat that as my reading of the chart. Nothing's wrong with the server, though: it's one GPU with a fixed amount of throughput, and the burst asked for more than that.

About a minute into the burst, the self-hosted median hits the 20-second timeout. Shortly after, its successful requests drop to zero. They stay at zero after the burst is over, even with traffic back down to 5 a second.

Locust's own charts fill in the rest. Failures on the self-hosted side start late in the burst and peak after the user count has dropped back to 20, at a rate well above the 5 requests a second still being sent. So most of those failures are frames from the burst, still timing out after it ended.

Serverless had its own wobble. Its median jumped to about 3 seconds early in the burst, then settled around 1.3 seconds while it answered roughly 16 to 24 requests a second. That's in line with how serverless scales: it brings more capacity online within seconds of a spike like this. Once the burst ended, it dropped to about half a second.

Over all three minutes, serverless returned 2,273 successful requests at a median latency of 1.4 seconds. The self-hosted server returned 1,095. Its whole-run median was 20 seconds, which is just the timeout, and Locust marked 50% of its requests as failures against 0% (rounded) for serverless.

Two ways to fix the self-hosted side

The first is to provision for the peak. Run enough GPUs to absorb the biggest burst you expect, plus some headroom. Most of that sits idle between bursts, and you pay for it anyway.

The second is to autoscale. When I set up this demo, getting the GPU VM booted, the inference container pulled and the server answering took about five minutes. The burst lasted 90 seconds, so even an autoscaler that reacted to the instant traffic jump would have delivered its new GPU long after the burst was over.

More cameras aren't the only thing that causes a spike. Maybe a new production line comes online, or someone kicks off a backfill of last night's footage. Or someone changes the Workflow to cut each frame into more tiles, which adds GPU work to every request without changing the request rate at all.

Self-hosting is a reasonable call when your traffic is steady and predictable. With bursty traffic it gets a lot harder to size.

What's behind the serverless side

Serverless runs the same open-source inference server you can run yourself. What's different is everything around it.

Requests go into a queue per model, and GPU workers pull from those queues. Workers prefer models they already have loaded in VRAM, because a warm model answers in about 5 ms and a cold load takes around 30 seconds.

The fleet scales on the total backlog across all models, and it keeps a warm floor of spare capacity so a burst has somewhere to land before any new GPU boots. When inference needs more GPUs, our own training jobs step aside. You pay per image, so that spare capacity isn't something you're paying to keep idle.

If you want the full engineering details, they're in Roboflow Serverless Inference: A Thousand Models on a Shared GPU Fleet.

If you run your own GPUs, size them for the day your traffic does something you didn't plan for. If you'd rather not plan for that day, the same Workflow runs on Roboflow Serverless with nothing changed but the URL.