How to Run Computer Vision Regression Tests with Roboflow

A model update can make a pill-counting application miss objects without causing a single execution error. Changing a confidence threshold or filtering step can have the same effect. The application still returns a result, but the count your system depends on is now wrong.

Computer vision regression testing helps catch these failures before deployment. You run an updated application against a fixed set of test images with reviewed answers, check its outputs, and compare the results with a baseline. You can then see where a change broke behavior that previously worked.

In this guide, we’ll use Roboflow Workflow Evals (opens in a new tab) to test a Workflow that counts five types of pills. The baseline passes all 85 count checks across 17 images. After swapping one model block, that drops to 35 of 85 checks, even though the Workflow still runs successfully.

We’ll walk through creating the test cases, defining correct outputs, comparing versions, and setting acceptance criteria for your next release.

What Is Computer Vision Regression Testing?

Computer vision regression testing checks whether a change to a vision model or application causes previously working behavior to fail. It involves rerunning a fixed set of images or videos, comparing the outputs with reviewed ground truth, and checking the results against an established baseline.

Each test needs an input, an expected result, and a rule for deciding whether the output is acceptable. For a pill-counting application, the input is an image, the expected result is the correct count for each pill type, and the rule might require every count to match exactly.

Run these tests after changes that could affect predictions or application logic, including model retraining, confidence threshold adjustments, preprocessing changes, and edits to filtering or counting steps. Compare versions on the same cases to find examples that newly fail.

This differs from screenshot-based visual regression testing, which checks changes to a rendered user interface. Computer vision regression testing checks the predictions and decisions an application makes from visual inputs.

Computer Vision Regression Testing vs. Model Evaluation

Model evaluation (opens in a new tab) measures prediction quality using metrics such as precision, recall, and mean average precision (mAP). Repeating those measurements across model versions on the same evaluation data is one form of regression testing.

Application-level regression testing extends that process to the outputs your system uses. A pill counter depends on the detector, confidence thresholds, class filters, and counting logic working together. When you evaluate the final counts, you test all of these steps together.

Roboflow Workflow Evals lets you check those outputs against expected answers. In this tutorial, each pill count must match its ground truth exactly. A detector may find most pills in an image, but one missed or extra pill is enough to fail the corresponding count Check.

Use model metrics to investigate prediction quality and Workflow Evals to verify that the complete application meets its output requirements.

What Roboflow Workflow Evals Checks

An Eval is a fixed test for a Workflow: a set of test images with known answers, plus the rules for judging what the Workflow returns. Four terms come up throughout this tutorial, so here is what each one means.

Each Run records its inputs, ground truth, Workflow definitions, and settings. Editing a Case later only affects future Runs, so earlier results stay as they were.

When to run it

Run the Eval any time something in the Workflow changes:

• A model swap or a retrain
• A confidence threshold change
• A new or reordered block

Because the test images and answers stay fixed, any difference between Runs comes from your change.

Build the Baseline Vision AI Workflow

This is the Workflow we will test, called Pill Count Per Class. It takes one image and returns five numbers, one for each pill type.

The image goes to a single object detection model. Its detections then split into five branches, one per class, from pill_01 to pill_05. Each branch filters the detections down to its class, and a Property Definition block turns the result into a count. Those five counts are the Workflow’s only outputs.

I set up the Workflow with the Eval in mind: every count is its own output, so each one can get its own Check, and the model sits in one block, so I can swap it without changing anything else.

Both models use the same architecture, RF-DETR Small (opens in a new tab), a real-time object detection model from Roboflow. We chose it in the training setup and changed only the training data.

The baseline is trained on the train split of the pill dataset (opens in a new tab). The dataset has 174 images, split into 122 for training, 35 for validation, and 17 for testing.

The weaker model uses a 7-image version of the same data: 2 images for training, 2 for validation, and 3 for testing. I gave it far fewer examples on purpose, so it would perform worse than the baseline.

The 17 Eval images are the baseline’s test split, so it has never seen them. Because neither model trained on these images, the Eval tests how each one handles images it hasn’t seen. The models themselves are only for this demo, though: for your own Workflow, retrain on your own images and validate on your own test set before relying on the results.

How to Build a Computer Vision Regression Test Set

Choose test images that represent the conditions your application needs to handle. Include everyday examples, difficult cases, and inputs where the correct answer is zero. Each should test a specific requirement of the Workflow.

For the pill counter, a broader regression test set could include:

For every image, record the expected count for all five pill types, including zeros. Review these answers carefully, and recount or remove any image with an unclear count before approving the ground truth.

Keep these images and their near-duplicates out of model training. When collecting frames from video, group related frames by recording session before assigning them to training or evaluation sets. Nearly identical frames on both sides can make the test look easier than deployment will be.

Keep the images, expected answers, and Check settings consistent when comparing Workflow versions. Record which test set and configuration produced each result. When you add Cases or correct ground truth, rerun the baseline and candidate against the updated set.

As you use regression results to improve the application, also maintain a separate held-out set for final validation. The regression suite checks whether known requirements still hold; fresh evaluation data helps assess whether improvements extend to examples you have not repeatedly used during development.

Create the Eval and Ground Truth

To create an Eval, open Workflows in the left sidebar, choose the Evals tab, and click Create Eval. Setup runs through five steps, and the first three build the test set.

Start by selecting the Workflow, Pill Count Per Class. Then upload the ground truth images. We used the 17 images from the dataset’s test split, which neither model trained on, through Use a Dataset test set. Next, choose which outputs to check. We kept all five counts, so each pill type gets its own Check later.

Now label the ground truth. For each image, enter the correct count for every pill type, then click Add to Ground Truth to approve it.

A wrong expected count will fail a correct Workflow, so count each pill by eye and zoom in on crowded images.

Define the Checks and Run the Baseline

Next comes Define Eval Checks. Roboflow drafts one Check for each output you selected, using your ground truth as the example, so we get five: Pill 01 count accuracy through Pill 05 count accuracy. Each Check asks three things: which output to evaluate, what to compare, and how to judge it.

For every Check, we compare the numeric value against ground truth and judge it with Matches ground truth exactly. A Case passes only when the Workflow’s count equals the expected count.

You can also use two other options. Custom rule lets you set your own metric and threshold, for example to allow a tolerance of one pill. To add a time limit, click Add in the Checks panel and pick Latency budget, which sets a limit per Case. The preview panel tests a Check on one image before you commit.

Click Next: Review Evaluation, check the setup and credit estimate, and start the Run. Run 1 passes all 85 Checks, which is 17 Cases with five Checks each, and no Checks fail. This is our baseline: every Case passes, so any failure in a later Run comes from the change we make.

The Metrics tab shows the same result per Check: 17 of 17 Cases pass each one, with a difference of 0. It also lists runtime metrics. Average latency is 592.6 ms, and the timeout and failure rates are 0%. That latency is the cloud request time, not model inference alone.

Vector Analysis plots the 17 Cases by image similarity and colors each by Checks passed. Right now every dot is the same color because nothing fails. It becomes useful when failures appear, since it groups similar images.

Change the Workflow and Rerun

Next, we change the Workflow and rerun the Eval. We duplicated the Workflow, named the copy Pill Count Per Class 2, and swapped only the object detection model for a deliberately weaker one. Every filter, count, and output stayed the same.

We then started a new Run in the same Eval, so both Workflows are graded against the same 17 Cases and five Checks. When a Run has two or more Workflows, it opens on an Overview tab, which plots Checks passed against latency. The original passes 100% of Checks and the copy passes 41.2%.

The heatmap below it shows how many of the five Checks each Workflow passed on each image. The original passes 5 of 5 on every Case. The copy ranges from 1 of 5 to 4 of 5, and passes all five on none.

On the Metrics tab, you can see which pill types broke. The copy passes Pill 01, 02, and 03 on only 5, 3, and 2 of 17 images, while Pill 04 and 05 hold up better at 13 and 12.

To compare Eval results with model quality, here are both models’ own metrics next to their Eval scores.

In plain terms, a recall of 83.3% means the weaker model finds about five of every six pills that are really there. A precision of 70.3% means about seven in ten of its detections are correct. Those numbers look passable, yet the Workflow built on that model passes less than half of the Eval’s Checks.

Each Check asks for an exact count, so one missed or extra pill fails it, even when most detections are right. Both Workflows use the same filters and counting steps, and only the model changed, so the drop comes from the model: decent model metrics can still produce wrong counts. These results only cover the 17 test images.

The copy was also slower, at 6.8 s average against 592.6 ms. Request time includes more than the model, so we treat that as a side note.

Vector Analysis has a toggle for each Workflow. For the copy, the dots that were all blue in Run 1 turn orange, and similar images sit close together, which helps you spot clusters of failures. Click any dot to open that Case.

Here is one of the copy’s failures. The baseline passes all five Checks on this image, but the copy passes only 1 of 5. Pill 05 passes, while Pill 01, 02, 03, and 04 fail. Each Check shows a pass or fail and a score.

Open the Pill 01 Check to see why it failed. The ground truth is 2 pills, but the copy returned 1. The rule requires a difference of 0, so the Check fails. The panel also reports the difference of 1 and a relative difference of 50%.

Both Runs used the same 17 Cases and five Checks on Roboflow’s Cloud API server (opens in a new tab), so the model is the only difference. The copy ran without errors, so you would only catch the drop by running the Eval.

From Tutorial to Production

Here’s how to turn Eval results into a decision about whether to ship.

Decide What Blocks a Release

Pick the rule before you run the Eval. Otherwise it is tempting to accept a weak result after you see it.

In our pill counter, a wrong count means the wrong number of pills in a pack. So we would block any release where a Check fails on any image. If you can accept some misses, write down how many before you run, and block any Run that goes over that number.

Look at each Check too, not only the overall score. In our demo, Pills 04 and 05 held up far better than Pills 01 to 03. A new version can score well overall and still be worse on one pill type, and that should block it too. If speed matters, add a Latency budget Check.

Block Workflow Updates That Fail Regression Tests

Once you have defined your release criteria, use a Roboflow deploy gate to enforce them. A deploy gate requires a saved Workflow version to pass an Eval before it can be published.

To configure one:

1. Open the Workflow you want to protect.
2. Click the dropdown next to Publish.
3. Under Publishing Requirements, select Require an Eval to pass.
4. Choose your Eval and set the minimum pass rate for each Check.
5. Click Save Checks.

For this pill counter, set the minimum pass rate to 100% for all five count Checks. Each Check already requires an exact count; the gate requires that Check to pass on every test image. The replacement Workflow’s 41.2% overall Check pass rate would therefore block publication.

After saving a Workflow change, click Run Checks to evaluate the latest saved version. Publish it once every configured requirement passes. Each new version needs its own passing run, and editing the Eval requires the checks to run again.

If you use Auto-Publish, each save runs the checks automatically, and the version publishes when it passes.

When Exact Match Is Too Strict

Exact match fits cases where one wrong count has a cost, like pills in a pack. It is too strict when a small error does not matter, such as a rough stock estimate. It can also be too strict when even careful human counts of a crowded image can differ by one.

In those cases, use a Custom rule to allow a tolerance, such as one pill. Choose the tolerance based on what the next step can accept, not on what makes the Run pass. If you set the tolerance too loose, you can miss small drops that exact match would catch.

Add Real Failures as Cases

When your Workflow gets an image wrong in real use, add that image to the Eval. Good examples are a crowded tray, glare, an unusual pill, or a blurry frame. Enter the correct counts and keep the image.

Expand the Test Set Over Time

Seventeen images from one setup do not cover other lighting, camera angles, trays, or pill types. A wrong expected count can also fail a correct Workflow. A pass means the Workflow handled these 17 images correctly. Keep adding images from new conditions, and spot-check live results against manual counts.

Automate Computer Vision Regression Testing

Add your Eval to your release pipeline so it runs whenever you prepare a Workflow update. Roboflow Workflow Evals (opens in a new tab) supports the Python SDK (opens in a new tab), CLI (opens in a new tab), HTTP API (opens in a new tab), and MCP (opens in a new tab). For a continuous integration (CI) pipeline, a Python script can start the evaluation, wait for results, and fail the job when the candidate does not meet your requirements.

For this pill counter, all five count Checks must pass on all 17 Cases. A completed Run means evaluation has finished; the script must also inspect whether the Checks passed.

Install the SDK:

pip install --upgrade "roboflow>=1.7.0"

Before running the example:

• Set ROBOFLOW_API_KEY as a secret in your CI environment and ROBOFLOW_WORKSPACE to your workspace slug.
• Replace the Eval ID and five Check IDs below with the IDs from your existing Eval.
• Prepare a validated Run request as run-request.json, following the Workflow Evals API contract (opens in a new tab). It should contain one candidate using savedWorkflowVersion and its bindingSet, which maps Workflow inputs and outputs to the Eval.

Save the following script as regression_check.py:

import json
import os
from pathlib import Path
from uuid import uuid4

from roboflow import Roboflow

EVAL_ID = "YOUR_EVAL_ID"
EXPECTED_CASES = 17
CHECK_IDS = [
    "CHECK_ID_FOR_PILL_01",
    "CHECK_ID_FOR_PILL_02",
    "CHECK_ID_FOR_PILL_03",
    "CHECK_ID_FOR_PILL_04",
    "CHECK_ID_FOR_PILL_05",
]

evals = Roboflow(
    api_key=os.environ["ROBOFLOW_API_KEY"]
).workspace(os.environ["ROBOFLOW_WORKSPACE"]).workflow_evals()

request = json.loads(Path("run-request.json").read_text())
if len(request["executions"]) != 1:
    raise SystemExit("Configure exactly one candidate Workflow.")
if request["executions"][0]["subject"]["kind"] != "savedWorkflowVersion":
    raise SystemExit("Select a specific saved Workflow version.")

request.update(
    caseSelection={"kind": "all"},
    executionReuse="never",
    inferenceServer={"kind": "serverless"},
)
started = evals.start_run(EVAL_ID, request, idempotency_key=str(uuid4()))
run_id = started["run"]["id"]
execution_id = started["executions"][0]["id"]
print(f"Eval Run: {run_id}")

finished = evals.wait_for_run(EVAL_ID, run_id, timeout=600)
state = finished.get("state") or finished.get("run", {}).get("state")
if state != "completed":
    raise SystemExit(f"Evaluation did not complete successfully: {state}")

all_passed = True
for check_id in CHECK_IDS:
    page = evals.list_case_results(
        EVAL_ID, run_id, execution_id,
        check_id=check_id, judgment="pass", state="completed", limit=200,
    )
    passed = len(page["items"])
    print(f"{check_id}: {passed}/{EXPECTED_CASES} Cases passed")
    if passed != EXPECTED_CASES or page.get("nextCursor"):
        all_passed = False

raise SystemExit(0 if all_passed else 1)

Run it from your CI job:

python regression_check.py

The script exits successfully only when every required Check passes on all 17 Cases. Failed Checks, missing results, API errors, and evaluation timeouts produce a nonzero exit status. Configure your deployment job to depend on this job succeeding.

Keep the Run ID in your build logs so you can inspect failures in Roboflow. When you intentionally expand the test set, update EXPECTED_CASES and rerun the baseline before comparing another candidate. For larger suites, paginate the results instead of relying on this example’s single-page check.

Regression Test with Workflow Evals

A Workflow can get worse without throwing an error. In this tutorial, swapping one model block dropped Checks passed from 100% to 41.2%, and the Eval showed which images and which pill types lost accuracy.

Keep your test images and expected counts fixed, and rerun the Eval after every change: a model swap, a retrain, a threshold edit, or a new block. Each Workflow execution costs 1/10,000 of a credit, plus any inference charges.

Next, build an Eval for your own Workflow. Start with a small set of images whose answers you have checked yourself, and add more as you find real failures.

Further Reading:

• Mean Average Precision in Computer Vision with Roboflow
• Object Counting with Computer Vision
• No-Code Computer Vision API Development