How to Pre-Filter Images Before Sending Them to an LLM

Sending every image in a dataset to a multimodal LLM means spending time and tokens on images that may never meet your selection criteria. To pre-filter images before sending them to an LLM, run an object detection model first, apply a relevance rule, and send only the images that pass for deeper analysis.

With Roboflow Workflows, you can connect these steps in one pipeline. An RF-DETR model detects objects, a Continue If block controls which images advance, and Google Gemini analyzes the remaining candidates and returns structured results.

In this tutorial, you’ll build a workflow for finding sports-car photos in a larger image library. The detector identifies potential candidates, while Gemini checks whether the vehicle is a sports car, whether it is the main subject, and whether alcohol is visible.

You’ll also learn how to tune the filter on labeled examples, measure how many relevant images it retains, and process a dataset with Python. The goal is to reduce unnecessary LLM calls while preserving the images your application needs.

Why Pre-Filter Images Before an LLM?

Imagine you need to select images for a sports-car marketing campaign from an existing media library of 10M images. The campaign requires photos that show a sports car with no visible alcohol, along with a short description of each selected image.

A direct approach is to send every image to a vision-capable LLM and ask it to check these conditions. However, the model must process each submitted image before deciding whether it is relevant. An instruction such as “ignore images without a sports car” does not avoid that processing.

For token-based APIs, inference cost includes image inputs, prompt text, and generated output. An irrelevant image can therefore incur charges even if the model returns only “no match.” The cost depends on the model, how the image is processed, and the amount of output generated.

When relevant images are rare, most requests analyze images that do not meet the selection criteria. Across millions of images, these requests consume inference budget and processing capacity without adding useful images to the campaign’s collection. The engineering goal is to reduce this unnecessary inference while retaining the relevant images.

How Pre-Filtering Images Before an LLM Works

The fix is to change when the LLM runs. In a dataset with rare positives, most images fail an easy check. An image with no car cannot contain a sports car, and a small object detector can run that check on every image at very low cost. Only the images that pass go on to the LLM, which handles the hard part of the task.

The pattern turns this idea into two stages, each with a clear job. Stage 1, the detector, must keep almost every positive image, so it is tuned for high recall. Stage 2, the LLM, sees far fewer images, so you can afford a large model for the final high-precision decision. Because the jobs are separate, each stage can also be evaluated and tuned on its own.

In the campaign example, a sports car is the positive condition and visible alcohol is the negative condition. For a hypothetical estimate, assume that about 1% of the 10M images in the library show a sports car.

Without Stage 1, Gemini processes all 10M images. For this hypothetical calculation, assume that a custom Stage 1 detector passes 1% of the images while retaining the relevant examples. Gemini would then process 100K candidates. This assumed pass rate must be measured in practice; having 1% relevant images does not guarantee that only 1% will pass the filter.

How to Pre-Filter Images Before Sending Them to an LLM

Let us now build a pre-filter workflow in Roboflow Workflows. Our example workflow screens images where a sports car is the main subject and no alcohol is visible. It avoids Gemini LLM calls on images where the detector finds no car. The blocks run in a fixed order.

First, the RF-DETR Nano detector looks for cars, bottles, and wine glasses at a confidence threshold of 0.3. A Property Definition block counts the cars, and an Expression block turns that count into a gate decision. The Continue If block then acts as the gate. An image moves on only if it has at least one car. Other images stop at the gate. Bottles and wine glasses do not stop an image, because detecting these objects does not establish that alcohol is present. Gemini checks for visible alcohol or alcohol branding in Stage 2.

For images that pass, the Google Gemini block looks at the full photo. It decides if the vehicle is a sports car or supercar. It checks if any alcohol or alcohol branding is visible. It also checks if the car is the clear subject. Then it writes a one-sentence description. The JSON Parser block turns this answer into separate fields that your application can use. First Non Empty Or Default blocks fill any empty field with a default value. A Custom Python block then checks every field and gives each image one of four outcomes. The following diagram shows how the workflow works.

The diagram groups the workflow into three stages, pre-filter the image, verify it with Gemini, and return a consistent result. Images without a detected car stop before Gemini is called. The four outcomes appear along the bottom. The steps below build this workflow one block at a time.

Step 1. Create a Workflow

In your Roboflow workspace, open the Workflows tab and click Create Workflow. This opens a blank workflow in the editor. In the default Auto mode, the editor connects compatible inputs and outputs as you add blocks. You can also describe Steps 2 to 5 to the Workflows AI Assistant in the chat panel of the editor. It adds the blocks for you. Check each block afterward, because some settings may need a manual fix. Each save creates a new version, so you can trace every run to its exact thresholds.

Step 2. Add the Detector for the Cheap Annotation Pass

Add an Object Detection Model block and connect it to the image input. Use RF-DETR Nano pretrained on COCO, with the model ID rfdetr-nano. COCO already includes the classes car, bottle, and wine glass. Then set two parameters.

• Class Filter: Set it to car, bottle, and wine glass. The gate uses only car, while the bottle and wine glass boxes stay in the output for the optional rule in Step 3 and for later reuse.
• Confidence Mode: Choose Custom and enter 0.3. A low threshold keeps recall high for cars, because an image with no detected car never reaches the LLM.

For better recall on small objects, move to the Small or Medium size of RF-DETR.

A COCO model knows the class car but not sports car. So this starter gate passes every car image, which is far more than the 1% of images that show a sports car. The share of images that pass the gate is the pass rate. To bring the pass rate closer to the positive rate, train a detector on a task class such as sports-car. A custom detector can take over the alcohol rule only if it learns alcohol-specific classes such as beer-bottle and beer-can. Validate it on labeled images before it drops any image. You can also train RF-DETR on a custom dataset or start from public datasets on Roboflow Universe. After that, change only the model ID in this step and the class name in the Step 3 count.

Step 3. Count the Cars and Add the Continue If Gate

The gate needs one clear yes or no for each image. Three small blocks create that decision and act on it.

Add a Property Definition block named car_count and set its data to the detector predictions. Give it two operations. The first is a Detections Filter that keeps only the car class. The second is SequenceLength, which counts the cars that remain.

Add an Expression block named gate_decision after car_count. Give it one case that returns true when car_count is 1 or more, and set the default result to false. This value is the gate decision, and the workflow returns it as gate_pass.

Add a Continue If block after gate_decision. For each image, it checks a condition. Map gate_pass in evaluation_parameters to the gate_decision output, and set the condition to gate_pass equals true. If the condition is true, the steps in next_steps run. If it is false, the block stops that branch.

The gate does not judge alcohol. A detected bottle or wine glass tells you what the object is, not what it holds. A COCO detector gives a water bottle and a beer bottle the same bottle class. A higher confidence threshold only makes the class more certain. Gemini decides whether alcohol is visible in Step 4.

Some campaigns ban every bottle or glass, whatever it holds. For those, add a second Property Definition block that counts bottle and wine glass detections with a confidence of 0.6 or higher. Pass that count to the Expression block too, and add a second statement with AND that requires it to be 0. Leave this rule out of the starter workflow, because it also drops images that meet the stated requirements. Step 2 covers the other option, a custom detector for alcohol-specific classes.

Set next_steps to the Gemini block after you add it in Step 4. Gate logic runs on CPU machines and is not billed per image on the Serverless API.

Step 4. Add the Google Gemini Block as Stage 2

Add a Google Gemini block, connect its image input to the workflow image, and select it as the next step of the Continue If block. Then set five parameters.

• Task Type. Choose Structured Output Generation (structured-answering). Each value in the output structure is the instruction for its key, and the model returns JSON only.
• Output Structure. Use the four keys below.
• Model Version. Choose Gemini 3 Flash (gemini-3-flash-preview).
• Thinking Level. Choose low, which minimizes latency and cost on simple tasks.
• API Key. Keep rf_key:account to use Roboflow's managed key with usage-based billing, or enter your own Google AI API key.

Start each instruction with its type, such as Boolean or String, so Gemini knows which kind of value to return.

{
  "is_sports_car": "Boolean: true if the main vehicle is a sports car or supercar, otherwise false.",
  "alcohol_visible": "Boolean: true only when visible evidence identifies an alcoholic drink, beer can, wine or liquor bottle, cocktail, or alcohol logo anywhere in the image; otherwise false. A generic bottle, water bottle, empty wine glass, or juice does not by itself establish alcohol. Assess visible evidence only, not hidden contents.",
  "car_is_subject": "Boolean: true if the car is the clear subject of the photo, otherwise false.",
  "description": "String: one sentence describing the car, its color, and the setting."
}

The alcohol_visible instruction also says that a plain bottle or an empty glass is not proof of alcohol. Gemini judges only what the image shows.

Stage 1 already removed the clear negatives, so the LLM task is small. It confirms the sports car class, checks every bottle, glass, can, and logo for alcohol, and writes a short caption you can search later.

Step 5. Parse and Validate the Answer

Add a JSON Parser block. Set raw_json to the Gemini output and add the four keys as expected fields. Each key becomes a workflow output. The block sets error_status to true when a key is missing, which gives you a parse error rate to monitor.

When the gate stops an image, the Gemini and JSON Parser steps do not run, so their outputs are null. Add five First Non Empty Or Default blocks, one for each parsed field and one for error_status. Each block passes its value on, or returns a default value when the input is null.

Set the default to missing for the three Boolean fields, to an empty string for description, and to true for error_status. None of these defaults can pass the check below, so a missing answer never becomes a match.

Add a Custom Python block named final_decision after these five blocks. Its inputs are gate_pass and the five filled fields, and its outputs are outcome, is_match, and response_valid. The block lives in the workflow JSON as a dynamic Python block. Custom Python blocks also run on the Serverless API, so the Step 8 script can call this workflow. Its run function holds the whole check.

def run(self, gate_pass, sports_car, alcohol, subject, description, parser_error):
    if gate_pass is not True:
        return {"outcome": "filtered_out", "is_match": False, "response_valid": False}
    valid = (
        parser_error is False
        and all(type(v) is bool for v in (sports_car, alcohol, subject))
        and isinstance(description, str)
        and bool(description.strip())
    )
    if not valid:
        return {"outcome": "error", "is_match": False, "response_valid": False}
    match = sports_car is True and subject is True and alcohol is False
    return {
        "outcome": "llm_match" if match else "llm_rejection",
        "is_match": match,
        "response_valid": True,
    }

The function returns filtered_out when the gate stopped the image. It returns error when the parser reports a problem, when a Boolean field is not a real true or false, or when the description is empty. A complete answer becomes llm_match only when is_sports_car and car_is_subject are true and alcohol_visible is false. Every other complete answer becomes llm_rejection.

After Step 5, the editor shows the full chain from the image input to the final_decision block. Your application can filter on outcome or is_match without reading the raw LLM text.

The Python script in Step 8 reads several of these outputs by name, so map them exactly as shown in the table.

The selectors assume the step names detector, car_count, gate_decision, gemini, verdict, and final_decision, so change them if your blocks use other names.

The workflow look like following.

Try this workflow to view it, run a preview, or fork it into your own workspace.

Step 6. Test the Gate in the Editor

Click Test Workflow at the top of the editor, add your test images, and click Run to see the JSON output for each image. Use three images that cover every branch. The first shows only a car. The second shows a car with a wine glass, which should still reach Gemini. The third shows no car. The table shows how the gate handles typical detector outputs.

When the gate_pass output gives these results for your test images, the gate logic is correct and ready for tuning.

Step 7. Tune the Gate on a Labeled Sample Without LLM Calls

The 0.3 car threshold from Step 2 is a starting point, so choose the final value with data. Label a sample of your dataset as positive or negative. Run only Stage 1 on it once, with a low threshold, through the Serverless API, and cache the detections. Then replay the gate rule offline for a grid of thresholds. Each setting gives a pass rate, which sets the LLM cost, and a recall, which shows how many positives survive. This step makes no LLM calls.

The labels go in a file named labels.csv, which is the answer key for the sample. It has one row per image. The image column holds the file name, and the match column holds yes when the image shows a sports car as the main subject with no visible alcohol. Every other image gets no. Decide each label by looking at the image, not from the workflow output.

image,match
input_1.png,no
input_2.png,no
input_3.png,no
input_4.png,yes
input_5.png,yes
input_6.png,no
input_7.png,no
input_8.png,yes
input_9.png,yes
input_10.png,yes
input_11.png,no
input_12.png,no
input_13.png,yes
input_14.png,no
input_15.png,no

Select the sample images at random, so the pass rate reflects your dataset. The sample also needs enough positives to measure recall well. With 10 positives, one missed image moves recall by 10 points. When positives are rare, add extra known positives to the sample, and expect the measured pass rate to come out a little high.

Put the labeled images in a folder named sample. Then save the script below as tune_gate.py, next to labels.csv and the sample folder.

import csv
import os
from pathlib import Path

from inference_sdk import InferenceConfiguration, InferenceHTTPClient

client = InferenceHTTPClient(
    api_url="https://serverless.roboflow.com",
    api_key="ROBOFLOW_API_KEY",
).configure(InferenceConfiguration(confidence_threshold=0.1, api_key_transport="header"))

# labels.csv has two columns, image and match, where match is "yes" or "no"
with open("labels.csv", newline="") as f:
    labels = {row["image"]: row["match"] == "yes" for row in csv.DictReader(f)}

# Run Stage 1 once at a low threshold and cache each class name and confidence
cache = {}
for name in labels:
    result = client.infer(str(Path("sample") / name), model_id="rfdetr-nano")
    cache[name] = [(p["class"], p["confidence"]) for p in result["predictions"]]


def gate(detections, car_conf):
    return any(c == "car" and s >= car_conf for c, s in detections)


positives = sum(labels.values())
for car_conf in [0.1, 0.2, 0.3, 0.4, 0.5]:
    passed = [name for name in labels if gate(cache[name], car_conf)]
    recall = sum(labels[name] for name in passed) / max(positives, 1)
    pass_rate = len(passed) / len(labels)
    print(f"car {car_conf}  pass rate {pass_rate:.1%}  recall {recall:.1%}")

To run the script, install the inference SDK and set your Roboflow API key as an environment variable.

pip install inference-sdk
python tune_gate.py

The script prints five lines, one for each car threshold. Each line shows the pass rate and recall for that threshold. You will see output similar to following.

Pick the operating point with the lowest pass rate that still meets your recall target. Copy the car threshold into the detector confidence. Then confirm the choice with one end-to-end run on the same sample. If you use the optional exclusion rule from Step 3, add its threshold to the grid in the same way.

Step 8. Run the Workflow on the Full Dataset

After tuning, process the whole dataset with a short Python script. The inference SDK can run a saved workflow with run_workflow. Each call sends one image to the Serverless API or to your own Roboflow Inference server.

import json
import os
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path

from inference_sdk import InferenceHTTPClient

client = InferenceHTTPClient(
    api_url="https://serverless.roboflow.com",
    api_key="ROBOFLOW_API_KEY",
)


def run(path):
    try:
        result = client.run_workflow(
            workspace_name="your-workspace",
            workflow_id="prefilter-workflow",
            images={"image": str(path)},
        )[0]
    except Exception as error:
        return path.name, {"error": str(error)}
    return path.name, result


images = sorted(
    path
    for path in Path("images").iterdir()
    if path.suffix.lower() in {".jpg", ".jpeg", ".png"}
)
with ThreadPoolExecutor(max_workers=8) as pool, open("results.jsonl", "w") as out:
    for name, r in pool.map(run, images):
        row = {
            "image": name,
            "gate_pass": r.get("gate_pass"),
            "outcome": r.get("outcome") or "error",
            "description": r.get("description"),
            "gemini_input_tokens": r.get("gemini_input_tokens"),
            "gemini_output_tokens": r.get("gemini_output_tokens"),
            "detections": r.get("detections"),
            "error": r.get("error"),
        }
        out.write(json.dumps(row) + "\n")

Set workspace_name and workflow_id to your own values. The script reads the JPEG and PNG files in the images folder and keeps six of the workflow outputs from the table in Step 5. It sends eight requests at a time and writes one JSON line per image to results.jsonl. To use your own GPU server, change api_url to its address, such as http://localhost:9001.

Every line in results.jsonl has the same structure. The example below shows the saved result for input_4.png.

{"image": "input_4.png", "gate_pass": true, "outcome": "llm_match", "description": "A bright yellow supercar with its headlights on is captured driving through a dimly lit tunnel.", "gemini_input_tokens": 1295, "gemini_output_tokens": 57, "detections": {"image": {"width": 827, "height": 789}, "predictions": [{"width": 553.0, "height": 345.0, "x": 426.5, "y": 439.5, "confidence": 0.9483212828636169, "class_id": 3, "class": "car", "detection_id": "c3fcc038-d388-43b0-9186-a1f07fa9744c", "parent_id": "image"}]}, "error": null}

Each line has eight fields.

• image is the file name.
• gate_pass is the gate decision. It is true when the image passed the gate, false when the gate stopped it, and null when the request failed.
• outcome is one of the four values from final_decision. filtered_out means the gate stopped the image. llm_match means Gemini returned a complete answer with is_sports_car and car_is_subject as true and alcohol_visible as false. llm_rejection means the answer was complete but failed that rule. error means the answer was incomplete or the request failed.
• description is the one-sentence caption from Gemini.
• gemini_input_tokens and gemini_output_tokens count the tokens of the Gemini call. They are null when Gemini did not run.
• detections holds the image size and the Stage 1 boxes, in the same predictions format that the Object Detection Model block returns. In each box, x and y give the center point in pixels, and width and height give the size. Each box also has a confidence, a class name with its class_id, a unique detection_id, and a parent_id for the source image.
• error is null unless the request failed. It then holds the error message, and every other field except image and outcome is null. Rerun only the images that show an error.

The above result was for following image.

In the 15-image sample, 13 images passed the car-only gate and two were filtered out. The gate therefore reduced the number of images sent to Gemini by 13.3%. Images containing both a detected car and bottles or wine glasses still passed, leaving Gemini to assess visible alcohol or alcohol branding.

The outcome field keeps the four results apart, so a missing caption never hides an error. The pass rate is the share of lines where gate_pass is true. This field comes straight from the gate decision, so an image with an incomplete Gemini answer still counts as a pass.

Keep the results file after the run. Its detections field holds the Stage 1 annotations for every image, including the images that stopped at the gate. Load these annotations into a table or a database such as BigQuery. A later campaign can then filter them with a new rule and send only its own matches to the LLM, without running the detector again. The saved annotations cover only the classes in the Class Filter, so add any classes a later campaign needs before the first run.

You can run the RF-DETR pre-filter on your own hardware with Roboflow’s free self-hosted inference. Local inference does not consume Roboflow credits; you cover the hardware and operating costs. Images that pass the filter still incur any applicable Gemini API charges.

Measured Results on a Labeled Sample

A labeled sample shows how the whole pipeline performs on your own data. Score it end to end with the Step 8 script and one short evaluation script.

Run the Step 8 script on the labeled sample from Step 7. Change Path("images") to Path("sample") and results.jsonl to sample_results.jsonl, so the results of the full run stay untouched. Then put the evaluation script below in a file named score.py, next to labels.csv and sample_results.jsonl, and run it with python score.py.

import csv
import json

with open("labels.csv", newline="") as f:
    labels = {row["image"]: row["match"] == "yes" for row in csv.DictReader(f)}

with open("sample_results.jsonl") as f:
    rows = [r for r in map(json.loads, f) if r["image"] in labels]

relevant = [r for r in rows if labels[r["image"]]]
known = [r for r in rows if r["gate_pass"] is not None]
passed = [r for r in known if r["gate_pass"]]
matches = [r for r in rows if r["outcome"] == "llm_match"]


def share(part, whole):
    return f"{len(part) / max(len(whole), 1):.1%}"


def total(field):
    return sum(r[field] or 0 for r in rows)


print("sample size", len(rows))
print("relevant images", len(relevant))
print("pass rate", share(passed, known))
print("filter recall", share([r for r in relevant if r["gate_pass"]], relevant))
print("pipeline recall", share([r for r in relevant if r["outcome"] == "llm_match"], relevant))
print("pipeline precision", share([r for r in matches if labels[r["image"]]], matches))
print("LLM requests", len(passed))
print("Gemini input tokens", total("gemini_input_tokens"))
print("Gemini output tokens", total("gemini_output_tokens"))
print("errors", sum(r["outcome"] == "error" for r in rows))

input_tokens = total("gemini_input_tokens")
output_tokens = total("gemini_output_tokens")

estimated_gemini_credits = (
    input_tokens / 4_000_000
    + output_tokens / 666_667
)

print("Estimated Gemini credits", f"{estimated_gemini_credits:.6f}")

Filter recall is the share of relevant images that pass the gate. Pipeline recall is the share that end as llm_match, so it also counts the mistakes Gemini makes. Pipeline precision is the share of llm_match images that are truly relevant. For the total cost, read the credits used during the run on the Credit Usage page in your workspace settings. With the managed rf_key:account key, Gemini is billed per token, so the two token totals explain the Gemini part of that cost. If the Gemini block uses your Google key instead, add the Gemini charges from your Google account.

The evaluation metrics and token counts come from the 15-image sample. Gemini credits are estimated from the recorded tokens, while the earlier 10M-image figures are hypothetical. Gemini credit usage was estimated as (input tokens ÷ 4,000,000) + (output tokens ÷ 666,667) using the published Gemini 3 Flash Preview rates, excluding RF-DETR inference costs.

These results demonstrate the workflow on a small 15-image sample. Evaluate it on a larger, representative dataset before scaling, and use a separate labeled test set to check the threshold selected during tuning.

Step 9. Reuse the LLM Output to Improve Stage 1

Every full run produces new training data. For each candidate, the LLM output gives an is_sports_car answer. That answer is an image-level label, which fits an image classifier. A detector also needs a bounding box for each object, which takes one more labeling step.

• Train an image classifier. Save the candidates with the Roboflow Dataset Upload block and review the is_sports_car answers. Use the reviewed answers as image labels and train a classification model. The Single-Label Classification Model block can then run it as Stage 1, and the gate checks its top class.
• Train a custom detector. Connect the detector predictions to the same Dataset Upload block and turn on persist_predictions, so the car boxes from RF-DETR arrive as pre-labels. In Roboflow Annotate, relabel each box as sports-car or car and fix missing or loose boxes. Then train RF-DETR on the reviewed boxes and swap its model ID into Step 2.

Also review a random sample of images with the outcome filtered_out. Gemini never sees these images, so its labels cannot show which relevant images Stage 1 misses. Every relevant image in this sample is a false negative to add to the training set.

Both paths are a form of knowledge distillation. A large teacher model generates labels, which are reviewed and used to train a smaller student model. When deployed as Stage 1, the new model can reduce false positives while maintaining recall, allowing fewer irrelevant images to reach the LLM on comparable future batches.

Another Example: Inspecting Finished-Product Packaging

The same pre-filtering approach can help manufacturers find cartons with missing labels or visible damage. A production-line image archive may contain empty conveyor frames, unrelated products, and finished cartons. Only images containing the target carton need detailed inspection. Here's how the whole process works.

In Roboflow Workflows, a custom RF-DETR model first detects the target carton. A Continue If block forwards images containing that detection to Gemini and skips the others before an LLM request is made. Gemini checks whether the required label is visible and looks for damage such as an open flap, tear, or crushed corner. It returns label_present, visible_damage, needs_review, and a short inspection_summary to support operator review.

The gate checks carton presence, so a detected carton with a missing label still reaches Gemini. This focuses LLM processing on relevant images. Teams can measure the reduction in requests alongside how reliably the pre-filter retains cartons that need review.

Conclusion

Sign up for a free Roboflow account, build the workflow in the editor, and tune the gate on a labeled sample of your own data before you scale.

Further reading:

• How to Add an LLM to a Vision Pipeline
• Building Vision-Language Pipelines with VLMs
• Gemini Computer Vision