Roboflow + Fastdup
Download a Roboflow dataset, run fastdup to find duplicates, outliers and other problem images, then upload the cleaned set back to train on.
How to connect Fastdup to Roboflow
5 stepsDownload a dataset version
Use the Roboflow Python SDK to download a version, for example project.version(1).download("yolov8"). Each split lands in its own images folder.
Run fastdup on each split
Install fastdup with pip, then run fastdup.create(work_dir=..., input_dir=...) and .run() for the train, valid and test images folders.
Review duplicates and outliers
Open the duplicates gallery and the outliers table to see clusters of near-identical images and images far from the rest of the dataset.
Remove the images you do not need
Keep one image from each duplicate cluster and drop outliers you judge to be noise. fastdup reports the files; you choose what to delete.
Upload the cleaned dataset
Upload the remaining images and labels to Roboflow, generate a new version and train on it to compare with the original.
What teams build with Fastdup
Shrink a dataset without losing accuracy
Remove near-duplicate images so training runs on fewer, more varied examples.
Find images that do not belong
Surface outliers, broken files and very dark or blurry frames before they reach training.
Questions
No. fastdup works on a local copy of the dataset. Changes reach Roboflow only when you upload the cleaned images as a new dataset.
fastdup reads image folders, so any format that keeps images in per-split folders works. Our guide downloads in YOLOv8 format.
The fastdup repository is licensed CC BY-NC-ND 4.0, a non-commercial license. Visual Layer sells commercial products for teams that need them.
Why are companies choosing Roboflow?
The end-to-end platform for vision AI: create datasets, train models and deploy to production.
Flexible deployment modes
Move from a scalable cloud API to edge hardware like NVIDIA Jetson without managing complex dependencies.
Integrated workflow builder
Combine multiple AI models, custom business logic and third-party integrations into one production-ready pipeline in a low-code editor.
Model monitoring and alerts
Dashboards track confidence, inference time and request volume for your deployed models, and email alerts tell your team when confidence suddenly drops.
