How Frontier AI Labs Acquire Visual Training Data for Multimodal Models

Training frontier multimodal models (opens in a new tab) requires massive amounts of high-quality visual data, including images, videos, documents, diagrams, and 3D environments. This data teaches models to recognize objects, understand scenes, interpret visual information, follow instructions, and reason about the physical world.

However, acquiring visual data at frontier-model scale presents significant engineering and operational challenges. AI labs cannot rely on a single source and instead combine multiple data acquisition strategies. Each strategy provides different types of visual information and supervision.

This guide explores how frontier AI labs acquire visual training data through web crawling, public datasets, commercial partnerships, human-collected and annotated data, and synthetic data generation, including the technologies and real-world examples behind each approach.

For the specialized, real-world data that web crawls and public datasets miss, Roboflow Labs (opens in a new tab) helps collect, annotate, and evaluate exactly the images and videos your model needs to close its performance gaps.

Sources of Visual Training Data for Frontier AI Models

AI labs use various strategies to acquire visual training data, often combining multiple sources to build large-scale, diverse, and high-quality datasets. The main data acquisition strategies include:

1. Web Crawling: Collecting Visual Data From the Open Web

Web crawling is the automated process of discovering and retrieving content from websites across the internet. It uses software programs called crawlers or spiders to follow links, access webpages, and extract information such as images, videos, and associated text.

For frontier AI labs, web crawling provides the scale and diversity needed to build large visual training datasets. Labs collect billions of images and their associated text from the open web to create candidate image-text pairs for multimodal model training.

However, raw crawled data is often noisy, redundant, or irrelevant. AI labs rely on a data curation pipeline to transform this content into high-quality, diverse, and relevant training data. This makes curation just as important as the scale of the crawling process.

The following steps outline a typical web crawling pipeline, from discovering relevant websites and extracting visual content to filtering, processing, and preparing images and videos for multimodal model training.

Step 1. Discovering Websites and URLs

AI labs identify potential sources of visual training data using Common Crawl indexes, which help them find previously crawled webpages containing useful images and videos without discovering every URL from scratch.

They also use XML sitemaps to locate website content, curated domain lists to target relevant websites, and hyperlinks to discover additional webpages across the internet. These sources help crawlers build a large collection of URLs for acquiring visual training data.

Step 2. Building a Distributed URL Frontier

After discovering potential data sources, AI labs use a distributed URL frontier to manage and prioritize URLs for crawling. It distributes URLs across multiple crawling workers, prioritizing webpages likely to contain valuable visual content while handling duplicate URLs, domain-based scheduling, retries, and crawl politeness.

This allows labs to retrieve images and videos efficiently at large scale. Tools used for building URL frontiers include Apache Nutch, Apache StormCrawler, and URLFrontier.

Step 3. Fetching and Parsing Webpages

Once URLs are prioritized, distributed crawlers retrieve webpages using HTTP-based tools such as Scrapy, Crawlee, and aiohttp. For JavaScript-heavy websites, browser automation tools such as Playwright and Puppeteer render webpages to access dynamically loaded images, videos, and embedded media.

This process allows AI labs to collect the webpage content needed to extract visual assets and their associated text for model training.

Step 4. Extracting Images and Associated Text

After retrieving webpages, crawlers parse their HTML to extract image URLs from elements such as <img>, along with associated text such as alt text, captions, page titles, and surrounding descriptions.

Tools such as BeautifulSoup and lxml support HTML parsing, while JSON-LD and Open Graph metadata provide additional image references and contextual information. This helps AI labs collect image-text pairs that can be used to train vision-language models to associate visual content with natural language.

Step 5. Preserving Raw Crawl Data

AI labs preserve collected webpage content and metadata for downstream processing using formats such as WARC (Web ARChive). WARC files store raw web content, including HTTP responses and webpage data.

Common Crawl also provides the WAT (Web Archive Transformation) and WET (Web Extracted Text) file formats. WAT files contain extracted webpage metadata, while WET files contain extracted plaintext. These formats allow AI labs to revisit and process previously collected data, extract additional visual information, and build training datasets without repeatedly crawling the original websites.

Step 6. Downloading Images at Scale

After extracting image URLs, AI labs use distributed workers to download images concurrently through asynchronous HTTP requests, retries, and connection management. Tools such as LAION's img2dataset support large-scale image downloading, validation, and resizing.

The collected images and associated metadata can then be packaged into formats such as WebDataset and Parquet for downstream processing and model training.

Step 7. Removing Duplicate Images

After downloading images, AI labs apply deduplication to remove exact duplicates and visually similar images from their training datasets. They use URL normalization, cryptographic hashes such as SHA-256, and perceptual hashing techniques such as pHash to identify repeated images.

Image embeddings and approximate nearest-neighbour search tools such as FAISS can also detect visually similar images, even when they have different resolutions, crops, or compression levels. This reduces dataset redundancy and helps maintain visual diversity during model training.

Step 8. Filtering Images Using CLIP Similarity

After removing duplicates, AI labs use CLIP-based filtering to select high-quality image-text pairs for vision-language model training. CLIP, a multimodal AI model, maps images and their associated captions into a shared embedding space, allowing systems to measure how closely the text describes the visual content using cosine similarity. Based on the resulting similarity scores, AI labs can discard image-text pairs that fall below a predefined threshold and retain those with stronger semantic alignment.

For example, LAION-5B, an open large-scale image-text dataset, used OpenAI's CLIP ViT-B/32 model to calculate image-text similarity scores for English samples. Image-text pairs with cosine similarity scores below a threshold were discarded, while those meeting or exceeding the threshold were retained for the dataset.

Step 9. Filtering Unsafe and Low-Quality Images

AI labs apply additional filtering to ensure their visual training datasets are safe and suitable for model training. Automated classifiers detect unsafe content, graphic violence, watermarks, and personally identifiable information (PII). Image quality checks remove blurred, corrupted, or low-resolution samples, while human reviewers assess uncertain cases.

Step 10. Packaging Training-Ready Datasets

After filtering, AI labs organize images, captions, and associated metadata into training-ready datasets. Formats such as WebDataset store samples in TAR archives, enabling efficient data streaming and distributed model training. Parquet stores structured metadata, including image identifiers, captions, source URLs, dimensions, similarity scores, and filtering results. These formats make large visual datasets easier to manage, access, and process during multimodal model training.

Step 11. Extending the Pipeline to Video Data

AI labs extend the crawling pipeline to collect video data. Crawlers discover video URLs and metadata through HTML elements, embedded players, and structured webpage metadata.

Tools such as yt-dlp retrieve supported video resources, while FFmpeg and PyAV extract video frames and audio. Scene detection, speech recognition, and video captioning generate additional visual and textual information, creating training data for video understanding and multimodal models.

The 11-step pipeline enables frontier AI labs to systematically acquire, process, filter, and curate large-scale web-based visual datasets for multimodal model training.

2. Publicly Available Datasets and Open Data Sources

Publicly available datasets provide frontier AI labs with a foundation of existing visual training data, reducing the need to build every dataset from scratch. Labs further expand these collections by incorporating web-crawled media and other sources of visual data to train increasingly capable models.

The value of these datasets lies not just in their scale but in the specialized supervision they provide. Different datasets help models develop distinct capabilities, including visual recognition, object detection, segmentation, OCR, document understanding, spatial reasoning, and temporal video comprehension.

Various categories of publicly available datasets include:

1. Large-Scale Image-Text Datasets for Visual Pretraining

Large-scale image-text datasets provide the visual and linguistic knowledge required to train general-purpose vision-language models (VLMs). They contain images paired with captions, descriptions, metadata, or surrounding webpage text.

During pretraining, models learn to associate visual content with natural language, enabling them to recognize objects, scenes, activities, attributes, and relationships between visual elements.

For example, an image captioned "a red car parked beside a building" teaches the model to connect visual elements with their corresponding language.

Some publicly available large-scale image-text datasets include:

• LAION-5B: Contains 5.85 billion CLIP-filtered image-text pairs, including 2.32 billion English-language pairs. It provides large-scale, multilingual visual data covering diverse objects, scenes, concepts, and artistic styles. Its metadata includes NSFW and watermark scores, enabling additional safety and quality filtering. LAION-5B was used in research and model development involving LLaVA and Stable Diffusion.
• Conceptual Captions 3M (CC3M): Contains approximately 3 million webpage-derived image-caption pairs. It provides processed natural-language descriptions that support visual recognition and image-text learning. CC3M was used in research and model development involving PaliGemma 2 and LLaVA.
• WIT (Wikipedia-based Image Text): Contains approximately 11.5 million unique images and 37.5 million image-text examples across 108 languages. It provides multilingual, entity-rich image descriptions enriched with contextual information from Wikipedia. WIT was used in the pretraining data mixtures of Google's PaliGemma and PaliGemma 2 models.

Other notable large-scale public image-text datasets include Re-LAION-5B, DataComp CommonPool, Conceptual Captions 12M (CC12M), and LAION-COCO.

2. Object Detection, Localization, and Segmentation Datasets

Image-caption datasets generally teach models what is present in an image, but captions do not necessarily specify the exact location, boundary, or relationship of individual objects.

Object detection and segmentation datasets provide more precise annotations, such as bounding boxes, object classes, pixel-level masks, and relationships between objects. These datasets are particularly useful for training models to identify objects, count instances, locate specific regions, and understand spatial relationships.

Some publicly available object detection, localization, and segmentation datasets include:

• Microsoft COCO: Contains approximately 330K images across 80 object categories, with 1.5 million object instances annotated using bounding boxes, segmentation masks, captions, and human keypoints. COCO was used in research and model development involving Qwen3-VL (opens in a new tab) for box-based visual grounding.
• SA-1B (Segment Anything 1 Billion): Contains 11 million images and 1.1 billion class-agnostic segmentation masks. Developed using Meta's data engine, it was used to train the Segment Anything Model (SAM). It supports general-purpose object and region segmentation without predefined object categories.
• Open Images V7: Contains approximately 9 million images, 600 object detection classes, 16 million bounding boxes, and 2.8 million segmentation masks. It was used in research and model development involving Qwen3-VL and Google's PaliGemma 2.
• Objects365: Contains approximately 600K images across 365 object categories, with more than 10 million bounding boxes. It provides broad object-category coverage for object detection and localization and was used in research and model development involving Qwen3-VL for box-based visual grounding.

Other publicly available datasets for object detection, localization, and segmentation include Visual Genome, RefCOCO, RefCOCO+, and RefCOCOg.

Roboflow Universe also hosts a wide range of class-specific object detection and segmentation datasets contributed by researchers and the computer vision community, covering applications such as license plate recognition, (opens in a new tab) construction site safety (opens in a new tab), and more.

3. OCR, Documents, Charts, and Diagram Datasets

Specialized OCR and document-understanding datasets provide annotations for text recognition, spatial layout analysis, information extraction, and visual question answering. These datasets help models read text embedded in images, understand tables and charts, identify document fields, and interpret structured visual information.

Some publicly available OCR, Document, Chart, and Diagram Datasets include:

• TextVQA: Contains 28,408 images and 45,336 questions. It focuses on recognizing and interpreting text embedded in real-world images, including signs, product labels, storefronts, and notices. It supports OCR-based visual question answering and scene-text understanding.
• DocVQA: Contains more than 12K document images and 50K questions. It provides document-level question-answer annotations, OCR transcriptions, and spatial information. It specializes in document layout understanding and information extraction from scanned documents. It was used to evaluate InternVL2.
• ChartQA: Contains approximately 9.6K human-written questions and 23.1K generated questions about charts. It specializes in chart interpretation, numerical value extraction, data comparison, and mathematical reasoning. It was used to evaluate InternVL2.

Other notable publicly available OCR, document, chart, and diagram datasets include AI2D, FUNSD, SROIE, and POIE.

4. Video and Egocentric Datasets

Image datasets provide static visual examples, while video datasets allow models to learn how objects, people, and scenes change over time. Essentially, video datasets provide temporal supervision for understanding sequences of actions, recognizing human activities, identifying interactions between people and objects, and reasoning about events across multiple frames.

Some datasets also combine video with additional modalities, such as audio, depth maps, 3D skeletal information, eye gaze, and natural-language descriptions. These additional signals enable models to learn richer representations of human activities, physical interactions, and real-world events.

Some publicly available video and egocentric datasets include:

• Ego4D: Contains approximately 3,670 hours of first-person video covering daily activities across household, workplace, outdoor, and leisure environments. It includes additional modalities such as audio, eye gaze, depth, 3D meshes, stereo, and synchronized camera recordings. It specializes in egocentric perception, human-object interactions, activity forecasting, and understanding everyday tasks.
• EgoTaskQA: Provides first-person video question-answer annotations focused on understanding human activities, object states, tasks, and goals. It teaches models to identify what people are doing, which objects they are interacting with, and how their actions relate to ongoing tasks.
• Panda-70M: Contains approximately 70.7 million video-caption samples representing around 167,000 hours of video. It provides large-scale video-caption pairs covering diverse scenes and activities, supporting temporal visual-language pretraining and video-text alignment.
• NTU RGB+D 120: Contains 114,480 video samples covering 120 human action categories. It provides synchronized RGB videos, depth sequences, 3D skeletal data, and infrared recordings. It specializes in human activity recognition, movement understanding, and 3D body-motion analysis.
• STAR: Contains approximately 23K real-world video clips, 60K situated reasoning questions, and 140K situation hypergraphs. It specializes in understanding object interactions, action sequences, event prediction, and physical feasibility.

Other notable publicly available video datasets include VideoChat2-IT, LSMDC-QA, and LAION-BVD.

5. Public Multimodal Instruction and Specialized Reasoning Datasets

Image-caption datasets teach models to associate visual content with natural language, but they do not necessarily teach models how to follow instructions, answer questions, explain visual content, or perform specific reasoning tasks. Public multimodal instruction datasets provide structured examples containing images or videos, natural-language questions, instructions, and corresponding answers.

These datasets are particularly important during supervised fine-tuning (SFT), where models learn to follow user instructions and perform tasks such as visual question answering, document understanding, and temporal reasoning.

Some publicly available multimodal instruction and specialized reasoning datasets include:

• M³IT (Multi-Modal Multilingual Instruction Tuning): Contains approximately 2.4 million instances collected from 40 curated datasets, with 400 manually written task instructions covering 80 languages. It supports multilingual visual instruction following, image question answering, and diverse vision-to-text tasks.
• PixMo: Comprises seven datasets combining human annotations and synthetic examples. It covers detailed image captioning, visual question answering, point-based object localization, counting, document and chart understanding, and clock reading. It provides detailed visual descriptions and spatial annotations for image understanding and reasoning. It was used to train the Ai2's Molmo models. Qwen3-VL also incorporates publicly available PixMo pointing and counting annotations into its visual grounding data.
• Mementos: Contains 4,761 image sequences spanning daily life, robotics, and comic books, with annotations describing objects and their behaviors across sequences. It supports sequential visual reasoning by helping models understand how objects, actions, and events change across multiple frames.

3. Licensed Data and Commercial Partnerships

Licensed data partnerships enable frontier AI labs to acquire large-scale, commercially usable image, video, and 3D datasets directly from content owners, stock media platforms, production studios, and creator networks.

Rather than relying solely on publicly available internet content, AI companies negotiate access to proprietary archives and professionally curated visual libraries that are not typically accessible through conventional web crawling.

These partnerships provide access to high-quality visual data with clearer licensing rights, established provenance, and associated metadata. They also help reduce copyright and legal uncertainties while providing a more reliable foundation for commercial AI training and development. Notable partnerships include:

• Shutterstock: Shutterstock has established licensing partnerships with several major AI companies, covering its image, video, music, and 3D libraries. OpenAI (2023) signed a six-year licensing agreement with Shutterstock. Other reported partnerships include Meta, Google, Amazon, and Apple (2024), as well as LG AI Research (2022) for image generation and captioning, NVIDIA (2023) for text-to-3D development, and Reka AI (2024) for multimodal AI development.
• Getty Images: Getty Images partnered with NVIDIA (2023) through NVIDIA Picasso to develop text-to-image and text-to-video models using licensed visual content. Runway (2023) also partnered with Getty Images to develop the Runway-Getty Images Model (RGM), an enterprise video-generation model trained on licensed creative assets.
• Lionsgate: Runway and Lionsgate (2024) established a partnership to develop a customized AI model using Lionsgate's proprietary film and television library. This agreement provides access to cinematic footage for developing specialized video-generation capabilities.

4. Human-Collected and Human-Annotated Data

Human-collected and human-annotated data enables frontier AI labs to acquire high-quality visual training data that cannot be reliably obtained through ordinary web crawling. It involves recruiting people to capture images, videos, and real-world sensor recordings, followed by human annotators and domain experts adding structured labels, descriptions, and contextual information.

For example, a cooking video with detailed annotations can identify individual actions, hand-object interactions, and changes in ingredients over time. In contrast, a web-crawled image of someone cooking might show them holding a knife but cannot capture the sequence of actions or how objects and ingredients change throughout the process.

Notable methods used by frontier AI labs to collect human-generated visual data include:

• Egocentric Video Collection: Recruit participants to wear head-mounted cameras, smart glasses, or chest-mounted cameras while performing everyday activities. For example, Meta's Ego4D project brought together 13 universities and research laboratories across nine countries. More than 700 participants initially contributed over 2,200 hours of unscripted, first-person footage.
• Exocentric and Multi-Camera Video Collection: Install multiple cameras around participants to record activities from different perspectives. Combining external camera views with first-person recordings provides complementary information about body movements, object interactions, and surrounding environments.
• Human-Robot Teleoperation: Ask human operators to remotely control physical robots using controllers, motion-tracking devices, or VR headsets. These systems record synchronized camera feeds, robot joint positions, gripper movements, and human control inputs to generate demonstrations for robot learning.
• Crowdsourced Image Collection and Annotation: Distribute image annotation tasks through crowdsourcing platforms, allowing multiple contributors to label and describe images at scale. Microsoft COCO is a notable example, using crowdsourced workers to annotate images with object instances, segmentation masks, and captions.

5. Synthetic Data and AI-Generated Training Examples

Synthetic data enables frontier AI labs to expand their vision training datasets by generating artificial images, videos, captions, visual question-answer pairs, and other forms of supervision using generated graphics, generative AI models, and simulation environments.

It supplements real-world data by providing targeted training examples for visual capabilities that are difficult, expensive, or impractical to capture and annotate through conventional data collection methods.

Importantly, synthetic data does not necessarily involve generating entirely artificial images or videos. It can also involve enriching existing real-world visual data by generating captions, descriptions, questions, answers, and structured annotations. Common methods of generating synthetic data include:

1. Generating Synthetic Images and Videos Using 3D Simulations

AI labs use 3D engines and simulation environments to generate photorealistic images and videos of objects, people, vehicles, buildings, and other environments.

Tools such as NVIDIA Omniverse allow developers to construct virtual scenes with configurable 3D objects, materials, lighting, and cameras. They can then generate thousands or millions of visual examples by varying individual scene parameters.

Because these environments maintain complete information about virtual scenes, they can automatically generate ground-truth annotations alongside rendered images and videos. This enables AI labs to produce large-scale visual datasets with precise labels and supervision while significantly reducing the need for manual annotation.

For example, NVIDIA's Isaac GR00T N1, a vision-language-action foundation model for humanoid robots, was trained using a combination of internet-scale human videos, real-world robot demonstrations, and synthetic data generated using NVIDIA Omniverse.

b. Generating Synthetic Annotations Using Vision-Language Models

Frontier AI labs use existing images and videos as visual inputs, leveraging powerful language models and vision-language models (VLMs) to generate additional training annotations and supervision.

Rather than relying entirely on manual annotation, AI labs use VLMs to automatically generate synthetic captions, detailed descriptions, questions, answers, and other forms of language-based supervision for existing visual datasets.

For example, a frontier AI lab could use a teacher VLM to analyze a cooking video and generate multiple training examples, including descriptive captions, visual question-answer pairs, timestamps for specific actions, activity labels, object interactions, and descriptions of action sequences.

This approach allows a single image or video to generate multiple training examples targeting different visual understanding tasks.

Meta demonstrated this approach through its Perception Language Model (PLM) research, released on April 17, 2025. The research used Llama 3.2 to generate synthetic image captions, questions, and answers for existing visual datasets, including SA-1B and Objects365.

These synthetic annotations were then used to train the Perception Language Model, which focuses on detailed image and video understanding. Its training pipeline combines AI-generated annotations with human-labeled data and existing publicly available datasets, expanding the diversity and depth of visual supervision.

Learn more about how Roboflow supports synthetic dataset creation for vision model training.

3. Generating Variations of Real-World Images and Videos

Frontier AI labs also use generative AI models to create modified versions of existing real-world images and videos under different environmental conditions.

Rather than generating entirely new scenes, AI labs use image and video generation models to modify selected visual attributes while preserving the original scene structure, objects, and spatial relationships. This allows them to expand existing datasets with additional examples representing environmental conditions and scenarios that may be underrepresented in real-world training data.

For example, NVIDIA's Cosmos Transfer uses generative world models to transform existing autonomous driving footage into variations featuring different weather, lighting, and environmental conditions while preserving the original scene structure.

How Roboflow Labs Helps Frontier AI Labs Acquire and Prepare Visual Training Data

Roboflow Labs (opens in a new tab) helps frontier AI companies and enterprise teams collect, annotate, and prepare visual training data. It helps source specialized images and videos, coordinate human data collection and annotation, and evaluate models to identify gaps in existing datasets so that missing data can be collected to improve model performance.

Rather than simply collecting large amounts of data, Roboflow Labs focuses on gathering high-quality, task-specific visual data that helps AI models perform better in real-world environments. Its work includes:

• Custom Visual Data Collection: Roboflow Labs works with data partners, domain experts, and contributors to collect images and videos for specific objects, environments, activities, and operating conditions. This includes specialized industrial environments, unusual camera viewpoints, and real-world scenarios that are difficult to capture through web crawling.
• Human-Generated Data Through Roboflow Nebula: Roboflow Nebula (opens in a new tab)connects engineers, researchers, and domain experts with data collection projects designed to address the limitations of frontier AI models. Contributors can capture videos, create detailed annotations such as bounding boxes, segmentation masks, keypoints, and object tracks, generate multimodal questions and answers, and verify AI-generated annotations.
• Model Evaluation and Failure Analysis: Roboflow Labs helps teams evaluate vision models on real-world tasks and conditions. By analyzing model failures across object categories, lighting conditions, camera viewpoints, and occlusions, teams can identify missing training examples and collect additional data to address these weaknesses.

Challenges in Acquiring Vision Training Data for Frontier AI Labs

• High Data Acquisition and Processing Costs: Collecting, storing, processing, and annotating billions of images and videos requires substantial computational infrastructure and operational resources.
• Copyright, Licensing, and Data Provenance: Images and videos publicly available on the internet may still be protected by copyright. AI labs must establish appropriate training permissions through licensing agreements and maintain records of data sources and usage rights, which becomes increasingly difficult when datasets contain billions of samples from different sources.
• Privacy and Personally Identifiable Information (PII): Visual datasets may contain identifiable faces, vehicle registration plates, private conversations, computer screens, and other sensitive information. AI labs need privacy-aware collection pipelines that detect and obscure sensitive details and enforce access restrictions.
• Limited Data Availability: Large web-crawled datasets tend to overrepresent common objects, popular locations, and frequently occurring activities. Rare scenarios, such as unusual industrial machinery, uncommon medical conditions, and dangerous road situations, remain underrepresented.
• Annotation Accuracy and Scalability: Manual annotation becomes expensive as datasets grow and labeling requirements become more complex. Automated annotation models can accelerate the process but may introduce errors.
• Bias and Geographic Representation: Web-crawled datasets often reflect the geographic and cultural distribution of their sources, potentially overrepresenting certain countries, populations, and environments. This can limit model performance across underrepresented regions.
• Synthetic Data Quality and Distribution Mismatch: AI-generated images, videos, and simulated environments may contain unrealistic object interactions, incorrect lighting, distorted details, or physically implausible movements. Models trained extensively on such examples may struggle with real-world conditions.

Conclusion

Training frontier multimodal models requires combining multiple sources of visual information rather than relying on a single dataset or acquisition strategy. Web crawling and public datasets provide scale, commercial partnerships offer access to licensed content, human-collected data captures real-world interactions, and synthetic data expands existing training datasets with underrepresented scenarios. Roboflow Labs (opens in a new tab) supports frontier AI labs throughout this process by helping them address data gaps, improve training data quality, and develop more capable models.