Top Multimodal Vision Models

Grounding DINO is a zero-shot object detection model made by combining a Transformer-based DINO detector and grounded pre-training.

PaliGemma is a vision language model (VLM) by Google that has multimodal capabilities.

GPT-4o is OpenAI’s third major iteration of GPT-4 expanding on the capabilities of GPT-4 with Vision

CLIP (Contrastive Language-Image Pre-Training) is an impressive multimodal zero-shot image classifier that achieves impressive results in a wide range of domains with no fine-tuning. It applies the recent advancements in large-scale transformers like GPT-3 to the vision arena.

LLaVA is an open source multimodal language model that you can use for visual question answering and has limited support for object detection.

GroundedSAM combines Grounding DINO with the Segment Anything Model to identify and segment objects in an image given text captions.

CogVLM shows strong performance in Visual Question Answering (VQA) and other vision tasks.

Qwen-VL is an LMM developed by Alibaba Cloud. Qwen-VL accepts images, text, and bounding boxes as inputs. The model can output text and bounding boxes. Qwen-VL naturally supports English, Chinese, and multilingual conversation.

SigLIP is an image embedding model defined in the "Sigmoid Loss for Language Image Pre-Training" paper.

MetaCLIP is a zero-shot classification and embedding model developed by Meta AI.

The 4M model is a versatile multimodal Transformer model developed by EPFL and Apple, capable of handling a handful of vision and language tasks.

BakLLaVA is an LMM developed by LAION, Ontocord, and Skunkworks AI. BakLLaVA uses a Mistral 7B base augmented with the LLaVA 1.5 architecture.

Kosmos-2 is a multimodal language model capable of object detection and grounding text in images.

BLIPv2 is a multimodal model developed by Salesforce Research.

GPT-4 with Vision is a multimodal language model developed by OpenAI.

Gemini is a family of Large Multimodal Models (LMMs) developed by Google Deepmind focused specifically on multimodality.

Florence-2 is a lightweight vision-language model open-sourced by Microsoft under the MIT license.