Overview

GroundedSAM combines Grounding DINO with the Segment Anything Model to identify and segment objects in an image given text captions.

‍

This provides GroundedSAM with the ability to caption detections and segmentations. Additionally, it can segment objects based off text prompts.

‍

GroundedSAM is also a zero-shot model, which allows the model to detect images that it was not trained on. This is perfect for auto-labeling tasks. Learn how to use GroundedSAM to autolabel your dataset.

Performance

‍

Using a variety of day-to-day objects, researchers found the three versions of GroudnedSAM to be exceptional at zero-shot predictions on the data.

Use This Model

First, install Autodistill and Autodistill Grounded SAM:


pip install autodistill-grounded-sam autodistill-yolov8

Then, run:


from autodistill_grounded_sam import GroundedSAM
from autodistill.detection import CaptionOntology
from autodistill.utils import plot
import cv2

# define an ontology to map class names to our GroundedSAM prompt
# the ontology dictionary has the format {caption: class}
# where caption is the prompt sent to the base model, and class is the label that will
# be saved for that caption in the generated annotations
# then, load the model
base_model = GroundedSAM(
    ontology=CaptionOntology(
        {
            "person": "person",
            "shipping container": "shipping container",
        }
    )
)

# run inference on a single image
results = base_model.predict("logistics.jpeg")

plot(
    image=cv2.imread("logistics.jpeg"),
    classes=base_model.ontology.classes(),
    detections=results
)
# label all images in a folder called `context_images`
base_model.label("./context_images", extension=".jpeg")

Label Data Automatically with Grounded SAM

You can automatically label a dataset using Grounded SAM with help from Autodistill, an open source package for training computer vision models. You can label a folder of images automatically with only a few lines of code. Below, see our tutorials that demonstrate how to use Grounded SAM to train a computer vision model.

Grounded SAM