Use Grounded SAM to train a Instance Segmentation model

Overview

To label data for a model with Grounded SAM, you will:

1. Install Autodistill
2. Create a dataset
3. Build a prompt to label images in the dataset
4. Label our data on Autodistill
5. Train a new model
6. (Optional) Train a model or export your data

Let's get started!

Step 1: Install Dependencies

First, install Autodistill and the required model dependencies:

pip install autodistill autodistill-grounded-sam

Step 2: Create a Dataset

Before you can label a dataset, you need a dataset with which to work.

Roboflow has a few resources that can help you create a dataset for your project:

• Roboflow Collect: Collect images in the background for use in your project.
• Roboflow Universe: Collect images from datasets made by the Roboflow community.

You can use any folder of images you have on your local machine with Autodistill, too.

Step 3: Label Images

Autodistill has two model types:

1. A Base Model, which automatically labels your data, and;
2. A Target Model, which trains on your labeled data.

To label your dataset with a Base Model, you need to provide prompt(s) that are relevant to the classes you want to label.

Replace "example" below with the prompt you want to use. Replace "class" with the name of the class you want the prompt results to be saved as in your dataset. Also, replace the IMAGE_NAME with an image from your dataset.

The code cell below loads the base model with your prompt on the provided image, then visualizes the results.

You may need to experiment with a few prompts.

from autodistill_grounded_sam import GroundedSAM
import supervision as sv

# define an ontology to map class names to our GroundingDINO prompt
# the ontology dictionary has the format {caption: class}
# where caption is the prompt sent to the base model, and class is the label that will
# be saved for that caption in the generated annotations
# then, load the model
base_model = GroundedSAM(ontology=CaptionOntology({"shipping container": "container"}))

# label all images in a folder called `context_images`
base_model.label("./context_images", extension=".jpeg")

box_annotator = sv.BoxAnnotator()

image = cv2.imread(image_name)

classes = base_model.ontology.classes()

detections = base_model.predict(image_name)

labels = [
    f"{classes[class_id]} {confidence:0.2f}"
    for _, _, confidence, class_id, _
    in detections
]

annotated_frame = box_annotator.annotate(
    scene=image.copy(),
    detections=detections
)

sv.plot_image(annotated_frame, size=(8, 8))

To start labeling your images, run the following code:

base_model.label(input_folder="./images", output_folder="./dataset")

Step 4: Train a Model

Step 5: Upload Model to Roboflow (Optional)

Fully managed, infinitely-scalable APIs for deployed models

SDKs for common deployment targets (NVIDIA Jetson, Luxonis OAK, Docker, and more)

SOC II Type 1 Compliant

Trusted by 250,000+ developers