

Grounded SAM and YOLOv8

GroundedSAM combines GroundingDINO with the Segment Anything Model to identify and segment objects in an image given text captions.

Ultralytics YOLOv8 is a convolutional neural network that supports realtime object detection, instance segmentation, and other tasks. It can be deployed to a variety of edge devices.

In this guide, we show how to train a computer vision model without labelling using these two models.

Use Grounded SAM to train a YOLOv8 Instance Segmentation model

Autodistill allows you to use state-of-the-art foundation models that know a lot about a variety of objects to label data for your project. You can then train a new model with your labeled data. This whole process uses around a dozen lines of code.

To learn more about how autodistill works, read our overview guide or watch our YouTube tutorial.

In this guide, we will show you how to use

Grounded SAM

to train a

YOLOv8

model.

To use

Grounded SAM

to train a

YOLOv8

model, we will:

1. Install autodistill
2. Create a dataset
3. Find a prompt to label images in the dataset
4. Label our data with autodistill
5. Train a new

YOLOv8

model

Let's get started!
‍

Grounded SAM

To label data with

Grounded SAM

, we will:

1. Install autodistill
2. Create a dataset
3. Find a prompt to label images in the dataset
4. Label our data with autodistill

Let's get started!
‍

Step #1: Install Dependencies

First, install the following packages:

pip install autodistill autodistill-yolov8 autodistill-grounded-sam supervision

Step #2: Create a Dataset

Before you can label a dataset, you need a dataset with which to work.

Roboflow has a few resources that can help you create a dataset for your project:

Roboflow Collect: Collect images in the background for use in your project.
Roboflow Universe: Collect images from datasets made by the Roboflow community.

You can use any folder of images you have on your local machine with Autodistill, too.

Step #3: Label Images with Grounded SAM

Autodistill has two model types:

A Base Model, which automatically labels your data, and;
A Target Model, which trains on your labeled data.

To label your dataset with a Base Model, you need to provide prompt(s) that are relevant to the classes you want to label.

Replace "example" below with the prompt you want to use. Replace "class" with the name of the class you want the prompt results to be saved as in your dataset. Also, replace the IMAGE_NAME with an image from your dataset.

The code cell below loads the base model with your prompt on the provided image, then visualizes the results.

You may need to experiment with a few prompts.


from autodistill_grounded_sam import GroundedSAM
import supervision as sv

# define an ontology to map class names to our GroundingDINO prompt
# the ontology dictionary has the format {caption: class}
# where caption is the prompt sent to the base model, and class is the label that will
# be saved for that caption in the generated annotations
# then, load the model
base_model = GroundedSAM(ontology=CaptionOntology({"shipping container": "container"}))

# label all images in a folder called `context_images`
base_model.label("./context_images", extension=".jpeg")

box_annotator = sv.BoxAnnotator()

image = cv2.imread(image_name)

classes = base_model.ontology.classes()

detections = base_model.predict(image_name)

labels = [
    f"{classes[class_id]} {confidence:0.2f}"
    for _, _, confidence, class_id, _
    in detections
]

annotated_frame = box_annotator.annotate(
    scene=image.copy(),
    detections=detections
)

sv.plot_image(annotated_frame, size=(8, 8))

Step #4: Label Data

To start labeling your images with

Grounded SAM

, run the following lines of code:
‍

base_model.label(input_folder="./images", output_folder="./dataset")

Step #5: Train a YOLOv8 Model

To train a

YOLOv8

model using your newly-labeled dataset, run the following code:
‍


from autodistill_yolov8 import YOLOv8

target_model = YOLOv8("yolov8n.pt")
target_model.train("./dataset/data.yaml", epochs=200)

# run inference on the new model
pred = target_model.predict("./dataset/valid/your-image.jpg", confidence=0.5)

print(pred)

After running this cell, you will have model weights that you can use to run inference on your new model.

Step #5: Upload Model to Roboflow (Optional)

You can deploy your trained model to Roboflow. By deploying your model to Roboflow, you can run inference on our infinitely-scalable API. As your inference demands grow, our you will continue to see high levels of performance thanks to autoscaling infrastructure that is always on.

Roboflow offers:

Fully managed, infinitely-scalable APIs for deployed models

SDKs for common deployment targets (NVIDIA Jetson, Luxonis OAK, Docker, and more)

SOC II Type 1 Compliant

Trusted by 250,000+ developers

To deploy your model to Roboflow, run the following code:


import roboflow

PROJECT_ID = ""
DATASET_VERSION = 1

roboflow.login()

rf = roboflow.Roboflow()

project = rf.workspace().project(PROJECT_ID)
project.version(DATASET_VERSION).deploy(model_type="yolov8", model_path=f"./runs/detect/train/")

Deploy your model with Roboflow

Through Roboflow, you can deploy your model to a range of targets. Below, we have listed devices to which you can deploy your model using Roboflow SDKs.

Deploy YOLOv8 Object Detection Models to Azure Virtual Machines

Deploy YOLOv8 Object Detection Models to AWS EC2

Deploy YOLOv8 Object Detection Models to GCP Compute Engine

Deploy YOLOv8 Object Detection using Docker (CPU)

Deploy YOLOv8 Object Detection Models to the NVIDIA Jetson

Deploy YOLOv8 Object Detection Models to the Raspberry Pi