Train, Evaluate, and Deploy Action Recognition Models with Roboflow

We're announcing support for action recognition in Roboflow, so you can train and deploy vision models that understand what happens over time in videos. An action recognition model can watch a video clip and return a timeline of the actions it detects, with a start and end time for each one. That means you can turn hours of video into a record of what happened, when it happened, and in what order.

Turn videos into insights

Roboflow helps teams build vision systems that see the world, understand what's happening, and take action. With action recognition, those systems can detect specific types of actions and record when each one starts and ends in the video. This is useful for teams that need to answer questions about physical actions, like:

• "Did a robot finish every step of its program, in the right order and within its cycle time?"
• "Did a pit crew complete each step of the stop, and which step took the longest?"
• "Did a gymnast perform every element of the routine, in sequence?"

An action recognition model helps answer those questions by watching video. From there, you can send the output to another system to analyze the actions over time or to trigger alerts and automations based on what the model finds.

See it work on real tasks

We trained action recognition models on three public datasets. In each video, the timeline you see at the top shows the ground truth, and the timeline at the bottom is the model's prediction.

Robot manipulation, two camera views

The model watches two camera views side by side and tracks six short manipulation primitives, including a regrasp, where the robot lets go of a part and picks it up again in a new position.

Robot assembly

The model splits a long robot session into pick, insert, remove, and place, without being told how many steps to expect.

Gear train assembly

The model follows all 12 steps of a gear train assembly, recorded by a person it never saw in training.

When to use action recognition models

Many of the questions operations teams ask are about what happens over time. Did the operator do every step, in order? How long did each step take? Did the robot finish its sequence, or retry? Object detection tells you what is in a frame, but a single frame can't answer these. Action recognition looks at a window of frames, so it sees the motion. It works well for:

• SOP compliance: confirm every step happened, in order, and flag the clip when one didn't
• Cycle time: time each step automatically and find the station that slows the line
• Kitting, packing, and changeovers: check the full sequence ran before the batch moves on
• Robotics: verify each step of a robot program, catch retries, and label hours of robot recordings

It also turns footage you already record into data: run a model over a week of video and you get a timestamp for every step of every cycle.

How action recognition models work

Roboflow supports two action recognition models, Cosmos 3 Edge and V-JEPA 2.1. Both look at a video in 16-second windows and label each moment in a window with one of your actions. On longer videos, the window slides along the clip, and the results are merged into one timeline.

• V-JEPA 2.1 is a video encoder from Meta that learned how scenes move from a large amount of unlabeled video. It's the smaller of the two, trains faster, and returns a confidence score for every action.
• Cosmos 3 Edge is a video language model from NVIDIA. It uses more motion and time context, which helps when two steps look alike in a single frame.

Cosmos 3 Edge vs. V-JEPA 2.1: accuracy and training time

We trained both on three public datasets and scored them on test videos the models hadn’t seen in training. The F1 score below counts a predicted action as correct when it overlaps the true action by at least half, and mIoU (mean intersection over union) measures how closely the predicted start and end times match the true ones.

V-JEPA 2.1 was more accurate on every dataset. Cosmos 3 Edge often placed the start and end of each action a bit more precisely, which can matter if you care about exact step timing. Results on your data will differ, so try both.

Training time

Here is how long our runs took:

Roboflow Training bills 2 credits per GPU hour, so the gear train run cost about 16 credits with V-JEPA 2.1. Training either model yourself means a 24 GB or larger GPU and hours to days of babysitting a run.

How to train and deploy an action recognition model

Here's how to go from raw videos to a deployed model.

Step 1: Create an Action Recognition project

In app.roboflow.com, click "New Project" and pick "Action Recognition" as the project type.

Step 2: Upload and label videos

Upload your videos, open one from "Annotate", and mark time ranges instead of drawing boxes. The editor shows the clip on top and one timeline row per action. Drag across a row from where the action starts to where it ends, and drag either edge to adjust.

Two things matter most:

• Use the camera you'll run in production. Train on clips with the same angle and lighting, and make sure each one shows the full task.
• Label a step that repeats back to back as one segment. Tightening three screws in a row is one action, not three. On the gear train data, that raised accuracy from 59.8% to 83.4%.

To go faster, run Auto Label on a batch: name the actions, describe when each one happens, and review the segments it finds.

Our gear train dataset had 82 videos from 41 people: 29 for training, 6 for validation, and 6 for testing.

Step 3: Train Cosmos 3 Edge or V-JEPA 2.1

Create a dataset version, click "Train on This Version", and pick a model. V-JEPA 2.1 is in Beta on paid plans.

We trained both on the same version: on the gear train test set, V-JEPA 2.1 reached 87.3% accuracy after about 8 hours of training, and Cosmos 3 Edge 80.0% after about 12. The "Ready to train" panel shows a cost estimate, and you can set a credit cap.

Step 4: Run your model

You can try the endpoint before you train anything. Install the SDK with pip install -U inference-sdk and send a clip to the base Cosmos 3 Edge model:

import os
from inference_sdk import InferenceHTTPClient, InferenceConfiguration

client = InferenceHTTPClient(
    api_url="https://serverless.roboflow.com",
    api_key=os.environ["ROBOFLOW_API_KEY"],
).configure(InferenceConfiguration(api_key_transport="header"))

result = client.infer_on_video(
    "assembly-clip.mp4",
    model_id="nvidia/cosmos-3-edge-action-recognition",
)

for segment in result["timeline"]:
    start = segment["start_frame_idx"] / result["source_fps"]
    end = (segment["end_frame_idx"] + 1) / result["source_fps"]
    print(f"{start:5.1f}s - {end:5.1f}s  {segment['class']}")

On the first 45 seconds of a gear train assembly, the base model has no classes of its own, so it describes what it sees:

  0.0s -  18.8s  A woman wearing a black top, blue jeans, and a face mask is standing behind a black table ...
 19.0s -  37.8s  ... She picks up small white objects from transparent containers ...
 38.0s -  44.8s  ... She carefully places a white circular object onto one of the containers ...

When your training finishes, replace model_id with your model's ID (it looks like your-project/2). Our fine-tuned model on the same clip:

  1.5s -   6.3s  place carrier
  6.0s -  13.8s  place gear bearing
 13.5s -  22.0s  place planet gear
 21.8s -  25.0s  place carrier shaft
 24.8s -  28.8s  place sun shaft
 28.5s -  31.8s  place sun gear
 32.0s -  35.8s  place sun gear bearing
 35.5s -  41.0s  place ring bear
 40.8s -  44.8s  place block 2 on block 1

Every step is named with your classes, in order, with start and end times. For live cameras, add the Action Recognition Model (opens in a new tab) block to a Workflow, or run the model on your own GPUs with Roboflow Inference (opens in a new tab).

Tips for production

• Fit the longest action in one window. With the defaults, the model sees 16 seconds at a time, so an action longer than that gets split.
• Lower the stride on live streams. The Workflow block classifies the stream in sliding windows. A smaller stride gives tighter start and end times, at the cost of more model calls.
• Check the sequence, not just each step. Compare the timeline to your SOP and send a missing or out-of-order step to a dashboard, a line alert, or your MES.

Get started

Create an Action Recognition project, label a few dozen clips, and train your first model today. Sign up for Roboflow to get started, or talk to sales about running it across your lines and sites. Learn more in the Action Recognition docs (opens in a new tab).