Get Started with the Video Labeler

The Video Labeler app provides an easy way to mark rectangular region of interest (ROI) labels, polyline ROI labels, pixel ROI labels, and scene labels in a video or image sequence. This example gets you started using the app by showing you how to:

  • Manually label an image frame from a video.

  • Automatically label across image frames using an automation algorithm.

  • Export the labeled ground truth data.

Load Unlabeled Data

Open the app and load a video of vehicles driving on a highway. Videos must be in a file format readable by VideoReader.


Alternatively, open the app from the Apps tab, under Image Processing and Computer Vision. Then, from the Load menu, load a video data source.


The app also enables you to load image sequences, with corresponding timestamps, by selecting Load > Image Sequence. The images must be readable by imread.

To load a custom data source that is readable by VideoReader or imread, see Use Custom Data Source Reader for Ground Truth Labeling.

Set Time Interval to Label

You can label the entire video or start with a portion of the video. In this example, you label a five-second time interval within the loaded video. In the text boxes below the video, enter these times in seconds:



  3. In the End Time box, type 10.

Optionally, to make adjustments to the time interval, click and drag the red interval flags.

The entire app is now set up to focus on this specific time interval. The video plays only within this interval, and labeling and automation algorithms apply only to this interval. You can change the interval at any time by moving the flags.

To expand the time interval to fill the entire playback section, click Zoom in Time Interval.

Create Label Definitions


Create ROI Labels

An ROI label is a label that corresponds to a region of interest (ROI). You can define these types of ROI labels.

ROI LabelDescriptionExample: Driving Scene
RectangleDraw rectangular ROI labels (bounding boxes) around objects.

Vehicles, pedestrians, road signs

LineDraw linear ROI labels to represent lines. To draw a polyline ROI, use two or more points.

Lane boundaries, guard rails, road curbs

Pixel labelAssign labels to pixels for semantic segmentation. You can label pixels manually using polygons, brushes, or flood fill. See Label Pixels for Semantic Segmentation.

Vehicles, road surface, trees, pavement

In this example, you define a vehicle group for labeling types of vehicles, and then create a Rectangle ROI label for a Car and a Truck.

  1. In the ROI Label Definition pane on the left, click Label.

  2. Create a Rectangle label named Car.


  4. Click OK.

    The Vehicle group name appears in the ROI Label Definition pane with the label Car created. You can move a labels to a different position or group by left-clicking and dragging the label.

  5. Add a second label. Click Label. Name the label Truck and make sure the Vehicle group is selected. Click OK.

  6. In the first video frame within the time interval, use the mouse to draw rectangular Car ROIs around the two vehicles.

Create Sublabels

A sublabel is a type of ROI label that corresponds to a parent ROI label. Each sublabel must belong to, or be a child of, a specific label defined in the ROI Label Definition pane. For example, in a driving scene, a vehicle label might have sublabels for headlights, license plates, or wheels.

Define a sublabel for headlights.

  1. In the ROI Label Definition pane on the left, click the Car label.

  2. Click Sublabel.


    The headlight sublabel appears in the ROI Label Definition pane. The sublabel is nested under the selected ROI label, Car, and has the same color as its parent label.


  4. In the ROI Label Definition pane, select the headlight sublabel.


    Draw headlight sublabels for each of the cars.

  6. Repeat the previous steps to label the headlights of the other car. To draw the labels more precisely, use the Pan, Zoom In, and Zoom Out options available from the toolstrip.


Create Attributes

An attribute provides further categorization of an ROI label or sublabel. Attributes specify additional information about a drawable label. For example, in a driving scene, attributes might include the type or color of a vehicle.

You can define these types of attributes.

Attribute TypeSample Attribute DefinitionSample Default Values

Numeric Value




Add an attribute for the vehicle type.

  1. In the ROI Label Definition pane on the left, select the Car label and click Attribute.


  3. In the List Items section, type different types of cars, such as Sedan, Hatchback, and Wagon, each on its own line. Optionally give the attribute a description, and click OK.

  4. In the first frame of the video, select a Car ROI label. In the Attributes and Sublabels pane, select the appropriate carType attribute value for that vehicle.



  1. In the ROI Label Definition pane on the left, select the headlight sublabel and click Attribute.





Create Scene Labels


Create a scene label to use in the video.

  1. In the Scene Label Definition pane on the left, click the Define new scene label button, and create a scene label named sunny. Make sure Group is set to None. Click OK.

    The Scene Label Definition pane shows the scene label definition. The scene labels that are applied to the current frame appear in the Scene Labels pane on the right. The sunny scene label is empty (white), because the scene label has not yet been applied to the frame.

  2. The entire scene is sunny, so specify to apply the sunny scene label over the entire time interval. With the sunny scene label definition still selected in the Scene Label Definition pane, select Time Interval.

  3. Click Add Label.

The sunny label now applies to all frames in the time interval.

Label Ground Truth


Label Ground Truth Manually



Label Ground Truth Using Automation Algorithm





  3. (optional) Configure the automation settings. Click Configure Automation. By default, the automation algorithm applies labels from the start of the time interval to the end. To change the direction and start time of the algorithm, choose one of the options shown in this table.

    Direction of automationRun automation fromExample

    The Import selected ROIs must be selected so that the Car labels you selected are imported into the automation session.



  6. Examine the results of running the algorithm.






Export Labeled Ground Truth




In this example, you export the labeled ground truth to the MATLAB workspace. From the app toolstrip, select Export Labels > To Workspace. The exported MATLAB variable, gTruth, is a groundTruth object.


gTruth = 

  groundTruth with properties:

          DataSource: [1×1 groundTruthDataSource]
    LabelDefinitions: [3×5 table]
           LabelData: [531×3 timetable]

Data Source

DataSource is a groundTruthDataSource object containing the path to the video and the video timestamps. Display the properties of this object.

ans = 

groundTruthDataSource for a video file with properties

    TimeStamps: [531×1 duration]

Label Definitions



ans =

  3×5 table

     Name        Type         Group      Description     Hierarchy  
    _______    _________    _________    ___________    ____________

    'Car'      Rectangle    'Vehicle'        ''         [1×1 struct]
    'Truck'    Rectangle    'Vehicle'        ''         []          
    'sunny'    Scene        'None'           ''         []  


Display the sublabel and attribute information for the Car label.

ans = 

  struct with fields:

        carType: [1×1 struct]
      headlight: [1×1 struct]
           Type: Rectangle
    Description: ''

Display information about the headlight sublabel.

ans = 

  struct with fields:

           Type: Rectangle
    Description: ''
           isOn: [1×1 struct]

Display information about the carType attribute.

ans = 

  struct with fields:

      ListItems: {3×1 cell}
    Description: ''

Label Data


Display the first few rows of the timetable. The first few timestamps indicate that no vehicles were detected and that the sunny scene label is false. These results are because this portion of the video was not labeled. Only the time interval of 5–10 seconds was labeled.

labelData = gTruth.labelData;
ans =

  8×3 timetable

       Time           Car            Truck        sunny
    __________    ____________    ____________    _____



ans =

  8×3 timetable

       Time           Car            Truck        sunny
    __________    ____________    ____________    _____



Display the bounding box positions for the vehicles at the start of the time interval. Your bounding box positions might differ from the ones shown here.

ans =

  1×4 single row vector

  415.8962   82.4737  130.8474  129.3805

ans =

  1×4 single row vector

  235.2182    1.0000  117.0611   55.3500

Save App Session



See Also



Related Topics