Using Label Studio to display labels for videos

I was recently tasked to present some form of classifier outputs to my stakeholders. Typically, when it is difficult to present the findings to stakeholders without some kind of visualization tools. Showing them the console outputs doesn’t make much sense either.

I need some kind of visualization tools that could help to playback the video as well as synchronise the classifier outputs as the video is playing. After doing some research, I came across two tools: i) Label Studio, ii) sed_vis and iii) Temporal Event Annotator.

Issue with (ii) is that the interface is kinda laggy and you can’t really seek to any particular timestep without the app crashing. Issue with (iii) is that I couldn’t easily find a way to insert the annotation in.

So therein lies LabelStudio. I had already been messing around with LabelStudio for annotation just did’t have time to explore further. Also, it seems that the annotation template has a wide variety, is able to support video and audio and has the ability to import pre-annotations.

Start Label Studio

Follow their guide to run Label Studio locally. There are various methods including docker run, docker-compose and pip install etc.

# Command to start Label Studio
LABEL_STUDIO_LOCAL_FILES_SERVING_ENABLED=true LABEL_STUDIO_LOCAL_FILES_DOCUMENT_ROOT=<specify_root_folder> label-studio

For docker-compose, include the following environment variables.

LABEL_STUDIO_LOCAL_FILES_SERVING_ENABLED=true
LABEL_STUDIO_LOCAL_FILES_DOCUMENT_ROOT=/home/user (or 
LABEL_STUDIO_LOCAL_FILES_DOCUMENT_ROOT=C:\\data\\media for Windows).

Specify a labelling template

<View>
    <Video name="video" value="$video_url"/>
    <Labels name="label" toName="audio" choice="single">
        <Label value="Neutral" background="#ddd5d5"/>
        <Label value="Happy" background="#d7ff0f"/>
        <Label value="Sad" background="#6b9cff"/>
        <Label value="Angry" background="#f40b0b"/>
    </Labels>
    <AudioPlus name="audio" value="$video_url" speed="false" zoom="false"/>
</View>

Upload pre-annotations via their UI.

The video_url can be some hosted URL for your media file. You can also upload via their UI and get the corresponding URL.

[
  {
    "id": 1,
    "annotations": [
      {
        "result": [
          {
            "original_length": 1501.9014965986394,
            "value": {
              "start": 124.99180231346867,
              "end": 184.9878674239336,
              "labels": ["Neutral"]
            },
            "from_name": "label",
            "to_name": "audio",
            "type": "labels"
          },
        ],
        "prediction": {}
      }
    ],
    "data": {
      "video_url": "/data/local-files/?d=<path_to_file>"
    }
  }
]