Joint Visual-Temporal Embedding for Unsupervised Action Segmentation

Unsupervised action segmentation in untrimmed video sequences using joint visual-temporal embeddings (WACV 2021, MIT-IBM Watson AI Lab).

The unsupervised action segmentation pipeline: (1) Frame-level joint visual-temporal feature embedding, (2) Gaussian Mixture Model (GMM) clustering into k sub-activity classes, and (3) Viterbi decoding with length modeling to generate continuous, ordered action segmentations.

Overview

Temporal action segmentation in untrimmed videos is important for procedural understanding, surveillance, and robotics. While supervised methods achieve high accuracy, they require dense, frame-level temporal boundary annotations that are labor intensive to collect.

Weakly supervised approaches rely on accompanying transcripts or video captions, but unconstrained video data rarely includes descriptive metadata. Early unsupervised methods relied on temporal regularizers (such as relative timestamp regression), but often neglected visual appearance, leading to misclassifications when different sub-actions occurred at similar relative points in time.

Developed during an internship at the MIT-IBM Watson AI Lab and continued as doctoral research at the University of Notre Dame, this project proposed a joint visual-temporal embedding for unsupervised action segmentation. Published at the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV 2021), the framework combines a predictive U-Net visual embedding with a continuous temporal timestamp discriminator to capture both visual appearance and chronological order without manual labels.

Architecture and joint embedding

Given a set of untrimmed videos $\mathcal{D}A = {v_i}{i=1}^V$ depicting a multi-step task $A$, the objective is to learn the sequence of $k$ sub-activities ${C_1, \dots, C_k}$ and assign a sub-activity label to every video frame $f_t \in v_i$ without supervision.

Two-stage joint training framework. (a) Stage 1: Standalone pre-training of the U-Net with Visual Loss and the MLP with Temporal Timestamp Loss. (b) Stage 2: Freeze-unfreeze iterative joint training: optimizing the U-Net with Visual Prediction + Temporal Reconstruction Loss while the MLP is frozen, followed by refining the MLP discriminator while the U-Net is frozen.

1. Predictive U-Net visual encoder

Rather than using a standard autoencoder to reconstruct the current frame, the network predicts future frame features:

  • Future prediction ($s=1$): Given frame features $f_t$ at timestamp $t$, the network predicts the representation of future frame $\hat{f}_{t+s}$. Shifting the target forward prevents trivial identity mappings and encourages the network to learn transition dynamics between actions.
  • U-Net structure: Consists of three convolutional down-sampling layers and three up-sampling layers connected by skip connections. The bottleneck feature $x_t$ from the final down-sampling layer serves as the joint visual-temporal representation for frame $f_t$.

  • Visual prediction loss:

    \[\mathcal{L}_{vis} = \frac{1}{N} \sum_{n=1}^N \left(\hat{f}_{n+s} - f_{n+s}\right)^2\]

2. Continuous temporal discriminator

To maintain temporal order in the embedding, we integrate an auxiliary temporal discriminator:

  • Timestamp predictor: A 3-layer Multilayer Perceptron (MLP) trained to predict the normalized relative timestamp $T(f_t) \in [0, 1]$ of each frame:

    \[\mathcal{L}_{MLP} = \frac{1}{N} \sum_{n=1}^N \left(T(f_n) - t_n\right)^2\]

  • Temporal Quality ($TQ$) loss: Evaluates the difference between the predicted frame’s estimated timestamp $T(\hat{f}{t+s})$ and the ground-truth frame’s timestamp $T(f{t+s})$:

    \[\mathcal{L}_{TQ} = \frac{1}{N} \sum_{n=1}^N \left(T(\hat{f}_{n+s}) - T(f_{n+s})\right)^2\]

  • Combined U-Net loss:

    \[\mathcal{L}_{UNet} = \mathcal{L}_{vis} + \mathcal{L}_{TQ}\]

3. Freeze-unfreeze training protocol

Joint end-to-end training of the U-Net and MLP discriminator is prone to instability: because the MLP has fewer parameters than the convolutional U-Net, it overfits quickly when updated simultaneously. We used a freeze-unfreeze training scheme:

  • Stage 1 (Standalone training): Pre-train the U-Net with $\mathcal{L}{vis}$ and the MLP with $\mathcal{L}{MLP}$ until initial convergence.
  • Stage 2 (Iterative refinement):
    • Freeze the MLP discriminator and train the U-Net with $\mathcal{L}_{UNet}$ for 20 epochs.
    • Freeze the U-Net encoder and train the MLP discriminator on observed and predicted timestamps for 5 epochs.
    • Repeat until global convergence.

Sub-activity clustering and Viterbi decoding

After extracting the embeddings $x_t$, frames are segmented into sub-actions using clustering and temporal decoding:

+------------------+      +-------------------+      +----------------------+
|  Joint Embeddings | ---> |   Global GMM      | ---> |  Viterbi Decoding    | ---> Sub-Action
|      x_t         |      | Cluster Modeling  |      | with Length Modeling |      Segments
+------------------+      +-------------------+      +----------------------+

1. Gaussian Mixture Model (GMM) clustering

  • Initial cluster centers are established via K-Means across all task videos.
  • A global Gaussian Mixture Model is fitted across the task feature space, outputting a $k$-dimensional probability vector $(p_{t,1}, \dots, p_{t,k})$ for each frame $f_t$.
  • Global GMM fitting handled class imbalances better than per-cluster K-GMMs, especially for activities where two or three dominant sub-actions account for over 50% of total frames.

2. Viterbi decoding with length modeling

  • To avoid noisy frame-by-frame assignments, we applied Viterbi decoding with an explicit action length model.
  • Constrains frame labeling to follow a fixed topological order $(c_1, \dots, c_k)$ determined by the mean timestamps of each cluster.
  • Frame $f_t$ either remains in its current cluster $c_i$ or advances to the next cluster $c_j$.

3. Sampling stride ($\gamma$) adjustment

  • For continuous, densely annotated actions (Breakfast Actions), a fine frame stride ($\gamma = 1$) preserved short sub-action transitions.
  • For unconstrained web videos with background frames (YouTube Instructions), a wider stride ($\gamma \in [70, 200]$) prevented background noise from fragmenting the Viterbi sequence.

Datasets and evaluation metrics

We evaluated the framework on standard video segmentation benchmarks:

Datasets

  • Breakfast Actions: 70 hours of video depicting 10 cooking tasks across 1,712 video sequences performed by 52 actors, divided into 48 sub-activities. Densely labeled without intermediate background classes.
  • INRIA YouTube Instructional Videos (YTI): 5 instructional tasks (making coffee, changing a tire, CPR, jumping a car, potting a plant) divided into 47 sub-activities. Videos include camera cuts, viewpoint changes, and unannotated background segments.
  • 50 Salads: Fine-grained culinary preparation sequences with dense temporal annotations.

Metrics

  • Global Hungarian matching: Unsupervised clusters are mapped to ground-truth labels using one-to-one Hungarian bipartite matching computed across all videos of a given activity simultaneously.
  • Mean over Frames (MoF): Percentage of correctly labeled frames over the task (evaluates dominant action accuracy).
  • Intersection over Union (IoU): Mean Jaccard index across all sub-activity classes (evaluates accuracy on less frequent sub-actions).

Experimental findings and observations

1. Value of joint embedding

Ablation experiments showed that training with either visual loss or temporal loss alone resulted in lower segmentation accuracy:

  • The visual U-Net alone captured appearance but lacked consistent temporal progression.
  • The temporal MLP alone enforced chronological ordering but struggled to differentiate visually distinct steps that occurred at similar relative timestamps.
  • The joint visual-temporal space achieved the best trade-off between MoF and IoU.

2. Effect of future frame prediction step size

  • When trained to reconstruct the contemporaneous frame ($s=0$), MoF increased while IoU dropped significantly, as the model produced overpopulated clusters dominated by one or two actions.
  • Predicting the next frame ($s=1$) balanced cluster sizes and improved IoU across sub-actions.
  • Larger step sizes ($s > 1$) degraded future frame reconstruction quality.

3. Comparison with baselines

  • Breakfast Actions: Achieved a 4% improvement in MoF over prior unsupervised methods (Kukleva et al., Sener et al.).
  • Comparison to weakly supervised methods: Performed competitively against methods requiring ordered transcript supervision, showing that predictive visual-temporal cues can provide meaningful segmentations without text annotations.

Project details and publications

  • Role: Graduate Research Intern (MIT-IBM Watson AI Lab) and Lead PhD Researcher (University of Notre Dame)
  • Collaborators and Mentors: Dr. Hilde Kuehne, Prof. Walter J. Scheirer, Dr. David D. Cox, Anna Kukleva
  • Affiliation: MIT-IBM Watson AI Lab and Computer Vision Research Laboratory (CVRL), University of Notre Dame
  • Recognition: Awarded “Best of Show” at the IBM Research Intern Showcase out of 50+ research intern projects.

Publication

  • Title: Joint Visual-Temporal Embedding for Unsupervised Learning of Actions in Untrimmed Sequences
  • Authors: Rosaura G. VidalMata, Walter J. Scheirer, Anna Kukleva, David Cox, Hilde Kuehne
  • Venue: IEEE/CVF Winter Conference on Applications of Computer Vision (WACV 2021)
  • Links: arXiv:2001.11122 Presentation on YouTube