Image Restoration & Enhancement for Visual Recognition (UG² / UG²+)

Benchmarking image restoration and visual recognition under severe weather, distance, and motion blur across 150K+ frames (IEEE TPAMI, CVPR UG²+).

Overview

Visual recognition models deployed on robotic platforms, unmanned aerial vehicles (UAVs), and ground vehicles encounter substantial real-world degradations, including motion blur, platform vibration, compression artifacts, distance variations, and adverse weather conditions. Traditional computational photography algorithms are typically designed to produce aesthetically pleasing images for human observers, which often smooths textures or introduces artifacts that degrade the edge features needed by deep neural networks.

Funded by IARPA with hardware support from NVIDIA, this multi-year research project investigated the relationship between computational photography and automatic visual recognition. Directed by Prof. Walter J. Scheirer at the University of Notre Dame, the project created the $\text{UG}^2$ video benchmark dataset, resulted in publications in WACV, IEEE TPAMI, and CVIU, and led to the recurring CVPR $\text{UG}^2$ and $\text{UG}^2$+ Challenge Workshop series.

Visual recognition performance across restoration algorithms under real-world motion blur and atmospheric degradation on the UG2 benchmark.

Dataset and challenge contributions

1. The $\text{UG}^2$ benchmark dataset

  • Dataset scale: We created a video benchmark specifically designed to assess how image restoration and enhancement algorithms impact automatic visual recognition models.
  • Three acquisition scenarios:
    • UAV video: Aerial captures with platform vibration, scale shifts, and oblique perspectives.
    • Glider video: High-altitude sequences with variable flight trajectories.
    • Ground collection: Vehicle-mounted camera captures recorded in rain, cloud cover, and direct sunlight.
  • Annotations: Over 289 video sequences comprising approximately 150,000 annotated object frames organized across 37 ImageNet-derived classes.
  • Superclass evaluation protocol: Defined superclasses and top-5 retrieval metrics to mitigate extreme perspective and scale ambiguity in aerial footage.

2. The CVPR $\text{UG}^2$ and $\text{UG}^2$+ Prize Challenge Workshops

We co-organized the CVPR 2018 and CVPR 2019 workshop prize challenges sponsored by IARPA ($75,000 prize purse in 2018):

  • CVPR 2018 tracks: (1) Image Enhancement for Manual Human Inspection (benchmarked via crowdsourced psychophysics and rater reaction times); (2) Image Enhancement to Improve Automatic Object Classification (24 unique algorithm submissions).
  • CVPR 2019 tracks: Expanded to $\text{UG}^2$+, introducing Track 1 focused on video object detection and classification from unconstrained mobile platforms (16 unique algorithm submissions).
  • Key experimental takeaway: Classical enhancement metrics (PSNR, SSIM, and human visual preference) do not correlate directly with machine task accuracy. Restoration algorithms that preserve feature representations or optimize end-to-end task loss consistently outperformed conventional deblurring and super-resolution techniques.

Project details

  • Role: Lead Graduate Researcher, Dataset Creator, and CVPR Workshop Co-Organizer
  • Principal Investigator: Prof. Walter J. Scheirer
  • Affiliation: Computer Vision Research Laboratory (CVRL), University of Notre Dame
  • Sponsors: Intelligence Advanced Research Projects Activity (IARPA) and NVIDIA Corporation
  • Timeline: 2016 – 2019
  • Technologies and frameworks: Python, PyTorch, Caffe, MATLAB, Deep Neural Networks (VGG-16, ResNet, Faster R-CNN, SSD, YOLO), Video Processing, Mechanical Turk API

Publications

1. WACV 2018 (Benchmark Paper)

  • Title: $\text{UG}^2$: a Video Benchmark for Assessing the Impact of Image Restoration and Enhancement on Automatic Visual Recognition
  • Authors: R. G. VidalMata*, S. Banerjee*, K. Grm, V. Štruc, W. J. Scheirer
  • Links: arXiv:1710.02909 Presentation Video (YouTube)

2. IEEE TPAMI 2020 (Journal Evaluation)

  • Title: Bridging the Gap Between Computational Photography and Visual Recognition
  • Authors: R. G. VidalMata*, S. Banerjee*, B. RichardWebster, M. Albright, P. Davalos, S. McCloskey, D. Miller, A. Tambo, S. Joshi, D. C. Cavalcanti, et al.
  • Links: arXiv:1901.09482 IEEE Xplore (doi:10.1109/TPAMI.2020.2996538)

3. Computer Vision and Image Understanding (CVIU 2020)

  • Title: Report on $\text{UG}^2$+ Challenge Track 1: Assessing Algorithms to Improve Video Object Detection and Classification from Unconstrained Mobility Platforms
  • Authors: S. Banerjee*, R. G. VidalMata*, Z. Wang, W. J. Scheirer
  • Links: arXiv:1907.11529

Code and resources