(CVPR 2018 Best Paper) Taskonomy:Disentangling Task Transfer Learning

Zamir A R, Sax A, Shen W, et al. Taskonomy: Disentangling Task Transfer Learning[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 3712-3722.

1. Overview

1.1. Motivation

  • relationships among different visual tasks

In this paper

  • proposes a fully computational approach for modeling the structure of space of visual task
  • exploit it to reduce the demand for labeled data
  • self-supervised learning
  • unsupervised learning
  • meta-learning
  • multi-task learning
  • domain adaption

2. Methods

2.1. Definitions

  • γ. limited supervision budget
  • T (target). set of task want to solve
  • S (source). set of task can be trained
  • V=T∪S. task dictionary
  • T∩S. task want to solve but can play as source
  • T-T∩S. task can not trained (target only)
  • S-T∩S. (source only)

  • edge. between a group of source and target tasks, represent feasible case

  • weight. prediction of its performance
  • use these edges to estimate the globally optimal transfer policy to solve T

Four Steps

2.2. Stage I: Task-Specific Modeling

  • Encoder-Decoder

2.3. Stage II: Transfer Modeling

  • learn a readout function

  • E_s. encoder

  • D_Θ. readout function
  • f_t. gt of task t for image I.
  • the performance of D_{s→ t} is a useful metric as task affinity
  • use shallow fully convolutional network
  • combinatorial explosion of higher-oder transfer. using beam search to sample

2.4. Stage III: Ordinal Normalization using Analytic Hierarchy Process (AHP)

  • aggregating the raw losses/evaluations L_{s→ t} from transfer function into a matrix is problem. vastly different scale and live in different spaces
  • naive solution of linearly rescale each fail

2.5. Step IV: Computing the Global Texonomy

  • edge (hypergraph)

  • Boolean Integer Programming (BIP) to find global policy

2.6. Constrains

  • each target task has exactly one transfer in
  • supervision budget is not exceeded