(CVPR 2017) The more you know:Using knowledge graphs for image classification

Keyword [GSNN] [GGNN]

Marino K, Salakhutdinov R, Gupta A. The more you know: Using knowledge graphs for image classification[J]. arXiv preprint arXiv:1612.04844, 2016.

1. Overview

1.1. Motivation

  • human has ability to acquire knowledge about the world and use knowledge to reason about the visual world
  • Gated Graph Neural Network (GGNN) suffers computational issues when graph is large

In this paper

  • proposed Graph Search Neural Network (GSNN) mitigates the computational issues
  • using knowledge improves performance on image classification

1.2. Contribution

  • GSNN. incorporate potentially large knowledge graph into end-to-end system

2. Methods

2.1. GGNN

  • input. graph of N nodes
  • output. every graph node or global output

  • h_{v}^{t}. hidden state for node v at time step t

  • x_v. problem specific annotation
  • A_v. adjacency matrix of graph for node v
  • W, U. learned parameters
  • (1). Initialization
  • (2). propagation updates from adjacent nodes
  • (3-6). combine the information from adjacent nodes and current hidden state of the nodes to compute the next hidden state

  • output network (g). FC network

2.2. GSNN

  1. propagation net
  2. importance net
  3. output net
  • start with initial nodes in the graph based on the likelihood of the concept being present (faster R-CNN)
  • add adjacent node in, learn a per-node score function to estimates the importance of these nodes

  • importance network. assign target importance value to each node, correspond to gt is 1, neighbours are γ, two-hop away γ^2

  • add top P nodes to expanded set
  • at final step T, compute the per-node-output and re-order and zero-pad the outputs into the final classification net