0%

(ICCV 2017) Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks

Zhu J Y, Park T, Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//Proceedings of the IEEE international conference on computer vision. 2017: 2223-2232.

论文提出Cycle GAN结构,基于unpaired data (Figure 2),学习domain X到domain Y的映射关系 (Figure 1)

  • Cycle GAN能够应用到不同任务上:style transfer, object transfiguration, attribute transfer, and photo enhancement等。
  • Cycle GAN包含adversarial loss, cycle consistency loss.
  • 对于某些特定任务,Cycle GAN额外包含一个identity loss.




  • 论文假设两个不同domain之间存在underlying relationships,Cycle GAN (Figure 3)从一个image collections X中学习到一些特征,并将这些特征转换到另一个image collections Y上。


  • 仅使用adversarial loss,无法保证生成的图像是有意义的,例如,G可能生成rubbish fool D。此外,生成图像不一定是desired。另一方面,标准的GAN过程可能会导致mode collapse问题:所有输入图像都会被映射到同一个输出图像。因此,模型引入了cycle consistent loss.
  • 对于painting->photo的任务,为了保留输入- painting的颜色 (Figure 9),模型引入了identiy loss.


  • Circle GAN可看作是两个auto-encoder:GF和FG。

1. Related work


  • Image-to-Image Translation (CGAN)
    1. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In NIPS, 2014
    2. P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. arXiv preprint arXiv:1611.07004, 2016.
  • Unpaired Image-to-Image Translation (VAE+GAN)
    1. M.Y. Liu, T. Breuel, and J. Kautz. Unsupervised image-to-image translation networks. arXiv preprint arXiv:1703.00848, 2017.
  • Neural Style Transfer
    学习两张特定图片之间的映射,Cycle GAN学习的是两个domain之间的映射。

2. Loss


  • Adsersarial Loss


  • Loss


  • Identity Loss


3. Implementation


  • D使用70*70 PatchGAN更少参数,能判别任意大小图像
  • 将adversarial loss从negative log likelihood改为least square loss.


  • History Buffer of generated images.
  • batch size 1 of scratch.

4. Experiments


  • CoGAN
    M.-. Liu and O. Tuzel. Coupled generative adversarial networks. In NIPS, pages 469–477, 2016.
  • Pixel loss + GAN
    SimGAN (self-regularision loss).
  • Feature loss + GAN
    SimGAN (vgg16 feature loss, instead of RGB loss).
  • BiGAN
    V. Dumoulin, I. Belghazi, B. Poole, A. Lamb, M. Arjovsky, O. Mastropietro, and A. Courville. Adversarially learned inference. arXiv preprint arXiv 2016.
  • Pix2pix
    CGAN.
  • AMT


  • FCN score


(Cycle GAN用的是unsupervised, pix2pix用的是supervised)

  • Pixel classification


  • Ablation


5. Dataset


  • Labels-photo: Cityscapes dataset (Figure 5)


  • Map-aerial photo: Google Maps (Figure 6)


  • Labels-photo: CMP Facade database (Figure 8)
  • Edges-shoes: UT Zappos50K dataset (Figure 8)


  • Style transfer: Flickr, WikiArt (Figure 10)


  • Object transfiguration&season transfer: ImageNet, Flickr (Figure 13)


  • Photo generation from paintings: Monet’s painting, Flickr (Figure 12)


  • Photo enhancement:Flickr (Figure 14)


  • Comparison with Gatys (Figure 15, 16)




  • Failure cases (Figure 17)