Zhu J Y, Park T, Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//Proceedings of the IEEE international conference on computer vision. 2017: 2223-2232.
论文提出Cycle GAN结构,基于unpaired data (Figure 2),学习domain X到domain Y的映射关系 (Figure 1)。
- Cycle GAN能够应用到不同任务上:style transfer, object transfiguration, attribute transfer, and photo enhancement等。
- Cycle GAN包含adversarial loss, cycle consistency loss.
- 对于某些特定任务,Cycle GAN额外包含一个identity loss.
- 论文假设两个不同domain之间存在underlying relationships,Cycle GAN (Figure 3)从一个image collections X中学习到一些特征,并将这些特征转换到另一个image collections Y上。
- 仅使用adversarial loss,无法保证生成的图像是有意义的,例如,G可能生成rubbish fool D。此外,生成图像不一定是desired。另一方面,标准的GAN过程可能会导致mode collapse问题:所有输入图像都会被映射到同一个输出图像。因此,模型引入了cycle consistent loss.
- 对于painting->photo的任务,为了保留输入- painting的颜色 (Figure 9),模型引入了identiy loss.
- Circle GAN可看作是两个auto-encoder:GF和FG。
1. Related work
- Image-to-Image Translation (CGAN)
- I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In NIPS, 2014
- P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. arXiv preprint arXiv:1611.07004, 2016.
- Unpaired Image-to-Image Translation (VAE+GAN)
- M.Y. Liu, T. Breuel, and J. Kautz. Unsupervised image-to-image translation networks. arXiv preprint arXiv:1703.00848, 2017.
- Neural Style Transfer
学习两张特定图片之间的映射,Cycle GAN学习的是两个domain之间的映射。
2. Loss
- Adsersarial Loss
- Loss
- Identity Loss
3. Implementation
- D使用70*70 PatchGAN:更少参数,能判别任意大小图像。
- 将adversarial loss从negative log likelihood改为least square loss.
- History Buffer of generated images.
- batch size 1 of scratch.
4. Experiments
- CoGAN
M.-. Liu and O. Tuzel. Coupled generative adversarial networks. In NIPS, pages 469–477, 2016. - Pixel loss + GAN
SimGAN (self-regularision loss). - Feature loss + GAN
SimGAN (vgg16 feature loss, instead of RGB loss). - BiGAN
V. Dumoulin, I. Belghazi, B. Poole, A. Lamb, M. Arjovsky, O. Mastropietro, and A. Courville. Adversarially learned inference. arXiv preprint arXiv 2016. - Pix2pix
CGAN. - AMT
- FCN score
(Cycle GAN用的是unsupervised, pix2pix用的是supervised)
- Pixel classification
- Ablation
5. Dataset
- Labels-photo: Cityscapes dataset (Figure 5)
- Map-aerial photo: Google Maps (Figure 6)
- Labels-photo: CMP Facade database (Figure 8)
- Edges-shoes: UT Zappos50K dataset (Figure 8)
- Style transfer: Flickr, WikiArt (Figure 10)
- Object transfiguration&season transfer: ImageNet, Flickr (Figure 13)
- Photo generation from paintings: Monet’s painting, Flickr (Figure 12)
- Photo enhancement:Flickr (Figure 14)
- Comparison with Gatys (Figure 15, 16)
- Failure cases (Figure 17)