Keyword [Partial Convolution]
Liu G, Reda F A, Shih K J, et al. Image inpainting for irregular holes using partial convolutions[C]//Proceedings of the European Conference on Computer Vision (ECCV). 2018: 85-100.
1. Overview
1.1. Motivation
- Conv responses conditioned on both valid pixels as well as the substitute values (mean value) in masked holes often lead to artifacts such as color discrepancy and blurriness
- post-processing used to reduce artifacts, but expensive and may fail
- previous methods focus on rectanglar regions
In this paper, it proposed partial convolutionss
- convolution is masked and renormalized to be conditioned on only valid pixels
- generate update mask for the next layer
- focus on irregular mask
1.2. Contribution
- partial Conv (PConv)
- replace Conv with PConv get state-of-art
- first on irregularly shaped holes
- proposed a large irregular mask dataset
1.3. Related Work
1.3.1. Non-learning Approach
- neighbouring pixel. can only handle narrow holes, where color and texture variance is small
- patch based. PatchMatch (faster similar patch searching algorithm)
1.3.2. Deep Learning Based
- initialize the holes with some constant placeholder values (mean pixel value of ImageNet)
- post-processing
- replace post-processing with refinement network
- ignore the mask placeholder values
- search closest encoding in a latent space
1.4. Dataset
- ImageNet
- Places2
- CelebA-HQ
1.4.1. Augmentation
- dilation, rotation, cropping
- masks with and without holes close to border
1.5. Do Inpainting on SR
extension experiments.
- offset pixels and insert holes (scaling factor k)
2. Methods
2.1. Partial Convolution (PConv)
- M. binary mask
- 1/sum(M). scaling factor to adjust for the varying amount of valid (unmasked) inputs
After each PConv, update the mask
- mask will eventually be all ones
2.2. Network
- U-Net (8 encoder + 8 decoder)…
- No padding
2.3. Loss Function
2.3.1. Total Loss
2.3.2. Pixel Loss
2.3.3. Perceptual Loss
- I_out. output image
- I_comp. I_out of hole + Input of non-hole
- ψ_n. nth layer of VGG16
2.3.4. Style Loss
- K_n. 1/chw
2.3.5. Total Variation (TV) Loss
- P. region of 1-pixel dilation of the hole region
- smoothing penalty. deal with the checkerboard artifacts of perceptual loss
3. Experiments
3.1. Comparison
