Video Inpainting of Complex Scenes

Video Inpainting of Complex Scenes

Alasdair Newson, Andrés Almansa, Matthieu Fradet, Yann Gousseau, Patrick Pérez

Abstract

We propose an automatic video inpainting algorithm which relies on the optimisation of a global, patch-based functional. Our algorithm is able to deal with a variety of challenging situations which naturally arise in video inpainting, such as the correct reconstruction of dynamic textures, multiple moving objects and moving background. Furthermore, we achieve this in an order of magnitude less execution time with respect to the state-of-the-art. We are also able to achieve good quality results on high definition videos. Finally, we provide specific algorithmic details to make implementation of our algorithm as easy as possible. The resulting algorithm requires no segmentation or manual input other than the definition of the inpainting mask, and can deal with a wider variety of situations than is handled by previous work.

The paper explaining this work may be found here : Download paper

Source code

The Matlab source code corresponding to this work may be downloaded here : Download source code  (new version with some bugs corrected, 06/2018)
 You will need to be able to compile mex files to use this code.
If you wish to use our code, please cite the following paper :

Video Inpainting of Complex Scenes
Alasdair Newson, Andrés Almansa, Matthieu Fradet, Yann Gousseau, Patrick Pérez
SIAM Journal of Imaging Science 2014 7:4, 1993-2019

Comparison with previous methods


The comparison of our results with those of Granados et al. may be found here : Comparisons, Granados
Miguel Granados, KwangIn Kim, James Tompkin, Oliver Grau, Jan Kautz and Christian Theobalt
How not to be seen: Object removal from videos of crowded scenes
Computer Graphics Forum (EUROGRAPHICS), 2012

The comparison of our results with those of Wexler et al. may be found here : Comparisons, Wexler
Yonatan Wexler, Eli Schechtman and Michal Irani
Space-time completion of video
IEEE. Trans. Pattern Analysis and Machine Intelligence, 29 (2007), pp. 463-476.

Video inpainting results


> Here are several examples of inpainting results in challenging situations such as moving objects, dynamic background and video textures.

Dynamic background example: Fountain, chatelet


Original

Download the video !
Download the mask (.avi file)
Download the mask (.mat file)


Our result
Inpainting result Inpainting result, occlusion border in green
Download the video ! Download the video !

Complex example (complex motion, background motion, video textures): Les loulous

Original

Download the video !
Download the mask (.avi file)
Download the mask (.mat file)


Our result
Inpainting result Inpainting result, occlusion border in green
Download the video ! Download the video !


Video texture example : Young jaws


Original

Download the video !

Our result, with texture features Our result, with texture features - border occlusion in green
Download the video ! Download the video !

Our result, without texture features Our result, without texture features - occlusion border in green
Download the video ! Download the video !

Still frame example (for the comparison of inpainted textures)

Without features With features
Without features With features


Video texture example : waves


Original

With features With features - border

Without features Without features - border