Texture Synthesis with
Patch Optimal Transport

Presentation   -   Papers   -   Results   -   Git   -   Source codes
DPP Acceleration   -   Multilayer

Presentation

  In this work, we propose to address exemplar-based texture synthesis with a model based on optimal transport in the patch space. More precisely, starting from a Gaussian random field at coarse scale, we propose to apply several local transformations in order to reimpose geometric features in a way which is statistically compliant to the patch distribution of the input texture. These local transformations operate on the 3x3 patches at multiple scales and are designed to solve a semi-discrete optimal transport problem in the patch space. They essentially consist of weighted nearest neighbor assignments, that are able to reimpose the empirical patch distribution of the exemplar at multiple scales.

Papers

  The texture model is described in the papers

"A Texture Synthesis Model Based on Semi-discrete Optimal Transport in Patch Space"
(Bruno Galerne, Arthur Leclaire, Julien Rabin),
SIAM Journal on Imaging Sciences, vol. 11, no. 4, pp. 2456-2493, 2018.
Hal preprint. doi.

"Semi-Discrete Optimal Transport in Patch Space for Enriching Gaussian Textures"
(Bruno Galerne, Arthur Leclaire, Julien Rabin), Geometric Science of Information (GSI).
Revised Preprint. Hal.

  The experiments shown in this paper can be reproduced using the codes available below.

Results

  Several high-resolution texture synthesis results can be seen here.

Git Repository

  This project is linked to a Git repository.

Source codes

  The Python source codes can be downloaded from the Git repository.

  The archive release.zip contains Matlab source codes for texture synthesis. See readme.txt for details about the usage.

  This archive contains the main functions

• texto_estimate.m : estimate the model from an input texture
• texto_synthesize.m : sample the texture model
• semi_discrete_ot_asgd.m : compute semi-discrete optimal transport plan
• sample_transport_plan.m : sample the optimal transport plan

  and also

• a script file texto_test.m which allows to test the main functions
• a folder textures containing input texture images
• a folder models containing pre-estimated texture models

  Besides, we provide a fast implementation based on CUDA/Matlab:

• texto_synthesize_cuda.m : sample the texture model
• patch_transport.cu : apply patch transport plan
• patch_transport_upsample.m : apply patch transport plan and upsample

  If you find some bugs or mistakes in those codes, you can report them by email to Arthur Leclaire.

DPP Acceleration

  The synthesis algorithm presented above can be accelerated by exploiting a subsampling of the target patch distribution based on determinantal point processes (DPP). This acceleration was studied in the following paper.

"Determinantal Patch Processes for Texture Synthesis"
(Claire Launay, Arthur Leclaire), Actes du GRETSI, 2019. Hal preprint.

  The archive release_dpp.zip contains the corresponding source codes. This file has the same structure that release.zip, with some additional files for the DPP simulation. The main script is texto_test_dpp.m. Some pre-estimated models are available in the textures folder. See readme.txt for details about the usage.