Projets du

Cours MVA : Modèles génératifs pour l’image

(année 2023-2024)

Master 2 MVA, ENS Paris-Saclay

• Projects by Groups of 2 students.
• Report due for March 25th:
  • 4 to 8 pages in simple column format.
  • Figures and References can be added in appendix (not counted in the 8 pages).
  • Introduction should place the context and connect with the course sessions.
  • The practical part should include experiments which are not in the original paper
    (other data, other inverse problem, relevant low-dimensional experiments, etc).
• Codes for your experiments should be sent as a zip archive (not including large database).
• Each project comes with a suggestion of 3 work tracks (which you may follow, or not).
• Defense: 20' presentation followed by 15' questions.
• Project Defense at Télécom Paris between March 25th and March 29th.

List of Projects

Project-01: A Neural-Network-Based Convex Regularizer for Image Reconstruction

• Explain convergence guarantees obtained with this regularizer.
• What is the interest of doing a multi-gradient-step instead of mono-gradient-step?
• Compare with Gradient-step Plug-and-Play (TP9).

Project-02: Demystifying MMD GANs

• Explain the differences between WGAN and MMD-GAN (for theory and training).
• Explain the gradient bias problem that may happen with GANs or WGANs.
• Train a MMD-GAN for a low-dimensional dataset (sklearn make_moons) and a large-scale dataset.

Project-03: Neural Optimal Transport

• Explain how weak optimal transport can be used for generative modeling.
• What particular problems may be expected when learning with weak optimal transport.
• Apply the proposed methodology to a synthetic 2D dataset and an image dataset.

Project-04: Plug-and-Play split Gibbs sampler: embedding deep generative priors in Bayesian inference

• In PnP split Gibbs sampling, can we use generative models with latent variables?
• Compare PnP split Gibbs sampling with Chung et al., 2023 (TP6, Exo3).
• Are there convergence guarantees for the PnP Gibbs sampling algorithm?

Project-05: Pseudoinverse-Guided Diffusion Models for Inverse Problems

• Experiments using the DDPM model used in TP6.
• Compare theoretically and experimentally with Chung et al. ICLR 2023 (TP6, Exo 3) for inpainting and Gaussian deblurring.
• Is the method stable in when the noise measurement increases for Gaussian deblurring?

Project-06: Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models

• Discuss the adequation between the algorithm and the theoretical results.
• For inpainting, does the gray color used to fill the holes has an influence on the algorithm?
• For super-resolution, compute the standard deviation of each pixels when running several times the algorithm and discuss the results.

Project-07: Denoising Diffusion Restoration Models

• Compare theoretically and experimentally with Chung et al. ICLR 2023 (TP6, Exo 3) for inpainting and Gaussian deblurring.
• Evaluate the stochastic aspect of the sampling on some example and discuss the results.
• Can you extend the model to non-linear problems?

Project-08: Consistency Models

• Train a diffusion model on a 2D toy dataset (eg sklearn make_moons), and then train a consistency model using the two different approaches (distillation of diffusion VS independent training).
• For inpainting, discuss the quality of the blending between known and unknwon pixels.
• Compare experimentally with Chung et al. 2023 (TP6).

Project-09: Convergence of SGD for Training Neural Networks with Sliced Wasserstein Losses

• Summarize the main results of the paper and explain the insight benefits with Sliced Wasserstein.
• Use Algorithm 1 to learn a generative network for a 2D continuous target distribution.
• Use the proposed algorithm to learn a generative network for a large-scale image dataset.

Project-10: Learning an Optimal Transport Brenier Map

• How can one constrain a neural network to be the gradient of a convex function?
• Use the algorithm for Brenier maps to compute a generative network for a low-dimensional example.
• Use the proposed algorithm to learn a generative network for a large-scale image dataset.

Project-11: Texture Generation using GMM Optimal Transport

• Explain the relation and differences between usual OT and GMM OT.
• Exploit the GMM OT cost to learn a generative model for a low-dimensional example.
• Use the proposed algorithm to learn a generative network for a large-scale image dataset.

Project-12: Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization

• Explain in detail the concept of Adaptive Instance Normalization and compare with simple BatchNorm.
• Implement AdaIN in order to perform style transfer between two images.
• Compare with other style transfer method (e.g. by Gatys et al.).

Project-13: Analyzing and Improving the Image Quality of StyleGAN

• Study and implement the inversion procedure for the StyleGAN network.
• Are you able to identify some image characteristics in the latent space of StyleGAN?
• Can the inversion method be used to address general imaging inverse problems?

Project-14: Posterior-Variance-Based Error Quantification for Inverse Problems in Imaging

• Can you test the error quantification method with a Total Variation (TV) regularization for image inverse problems?
• Can you test it with a deep regularization and compare the error quantification results (between TV and deep regularization)?
• How do you generalize this work for RGB images?

Project-15: Provable Probabilistic Imaging using Score-Based Generative Priors

• Can you test this reconstruction method with a pre-trained diffusion model?
• Can you compare this method with standard diffusion model sampling?
• Comment the different assumptions of the theoretical part. Can you test the optimilaty of Theorem 1 (in practice)?