SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces

Sumit Chaturvedi1, Mengwei Ren2, Yannick Hold-Geoffroy2, Jingyuan Liu2, Julie Dorsey1, Zhixin Shu2
1Yale University,2Adobe Research
Paper arXiv

SynthLight relights human portraits using an environment map lighting. By learning to re-render synthetic human faces, our diffusion model produces realistic illumination effects on real portrait photographs, including distinct cast shadows on the neck and natural specular highlights on the skin.

Abstract

We introduce SynthLight, a diffusion model for portrait relighting. Our approach frames image relighting as a re-rendering problem, where pixels are transformed in response to changes in environmental lighting conditions. Using a physically-based rendering engine, we synthesize a dataset to simulate this lighting-conditioned transformation with 3D head assets under varying lighting. We propose two training and inference strategies to bridge the gap between the synthetic and real image domains: (1) multi-task training that takes advantage of real human portraits without lighting labels; (2) an inference time diffusion sampling procedure based on classifier-free guidance that leverages the input portrait to better preserve details. Our method generalizes to diverse real photographs and produces realistic illumination effects, including specular highlights and cast shadows, while preserving the subject's identity. Our quantitative experiments on Light Stage data demonstrate results comparable to state-of-the-art relighting methods. Our qualitative results on in-the-wild images showcase rich and unprecedented illumination effects.

Main Results

Close-up Portraits

SynthLight relights close-up portraits by continuously rotating an environment map. Training on synthetic data helps us capture illumination effects such as specular highlights from skin, subsurface scattering in ears due to intense backlight, rim-effects in hair, hard cast-shadows, all while maintaining the identity of the original subject and the harmony of the entire composition.

Half-Body Portraits

We apply SynthLight on half-body portraits, a domain not present in our synthetic dataset, and find that it generalises surprisingly well to such portraits.

Portrait Illumination Effects

We demonstrate a variety of interesting portrait illumination effects for human portraits such as rim lighting in hair, subsurface scattering in ears, strong cast shadows and more!

Table illustrating portrait relighting effects

Generalization Capabilities

Face Paint

Our method works reliably, even when faces are painted. This is inspite the fact that our method uses only synthetic data for relighting supervision.

Miniatures and Figurines

We stress-test SynthLight by testing it on toys, figurines and non-photorealistic 3D characters, to elucidate it's generalization capabilities. We find that surprisingly, plausible shadows are cast on floor from various objects in the scene, in response to change in lighting.