The relatively hot temperature of the human body causes people to turn into long-wave infrared light sources. Since this emitted light has a larger wavelength than visible light, many surfaces in typical scenes act as infrared mirrors with strong specular reflections. We exploit the thermal reflections of a person onto objects in order to locate their position and reconstruct their pose, even if they are not visible to a normal camera. We propose an analysis-by-synthesis framework that jointly models the objects, people, and their thermal reflections, which combines generative models with differentiable rendering of reflections. Quantitative and qualitative experiments show our approach works in highly challenging cases, such as with curved mirrors or when the person is completely unseen by a normal camera.
Thermal reflection can be found in many common objects. Examples shown here include: glasses, white board, car, bowl, and mug. In addition, thermal radiation provides a robust signal of human activities regardless of illumination conditions.
High-level overview of our analysis-by-synthesis framework. We sample random initializations from the latent space of pretrained generative models of humans and objects in 3D. Through a differentiable rendering process, we synthesize a reflection image of a human body on object surfaces. This synthesized reflection is compared with the observed reflection with a loss. Gradients are backpropagated through differentiable rendering and generative models to the latent variables.
Differentiable Rendering of Reflection. Our method is an extension of the Soft Rasterizer to support reflection off of curved surfaces (see figure 5 in the paper). Ray direction shown reverses the physical propagation of light by Helmholtz reciprocity.
@misc{liu2023humans,
title={Humans as Light Bulbs: 3D Human Reconstruction from Thermal Reflection},
author={Ruoshi Liu and Carl Vondrick},
year={2023},
eprint={2305.01652},
archivePrefix={arXiv},
primaryClass={cs.CV}
}