Humans as Light Bulbs

3D Human Reconstruction from Thermal Reflection

CVPR 2023

Columbia University

Columbia University

TL;DR: With a thermal camera, we leverage “invisible reflections” created by infrared radiation of humans in order to reconstruct their location, orientation, and 3D pose, even under occlusion.

Abstract

The relatively hot temperature of the human body causes people to turn into long-wave infrared light sources. Since this emitted light has a larger wavelength than visible light, many surfaces in typical scenes act as infrared mirrors with strong specular reflections. We exploit the thermal reflections of a person onto objects in order to locate their position and reconstruct their pose, even if they are not visible to a normal camera. We propose an analysis-by-synthesis framework that jointly models the objects, people, and their thermal reflections, which combines generative models with differentiable rendering of reflections. Quantitative and qualitative experiments show our approach works in highly challenging cases, such as with curved mirrors or when the person is completely unseen by a normal camera.

RGB vs. Thermal

Thermal reflection can be found in many common objects. Examples shown here include: glasses, white board, car, bowl, and mug. In addition, thermal radiation provides a robust signal of human activities regardless of illumination conditions.


Method

High-level overview of our analysis-by-synthesis framework. We sample random initializations from the latent space of pretrained generative models of humans and objects in 3D. Through a differentiable rendering process, we synthesize a reflection image of a human body on object surfaces. This synthesized reflection is compared with the observed reflection with a loss. Gradients are backpropagated through differentiable rendering and generative models to the latent variables.

Differentiable Rendering of Reflection

Differentiable Rendering of Reflection. Our method is an extension of the Soft Rasterizer to support reflection off of curved surfaces (see figure 5 in the paper). Ray direction shown reverses the physical propagation of light by Helmholtz reciprocity.

Reconstruction Results

Object: SUV

Object: Sedan

Object: Bowl

Object: Mug

Arxiv

BibTeX

@misc{liu2023humans,
      title={Humans as Light Bulbs: 3D Human Reconstruction from Thermal Reflection}, 
      author={Ruoshi Liu and Carl Vondrick},
      year={2023},
      eprint={2305.01652},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}