Archive notice: This article was originally published on May 27, 2019. Links and embedded videos are preserved as part of the historical record.
The use of depth maps can be very useful in many areas of VFX. Only recently, Autodesk introduced a Z-depth generator in Flame 2020. Google has now published its own method in the paper “Learning the Depths of Moving People by Watching Frozen People.” The depth-map generation is based on footage in which both the camera and people are moving.
The Training Data
For training the AI model, the Google AI team used more than 2,000 YouTube videos of the “Mannequin Challenge,” which show groups of people “frozen” in time while the camera moves through the respective scene.
Because the video content does not move apart from the camera, the team was then able to create depth maps for the entire scene, including the people, using Structure from Motion (SfM) and Multi-View Stereo (MVS) algorithms, among other things. These are set as “ground truth” when training the network. You can view the dataset including results of the Mannequin Challenge here.
Depth Prediction
The AI model can then be applied to videos in which the people now also move. Instead of creating the map separately for each individual frame of the video, the team further improved the results, among other things, with the help of motion parallax (optical flow) and a human segementation network.
The input to the AI model consists in total of an RGB image (frame x), a B/W mask of the people and an initial depth map for the regions without people, which is calculated from the motion parallax. The model outputs a complete depth map for frame x via a regression network. You can view some of the application examples here.
Further information: Go to the Google website