Controllable Radiance Fields for Dynamic Face Synthesis.

3DV 2022


Peiye Zhuang1,3,4,    Liqian Ma2,    Sanmi Koyejo1,3,4,    Alexander G. Schwing1,   


1University of Illinois, Urbana-Champaign   
2ZMO AI Inc. 3Stanford University 4Google Inc.



Recent work on 3D-aware image synthesis has achieved compelling results using advances in neural rendering. However, 3D-aware synthesis of face dynamics hasn't received much attention. Here, we study how to explicitly control generative model synthesis of face dynamics exhibiting non-rigid motion (e.g., facial expression change), while simultaneously ensuring 3D-awareness. For this we propose a Controllable Radiance Field (CoRF): 1) Motion control is achieved by embedding motion features within the layered latent motion space of a style-based generator; 2) To ensure consistency of background, motion features and subject-specific attributes such as lighting, texture, shapes, albedo, and identity, a face parsing net, a head regressor and an identity encoder are incorporated. On head image/video data we show that CoRFs are 3D-aware while enabling editing of identity, viewing directions, and motion.


Overview


Controllable free-view dynamic head synthesis. Each row presents an identity sampled from a prior distribution and two expressions guided by a reference image (bottom left), viewed from multiple directions (column 1-3 and 4-6).



Pipeline




Videos


The following videos present the multi-view video generation of 3D content generated by CoRF.


Additional human body results


Human body motion control with novel motion representation. We generate 9 identities rendered in 3 views (columns 1-3; 4-6; 7-9) at a 256X256 resolution conditioning on 3 different motion representations (row 1-3) that are randomly sampled weights of the PCA bases as motion representation.



Bibtex

						
			@inproceedings{zhuang2022controllable,
				title = {Controllable Radiance Fields for Dynamic Face Synthesis},
				author = {Zhuang, Peiye and 
					  Ma, Liqian and 
					  Koyejo, Oluwasanmi and 
					  Schwing, Alexander},
				booktitle = {Proc. 3DV},
				year = {2022},
			}
						
					

Acknowledgements

Some of the work was completed while P.Z. was at Google. This work is supported in part by NSF under Grants 1718221, 2008387, 2045586, 2106825, 1934986, MRI #1725729, and NIFA award 2020-67021-32799. S.K. was supported by Google.