TaoAvatar: Real-Time Lifelike Full-Body Talking Avatars

Overview

TaoAvatar generates photorealistic, topology-consistent 3D full-body avatars from multi-view sequences. These avatars are fully controllable in pose, gesture, and expression, providing high-quality, real-time rendering with low storage requirements. They are compatible across various mobile and AR devices, including the Apple Vision Pro.

Applications

Realistic 3D full-body talking avatars have significant potential in augmented reality, with applications ranging from e-commerce live streaming to holographic communication. Despite advances, existing methods often struggle with fine-grained control of facial expressions and body movements, and they lack sufficient detail for real-time mobile device operation.

Our Approach

We present TaoAvatar, a high-fidelity, lightweight, 3DGS-based full-body talking avatar driven by various signals. Our method starts by creating a personalized clothed human parametric template that binds Gaussians to represent appearances. We pre-train a StyleUnet-based network to handle complex pose-dependent non-rigid deformation, capturing high-frequency appearance details.

To make this resource-efficient for mobile devices, we "bake" the non-rigid deformations into a lightweight MLP-based network using a distillation technique and develop blend shapes to compensate for details. Extensive experiments show that TaoAvatar achieves state-of-the-art rendering quality while running in real-time across various devices, maintaining 90 FPS on high-definition stereo devices such as the Apple Vision Pro.

Demos and Datasets

Demos on the Apple Vision Pro showcase a 3D digital human agent interacting with users through an ASR-LLM-TTS pipeline. Facial expressions and gestures are dynamically controlled by an Audio2BS model, allowing natural responses with synchronized speech, expressions, and movements.

Our dataset, TalkBody4D, contains eight multi-view image sequences captured with 59 well-calibrated RGB cameras at 20 fps, with a resolution of 3000×4000 and lengths ranging from 800 to 1000 frames. To request the dataset, please visit our HuggingFace repository, complete the required login information, and submit the corresponding request form. We will perform dual verification of your identity and grant access permissions on HuggingFace after validation.