Talk-to-Edit: Fine-Grained Facial Editing via Dialog

Deep generative models like GANs enable various facial editing techniques.

However, these often lack interactions with users. Thus, a recent paper proposes the first attempt towards a dialog-based facial editing framework. Here, editing is performed round by round via requests from the user and feedback from the system.

Face recognition – artistic interpretation in Hollywood CA. Image credit: YO! What Happened To Peace? via Flickr, CC BY-SA 2.0

The researchers propose to learn a vector field that describes location-specific directions and magnitudes for attribute changes in the latent space of GAN. That allows achieving more fine-grained and accurate facial editing. A large-scale visual-language dataset containing fine-grained attribute labels and textual descriptions complements the framework.

The proposed method achieves superior results with better identity preservation and smoother change compared to its counterparts.

Facial editing is an important task in vision and graphics with numerous applications. However, existing works are incapable to deliver a continuous and fine-grained editing mode (e.g., editing a slightly smiling face to a big laughing one) with natural interactions with users. In this work, we propose Talk-to-Edit, an interactive facial editing framework that performs fine-grained attribute manipulation through dialog between the user and the system. Our key insight is to model a continual "semantic field" in the GAN latent space. 1) Unlike previous works that regard the editing as traversing straight lines in the latent space, here the fine-grained editing is formulated as finding a curving trajectory that respects fine-grained attribute landscape on the semantic field. 2) The curvature at each step is location-specific and determined by the input image as well as the users' language requests. 3) To engage the users in a meaningful dialog, our system generates language feedback by considering both the user request and the current state of the semantic field.

We also contribute CelebA-Dialog, a visual-language facial editing dataset to facilitate large-scale study. Specifically, each image has manually annotated fine-grained attribute annotations as well as template-based textual descriptions in natural language. Extensive quantitative and qualitative experiments demonstrate the superiority of our framework in terms of 1) the smoothness of fine-grained editing, 2) the identity/attribute preservation, and 3) the visual photorealism and dialog fluency. Notably, user study validates that our overall system is consistently favored by around 80% of the participants. Our project page is this https URL.

Research paper: Jiang, Y., Huang, Z., Pan, X., Change Loy, C., and Liu, Z., "Talk-to-Edit: Fine-Grained Facial Editing via Dialog", 2021. Link: https://arxiv.org/abs/2109.04425