I trained a network to animate faces of anime characters. The input is an image of the character looking straight at the viewer and a pose, specified by 6 numbers. The output is another image of the character with the face posed accordingly.

What the network can do in a nutshell.

I created two tools with this network.

• One that changes facial poses by GUI manipulation: https://www.youtube.com/watch?v=kMQCERkTdO0
• One that reads a webcam feed and make a character imitates the user’s facial movement: https://www.youtube.com/watch?v=T1Gp-RxFZwU

Using a face tracker, I could transfer human face movements from existing videos to anime characters. Here are some characters impersonating President Obama:

https://reddit.com/link/e1k092/video/jqb6eziwgv041/player

The approach I took is to combine two previous works. The first is the Pumarola et al.’s 2018 GANimation paper, which I use to change the facial features (closing eyes and mouth, in particular). The second is Zhou et al.’s 2016 object rotation by appearance flow paper, which I use to rotate the face. I generated a new dataset by rendering 8,000 downloadable 3D models of anime characters.

You can find out more about the project at https://pkhungurn.github.io/talking-head-anime/.