REALTIME VFX and animation studio used DI4D’s PURE4D pipeline to drive the facial animation of five digital doubles and one digital character for the story mode cutscenes in Codemasters’ F1 2021. The finished game contained one hour of facial animation, making it the largest video game project that DI4D has processed so far.
From Braking Point to F1
F1 2021 has a new story mode called ‘Braking Point’ that allows players to improve their ranking in the game from Formula 2 for a chance to become Formula 1 stars. To create an authentic world off the track for the in-game cutscenes, REALTIME needed to feature believable characters that resemble real people as far as is digitally possible. To achieve this, REALTIME chose DI4D’s new pipeline, PURE4D, to track and recreate the exact facial performance of each character’s actor. PURE4D was developed for the animation of digital doubles.
The six characters include two competing drivers as the main protagonists. One is at the end of his career and the other is a young driver aiming to make a name for himself in the world of F1. Further characters include the team manager, a journalist and a driver’s wife – all five of whom are exact digital doubles of actors cast for their likeness, animated using PURE4D. A long-time F1 character, Devon Butler, was also animated in part using DI4D’s 4D facial performance data.
New Pipeline
The PURE4D pipeline works by combining facial performance data captured with DI4D’s head-mounted camera system, HMC, with the higher fidelity expression data captured with the DI4D PRO system. Both sets of data are processed using DI4D’s machine learning algorithms to increase efficiency, making it possible to use PURE4D on larger projects. F1 2021 is the first game to make full use of PURE4D’s capabilities.
All of the facial and performance data was captured in one week at Rebellion Film Studios in Didcot, near Oxford in the UK, with Audiomotion, a VFX facility specialising in performance capture and production services for film, TV and video games. The studio is an old converted paper factory with a dedicated performance-capture stage.
Expression Capture
Seated in front of the DI4D PRO system, the actors performed one minute of scripted facial expressions that produce 60 shapes. Further to these, phonetic pangrams – sentences designed to use as many consonances and sounds as possible – were captured from each actor. The actors received a video of the pangrams for reference.
It was important that they performed these facial performance sequences in character. This includes adopting the correct accent that will be used to portray the character. This makes sure that the most accurate data is captured, matching the final performance data recorded with the HMC.
DI4D PRO captures sequences of high resolution 3D facial performance data in colour, using nine synchronised 12-megapixel machine vision cameras. The process produces photoreal data from closeup shots, or can record the exact movements of facial performance. It only took the actors about 15 minutes each to capture the required DI4D PRO expression data, which they could complete during breaks in the performance capture shoots to avoid interfering with their work on-set.
Face and Body
The DI4D HMC is a wireless headset fitted with a stereo pair of 4MP cameras, designed to track accurate facial performance data simultaneously with full-body motion capture, audio recording and reference video. All of these data streams are recorded with the same timecode to maintain precise synchronisation. The complete process is referred to as performance capture and allows multiple actors to move about and interact with each other naturally during the capture session, resulting in a better performance.
For the F1 2021 performance shoots, the producers used a Vicon optical marker-based motion capture system to record full-body motion data. Audio was captured by Codemasters using a tiny lavalier mic fixed onto each DI4D HMC and connected to a recorder attached to the HMCs’ harnesses. Because each session took place on a large performance capture stage, several actors could be recorded simultaneously.
Using the DI4D HMC, REALTIME captured the exact facial performance of each actor as it would appear in the final game’s cutscene. This approach allowed the studio to drive the facial animation directly without building detailed animation rigs. The actors were free to perform their character’s dialogue and motion in the same way they would perform on-set for movie or television and REALTIME didn’t need to complete further animation in post-production.
The final level of fidelity and realism is very high. This was also the first time the HMC data had been used to track the exact movement of each actor’s eyes. Eye and jaw tracking are both new features available with PURE4D.
Machine Learning
PURE4D’s machine learning engine was used to process the DI4D HMC data efficiently. The resulting 4D facial performance data was then used to drive the DI4D PRO digital double assets, producing accurate facial animation and conveying authentic facial expressions in high fidelity.
The final F1 game required one hour of 4D facial capture data, which was considerably more than any previous project processed by DI4D. Nevertheless, by using PURE4D’s machine learning algorithms, DI4D was able to complete the post-processing for the project in only two months – while also improving the quality of the final animation.
PURE4D not only produces dense, high-fidelity 4D animation data, but it represents it in an extremely lightweight form, meaning that it can run directly inside modern game engines. This allowed REALTIME to render the final cutscenes for F1 2021 completely from Unreal Engine, and still achieve realistic, 120 fps digital human animation both in-game and during cinematics. di4d.com
