By Sean Young
‘Malaria Must Die, So Millions Can Live’ is a global campaign from the charity Malaria No More UK, supported by a number of organisations and celebrities who have joined the fight against malaria and want to encourage strong political action to permanently end the disease.
RSA Films Amsterdam produced a short film for the campaign that features former football champion and celebrity David Beckham. In the short, an older version of David Beckham is making a speech on the day malaria has ended, sending a hopeful message from decades into the future. As he speaks, the years fall away, leaving the younger version of today to make his final statement – a malaria-free future is possible in our lifetimes, but only if we keep up the fight.
Charlatan
Artists from Digital Domain used a combination of traditional VFX techniques and their proprietary facial manipulation system called Charlatan to create the ageing and de-aging effects – without performing a 3D scan. The transformation process required performance clips of both David and an older stand-in actor delivering the speech. These reference materials were then fed into Charlatan, which uses machine learning and video footage to produce realistic digidoubles.
Digital Media World spoke to Digital Domain’s Executive Creative Director Kevin Lau about why Charlatan was an especially good match for this project, and how the combination of machine learning and video can take the place of a 3D scan and modelling.
“In this case, our client needed a very specific output, and the time constraints were very tight,” he said. “Normally, you would want an accurate model of the hero subject and your goal would be a sculpted, art directed, digitally aged or de-aged version of the actor – similar to the process used for ‘The Curious Case of Benjamin Button’. That model would then be used as our new target, and its actions would be driven by our subject’s performance seen through Masquerade, our facial performance software.
“The newly rendered CG head would be composited back onto the performance and integrated into the original plate. But since this is a complex and costly endeavour, it’s usually reserved for higher budget motion pictures where you can spread the cost over many shots throughout the production.”
Mapping Youth to Age
Newer machine learning techniques are changing this whole process. In many cases, Charlatan’s method of ageing/de-aging is significantly faster than a full CG approach, and can give higher quality results. For Digital Domain, one of its best features is how well it retains the subtlety and nuance of the actor’s original performance.
Kevin said, “For this project, our ML process analysed our target (Beckham) and learned his performance mannerisms. Our older, secondary subject mimicked the performance live but, like any human performance, it was not frame accurate. That’s where machine learning was able to help. Charlatan was able to take these two faces – Beckham and the older stand in – create a mapping between them, and generate a variety of different characteristics that represented each actor.
“Once we had the mapping, we were able to feed in Beckham’s performance, then adjust the different characteristics, increasing the characteristics related to the age of the stand in, while keeping characteristics of bone structure, likeness and performance. It synthesised an entirely new image that kept all the features and the performance that were uniquely Beckham, while adding traits tied to an older age.”
Focus on Performance
The system was designed so that it can be trained completely through principal photography, allowing David to film his scenes normally, without an intensive data capture shoot. This non-invasive approach – that is, keeping the on-set impact very minimal – is important because it means that, within certain limitations, the focus can be entirely on the performance. Scenes can play out naturally without worrying if the technical aspects will interfere or cause distraction.
When work on a scene becomes highly technical, it’s very easy for the focus to shift from the creative side to the execution. “By combining ML techniques with traditional VFX artistry in post, we can truly step back, let directors do their thing and let the performances shine,” said Kevin. “In fact, we only had a limited time with David.
“The traditional workflow – a lengthy scanning process or applying tracking dots to the talent’s face – is demanding and would have been too cumbersome and time consuming. Charlatan was able to capture all the data we needed. Ideally, you will have a controlled environment so that your subject and targets will be shot in similar conditions. Creating favourable scenarios and getting solid input data for the ML process creates a solid foundation, and sets up everything else to succeed.”
As the Charlatan process is based on the actual performance, the team did not need any special equipment from a production standpoint, nor did the older stand-in actually look much like David at all. Instead, he was chosen for key characteristics like laugh lines and crow's feet. His overall proportions were similar, but he was not someone you might mistake for David Beckham. The delivery of the performance was not the main concern – they only needed it to be generally timed out, and the performance position and timing were then conformed to the hero take of David.
Automation with Control
Charlatan’s main advantage is giving artists a head start on such projects, intelligently merging the performances of the two subjects to automatically project what an older David Beckham would look like, giving the same speech. Key age characteristics like skin motion and specific wrinkles were woven into the likeness, while keeping all of David’s unique attributes that make him recognisable. The automation took months of sculpting time off the process, leaving the team with a natural-looking nose, laugh lines and other hard-to-animate features nearly finished before traditional VFX techniques were needed.
“As an artist, what you are really looking for is control. Details matter when it comes to faces, especially when your subject is recognised all over the world,” said Dan Bartolucci, VFX supervisor at Digital Domain. “We could see that blending in attributes from an older face was still going to give our team the nuances they needed.”
Kevin agreed, having found that with many machine learning techniques, control is a delicate balancing act. In this case, it proceeds from the mapping of facial features from the two faces, described above. “Once the mapping is established, we can take the source performance – Beckham’s in this case – and shift the weighting of different characteristics to keep the source actor’s face shape, while introducing the age characteristics of the second actor," he said.
“Obviously, it’s much more complex under the hood, but using universal knowledge of human features and cross referencing that with new input, we are able to create new hybrid performances that could only exist in the computer. Within Charlatan, ageing/de-aging encompasses a variety of different attributes that can be controlled when changing the appearance of one subject to another. You could, for example, focus more on one actor's face shape while keeping the skin details of another. Many of these attributes can be dialed in interactively to see the results quickly.
“To help refine this process, you could add more input, for example. So, for instance, if we wanted to have more complex or different features, we would want to cast and record additional stand-ins and feed more input into the system.”
Artistry Masters
Nevertheless, artistry continues to play a huge role to achieve the expected polish. With Charlatan’s high-resolution base to work from, artists began applying a human touch to the face, defining key parts of the ageing process with the help of the traditional techniques like matte painting and compositing that Digital Domain has mastered over the years. Because ageing cues are so subjective, artists need access to controls over the hair, skin and beard to make the transformation believable and allow them to truly art direct the vision of David in the future.
“The initial elements of David were supplied as ‘film’, and the output was an image sequence of the new hybrid face that was conformed to the original plate. Flame artists then took those elements and combined them with additional matte painted elements that were combined for the final look of the film. In other words, Charlatan is just one weapon in our greater arsenal of VFX tools.”
Without having to create or work with 3D geometry, they turned out a final asset in under eight weeks. Kevin reflected, “In a perfect world, Charlatan would spit out exactly what an artist was envisioning. In this case, it did a great job of giving us an image of how an older version of David would look, based on of his current form along with the stand-in, but in reality, the artistic interpretation of what he would look like depends on so many opinions – from director to agency to David himself – that we needed to employ more tricks to create something that was art directable.
“With other scenarios, such as straight face swaps or even driving a fully CG generated character with a live performance, the result would be closer to a one-to-one output. But in the end, each opportunity to deploy this type of process has unique constraints, and requires techniques that would support the final output. It’s really exciting to be on the cutting edge, developing these new visualisation tools. Over time, they are only going to enhance creative storytelling more.” www.digitaldomain.com
