The Digital Human Group at Digital Domain talks about using new developments in their Masquerade software, from machine learning to a streamlined workflow, to transform actors to vampires.
Marvel's most recent antihero to reach movie screens, Morbius is a character for audiences to both love and fear. Dangerously ill with a rare blood disorder and determined to save himself and other sufferers, Dr Michael Morbius undertakes a series of experiments that casts his lot with the vampire bats of Costa Rica. His work seems to succeed and he survives but soon realises he has only become a vampire himself. Inadvertently pulling his friend Milo along with him, he is forced to struggle endlessly between the good and evil within himself.
Morbius’ dilemma led the filmmakers and actor to consider how to express the emotions of his tortured human existence, while visualising alongside that the monstrous look of the vampire inside. To find the necessary authenticity, realism and consistency for the project, the production chose to work with the digital human experts at Digital Domain.
Digital Media World had a chance to learn about this project directly from Facial Modeling Supervisor Ron Miller, Facial Rigging Lead Ricky Cloudsdale and Animation Director Phil Cramer from Digital Domain’s Digital Human Group (DHG).
The first minutes of the team's breakdown reel below show the team's work in detail:
Stepping Toward the Dark Side
Their team began its work on 'Morbius' by designing the final look for each character’s vampiric forms, with the goal of creating a photorealistic, monstrous appearance without restricting the actors with prosthetics or requiring them to spend hours in a makeup chair each day. They had a rough idea of how Morbius should look because of his comic bookorigins, but transferring and finalising that look for the filmmakers required countless tweaks -- from the jagged teeth to the pigmentation of the skin. Milo as a vampire, on the other hand, was a new character, an original design that Digital Domain created for the actor from scratch.
Shooting sequences and incorporating them with the ideal facial and body performances was a straightforward process carried out in steps. Once the essential look of each vampire was determined, the actors first performed their scenes live on set wearing facial capture markers. Independently, each performer also recorded a full range of facial expressions and movements, in order to create a complete facial geometry for each subject.
The next step involved the director. “Unlike most VFX shows, for Morbius we captured most of the facial data separate from the body performances,” Animation Director Phil Cramer said. “This was mainly because the character of Morbius would constantly revert back to his human form. We ended up holding multiple facial-focused capture sessions throughout the production, which allowed both the actor Jared Leto and the director Daniel Espinosa to fine tune performances, adjust lines and ensure the character's vampiric moments blended seamlessly with his live action performances. This work was also useful because the character's design was not finalised before principal photography.”
With the facial data secured, Digital Domain then employed its facial capture system Masquerade, a proprietary tool designed to produce extremely high fidelity facial animation at speed, without limiting the performers’ on-set movements. From there, artists were able to turn the high-resolution face model into a high-resolution, photorealistic digital character.
Masquerade’s Learning Curve
For Morbius, advances made to Masquerade for version 2.0 allowed the team to record nearly double the geometry resolution and details used to create Thanos in the Avengers films completed earlier. By the time Masquerade 2.0 was released in 2020, the developers wanted to be able to let their custom machine learning processes take over and automatically determine where the markers are, track them through space and produce a very accurate moving mesh of that performance in every frame.
“In particular, one of DHG’s primary goals continues to be minimising the shooting requirements for clients,” said Facial Modeling Supervisor Ron Miller. “For instance, with Masquerade 2.0, we no longer need to capture 4D as training data from outside vendors, because we are now able to generate it in-house from an iterative process using the Head Mounted Camera system data. Using different combinations of techniques, the overall stabilisation has improved, and we also developed a way to automatically generate training data to use for marker tracking.”
The error detection process to automatically remove bad data has improved, and their HMC calibration technique is more robust. The Masquerade pipeline uses a 1536 x 2048 8-bit image sequence recorded from the two cameras on the head mounted camera rig. Synchronized witness cameras are also set up around the set for extra facial animation data, if needed, to fill in any blanks though it isn’t needed during the machine learning process.
Level of Accuracy
“All of this has resulted in a simplified, less costly shoot process for clients. But in fact, the biggest advantage Masquerade offers is the level of accuracy for the actor's performance coming through to the target asset. This gives the artists a great starting point to work from, so they are not beginning from scratch, which is time consuming and less precise,” Ron said.
The process gives artists what they need to make adjustments ensuring that the CG version of the character was an exact match to the subject’s original performance, including emotional human-to-human nuances that the equipment may have missed. This step became important for ‘Morbius’ because Digital Domain also needed to alter the CG faces to adjust the proportions at certain points, for instance, digitally removing and replacing the performers’ cheek bones to highlight the emaciated, sunken look.
Facial Rigging Lead Ricky Cloudsdale said, “This project required us to have the two facial performances, described above, per character – one based on the actor for matching the plate, and one vampire version that could be invisibly blended in. Masquerade contributed both versions beautifully, but due to the design differences between the human and vampiric forms, new techniques had to be developed to manage the structural differences in the face.
From ML to Artist – and Back Again
“Once the vampire versions are activated, the artistic interpretation, going from machine learning to facial animation, also needs to occur to ensure they are hitting the director's vision for the performance. The animators are then able to art direct the facial performances post capture, by offsetting or overwriting parts of a given performance. This is crucial to avoid endless reshoots for minor adjustments.”
Facial-focused capture sessions
Having two fairly different characters to transform gave the artists an interesting challenge, especially since Morbius had an historical start point, and Milo was entirely new. Marvel comics notwithstanding, from as early as the initial concept phase, Digital Domain operated under a certain logic and set of rules regarding the design process for the character of Morbius. One of the first rules was to ensure that the design featured as much of the actor as possible.
“We kept to those same logic and principles when developing Milo,” said Ron. “Although Matt Smith is quite different looking from Jared Leto, having rules to follow helped us quickly establish a look, ensuring Smith’s facial performance came through, matching as accurately as possible what was captured on set. For instance, a major difference between Jared and Matt’s characters was that Milo had not faced the same hardships as Morbius, partly due to his wealth. His performances were therefore happier in a lurid way, similar to what we have seen from the Joker. This provided an additional focus, as his smile became character defining.”
Bloom
Along with the core look of the vampiric characters, Digital Domain created multiple variations of the character Morbius, highlighting his evolution throughout the film. Five separate versions of the character were seen, from the earliest moments when just hints of the coming transformation are present to a final form where Michael Morbius breaks down and the audience takes a truly monstrous look at the vampire character. [See the top image in this article.]
The filmmakers were also able to introduce another storytelling device that they called the ‘bloom’, where the characters are seen to momentarily morph from their human visage to their vampiric alter ego. This subtle versioning was possible, mainly because of the ability to blend each character’s two facial performances – the one based on the actor to match the plate, and the vampire version. Added to this was the abundance of data the animators and 3D artists have, and the fact that Masquerade delivers a flexible set of data that, once captured, can be applied to many, varied situations. www.digitaldomain.com
