VFX Supervisor Sebastian von Overheidt talks about designing and animating the wicked ghosts and creepy characters who stalk the spooky environments inside Disney’s Haunted Mansion.
In the Disney film ‘Haunted Mansion’, directed by Justin Simien, a mother named Gabbie and her young son Travis purchase a gracious Southern mansion, but find it comes with a population of unfriendly ghosts and creepy characters. Joining forces with their new friends – a tour guide, a historian, a psychic and a priest – they try to take control of their haunted house.
As the lead VFX partner for ‘Haunted Mansion’, led by Production VFX Supervisor Edwin Rivera working with DNEG VFX Supervisors Sebastian von Overheidt and Gregor Lakner, DNEG delivered close to 600 VFX shots. The team was responsible for the majority of the environment work inside the mansion, as well as some of the story’s intriguing supernatural characters, including Madame Leota in her crystal ball and the wicked Hatbox Ghost, from concept development through to the final asset. They also created the many ghost effects seen throughout the movie.
Very early on during pre-production, Sebastian’s team and Edwin Rivera helped the filmmakers determine how the ghosts, environments and supernatural effects should ultimately look. That decision, in turn, helped formulate a plan on how to shoot the actors, organise the set pieces, handle the lighting and more, to achieve the film’s final look.
Into the Mansion
The shoot mainly took place at Trilith Studios in Georgia and on location in New Orleans, Louisiana. Production built a complete interior set of the 1st floor of the mansion on the studio stages, impressive in scale and attention to detail, combined with blue screen whenever it was needed or advantageous. “I had the opportunity to work on set for some key sequences, which allowed me to see it all up close and bring that first-hand knowledge back to the team at DNEG,” Sebastian said.
“We wanted to create an extensive library of LIDAR scans of the mansion’s interiors and each object and piece of furniture to prepare for a digital recreation of the rooms. This would support complex scenes involving CG characters, such as the stand-off between the floating Hatbox Ghost and a group of performers on wires.
"In the case of reshoots, since only parts of some rooms were rebuilt, we had to lean more heavily on bluescreens and the VFX teams to add digital set extensions. I assisted the great on-set VFX team to capture some key scenes and kept an eye on the camera angles the director and DoP were choosing. That helped the artists accurately match the measurements of the missing sections that had originally been built practically.”
Reference and Research
When it came to references for the movie’s effects work, the aesthetics of the Haunted Mansion attractions at Disneyland and Disney World weighed in heavily, of course, inspiring much of the art department’s work especially when designing the interior of the mansion. DNEG completed a lot of concept art work in addition to that, along with extensive character and shot development.
Sebastian said, “We definitely did a lot of research, studying imagery and clips from the internet, and when Edwin was given a comprehensive, behind-the-scenes tour of the original ride, he talked us through it afterwards. Quite a few of our artists had also experienced the ride themselves and shared their impressions. Even so, the brief wasn’t to match the ride exactly, but to get inspiration from the style and vibe.”
The two ghostly characters DNEG created, Madame Leota and the Hatbox Ghost, needed a special kind of research. Although the characters are entirely otherworldly, they are played by well-known actors and the production wanted their distinctive looks and personalities to shine through in the on-screen visualisation. It was a tough call to bring those two worlds together.
Madame Leota
Madame Leota, played by Jamie Lee Curtis, is a clairvoyant spirit whose floating head appears in a magic crystal ball, but whose look is perfectly photoreal. DNEG created all aspects of this head from the initial concepts to the final realisation in shots. She started with a full scan of the actress artist, captured by Clear Angle. “We modelled her full head to a high level of detail before working on concepts for her hair and make-up style as Madame Leota. DNEG lead compositor Eva von Overheidt and lookdev artist Haejin An eventually transferred the concept to the base model and refined the look.
"In order for the character to deliver dialogue, perform and still be unmistakably recognisable as Jamie Lee Curtis, DNEG worked with DI4D’s dual-camera stereo photogrammetry system, built into a proprietary HMC, to capture her facial data as she performed. Sebastian said, “The DI4D approach worked really well for us because we could transfer the captured data directly to our actual model and rig. The performance of her lips are in sync with the audio and the general expression has a high level of detail. It’s a huge benefit to be able to start animation from that baseline.”
Between Takes
The animation team, led by animation supervisor Andrew Doucette, still had some clean-ups and performance tweaks to do – for example, blending between multiple performances within one shot meant FACS shapes were needed to some degree – nevertheless, DI4D’s scans captured her facial performance down to the fine details of tiny twitches and eye movements.
Once they had a defined workflow with the scanned data applied to the assets, the work progressed faster without having to spend time on a believable animation that delivers the lines. It also meant they were able to accommodate a progressing edit, and editorial could even mix and match takes.
As they blended between the takes, the animators made sure the head rotation and eye-line worked in context for the shot and edit and manually accentuated small details in the performance. “We kept a very close eye on Jamie Lee Curtis’ actual video footage to catch certain expressions that wouldn’t come across the same way in a different lighting and make-up scenario,” said Sebastian. “The ultimate goal was to make sure her performance was fully transferred to the render and that she could still be recognised.”
Every Twitch
For more realism and believable dialogue, DNEG also applied skin and muscle simulations. DNEG creature supervisor Adam Vanner and his team built a separate rig to simulate these finer movements. “The animators still had good control to adjust facial movements on top of the 3D facial capture for Jamie Lee Curtis, but all the finer skin and muscle simulation was sent to our creature team once the animation was approved,” Sebastian said.
“Although the facial capture caches we received were very detailed, this finer detail was mostly lost. Each and every capture cache was accompanied by facial reference takes, which proved to be very helpful reference to recreate all the smaller movements.”
Madame Leota’s expressive, constantly moving hair was another side of her character. Extensive communication was needed between the groom and creature artists, led by lead creature TD Nandan Phansalkar, to persuade this stylised hairstyle to move and behave well within the simulation and confinements of the crystal ball. They started with a kind of underwater and zero gravity-type simulation for her hair, before landing on the final approved motion.
Convex
The whole refractive aspect of her render, caused by the convex surface of her crystal ball, was taken out of the equation and dealt with it in compositing. Sebastian said, “We knew what it should look like and basically matched that look in Nuke on the outer edges of the crystal ball. Early on in our tests, we would get the full spectrum of Kai’s Power Goo (for those who remember) effects from elongated faces to fish-bowl noses, all constantly changing when she moved her head.
“Another challenge was that the production’s lenses would sometimes change between scenes, resulting in a lot of warping to the crystal ball. We made sure to avoid an egg-shaped ball – imagine what that would have done to Madame Leota’s head! - and took the lens distortion out of the equation as well and dialled it in to taste in compositing.”
Hatbox Ghost
Compared to Madame Leota, creating Hatbox Ghost used a different approach altogether. Ever since his debut in the Disneyland attraction, this ghost has always been a more abstract character with a skull-like appearance, and he also received his voice later in a separate process, as an overdub from the actor Jared Leto.
“We started by designing him in concept using references from the production art department. During that process, DNEG concept artist Nikolai Razuev, was already modelling and sculpting the base for this head and face, stopping short of refining the details by painting on top,” said Sebastian.
“We received a maquette of the head from the production art department that inspired aspects of his eyes, teeth and hair. It was great to use as reference, but we also analysed images of extremely old people for inspiration. We wanted to get a better idea of skin complexion and how deep the eye sockets could be.”
Full Circle
At that early stage, they went full circle from too cartoon-like, over to a very clear resemblance with Jared Leto, to very creepy with huge fangs and dragonesque proportions. Finally, they landed in the middle. “That’s how a creative process goes sometimes,” said Sebastian.
“Once the concept was approved, we had a great visual reference in the right lighting context, and a very precise brief for finishing the character in modelling and the look development on his skin and eyes. The character would still progress later on as the filmmakers made changes to the features of his face, but we had a good process in place at that point.”
His facial animation had to be entirely based on keyframing, since the audio captured on set was going to change later during production. But it still served as good reference for many shot features like lighting and cloth behaviour, and his rig and expressions were bespoke to the style of the character.
His general body performance was a mix of body-tracked animation of the actor, fully keyframed, and in many cases only the head was actually replaced or removed and then added to the Hatbox Ghost.
Environments That Tell Stories
No haunted house is complete without spooky and terrifying environments for the ghosts to inhabit. In this mansion’s case, the characters find themselves trapped in endless hallways and a deadly stretching room. The ‘Endless Hallways’ needed to have a mechanical feel to them, especially as the audience watches them expand.
“It’s a moment that feels like an illusion meeting the reality of being trapped in the ghost realm,” Sebastian said. “You can run as long as you want, but there is no end and ghosts could come from anywhere. Anything could be a ghost. Ghosts can walk through mirrors as if they weren’t there, but our hero Ben can’t. They can hide in a candle but people can’t. DNEG’s DFX supervisors Nik Brownlee and Caine Dickinson made sure that all of these elements were integrated seamlessly into the captured footage.”
The ‘Stretching Room’ is another well-known part of the Disneyland attraction with its illusion of stretching walls. From a story point of view, the design and animation of the entire Stretching Room serve as choreography that leads the two characters Ben and Travis down a particular path. It’s also the most dangerous environment, with deadly elements everywhere.
Sebastian commented, “Tasks like defining the look and feel of such places are one of the many fun parts of being a visual effects artist. It all starts with the production design team and their amazing sets, of course. Then the previz work goes into the first drafts of the edit. LIDAR, texture references, tons of set photos, costume bible and more, all describe the mood baseline of the movie.
“From there, a lot comes from the inspiration that the artists take from those. Throughout the show, DNEG was a strong creative partner in the process, proposing many ideas for Edwin Rivera and director Justin Simien to respond to. It was a great relationship and collaboration in that way.”
Endless Geometry
The Endless Hallway scenes were shot on reconfigurations of the same practical short section of the hallway, which the team would extend with a digital environment afterwards. Logistically, not only does an ‘endless’ environment grow into a lot of geometry, very quickly, because of the foreshortened perspective, but the smallest misalignment of a wall or floor on the practical set also becomes a compounding problem once an artist starts repeating it into the distance.
CG supervisor Ummi Gudjonsson approached this challenge by deciphering and breaking it down into the parts that were reused, and also found the small irregularities between the sets. “Our build team led by build supervisors Nishad Shelar and Fellipe Beckman used the LIDAR scans and texture references to build highly detailed versions of these environments as digital assets, but kept them modular to give us freedom with the design and with cameras. Our layout team led by layout supervisor Sandra Murta did a fantastic job of putting this puzzle together in the shots,” said Sebastian.
“A big component of making a digital environment believable is the lighting and compositing. For instance, the far end of the hallways were kept somewhat neutral – they aren’t dark holes, but also aren’t too bright. Lighting supervisor Nicolas Guerrero Arceo and his team art-directed shots to look aesthetically pleasing in order to look believable.”
Compositing supervisors Norbert Ruf and Robert Willis would then emphasise the ‘sweet spots’ of each CG render, ranging from something simple like adding a rolling sheen moment on a flat surface to achieve perceived realism in a CG shot, to making sure the integration and look of volumetrics and lens artefacts was spot on.
Animated Paintings
In contrast, the stretching room took an approach that was more visual and less mechanical. Technically the room never stretches, however its infamous paintings do extend, revealing more canvas as they travel up the wall. The room itself is basically a long cylinder that starts at normal ceiling height and then moves up. The paintings start moving slightly faster, as do the gargoyles that the characters are holding onto.
Throughout the movie, in fact, paintings were an exciting challenge that the Vancouver team, led by DFX supervisor Nik Brownlee, looked into early on. Unsure of the scope of shots at that point, they looked for a solid approach that would allow creative freedom in animation. Moving paintings had been done before, Nik’s team wanted to keep the look very close to the original artwork without having to compromise.
The ‘Stretching Room’ paintings, in particular, were received at DNEG as fully extended, original paintings on canvas for processing in the studio. However, since cropped-in, printed copies of the originals had been used on set, the team had to compensate for this difference in looks in order to render CG scenes later on.
Competing Factors
“The next step was to do some secondary grading. We closely studied the original paintings from the Disney attraction as our guide – they have a simplicity and distinctive lighting that made them pop in a room with dimmed lighting,” Sebastian said.
“For the animation, we worked with DI4D, who captured the performances of actors for the lines the paintings would deliver. The challenge there was that afterwards, we had to change the models to match the likeness in the paintings, while keeping the performance intact.”
DNEG continuously worked through a series of competing factors – creating paintings that felt ‘alive’ in the ghost realm while maintaining a believable look for the painting itself, without sacrificing the style and displacement detail you would see on an old oil-painted canvas. There were also a variety of requirements in terms of source material.
“The paintings in the ‘Stretching Room’ were based on real paintings and had to match the captured footage. Some paintings in the hallway were shot practically as elements on greenscreen and we needed to recreate their look. For example, one painting was inspired by ‘Woman with a Parasol – Madame Monet and Her Son’ by Claude Monet. Another recalled a work by Rembrandt.
“The woman who steps out of her painting with her complex, fully modelled and simulated dress is another use-case. The entire asset, inspired by the work of Giovanni Boldini, is fully digital.”
EbSynth
During the R&D stage, they looked for a tool that would allow as much creative freedom on both the look of the painting and the animated performance as possible. Both were equally important. DeepFake came to mind, of course, but they also had to create paintings where the whole character and even the environment of the painting had to move.
EbSynth matched the agenda, giving them a standalone software that didn’t rely on cloud-based computing or internet access and could be used inside the DNEG pipeline. It also had parameters we could play with and optimize. The initial tests yielded convincing results beyond our requirements.
Sebastian said, “From testing and researching, we decided on a shader for the CG renders that helped the EbSynth software recognise the performance, and we could then apply that to our aligned static painting texture. Our team of generalists, led by lead environment and generalist TD Simona Ceci, handled the EbSynth part and its assembly as a working texture.
“Not every aspect of the creation of these paintings relied on EbSynth though. Some parts worked better by having the painting look in the shader for rendering. The approach that EbSynth is taking allows us the creative freedom to control the entire process and be able to make adjustments.
“Using EbSynth, we can split a painting into layers and process them while maintaining an alpha channel, meaning we could approach the foreground and background of a painting with different methodologies if needed. It also meant the combination of the video and painted keyframe would vary from painting to painting.
“Whenever we rendered our assets to become the source video for EbSynth, we didn’t worry about a realistic-looking shader, instead working on a purely technical shader that would help increase the precision of the software In other scenarios, the reference video was practically shot material and we created multiple key-frames in a procedural way.”
Dangerous
Creating and integrating the dangerous elements of the Stretching Room was a part of the project that required very close collaboration between departments, relying on the expertise of each type of artist. The build team had created a full digital recreation of the room in its normal state before the transformation, and then defined the ‘quadrants of hazards’ in layout, choreographed the arrangement of tombstones, alligators and other elements, and then had the EnvGen team – led by environment generalist supervisor Romain Simonnet – build the chasm, broken boards, outline the sand geometry and so on.
Sebastian talked through the steps. “Then our FX team led by FX supervisor Dan Finnegan simulated the waterfall, quicksand interaction and tombstones breaking through the floor with splintering and dust. Our lighting and comp teams found creative ways to make each shot look consistent – which isn’t easy in an ever-changing room. We couldn’t simply recreate the lighting on set because what would originally have been mounted on the sound stage ceiling, was now constantly moving higher and higher.
“The alligators, obviously, were entirely digital with fully keyframed animation and muscle and skin simulations. Working out how to portray such an animal climb up a vertical wall, while maintaining a sense of weight in their movement, was a challenge. The FX team simulated interaction and damage of their claws so we could see the alligators really dig into the wall.”
Flying Ghosts
As critical features of the Mansion environment, all of the ghosts the audience sees are based on real actors and costumes. The director’s idea was to have them walking about on set to give the movie a grounded feel. DNEG created digital doubles for many of the ghosts that feature as more distinct personalities and would also track their performance in the images.
Doing this gave each ghost a position in 3D space, needed to create a look they called the ‘ectoplasmic effervescence’ effect, inspired by bioluminescent algae. The ghosts react to the surrounding lighting and to movement, so when they walk into shadows the viewer sees an underlying skeleton as well as the background. When they stop moving, their ghost effect lessens.
Needing to get a clear impression in pre-production of how this effect would work in the images, they used generic, creepy CG characters to test it out. “Once approved, Production was really happy with the look, so we evolved this idea further, resulting in some of our ghosts becoming flying ghosts,” Sebastian said.
“Animation supervisor Sibi Suku, FX supervisor Goolzar Buchia and creature supervisor Amitkumar Tembhare together handled the bespoke requirements of those flying ghosts. Their appearance became somewhat fluorescent with a more ‘cartoony’ animation. Many of them are real, augmented actors, and any with fully digital counterparts have cloth and hair simulations as well.
“For the specific flying effects, we used Houdini but all the ghosts’ cloth simulations were set up in Maya. To keep enough control over the simulations to allow multiple ghosts to fly through the interiors, we spent some time upfront to block the movement of the characters within the tight spaces. That made the process much easier for FX once we got into simulation. All the development work was done in animation within our sets, leaving us with a pretty robust setup once we got into full-shot production.” www.dneg.com
Words: Adriene Hurst, Editor
Images: ©2023 Disney
