In 'The Nutcracker and the Four Realms', young Clara discovers an unexpected world of heroic mice, living toys and immense, snowy fantasy landscapes, hidden somewhere within her godfather Drosselmeyer's house. The story is based loosely on the famous Christmas fairy tale 'The Nutcracker and the Mouse King' written by German Romantic writer E T A Hoffmann. The film features beautiful visual effects work that relies on heavy use of blue and green screens on the sets during production.
“Because 'The Nutcracker' had so many scenes shot against blue/green screens that needed to be replaced with our virtual world, one of the first things we did was look for a pre-visualisation system,” said Max Wood, the production VFX Supervisor. “First, it would help the director to frame his shots more accurately – we didn’t want to shoot blind and find out later that we’d cropped the top of the palace off. We knew it would save time and money in post to have a better idea of the geography while on set.
“Second, it would give the post production teams a great head start when it came to production screenings to have a representation of the backgrounds while the final VFX were still being built – people would be able to understand what the movie would look like, and not need somuch explanation.”
Flexible Ncam
As a VFX Supervisor at MPC, Max had previous experience of receiving data from pre-vis systems but had never used one on set, so the team tested several different systems, and then chose Ncam. “All the systems we tried had pros and cons, but Ncam had the best all round functionality. We also found it very flexible and it seemed to be the most reliable. Most important, it wouldn’t impact on production times - it was important that factors outside the direct control of the VFX team did not interfere with using an accurately tracked, live set extension.”
For example, during the testing period, the team looked at a system involving an on-camera device that continuously monitors hundreds of reflective disks applied to rigid points around the set. The system worked extremely well, but the lighting department announced plans to cover the entire ceiling in massive silks. These would block the disks in their original positions, and the disks couldn't be applied to the silks as they are not rigid. Ncam, on the other hand, is self contained and does not have these kinds of restrictions.
The ability to make CG geographical changes on the fly was another major factor for the VFX team. “We knew that our pre-vis landscape matched to the art department layout, but also knew from experience that production design decisions can change very quickly while filming is underway. The flexibility of the Ncam set-up assured us that it would be possible to make quick changes.”
Pre-vis Asset Preparation
The system was primarily used for set extensions that ranged from palace walls to vast landscapes going off into the distance with waterfalls and rotating water wheels. The pre-viz assets to represent those elements – which were delivered as 3D scenes created in Autodesk Maya - were supplied to the Ncam team well before the shoot got underway in order to prepare them for use live on set.
“We tweaked the assets to optimise them for use in the Ncam system, for example cleaning up and streamlined the topology of models, removing non-essential detail and making sure that the textures were of a suitable resolution. The files could not be so big that they slowed down playback to less than 24 fps,” Max said.
“The models' pivots also needed to be in the correct places. The pivots are the points within a 3D scene at which each model can be moved, scaled and rotated with relative freedom relative to the camera position. On set, we often need to adjust items, such as the palace, to do some live framing with the director. If the pivot is in the wrong place, the palace or other element might rotate in an unexpected way and take precious time to fix. It's critical not to hold up the shoot”
Going On-Set
A new piece of equipment on set can be a challenge to integrate when people aren’t accustomed to using it, but Max and the Ncam team took measures to prevent the system from becoming intrusive. The production had four Ncam systems available – two systems were used for the A and B Panaflex Millennium XL2 cameras on the main shoot, with two extra ones available for a splinter unit that filmed additional elements.
“We rigged the Ncam sensor bar to our main cameras, plus a handheld Arriflex 235 and crane-mounted cameras,” said Max. “We had worked with Panavision in advance to make sure the Ncam bars would mount as securely as we needed them to, so by the time we got on set we could rig and derig efficiently.”
A small team from Ncam operated their equipment on set during the shoot from two workstations. One was running Ncam's Reality software, which handles the camera tracking and rendering of the final image, and the other was running Autodesk MotionBuilder, which was feeding out the 3D assets and animations. Maya was also loaded in case any assets needed modification.
Live Composites
Meanwhile, two monitors per Ncam system were positioned at the director's chair. One screen showed the green screen version as shot, with no Ncam output, and the other showed a live, real-time composite with the Ncam background behind the actors and live action set. The same live comps were broadcast to the crew's screens. They would often bring the actors to the director’s screen to see playback as well, so that they can see and keep in mind exactly what will be replacing the green screen.
“At the end of every take not only was the camera data saved but also the geometry of the models - this means that as we get into post, the VFX vendors have a 3D CG camera to work with, plus geometry that aligns exactly to the QuickTimes showing the on set composite. If the director or DP want to make minor adjustments after seeing the pre-vis, the artists already have the essentials in place and can go from there,” said Max.
The fact that this film was shot on film affected their process in certain ways. In terms of Ncam, we had to make some custom equipment to physically fix the Ncam bar to the cameras, since the bars are built for digital camera rigs, but we knew about these problems going in and made that we planned for them. We would also have captured a cleaner feed from the cameras, meaning that it would have been much easier to extract the green screen. Often our shots were very dark and on a film feed there really isn’t a lot of information to play with, whereas digital would have a much higher dynamic range.
Saving Time
“The visuals displayed in real time by the Ncam system were extremely useful to give the director and DoP the ability to frame for specific key elements, and also to give the actors an understanding of the geography, which was vital in many scenes. In one particular scene the actors were standing at the edge of a cliff in front of a blue screen. In the story Captain Philip Hoffman, the bridge keeper, is pointing out key elements of the VFX world to Clara, but on the set, he was just pointing at different areas on a large blue screen.
We used the Ncam system to accurately place markers onto the blue screen to represent the areas being described in the dialogue – in advance of the shoot. Without the Ncam system we would have had to guess where Philip needed to point and then retrofit our world in post to match. Not only could we see the shots in real time, but we saved a lot of post-production time by not having to adjust our world for one shot.”
As planned, the post team also used Quicktime files captured from the Ncam system as backgrounds for screenings while the final VFX were being created. “This saved us even more time in post because we had backgrounds already done to quickly place into the edit and help us to flesh out the movie. It also meant that we had very good starting points for our matchmoves in post-vis – the VFX assembly process carried out after the shoot - both huge advantages for the first screening of the movie,” said Max.
A Glimpse into the Future
By the end of principal photography, both the director and DP were enthusiastic about Ncam and felt that having a glimpse into the future of their movie was a huge advantage, which made them more accommodating to a new workflow.
Max’s main advice for a successful shoot with a pre-vis system like Ncam is to keep a good relationship with the camera team. “If something is not working for them, find a better solution,” he said. “Also, think carefully about where to place the Ncam operators’ workstations and make them easy to move. Things change quickly on a film set, and if space is at a premium, VFX needs may be overlooked. Looking ahead to the next set-up is vital, so as to not miss a take. We kept in constant communication with ADs to stay one step ahead.”
He also recommends meticulous preparation. “Prep time when our Ncam team and pre-vis asset team could check the assets was vital - not only to make sure they worked within the Ncam system, but also that they were grouped properly with correct pivots to help us to make changes efficiently while filming,” he said. “It’s absolutely essential not to hold up filming, and with proper preparation this can be avoided.” www.ncam-tech.com
