In 2020 the world's studios, sets and physical workflows suddenly failed. The idea of fault tolerance as we had previously known it became obsolete. But somehow, the world of digital media still found a way to thrive and improve.
According to Scott Carroll, External Communications Director for the Vizrt Group, broadcast graphics software developers, credit is owed to the industry’s innovators and inventors of the past year. “Creatives on both the content and technical sides of production found entirely new ways to overcome some of our greatest challenges to date,” he said.
“When our physical spaces were no longer tenable work environments, we found ways to produce content from a distance, even from home. Where many saw empty stadiums, others saw an opportunity to create a new experience. If you had said in 2019 that we would be producing huge amounts of high-quality content remotely, the response would have been skeptical at best.”
Nevertheless, producers are now looking back at some impressive techniques, strategies and production processes that came about through necessity, and are largely led by software.
Fault Tolerance Redefined
“Consider the concept of fault tolerance. Up to the year 2020, the term meant the ability to replace a device in a location with a backup device or a technical workaround. This was the operations standard, from the broadcast studio to manufacturing and production lines around the globe. Nevertheless, when our carefully architected physical spaces failed, we had to reconsider the concept,” said Scott.
The answer in production turned out to be software-based. That doesn’t mean there were no longer hardware components involved in production, but rather an increasing move to software-first mindsets, with far less specific requirements for the hardware, that allowed individuals to remotely connect to studio systems from their homes.
That transition expanded throughout the year as manufacturers developed and released more software-based tools. Overnight, the idea of directing a production from afar became a new standard. Live broadcasts, such as sports, may now employ camera operators and commentators doing their jobs from time zones away.
“This was a shift that won’t be going away any time soon. The ability to work remotely in broadcast and other forms of production is absolutely here to stay for fault tolerance alone, and other benefits have emerged as well,” Scott said.
Remote Participation
The ‘talking head’ idea of call-in participants in a production has traditionally been kept within the realms of news and sports talk shows. It was, for many producers, a necessary way of quickly bringing experts on wide-ranging topics – but was hardly the ideal scenario in production terms.
“That idea has now shifted by a large margin! What was once a secondary option is now considered far more acceptable – not just by productions, but by viewers as well,” said Scott. “This transition to acceptance has been smoothed in the second half of the year as individuals gained on-camera skills from home, and as we upgraded our at-home cameras.
“Newer model mobile devices featuring high-resolution cameras can be used as high quality webcams. Professional lighting is not terribly challenging to set up after all, and suddenly our homes became studios. Once video conferencing software was also developed over the year to a similar standard, participants could call into a central, virtual production space.
“These software advances, access to easy-to-use tools and general acceptance of the process means homes will increasingly be regarded as production spaces. With no need for large hardware-specific installations in the home, we should expect to see home studios and remote participants exist long term.”
Virtualising Togetherness
While system failure and remote production are largely process driven, advances in graphic forms of virtualisation also emerged in 2020.
Scott said, “The first is mostly seen in sports with the addition of fans to otherwise-empty stadiums. La Liga in Spain became one of the first to trial the concept in 2020 when it added computer generated fans to seats, relying on accurate compositing and real-time processing to achieve the desired effect.”
La Liga worked with broadcaster MediaPro and Vizrt, who created an accurate 3D copy of the stadium that would guide placement of the required graphics into the camera’s view, in this case, spectators on the stands.
Vizrt combined two of its graphics tools – Viz Engine compositing and render engine and the VizArena system, which processes the image from the stadium using key landmarks – pitch marking, goal, ball – and understands the boundaries of the field.
De-personalised miniature copies of people were placed in the stands. Virtual viewers only appear in the main upper plan, and cameras for replays and close-ups were positioned so that empty seats do not fall into the frame. This way, shots only show the stands with virtual fans, or with the field and players.
The NBA took a different approach by installing massive screens on three sides of the court, where the faces of hundreds of fans calling in via Microsoft Teams were displayed in real-time. The screens were then captured in camera along with the action.
Real-Time 3D Rendering
“In fact, graphics engines delivering real-time 3D rendering had plenty of use in 2020,” said Scott. “It will be intriguing to see how they are used moving forward. But keeping fans connected to the action – be it with CG images or through video wall installations – will remain a priority in sport, and software-based tools will be important in managing most of these systems.”
Another example of real-time rendering is still at an experimental stage for live production, but the overall idea is bringing two or more people, in different time zones, together on a single stage using green screen techniques.
These individuals don’t only appear to exist on one stage or screen, they can also interact with one another, handing off control of a display or virtualised set in real time. Furthermore, this doesn’t necessarily need to be done across multiple time zones. On-site productions can keep talent socially distant across a large space. Then, from their separate green screen streams, they can be composited into a single virtual set. The talent are still looking at each other in the room – they just appear closer on video than they are in real life.
So, this kind of ‘teleportation’ is possible and proven in a live environment, leaving the question of how far it will expand to more common uses. Time will tell, but it is a promising example of how virtualisation is advancing.
Robust Graphics
Graphics – and specifically augmented reality – became increasingly impressive toward the end of 2020 and was certainly amplified by the importance of the US presidential election, attracting massive audiences.
Scott noted, “Every four years, the election serves as a window into the direction cable and broadcast news shows are taking their digital storytelling. This election went beyond the standard video wall as broadcasters quickly and effectively communicated complex voting data to viewers. They displayed real-time images of the US House and Senate races, supplied granular voter data down to the county-level, and used data visualisation techniques that were entirely new to audiences.”
Behind broadcasters’ graphics engines, Vizrt’s software Media Sequencer engine allows the operator, producer and director to control exactly how the graphics play out. The Media Sequencer middleware defines and executes media elements and connects render engines and servers. It is integrated with Vizrt control applications like the Trio 3D character generator, Multiplay video wall controller and Mosart for rundown automation.
When used in custom workflows, these applications serve as interfaces to bring the election data into control software or directly into graphics engines. They give broadcasters the chance to display the same information in many different, eye-catching ways that make complex data readable and meaningful.
Dubai and Norway
Al Arabiya built a very large, elaborate set piece at Dubai Media City featuring landmarks from Washington DC. The production team recorded moves for three robotic cameras, two on cranes, and the graphic artists then pre-recorded and rehearsed 3D graphics, which played out during the live coverage.
They spent two months preparing sophisticated graphics in 3ds Max, with looks and textures baked in with V-Ray, that would play out through the Viz Engine compositor at specific points on the set as the camera moved through it. This approach meant they could take special care with reflections for realism, adjust the day/night lighting throughout the 20 hours of coverage and other effects.
At TV2 in Norway, the team used a single, comprehensive template for election data visualisation that they had developed for the Norwegian elections to handle lower thirds, AR elements and charts and graphs. A graphics operator used the template with various presets to display information during the programme. Once set up with data sources, it is placed into a rundown in Viz Mosart.
The team also wrote a Viz script that receives data on their presenter’s touch screen and triggers various AR graphics to appear on the set around him, which made the entire presentation interactive. The graphics could be used for analysis, or to update viewers with the latest voting results data. New for this election, TV2 applied outdoor AR graphics to tracked footage, and added virtual windows to expand the horizons of their virtual sets. Throughout the project, they were using the new Viz Engine 4.
Future Potential
“Software-defined visual storytelling is the future of production,” Scott said. “Whether for broadcast news, live sport, networked television or streaming services, the ability to create content from anywhere will be crucial in the years to come.
“No longer just an idea that we wish we had, it is a practice we must embrace for the fault tolerance component alone. Our physical studios may fail again, but that doesn’t mean we need to scramble to replace them. With software-based systems, every location is a potential studio.
“These examples serve as a starting-off point, as inspiration for what is to come. With software-based tools we can develop new methods to tell more creative stories. We can deploy our workforce strategically. We can make sure the story always comes first – something that never has and never will change – and that the tools and techniques are always supporting that effort.” www.vizrt.com
