Arcturus is the developer behind the post-production platform HoloSuite, software tools for creating, editing and distributing volumetric video of digital humans. The company’s founders are artists and developers devoted to storytelling innovation, and come from Pixar, Dreamworks, Autodesk, Google and YouTube. Their next critical volumetric step is delivering those videos to mobile devices.
HoloSuite combines two applications – HoloEdit is designed specifically to edit and compress volumetric video, and HoloStream is adaptive bitrate streaming software that streams polygons directly to a mobile device. Together, they help producers work creatively with the huge, complex files that result from volumetric video capture, processing images from multiple cameras that simultaneously record the actor’s performance from different angles inside a capture volume.
Moving Meshes
Digital Media World spoke to Piotr Uzarowicz, Head of Partnerships & Marketing at Arcturus, who talked about the differences between editing requirements of traditional video and those of volumetric video. “The linear approach of conventional editing software is good for regular video – clips are placed on a timeline representing your story, developed into a sequence, frames are selected for VFX and at the end you can export your EDL,” he said.
“Up to a point, editing a volumetric video is similar but it also needs automated 3D animation tools like those in HoloEdit, used to fix multiple meshes for characters, clean textures and compress data.”
HoloEdit has its own specialised type of automated character rigging as well. “This is because volumetric capture doesn’t involve a single mesh, but hundreds of them, coming from the production’s cameras,” Piotr said. “All of them will have different topologies, polygon counts and texture layouts per frame or every few frames and, taken together, these moving meshes are what create the motion.”
Finding the Audience
HoloEdit’s tools, described more fully later in this article, are critical but Arcturus believes that distributing digital humans is at least as important as creating them, to make sure they reach their audience. That is why HoloStream, the second part of Arcturus’ Holosuite, focusses on streaming uninterrupted volumetric experiences on demand.
Although the use of volumetric videos has been growing steadily, and the recent introduction of LIDAR cameras for mobile devices is making it possible for people with a compatible phone or tablet to create content for 3D applications, streaming the content has remained a challenge. The size and complexity of the captured files has meant that anything longer than a short clip hasn’t been suitable for streaming.
Volumetric On Demand
Arcturus’ most recent news is the formation of a partnership with Japan’s wireless carrier NTT DOCOMO, which now operates on a 5G network, aiming to help stream volumetric videos of any length over a mobile network. The two companies want to make it easier for producers to reach their audiences, and put volumetric video into the hands of anyone with a mobile device. DOCOMO has developed their 5G capabilities further than most other carriers, which Arcturus sees as an opportunity to really push the volumetric format forward.
The partnership is positioned to give 5G mobile users the ability to stream volumetric videos of any length without a drop in quality, and explore all captured angles of the video directly from their browser. Alternatively, users can access volumetric 3D models as AR elements or through a VR device, also in real-time.
DOCOMO is combining its 5G network speeds and CDN expertise with HoloEdit’s compression tools and HoloStream’s adaptive bitrate streaming and video player. Using these tools, playback length in volumetric videos no longer creates bottlenecks.
Viewers browsing online, or experiencing the video in AR or VR, can now explore travel destinations or retail locations, for example, or watch complete workout videos from a chosen point of view on mobile devices. On DOCOMO’s 5G network, ordinary individuals – or professionals in ecommerce, entertainment, education and healthcare – can stream interactive 3D video files directly to their viewers, with minimal latency and no need to download.
Adaptive
Most capture systems deliver short sequences of volumetric video for downloading. But, because volumetric data can be used in game engines, in WebGL, it can be delivered to devices, even phones, for viewers to discover and watch at any time. HoloStream uses this ability to stream the sequence, using adaptive bitrate techniques that make it possible to add streaming volumetric video to applications. It has an API for integrating streaming into Unity, Unreal Engine, native apps and the web.
HoloStream determines and delivers the highest quality stream that a user is able to view, continuously analysing and sending the best quality footage their internet connection and device can support. It also uses local caching to support continuous playback.
Interestingly, because it streams polygons instead of a 2D image, HoloStream maintains six degrees of freedom capabilities within characters, which allow digital humans to respond to viewers in real time. This kind of interactivity is Arcturus’ new frontier and where they are devoting their effort now.
5G will be able to cope better with variations in demand created by applications like volumetric video, not only through greater connectivity but also through management techniques like network slicing, which becomes easier on a 5G network. Network slicing splits the network to tune speed, capacity, coverage and security by reassigning resources from one slice of the network to another one that needs it. This improved performance comes with lower latency that is critical for interactivity and complex imagery.
Build a Better Skeleton
5G networks, network slicing and other developments are significant but, at the same time, improving streaming quality will always be about optimisation, especially when real-time interactivity is a goal.
Piotr said, “Character animation in game engines is created by rigging – building a skeleton to define how a character moves, and then applying the skeletal motion to the character’s geometry – enabling a single mesh to move by moving the skeleton with keyframe animation, motion capture or procedurally.
“Because existing rigging and animation software can’t easily create a skeletal rig and bind the large number of volumetric meshes to follow it, Arcturus created HoloEdit to analyse the movement of the volumetric capture, create an animated skeleton that precisely matches the movement of the capture data and automatically bind the capture to the skeleton. It correctly offsets the meshes so that major motion is driven by the newly created skeleton, and subtler motion still comes from the original capture.”
Once your character is skeletonised and rigged, you can alter his or her performance in real time or in post.
Trying to recreate this process for volumetric video using regular 3D animation software like Maya, 3ds Max, Cinema 4D and so on is difficult. Those applications treat each frame like a still image with one mesh per frame or every few frames. HoloEdit, however, corrects single keyframes that are then re-imported into Holoedit and applied across an entire sequence, saving time and effort trying to fix issues frame-by-frame. Skeletonisation and automatic rigging can either be used directly in HoloEdit or, for artists that want to continue using familiar tools, HoloEdit has plugins for 3rd party applications.
Coming to Life
“Now that you have a skeleton driving the capture motion, this skeleton can be dynamically altered in the game engine,” said Piotr. “For example, you can have the head and neck joints rotate to follow and look at a point, such as an AR camera, creating the illusion that the pre-recorded volumetric character is looking at you.”
As well as real-time head and limb re-targeting, the animator can manipulate and change physical performances in post – turning, looking, shaking hands – and bake the changes into a final product. Eventually these volumetric files are exported to game engines to use later in different kinds of project. arcturus.studio
