FilmLight’s Martin Tlaskal talks about the growing importance of machine learning in post-production today, and introduces Face Track ML-based face tracking in the new Baselight 6.0.
According to Martin Tlaskal, Head of Development at FilmLight, machine learning (ML) has become critical in the constantly evolving world of film and TV post-production as new ML models for image processing are released every single day.
Martin said, “The FilmLight development team continually monitors the market and customers’ requirements in order to update and improve the software so that it continues to serve as a platform that fulfils colourists’ needs. This is shown in the latest version of our grading software, Baselight 6.0, which we launched at the end of last year.”
Following an active beta programme, during which the new software was used in productions around the world, Baselight 6.0 is now available to all colourists. It comes after several years of development that included upgrading the timeline, a new primary grading tool, X Grade, and a new kind of look development tool, Chromogen.
Baselight’s First ML-based Tools
“This new Baselight release also features ML-based tools and features, for the first time. Our main goal was to come up with a platform to allow us to do two things,” said Martin. “First, where we felt it would be beneficial to our customers, we wanted to be able to train our own ML models and integrate them deeply into Baselight.
“But we also wanted to create a platform that allows us to easily and quickly integrate external models into Baselight – allowing ourselves and our customers to continue working at the edge of development, in line with the trends and new technology in this field.”
Face Track is the first example of a FilmLight in-house ML-based model which, following years of research and development led by Software Developer Patrice Lacour, has been deeply integrated in Baselight 6.0.
“Face Track was born out of a recognition of the significant amount of time colourists spend on tasks like facial match grading and digital beauty work. Our goal was to create a tool that streamlines and enhances this process, saving the colourist time and increasing productivity,” Martin said.
Face Track Unwrapped
Previous methods of face tracking involved using trackers, such as area or planar trackers, which, while effective in basic scenarios, faltered when faced with complex facial structures or occlusions.
Face Track works by first detecting all faces in a scene. It will then track each face through time, determining its pose. It will essentially unwrap the skin of the face into a constant coordinate space, or UV space. This space becomes the canvas for colourists to make corrections or enhancements, with the advantage that the corrections made there will automatically be applied to 3D geometry.
“This means that colour corrections are applied consistently to the face, regardless of how the face moves,” said Martin. “From there, because the system knows about faces and poses, the colourist can copy and paste their enhancements across multiple scenes, sequences or entire episodes. It will make a ‘best guess’ at matching faces and while it may not be perfect, it will get the colourist 90-95% of the way there, allowing them to focus on finessing the creative aspects rather than time-consuming manual tasks.”
The feedback so far has been extremely satisfying, and customers have already used the beta version of Baselight 6.0 to complete productions. Once they understood the tools and the efficiency they bring, they didn’t want to return to previous versions of the software.
Paint a matte ...
... copy and paste.
Flexi Architecture for ML Integration
Realising that they can't create all models themselves, the R&D team has also introduced Flexi architecture into Baselight 6.0. Many new ML-based models are released every day with licenses allowing commercial use. “However, they’re all written in Python, which means it’s not easy or fast to integrate them into most apps,” Martin said.
Flexi is a new architecture in Baselight 6.0 that allows FilmLight’s developers and customers to integrate external ML models written in Python into the Baselight environment. This flexibility makes it possible for users to adapt and run various open-source models within the application, creating a much greater potential for experimentation and creative exploration. For example, by using the Flexi architecture, they have been able to integrate the MiDaS depth mapper, widely used and well-known, into Baselight.
“Although MiDaS is perhaps not state of the art today, it is a good example of how an external ML model can be integrated,” said Martin. “An accurate depth mapper produces an estimation of the per-pixel depth of an image and allows the user to apply Bokeh – a cinematic blur of the background – to produce beautifully natural-looking shots with a more filmic depth of field. By taking the current version of MiDaS, including a Flexi Python script, and slightly altering the wrapper script, we were able to get it running inside Baselight in under a day.
“We may choose to integrate some models more deeply into Baselight, but for others we may publish an API and allow our customers – many of whom have their own internal scripting teams – to integrate models of interest directly into their Baselight system as required. These models are computationally expensive, requiring a lot of GPU and memory, so doing it this way allows them to use Baselight’s caching and rendering system and allow real-time interaction.”
Keeping Up with Flexi
The vision with Flexi is to keep up with the rapid advancements in machine learning, and to make it easy for users to incorporate the most up-to-date, innovative models into Baselight, whether it's image segmentation, video in-painting or other video-centric applications, revolutionary or creative. Flexi aims to be the bridge between open-source ML models and the post-production environment.
As the film and television industry evolves, filmmakers will be seeking tools that enhance both creativity and efficiency. Moving forward, Martin anticipates further improvements and integrations, but believes Baselight 6.0, with Face Track and Flexi, aligns with these trends. “Through the use of machine learning, colourists can automate and simplify the colour grading process, achieving stand-out results in less time,” he said. www.filmlight.ltd.uk/baselight6
