MTI developed an AI-assisted tool for frame-rate conversion for Cortex and camera motion analysis in DRS Nova, efficiently handling common issues in post-production and restoration.
Over time, MTI Film has been steadily improving and adding new features to its core applications, Cortex and DRS Nova, to simplify the management of content from set to screen, and make restoring film and television content faster, more effective and less costly.
Its research and design team is continuously developing new tools, often inspired by customer feedback or its own experience in post-production at its studio in Hollywood. As well as software development, MTI Film operates a full-service post facility in Hollywood for dailies, editorial, visual effects, colour correction and assembly for film, TV and commercials.
Frame Rate Conversion
The company’s most recent update to Cortex Enterprise, the full version of Cortex, is a tool for frame-rate conversion. Scheduled for release in May 2022, the tool uses AI-assisted machine vision to automate the process of converting content to a different frame rate while avoiding artefacts caused by camera motion. “It’s a common task to convert media frame rates, for example from 30fps to 24,” said MTI Film CEO Larry Chernoff, “but you run into problems with shots that include lateral motion. If you were to simply remove one frame out of each five, you would get a jumping effect.”
MTI Film began searching for a solution to this problem while remastering a television series from the 1990s. Source material from the show included a mix of 30- and 24-fps original camera negative that needed to be brought into alignment. The post team first tried eliminating every fifth frame from the 30fps media but, as mentioned, the camera motion resulted in too many jumps and other artefacts.
They also tried an existing automation tool that used AI to identify frames that could be safely eliminated, but it produced too many errors and was too slow. “It took 40 hours to process a 46-minute episode,” Larry said, “and that didn’t include the time it took in an edit bay to correct the artefacts, especially the ones occurring at each cut.”
Open-Source AI Machine Vision
From that point, MTI Film started working on its own system. The team started out using an open-source AI machine vision software, originally developed at Beijing University, that essentially reconstructed frames through interpolation. This approach was successful in creating high quality frame interpolation, but it still produced artefacts at each cut and required a lot of processing time.
MTI Film addressed that limitation through distributed processing. “Using an EDL, we parsed and distributed over 700 shots across ten machines and got them back in what resulted in real-time,” Larry said. “As part of the tool, we also created an algorithm that accounted for each shot’s time, thereby maintaining sync and producing clean cuts. After that, all that was left to do was QC and replacement of the few frames that needed more attention.”
MTI Film developed this process into a general tool for performing frame rate conversions, from and to any speed, with minimal error. The team is making it generally available by packaging it as a feature within Cortex Enterprise. “A lot of content exists that started its life as 30fps but now needs to be 24fps,” said Larry. “Previously, the work was simply too time-consuming and prone to error, but this tool makes it practical.”
Camera Motion Analysis
In version 5 of DRS Nova (released a year ago in 2021), MTI Film introduced Camera Motion Analysis, which divides clips into three types of motion -- static, low and high. Users can then discretely assign unique values to each motion segment, either globally or segment-by-segment within a shot. Motion segmentation is used to differentiate and extract significant motion from the background, and analyse it. The pixels are segmented depending on their relative movement over a period of time, for instance, the time of the video sequence.
Also, the new version now has the ability to virtually alter the timecode start point of a clip and change a clip’s frame rate. Last year, the company introduced Shine, a utility for DRS Nova, that automates the filtering of dirt and fleeting scratches, reducing the time required for such repairs by more than half. More recently, it added an alpha filter to Shine for replacing dirt and scratches found in the IR map.
“Many new features added to DRS Nova relate to automating common restoration tasks,” Larry said. “They not only eliminate a lot of tedious and repetitive work, they also significantly reduce overall restoration costs. As a result, it is now feasible to restore a lot of content that was previously cost-prohibitive.” www.mtifilm.com
