In this very intriguing experiment, an ARRI ALEXA Classic 2K footage were undergone AI ‘treatment’ to output 8K footage by using machine learning technologies. How much Artificial Intelligence methodologies can assist us in resolution upscaling, instead of perusing after 8K sensors?
ARRI ALEXA Classic 8K footage
In a video titled “This Is What an 8K ARRI ALEXA Would Look Like…”, a very neat demonstration of technology called AI upscaling was presented. The concept is very simple. Take the very high-quality 2K ProRes footage from the old but mighty ARRI ALEXA Classic, and by utilizing AI (Artificial Intelligence) upscaling machine learning technology, turn it into 8K footage. As you can figure out, the process demands some time (for a video from HD to 8K, it typically takes 2-3 seconds for each frame). This technology has been established by a company called Topaz. Its related product is called Video Enhance AI, which can enhance, and improve many aspects of videos, from implementing slow motion, to resolution upscaling.
Video Enhance AI
As stated by Topaz: “Using the power of machine-learning AI technology, Video Enhance AI is trained on thousands of videos and combines information from multiple input video frames. It can enlarge HD video to 8K resolution with correct details and motion consistency. For example, you can upscale SD video to HD or even 4K resolution”. The technique being presented here is known as “AI upscaling.” In essence, you feed an algorithm a low-resolution image, and, based on training data it’s seen, it spits out a version that looks the same but has more pixels in it. “We support up to 8K video output. Any size larger than 8K will cause a failure. But you can set the output format as a sequence of images (png/tif/tiff) which can support nearly any size” Topaz emphasizes.
Upscaling techniques with a focus on AI
Upscaling, as a general technique, has been around for a long time, but the use of AI has drastically improved the speed and quality of results. In the case of banal upscaling algorithms, these rules are often pretty simple. If you want to upscale a 50 x 50-pixel image to double its size, for example, a traditional algorithm just inserts new pixels between the existing ones, selecting the new pixels’ color based on an average of its neighbors. However, Video Enhance AI utilizes Artificial Intelligence algorithms created by machine learning by ‘looking’ at whole sections of images at a time.
Allowing Ultra-High resolution from HD
“We originally built Video Enhance AI to improve the most common quality issues affecting videos, including de-interlacing footage, intelligent upscaling up to 8K resolution, and seamless frame rate conversion,” says Topaz, and adds: “Because it treats video as a series of constant rate images, it led to problems when the app’s AI models had to process variable frame rate footage, frame repeats, and scene changes. It also complicated our ability to develop new models to address stabilization or scene detection because they require multiple passes over the video”. Hence, to develop these new features and to continue innovating, Topaz has built this next generation of the app at the top with an entirely new code base and a brand new user interface with numerous functionality improvements, such as stacking multiple AI models, executing parallel tasks, and project queue filtering.
Price and requirements
The price of the Topaz Video AI app is $300. You can try it for free as well. The system’s minimum requirements are:
- PC: A DirectX12 compatible GPU (NVidia or AMD). An Intel CPU from 2015 onwards or an AMD CPU from 2016 onwards are our minimum requirements for the CPU. On a PC with Nvidia GTX 1080 for a video from HD to 8K, each frame typically takes 2-3 seconds. If you upscale a video from SD to HD, it will take about 0.4 seconds for each frame.
- Mac: 8GB or more RAM, 2GB of VRAM. MacOS 10.14 (Mojave) to run on CPU only. MacOS 10.15 (Catalina) or newer to run on GPU. This gives the best performance.
Final thoughts
Back to the ALEXA Classic experiment. It’s not a secret that this camera owns a very solid big-screen history. Its 2K resolution footage was used in IMAX theaters (equivalent to 11K resolution). Hence, it’s not a big deal that its files would look good at 8K. However, we can discuss the 8K resolution race that many camera manufacturers are obsessed with. AI-upscaling solutions are here to stay. If so, a 4K sensor would be good enough for 8K enhancement without the need to develop an 8K sensor with impaired dynamic range capabilities. Thus, a 4K sensor plus machine learning software can do a better job than a native 8K sensor for 8K imagery. What are your takes on this?
I use Topaz regularly for my videos, and while I like his approach, it’s actually pretty flawed. Here’s the comment I wrote him as well:
“I love the concept behind your idea, I agree entirely with the point you’re making, and I can see you’ve put a lot of effort into this, but the way you’ve been doing it is pretty flawed. Most importantly, because you used a vintage lens that creates a lot of digital deficiencies in the image, as well as have a very soft resolve. The inherent “problem” with AI upscaling is that while it can create detail where there isn’t enough contrast to imply said detail, it can’t create the same for parts of the images that are soft or out of focus. In cases like this, sharpening is actually more useful, as it creates more per-pixel contrast. The perfect example of this is your comparison around the 4:54 mark. This test would be much more scientific if you eliminated the resolution issues of the lens. Maybe by using a newer and cleaner lens. Or if you don’t have one, at least stop down the lens to its sharpest aperture.
Either way, thanks for making this comparison.
I would love to see BlackMagic implement AI upscaling features natively into Resolve Studio as an effects node. I’d even pay for that feature. Or maybe include Topaz as a plugin.”
PS: I love the work your doing. Keep it up.
THX!
You’re right about Resolve adding this feature – would be a smart move. Thought about this also. Although this tool is far from perfect – the potential is there for sure.
Yossy
Blackmagic already has a built-in function in Resolve for upscaling. Of course there is always room for enhancement and AI implementation.
P.s : I have used Topaz for treating a tricky footage and i got good results
Topaz is great but my guess is the AI thing is mostly for marketing. 10 years ago there was an uprez software “perfect resize” and it used “fractals”… it was very good and “fractals” sounded even better.
The video you cite has a suboptimal colour correction and the upscaling results are nothing to get excited about. Click bait IMO.
Topaz is handy because it makes edges smaller and smoother, and cleans up the image so better textures seem to appear. Plus good sharpening, good noise reduction. That is not rocket science let alone AI but a couple of well done algorithms. I love it for improving dodgy HD or UHD in bad codecs from consumer cams or underexposed, and grainy shots. Artifacts are rare but they sometimes appear in unsharp parts and make the render obsolete. And some footage is very resistant against Topaz improvements.
We are still very far away from upscaling 2K to something close to 8K.
Besides, considering the comment by Visar above about the imperfection of the technology used by Topaz to upscale the footage, this seems to be a promising technology.
The mass distribution of 8K displays and camera sensor acquisition is still a niche. Topaz may have the chance to improve and catch up in the future with more efficient technology. The current effort is still plausible. My question is: does upscale a 2K camera sensor to 4K or to 8K, like Alexa Classic, allow reframing a subject in the shot (zoom in, panning) as a native 4K+ sensor allows? If this is possible, some cinematographers and OTT networks requiring 4K resolution might accept films shot on Alexa Classic. This will justify recycling this off-date camera.