Note (February 5): In the original version of this story I was comparing a low-quality copy of the 1896 film to the upscaled version. Shiryaev actually started with a higher-quality scan of the film, and many of the differences I observed actually reflected his better source material, not the upscaling algorithm. I've updated the first video below to the one Shiryaev used, but I left the text of the story as-is.
Arrival of a Train at La Ciotat is one of the most famous films in cinema history. Shot by French filmmakers Auguste and Louis Lumière, it achieved an unprecedented level of quality for its time. Some people regard its commercial exhibition in 1896 as the birth of the film industry. An urban legend—likely apocryphal—says that viewers found the footage so realistic that they screamed and ran to the back of the room as the train approached. I've embedded a video of the original film above.
Of course, humanity's standards for realism have risen dramatically over the last 125 years. Today, the Lumière brothers' masterpiece looks grainy, murky, and basically ancient. But a man named Denis Shiryaev used modern machine-learning techniques to upscale the classic film to 21st-century video standards.
The result is remarkable. Watching the upscaled version makes the world of our great-great-great-grandparents come to life. Formerly murky details of the train, the clothing, and the faces of the passengers now stand out clearly.
How did Shiryaev do it? He says he used commercial image-editing software called Gigapixel AI. Created by Topaz Labs, the package allows customers to upscale images by up to 600 percent. Using sophisticated neural networks, Gigapixel AI adds realistic details into an image to avoid making it look blurry as it's scaled up.
As the name implies, neural networks are networks of artificial neurons—mathematical functions that transform a set of input values into an output value. The key feature of neural networks is that they can be trained: if you have a bunch of example inputs whose "correct" outputs are known, you can tune the parameters of the network to make it more likely to produce correct answers. The hope is that this training will generalize—that once you've trained it to produce the right answer for inputs the network has seen before, it will produce good answers for inputs it hasn't seen, too.
To train a network, you need to have a database of examples where the right answer is already known. Sometimes AI researchers have to hire human beings to produce these right answers by hand. But for image upscaling, there's a convenient shortcut: you start with high-resolution images and downsample them. The low-resolution images become your inputs, and the high-resolution originals serve as the "correct" answer the network is aiming to produce.
"A neural network analyzes thousands of photo pairs to learn how details usually get lost," Topaz Labs explains on their product page for Gigapixel AI. "The algorithm learns to 'fill in' information in new images based on what it has learned, effectively adding new detail to your photo."
Show the neural network a low-resolution image of a face and it will figure out that it's a face and fill in the right details for the subject's eyes, nose, and mouth. Show the neural network a low-resolution brick building and it will add a suitable brick pattern in the high-res version.
An obvious next step would be to colorize the video. Neural networks can do that, too, using the same basic technique: start with a bunch of color photos, convert them to black and white, and then train a neural network to reconstruct the color originals.
I dropped a frame from Shiryaev's video into the Colorize Images app for Android, which uses machine learning to automatically colorize images. As you can see, it does a pretty good job, correctly inferring that trees should be green, gravel should be a brownish color, and that men's coats should be black. I would love to see someone with more time and better tools colorize Shiryaev's upscaled version of the Lumière Brothers' classic.
reader comments
232