China re-rendered

This article was originally published on Neocha and is republished with permission.

How did the socialites of Shanghai dress a century ago? What did the Great Wall look like before a century’s worth of human and natural damage? What would one of the terracotta soldiers of Xi’an look like in real life?

New York-based artist Hu Wengu, better known by the online handle of DGSpitzer, has pondered long and hard on these questions. In his search for answers, he’s using AI and machine-learning software to restore archival footage of China, offering audiences a high-definition glimpse into the country’s past.

Hu works full-time as a game developer, and his original impetus for learning AI was to learn how the technology can be incorporated into his game-design workflow. He quickly realized that it could be applied in nearly every area of game development, whether it be training in-game NPCs, generating vast maps, or even composing original music. “It actually has interesting applications in a wide range of fields, even social and human sciences,” Hu says. “So this year I made a series of videos introducing people to how AI can compose music, paint, generate faces, and more.”

As Hu’s understanding of AI deepened, he began exploring how the technology could carry over into other domains. This interest eventually led him to AI-enhanced video. Since then, he’s leveraged the power of AI to create a stunning collection of color-restored videos showcasing China’s past.

Three of Hu’s AI-enhanced works include footage sourced from the University of South Carolina Moving Image Research Collections, which showcase a Beijing streetside barber, a Shanghai fashion show, and the construction of the Broadway Mansion Hotel. Hu discovered their footage archives by chance, and as he dug through the video library, he became more and more fascinated. “MIRC has recordings of China from the 1930s, which is extraordinary,” he says. “In the old Beijing barber video, there was a bald, charismatic guy who left a profound impression on me. So much so that, after I watched it, I got in touch with the university and asked if I could use their black-and-white videos in my AI project. They responded quickly and were happy to oblige. They even helped look into the exact dates of when the films were shot.”

Hu’s video restorations happen in three stages. The first involves adding in extra frames, then the footage is colorized, and finally, the resolution is upscaled. His arsenal of AI software, which includes DAIN, DeOldify, DeepRemaster, and PaddleGAN, all tackle these tasks independently. “For example, the AI that adds missing frames, it analyzes the first frame and last frame to logically work out what should fit between them to smooth out the framerate,” he explains. “As for the colorizing AI, it almost like a human toddler. It examines large amounts of black-and-white photos, eventually learning that when a plant-shaped object appears, it should be green; when the sky and clouds appear, they should be blue and white; and when a human face appears, it should be flesh colored. The resolution upscaling AI works in a similar way, and by combining the output of these AI software, the footage is restored.”

Despite the amount of work and time Hu has invested in the project, he confesses that he’s not exactly a history buff. Instead, he considers himself a sci-fi nerd. “When I watched black-and-white videos in the past, they felt like history lessons,” he shrugs. “I would fast forward it to get through it quicker. They just didn’t catch my interest.”

However, to restore a piece of footage, sped-up viewing was no longer an option. Watching at regular speeds, he was able to discover a ton of details that he overlooked on the initial watch, such as a young man staring at the cameraman from behind a tree, a group of carefree kids playing in the street, or a curious worker studying the camera as he passes by.

These little moments of connection fascinate Hu. “It reminds me of the fact that even though we’re living centuries apart, we’re still under the same sky and walking the same earth,” he says. “It’s like watching a century-old vlog.”

Hu has also begun looking into restoring historical footage from other countries. Recently, he’s working with a German artist to restore a documentary shot in Dresden in 1936, even taking it a step further with the implementation of AR technology that will allow viewers to overlay the AI-enhanced scenes atop locations in real-life to see how the area has changed over the years.

Aside from video-related endeavors, he’s also begun using AI to colorize and upscale old portrait photography from China. Through his use of neural networks, Hu hopes for viewers to experience the simplicity of the past and immerse themselves in the lives of these people long gone—if only for a few short minutes. “It can be a moment of respite when you allow people to look into the past,” he says. “Stepping into the shoes of these people on camera and  looking at history through the lens of the present can perhaps help us gain a greater understanding of culture and what it means to be human.”

History can often seem dull, appearing as unrelatable lines of text tucked inside of course books, but Hu believes it doesn’t have to be. Through his AI-enhanced videos, Hu hopes that people who may have previously dismissed how interesting history can be are able to find renewed excitement for the subject, and in turn, develop a healthy appetite for knowledge in all forms and shapes.

Like this story? Follow Neocha on Facebook and Instagram.

Youtube: ~/DGSpitzer
Weibo: ~/大谷Spitzer
Bilibili: ~/大谷的游戏创作小屋

Contributor: Chen Yuan