NVIDIA’s AI Can Turn Doodles Into Landscape ‘Photos’

Impressed by NVIDIA’s AI creating photorealistic portraits of people who don’t exist? Check out what the company is doing with landscapes. NVIDIA has developed a powerful AI that can turn your doodles into photorealistic landscape images in real-time.

The AI is named GauGAN, a nod to the post-impressionist painter. Here’s a 2-minute video that shows the technology in action:

“A novice painter might set brush to canvas aiming to create a stunning sunset landscape — craggy, snow-covered peaks reflected in a glassy lake — only to end up with something that looks more like a multi-colored inkblot,” NVIDIA writes. “But a deep learning model developed by NVIDIA Research can do just the opposite: it turns rough doodles into photorealistic masterpieces with breathtaking ease.”

NVIDIA is showing off a demo of the AI packaged in a program that’s as easy to use as MS Paint. To create lifelike landscape “photos,” all you need to do is paint rough brushstrokes and colored sections onto the canvas on the left. The generated landscape is instantly updated in the panel on the right.

The colors you choose correspond to what the section should be filled in with — things like sky, trees, clouds, mountains, flowers, water, and rock.

And as you edit the doodle, all the elements in the scene are updated for maximum realism. For example, if you have a simple photo of a lake scene with trees…

…and you’d like to add a mountain range in the distance, the trees and water are also updated to match the addition of the mountains.

Similarly, changing the foreground in a scene from grass to snow…

…also changes the trees and sky.

Here are a couple more before-and-after examples showing the types of scenes you can create in just seconds using the tool.

The AI was trained using a million photos to learn how to realistically fill in sections of landscapes.

“It’s like a coloring book picture that describes where a tree is, where the sun is, where the sky is,” says NVIDIA VP Bryan Catanzaro. “And then the neural network is able to fill in all of the detail and texture, and the reflections, shadows and colors, based on what it has learned about real images.”

There’s no word on when we might see this technology available to the public, but NVIDIA says it could one day help everyone from architects to people who create virtual worlds.