Impressive Adobe Algorithm Transfers One Photo’s Style Onto Another

Two pairs of researchers from Cornell University and Adobe have teamed up and developed a “Deep Photo Style Transfer” algorithm that can automatically apply the style (read: color and lighting) of one photo to another. The early results are incredibly impressive and promising.

The software is an expansion on the tech used to transfer painting styles like Monet or Van Gogh to a photograph like the app Prisma. But instead of a painting, this program uses other photographs for reference.

“This paper introduces a deep-learning approach to photographic style transfer that handles a large variety of image content while faithfully transferring the reference style,” says the rather technical abstract of the Deep Photo Style Transfer paper.

Put more plainly: when you put in two photographs, the neural network-powered program analyzes the color and quality of light in the reference photo, and pastes that photo’s characteristics onto the second. This includes things like weather, season, and time of day—theoretically, a winter’s day can be turned into summer, or a cloudy day into a glorious sunrise.

The team’s early examples show the program in action. So this original photo:

Plus this reference photo:

Equals this final photo:

It’s important to note that the software does not alter the structure of the photo in any way, so there’s no risk of distorting the lines, edges or perspective. The entire focus is on mimicking the color and light in order to copy the “look” or “style” of a reference photograph onto a new shot.

Since this is a lot easier said than done, the program has to intelligently compensate for differences between the donor and receiving image. If there is less sky visible in the receiving image, it will detect this difference and not cause the sky to spill over into the rest of the original shot, for example.

The software even attempts to “achieve very local drastic effects,” such as turning on the lights on individual skyscraper windows, all without altering the original photo by moving windows around or distorting edges.

Let’s look at one more example. Here’s the original winter landscape shot:

When you plug it into the Deep Style Transfer program with this summer landscape as a reference image:

You get this odd creation:

In the future, a perfected version of this technology could make its way into Photoshop as a tool, or run as a separate program or plug-in. Not that you should bank on this tech fixing the photos from your upcoming trip; like any other new technology, there is work to be done.

“The study shows that our algorithm produces the most faithful style transfer results more than 80% of the time,” the paper cautions. So maybe you can’t change Ansel Adam’s Moonrise, Hernandez to a Sunrise, Hernandez, but you get the picture (no pun intended) and it is very promising.

If you’re interested in digging into the tech behind this creation, read the full paper at this link.

(via DPReview)