Life through Lens: inside Google’s plan to re-skin reality

“It’s the ultimate try before you buy,” says Amit Singh, vice president of AR and VR at Google.

He’s referring to the Porsche Mission E Concept that’s being driven around in front of us. But it’s not really there, it’s part of an application developed using ARCore, Google’s powerful software kit that allows developers to create augmented reality experiences for Android phones.

Originally locked to the Google Pixel range, Google has opened up the SDK called ARCore 1.0. It’s now out of preview and comes complete with improvements and learnings Google has amassed since first announcing ARCore in August 2017 and before that with its Project Tango platform.

“We have scaled it so that it is across 100 million phones and growing,” says Singh.

At MWC 2018, it was included in the Samsung Galaxy S9 reveal and it’s available for other Samsung handsets, as well as some LG phones and the OnePlus 5. You can also expect it to come to Asus, Nokia, Huawei and ZTE phones in the near future.

“We want AR to be a core feature on Android, at least for the high end phones and then over time for many more,” explains Singh.

Scaling is all very well, but if the apps aren’t there it’s moot. Google knows this and that’s why it is looking to tap into different markets with Lens, in particular China.

AR for everyone

“We’re partnering with Huawei, Xiaomi and Samsung to bring ARCore to this market,” says Singh. Google in its blog post emphasises this, noting, "Everyone should get to experience augmented reality."

From the demos TechRadar had, everyone should. If driving a car around wasn’t impressive enough, we ducked and got into the interior, analysed the thermodynamics of the vehicle, then color picked from the shirt Singh was wearing and gave the car a coat of paint in that shade.

“The ability to re-skin reality is compelling here,” says Aparna Chennapragada, Product Lead of Lens. “But to even create that space you need to know what the semantic meaning is of what you are looking at, and that is why we started Lens.

“Yes, we’re putting digital into the real world but we’re attaching semantic meaning to it. If you want to know something about an object, you tap into the rest of Google. The knowledge graph, maps, the whole shebang.

"Lens is a powerful visual intelligence layer.”

One of the most impressive showcases of this is the cutting-and-pasting of real-life text. Look around you, wherever you are there is some text. But, for obvious reasons, the text isn’t searchable.

Google Lens changes this. We were shown this working on a menu - Lens allowed you to take that text, highlight it and explore certain words. It even worked, to some extent, with handwriting.

Deceptively simple, incredibly powerful

“All those years of understanding different text on web pages comes in really handy,” says Chennapragada. “On the web, you know something in a bigger font is important, something that starts a paragraph is another thing. We use all of that experience and apply it to being able to browse the real world.”

“It sounds deceptively simple but it’s incredibly powerful,” continues Chennapragada. “Say, for something like Coachella - you have a lineup on a poster and you want to see what each of these bands are like. You can use Lens, immediately find them on YouTube, and instantly get a sense of who is playing there.”

Other demos we were shown included a Snapchat Lens that allowed you to tour Barcelona’s stadium Camp Nou and an app that allowed you to furnish a room.

It’s here where ARCore’s technology shows most promise - it doesn’t just have the ability to ‘add’ furniture to a room, but if there is real furniture around, it also has the ability know what that furniture is and may even know the model and make.

Search has expanded from typing into a search box to speaking and it’s now clear that the next medium for Google is the camera.

“We think about it in two parts: there’s ‘camera in,'” says Chennapragada. That’s what we think about Google Lens when you point the camera at something, make some sense of it, put some meaning to it - is it a building, a business card etc?

Then there’s ‘camera out’ which is all of the AR stuff. So, we can put that sticker over there, place a virtual furniture couch in the real world. That is how we are thinking about both and how they come together.

“For us that is the evolution of search and information discovery.”

MWC (Mobile World Congress) is the world's largest exhibition for the mobile industry, stuffed full of the newest phones, tablets, wearables and more. TechRadar is reporting live from Barcelona all week to bring you the very latest from the show floor. Head to our dedicated MWC 2018 hub to see all the new releases, along with TechRadar's world-class analysis and buying advice about your next phone.