Sony's futuristic take on AR shows depth-sensing taken to another level

In 2015, Sony acquired the Belgian company SoftKinetic, a firm boasting Kinect-style depth-sensing, including tracking light via Time of Flight (ToF). SoftKinetic’s ToF tech ‘DepthSense’ just re-emerged in a video Sony just uploaded, which shows depth-sensing coming to the gadget giant’s line of smartphones.

The video (embedded below) shows folks looking through a nondescript phone’s display to see how DepthSense can liven up their lives: 3D neon writing and drawing suspended in the air, a game where players use their finger-guns to shoot at enemies, seeing a shopping item displayed in real space, and so on.

Days ago, a leaked photo hinted that the upcoming Sony Xperia XZ4 could pack a 0.3MP ToF camera – so this new DepthSense showcase on a generic (i.e. probably not the XZ4) phone is likely not a coincidence, GSMArena pointed out.

In any case, this is tech that could end up in any phone, given how many OEMs install Samsung cameras on their devices.

And while we've seen AR effects demonstrated at phone launches before – Apple and Huawei come to mind – Sony is showing off another level of functionality and applications. Instead of sticking cute animated animals alongside people on stage, Sony's tech enables interactive annotation in the AR-space with a phone-based camera setup.

Why Time of Flight?

Time of Flight is a method of sensing depth using a 2D camera, essentially by tracking the time it takes for light to rebound off surfaces and reach the camera. It’s a scrappy way to determine where objects are in relation to each other – say, for AR games and experiences – without requiring extensive cameras or sensors.

Rumors have suggested Apple is lining up to buy Sony’s 3D-sensing tech, which could end up in this year’s iPhone 11. Another leak suggests only a triple-camera setup is coming in 2019’s iPhone crop, but no 3D sensing until the following model in 2020, which might use lasers to sense depth.

3D sensing could be a good feather in any smartphone’s cap – especially since this year’s other phone innovations, like a notch-avoiding punch hole and unfinished folding tech, aren’t terribly exciting. The smartphone market needs novel new ideas: lack of device innovation was a factor that made 2018 the first full year that global smartphone shipments declined.