This smartphone app plays smooth jazz when you put it on the sofa

Computer scientists from the University of St Andrews have developed a smartphone app that uses the device's front-facing camera to recognise what kind of surface it's sitting on.

The app is called SpeCam, and links up with an existing database of materials. When it detects a particular surface, it can trigger a specific action. The researchers suggest that you could trigger it to play jazz when placed on the sofa, or dance music if on a coffee table.

Multi-Spectral

Its actions aren't limited to music. If your phone rings and you place it face-down on your laptop, it could send a message reading "sorry, I'm in a meeting", but if you put it in your pocket instead it could send a different message - perhaps suggesting that the caller contacts a colleague.

It works by using the device's screen as a multi-spectral light source, and the front-facing camera to then capture the reflection of the material that it has been placed on. After cycling the screen through different colors, the material can be detected. You can see it in action here:

Discreet Computing

"This is an example of what we call Discreet Computing or discreet interaction, where subtle and inconspicuous user actions can result in entirely new forms of interaction," said Lead researcher Professor Aaron Quigley, Chair of Human Computer Interaction at St Andrews.

"By training and hence recognising materials all the surfaces around us become a canvas for our imagination."

Hui Yeo, who contributed to the research, added: "The goal is to explore novel forms of interaction for one-handed use which can leverage what people already do with devices, without adding new hardware to the device."

A paper describing the technology will be presented at the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services, ACM SIGCHI MobileHCI 2017 on 6 September in Vienna, Austria.