From pixels to pixies: the future of touch is sound

My piece on using sound and lasers to create 3-dimensional interfaces. It’s still some ways off, but it’s funky.

Screenshot 2015 10 01 10 49 33

Screenshot from Ultrahaptics video demo

From pixels to pixies: the future of touch is sound | Reuters:

SINGAPORE | BY JEREMY WAGSTAFF

(The video version: The next touchscreen is sound you can feel | Reuters.com)

Ultrasound – inaudible sound waves normally associated with cancer treatments and monitoring the unborn – may change the way we interact with our mobile devices.

Couple that with a different kind of wave – light, in the form of lasers – and we’re edging towards a world of 3D, holographic displays hovering in the air that we can touch, feel and control.

UK start-up Ultrahaptics, for example, is working with premium car maker Jaguar Land Rover [TAMOJL.UL] to create invisible air-based controls that drivers can feel and tweak. Instead of fumbling for the dashboard radio volume or temperature slider, and taking your eyes off the road, ultrasound waves would form the controls around your hand.

‘You don’t have to actually make it all the way to a surface, the controls find you in the middle of the air and let you operate them,’ says Tom Carter, co-founder and chief technology offjauiclinkeer of Ultrahaptics.

Such technologies, proponents argue, are an advance on devices we can control via gesture – like Nintendo’s Wii or Leap Motion’s sensor device that allows users to control computers with hand gestures. That’s because they mimic the tactile feel of real objects by firing pulses of inaudible sound to a spot in mid air.

They also move beyond the latest generation of tactile mobile interfaces, where companies such as Apple and Huawei [HWT.UL] are building more response into the cold glass of a mobile device screen.

Ultrasound promises to move interaction from the flat and physical to the three dimensional and air-bound. And that’s just for starters.

By applying similar theories about waves to light, some companies hope to not only reproduce the feel of a mid-air interface, but to make it visible, too.

Japanese start-up Pixie Dust Technologies, for example, wants to match mid-air haptics with tiny lasers that create visible holograms of those controls. This would allow users to interact, say, with large sets of data in a 3D aerial interface.

‘It would be like the movie ‘Iron Man’,’ says Takayuki Hoshi, a co-founder, referencing a sequence in the film where the lead character played by Robert Downey Jr. projects holographic images and data in mid-air from his computer, which he is then able to manipulate by hand.

BROKEN PROMISES

Japan has long been at the forefront of this technology. Hiroyuki Shinoda, considered the father of mid-air haptics, said he first had the idea of an ultrasound tactile display in the 1990s and filed his first patent in 2001.

His team at the University of Tokyo is using ultrasound technology to allow people to remotely see, touch and interact with things or each other. For now, the distance between the two is limited by the use of mirrors, but one of its inventors, Keisuke Hasegawa, says this could eventually be converted to a signal, making it possible to interact whatever the distance.

For sure, promises of sci-fi interfaces have been broken before. And even the more modest parts of this technology are some way off. Lee Skrypchuk, Jaguar Land Rovers’ Human Machine Interface Technical Specialist, said technology like Ultrahaptics’ was still 5-7 years away from being in their cars.

And Hoshi, whose Pixie Dust has made promotional videos of people touching tiny mid-air sylphs, says the cost of components needs to fall further to make this technology commercially viable. ‘Our task for now is to tell the world about this technology,’ he says.

Pixie Dust is in the meantime also using ultrasound to form particles into mid-air shapes, so-called acoustic levitation, and speakers that direct sound to some people in a space and not others – useful in museums or at road crossings, says Hoshi.

FROM KITCHEN TO CAR

But the holy grail remains a mid-air interface that combines touch and visuals.

Hoshi says touching his laser plasma sylphs feels like a tiny explosion on the fingertips, and would best be replaced by a more natural ultrasound technology.

And even laser technology itself is a work in progress.

Another Japanese company, Burton Inc, offers live outdoor demonstrations of mid-air laser displays fluttering like fireflies. But founder Hidei Kimura says he’s still trying to interest local governments in using it to project signs that float in the sky alongside the country’s usual loudspeaker alerts during a natural disaster.

Perhaps the biggest obstacle to commercializing mid-air interfaces is making a pitch that appeals not just to consumers’ fantasies but to the customer’s bottom line.

Norwegian start-up Elliptic Labs, for example, says the world’s biggest smartphone and appliance manufacturers are interested in its mid-air gesture interface because it requires no special chip and removes the need for a phone’s optical sensor.

Elliptic CEO Laila Danielsen says her ultrasound technology uses existing microphones and speakers, allowing users to take a selfie, say, by waving at the screen.

Gesture interfaces, she concedes, are nothing new. Samsung Electronics had infra-red gesture sensors in its phones, but says ‘people didn’t use it’.

Danielsen says her technology is better because it’s cheaper and broadens the field in which users can control their devices. Next stop, she says, is including touchless gestures into the kitchen, or cars.

(Reporting by Jeremy Wagstaff; Editing by Ian Geoghegan)

Cook: 3D Touch a Game Changer

I think 3D Touch is the most important thing that Apple has done for a while, and I think as with all such things we don’t really see it until later. Cook seems to agree: 20 Minutes With Tim Cook – BuzzFeed News:

“But he’s most excited by 3D Touch. ‘I personally think 3D Touch is a game changer,’ he says. ‘I find that my efficiency is way up with 3D touch, because I can go through so many emails so quickly. It really does cut out a number of navigational steps to get where you’re going.’ Even with just a quick demo, it’s easy to see his point. It’s a major new interface feature, one that threatens to upend the way we navigate through our phones, especially once third-party developers begin implementing it in their applications. Apple has engineered the hell out of this 3D Touch to ensure they’ll do just that.

For Cook, 3D Touch is a tentpole feature of not just the iPhone 6s series, but of the iPhone itself and one that shows the company isn’t saving marquee innovations for those ‘tick’ years. ‘As soon as products are ready we’re going to release them,’ Cook explains. ‘There’s no holding back. We’re not going to look at something and say ‘let’s let’s keep that one for next time.’ We’d rather ship everything we’ve got, and put pressure on ourselves to do something even greater next time.’”

Force field: Apple’s pressure-based screens promise a world beyond cold glass

A piece looking at the technology behind the pressure sensing. My prediction: once people play with it they’ll find it hard to go back to the old way of doing things. Maybe typing on an touchscreen may one day feel natural, and maybe even enjoyable. 

Force field: Apple’s pressure-based screens promise a world beyond cold glass | Reuters:

SINGAPORE/TAIPEI | BY JEREMY WAGSTAFF AND MICHAEL GOLD

By adding a more realistic sense of touch to its iPhone, Apple Inc may have conquered a technology that has long promised to take us beyond merely feeling the cold glass of our mobile device screens.

In its latest iPhones, Apple included what it calls 3D Touch, allowing users to interact more intuitively with their devices via a pressure-sensitive screen which mimics the feel and response of real buttons.

In the long run, the force-sensitive technology also promises new or better applications, from more lifelike games and virtual reality to adding temperature, texture and sound to our screens.

‘Force Touch is going to push the envelope of how we interact with our screens,’ says Joel Evans, vice president of mobile enablement at Mobiquity, a mobile consultancy.

The fresh iPhones, unveiled on Wednesday, incorporate a version of the Force Touch technology already in some Apple laptop touchpads and its watches. Apple also announced a stylus that includes pressure sensing technology.

As with previous forays, from touch screens to fingerprint sensors, Apple isn’t the first with this technology, but by combining some existing innovations with its own, it could leverage its advantage of control over hardware, interface and the developers who could wrap Force Touch into its apps.

‘Here we go again. Apple’s done it with gyroscopes, accelerometers, they did it with pressure sensors, they’ve done it with compass, they’ve been great at expediting the adoption of these sensors,’ said Ali Foughi, CEO of US-based NextInput, which has its own technology, trademarked ForceTouch. ‘Apple is at the forefront.’

TOUCHY FEELY

Haptic technology – a tactile response to touching an interface – isn’t new, even in mobile devices. Phones have long vibrated to alert users of incoming calls in silent mode, or when they touch an onscreen button.

But efforts to go beyond that have been limited.

BlackBerry incorporated pressure sensing into its Storm phone in 2008. And Rob Lacroix, vice president of engineering at Immersion Corp, said his company worked in 2012 with Fujitsu on the Raku-Raku Smartphone, an Android phone that could distinguish between a soft and firm touch to help users unfamiliar with handheld devices.

But most efforts have been hamstrung by either a poor understanding of the user’s needs, or technical limitations. A vibrating buzz, for instance, has negative connotations, causing most people to turn off any vibration feature, says James Lewis, CEO of UK-based Redux, which has been working on similar touch technology for several years.

The technology powering vibrations is also primitive, he said, meaning there’s a slight delay and a drain on the battery. Early versions of pressure-sensing technology also required a slight gap between screen and enclosure, leaving it vulnerable to the elements.

Apple seems to have solved such problems, experts said, judging from their trackpads and the Apple Watch. Indeed, the trackpad carries the same sensation of a physical click of its predecessors, but without the actual pad moving at all.

The result: In the short term, Force Touch may simply make interacting with a screen more like something we’d touch in real life – a light switch, say, or a physical keyboard. With Force Touch, the device should be able to tell not only whether we are pressing the screen, but how firmly. It should in turn respond with a sensation – not just a vibration, but with a click – even if that click is itself a trick of technology.

‘What we’re going to see initially is putting life back into dead display,’ said Redux’s Lewis. ‘We just got used to the cold feel of glass.’

HARD PRESSED

To be sure, mobile is not the first industry to flirt with haptics.

For example, for car drivers, Redux demonstrates a tablet-like display which creates the illusions of bumps and friction when you run your fingers over the glass, mimicking physical buttons and sliders so your eyes don’t need to leave the road.

Mobiquity’s technical adviser Robert McCarthy points to several potential uses of Apple’s technology – measuring the force of touch when entering a password, say, to indicate how confident the user is of their selection, or keying in a numeric passcode using different pressure levels as an extra layer of security.

While Apple’s adoption of the technology has awoken the mobile industry to its possibilities, it was pipped to the post by Chinese handset maker Huawei, which this month unveiled one model with what it also tagged Force Touch technology. Pressing harder in a photo app, for example, allows you to zoom in on a picture without the usual two-finger spread.

Other manufacturers are exploring how to make touching a device more friendly, and more advanced, says Freddie Liu, CFO of Taiwan-based TPK Holding Co Ltd, an Apple supplier.

‘This is just the beginning for Force Touch,’ he said.

(Reporting by Jeremy Wagstaff and Michael Gold, with additional reporting by Reiji Murai in TOKYO; Editing by Ian Geoghegan and Raju Gopalakrishnan)”

Factbox: iPhone 3D Touch suppliers and haptics companies | Reuters