Microsoft shows OmniTouch: its vision for the future of touch interfaces (hint: they’re everywhere)

Microsoft shows OmniTouch: its vision for the future of touch interfaces (hint: they’re everywhere)
Now, here’s what we want to see more from Microsoft - innovation. Two teams at Microsoft Research have been working on two very exciting projects and presented their findings at UI symposium UIST 2012. The first project, OmniTouch, is the most eye-catching with its futuristic idea - turn every surface into a touch-enabled space. 

How did they do it? How about plant a smaller Kinect-like device on your shoulder which allows you to interact with all surfaces using multitouch, tapping, dragging and even pinch to zoom. Sounds almost incredible, and the researchers admit that the first three weeks of developing the project were the hardest.

"We wanted to capitalize on the tremendous surface area the real world provides,” explains Hrvoje Benko, of the Natural Interaction Research group. “The surface area of one hand alone exceeds that of typical smart phones. Tables are an order of magnitude larger than a tablet computer. If we could appropriate these ad hoc surfaces in an on-demand way, we could deliver all of the benefits of mobility while expanding the user’s interactive capability."

In order to do that they constructed that Kinect-like device consisting of a laser pico projector which would project images on all surfaces and a depth sensing camera, responsible for the magic. Tweaking the depth camera to recognize human fingers as the source of input, as well as adjusting the accuracy of depth recognition was key to the success of the project.

"Sensing touch on an arbitrary deformable surface is a difficult problem that no one has tackled before. Touch surfaces are usually highly engineered devices, and they wanted to turn walls, notepads, and hands into interactive surfaces—while enabling the user to move about."

So while by now we’re used to the Kinect user recognition, this was a tougher problem to face as it required not only sensing where and how the user moves, but whether the user taps on a surface. Since those surfaces differ, the depth camera performance was essential. The researchers managed to get accurate feedback and detect when the finger is 0.4” (1cm) away from a surface. At that distance a tap is assumed, and they even managed to maintain that state for actions like dragging and pinning. But with no further ado, here’s the video about it, visualizing everything we’ve said so far:

The prototype you see on the image is not small at all, actually it’s ridiculously big for use in public spaces, but the research team agrees that there are no big barriers to miniaturizing it to the size of a matchbox. It could also be conveniently placed as a watch or a pendant.

Now, there was also a second ambitious project, PocketTouch, which might not look that exciting but if you think about it’s equally thought provoking. PocketTouch, as it’s called, aims to allow you interaction with your device through different fabrics. This allows “eyes-free” input, so you can use all kinds of gesture without having to look at your device at all times. 

The results exceeded expectations, and the only thing we have left now is wish these technologies arrive sooner to the mainstream.

Update: And if you've just had a deja vu, here's why:




13. cncrim

Posts: 1591; Member since: Aug 15, 2011

very cool, but impractical.

10. Forsaken77

Posts: 553; Member since: Jun 09, 2011

This would be great if they could incorporate it into a bluetooth headset.

8. alx33

Posts: 32; Member since: Oct 07, 2011

I like the concept but as arcq12 said, This probably won't come out for a long time. and I don't like the UI very much it's a little 2 clunky for my test. I presume that can be changed though :)

7. arcq12

Posts: 733; Member since: Oct 13, 2011

Can't see this until the 2020's.

6. lubba

Posts: 1313; Member since: Jan 17, 2011

Nano size the technology and put it on a phone or tablet.

5. Cyd07

Posts: 83; Member since: Oct 03, 2011

Awesome. in some years, no more need for a computer : strenght smartphone with big battery and this : the screen is as tall or small as you want, and you resolve the biggest problem : input informations in a smartphone, typically text. You've voice solutions (Siri and others), and you will have this, better in public. Want a full qwerty keyboard of 70cm/20cm ? You have. And we could have more little phone (3.5" max, maybe less), just for speed viewing ans keeping battery like longer, since we have the screen we want (beware of confidentiality). Or maybe no screen at all, just a box with processing power in the phone (and again big big battery life). But, again, there are matters of confidentiality if you must project the screen somewhere, whareas if it is in your hand , no more problem than with a phone. Briefly putted : absolutely awesome.

4. belovedson

Posts: 1061; Member since: Nov 30, 2010


3. Johnny_Mnemonic

Posts: 240; Member since: Oct 05, 2011

Microsoft Great job as always, beating Apple's In new Technology

12. Jeromeo

Posts: 135; Member since: Jan 11, 2010

14. rabbit1

Posts: 3; Member since: Oct 17, 2011

i think they hired him, but if you look at some of the konect hacks people have done some great stuff

2. GoodFella

Posts: 112; Member since: Oct 18, 2011

That is absolutely phenomenal!!! I am not sure what to think (if we all these things strapped to our shoulders) we'd all look like a bunch of crazy mimes. Cool concepts though. This has so much potential- Nice Job Microsoft!!

1. Phullofphil

Posts: 1846; Member since: Feb 10, 2009

I think those can actually usfull eventually to a point of mass use as a smart phone acessory and maybe home computing. They can go in alot of kinda amazing directions

9. ayephoner

Posts: 858; Member since: Jun 09, 2009

i like the design and functionality of the 6th sense much more. and it was from a few years back. anyone know what happened to that? i figured we'd see a device launched by now.

11. Jeromeo

Posts: 135; Member since: Jan 11, 2010

I knew someone else had seen this before, too! Check out this TED video from 2009 and the brilliance Microsoft copied:

15. jdrevolution

Posts: 87; Member since: Oct 07, 2011

Actually the Microsoft Surface (which never really became mainstream expect for the rich and famous) was actually incredibly similar to a multitouch screen created by a professor and his team at NYU and was also intruduced at a TED convention. The video should still be on youtube if you search "multi-touch" and "TED". Now, I don't know whether microsoft bought it from this professor but they definately came out very close together, one right after the other

Latest Stories

This copy is for your personal, non-commercial use only. You can order presentation-ready copies for distribution to your colleagues, clients or customers at or use the Reprints & Permissions tool that appears at the bottom of each web page. Visit for samples and additional information.
FCC OKs Cingular's purchase of AT&T Wireless