eBay intern uses ARKit and iPhone X TrueDepth camera to develop potentially groundbreaking AR tool

eBay intern uses ARKit and iPhone X TrueDepth camera to develop potentially groundbreaking AR tool
Augmented reality has been constantly hailed as the single most groundbreaking new technology over the past half a decade or so by both industry pundits and companies as diverse (and huge) as Google, Apple, Samsung, and Microsoft. But in all honesty, we’ve yet to see a truly life-changing AR-powered experience or app. 

While waiting for the necessary hardware to drive innovation forward, you’ll be happy to hear existing experimental software has been used to create a potentially revolutionary tool called HeadGaze.

This is a new type of open-source technology developed by a young eBay intern and University of California PhD Candidate, leveraging Apple ARKit resources and the iPhone X TrueDepth camera to support head motion tracking for hands-free mobile device navigation.

As you can imagine, everything’s still a little rough around the edges, but a sample of HeadGaze’s incredible potential is showcased in “example app” HeadSwipe, also available as a free GitHub download.

Basically, what reusable technology library HeadGaze does is create a “virtual stylus that follows the motion of your head (up, down, side to side)”, applying a complex method called “3D geometry mapping” to get the location of the “cursor” on the screen.

In other words, a mouse-style desktop navigation experience is enabled, with your head pointing to any location on the iPhone X screen to activate designated “buttons.” As a proof of concept, HeadSwipe lets you swipe eBay deals... with your head, and although the app looks fairly slow in a short demo video, it could mark the beginning of something big.

As lead developer Muratcan Cicek points out, HeadGaze could be used in the not-so-distant future for a number of actually convenient, non-gimmicky AR-powered tasks like scrolling through a recipe while cooking, following on-screen instructions during a messy car engine tinkering session, or simply operating your phone with normal gloves on. 

All of that could be possible without getting your hands dirty, which is especially important for people with motor impairments like Cicek himself. That’s right, this guy (along with his “team”, of course) created HeadGaze and HeadSwipe without full control of his limbs. While pursuing a PhD and completing an eBay internship. What have you been doing with your life?

source: eBay

Related phones

iPhone X
  • Display 5.8" 1125 x 2436 pixels
  • Camera 12 MP / 7 MP front
  • Processor Apple A11 Bionic, Hexa-core, 2390 MHz
  • Storage 256 GB
  • Battery 2716 mAh(21h 3G talk time)



1. bucknassty

Posts: 1345; Member since: Mar 24, 2017

sooooo dexterity and interaction under the accessibility menu in my phone means nothing... or all those gimmicks used to interface with your phone without the use of my fingers did not exist before... interesting

3. PhoneCritic

Posts: 1354; Member since: Oct 05, 2011

I had the same thoughts. This is not new I use these features on my Note 8 to browse the web. its cal Eye tracking. If I gaze down the page scrolls down I look either right or left to view any extra content that is not displayed totally on the screen. I mean I don't see whats new here maybe I am missing something, and I am asking an honest question, can someone explain it to me?

4. bucknassty

Posts: 1345; Member since: Mar 24, 2017

Yep... and I was using this in my note 3 along with being able to wave my hand up and down to scroll through web pages and pictures... but it was a gimmick and some sites.... thought it wasn't important... We all know why it's 'groundbreaking' now don't we??

14. Phullofphil

Posts: 1799; Member since: Feb 10, 2009

If you have to ask you don’t know why

15. Phullofphil

Posts: 1799; Member since: Feb 10, 2009

If you have to ask you don’t know why

8. NarutoKage14

Posts: 1324; Member since: Aug 31, 2016

I knew I had seen this elsewhere before and was going to ask about it. It's nice to know that technology is not thought of as important or relevant until it can be done with iDevices.

12. Finalflash

Posts: 4063; Member since: Jul 23, 2013

Even with iDevices and Apple's marketing puppets pushing it, it will remain a gimmick. The issue with eye or head tracking is always error cases. You move your eyes and head too much for it to be course controls much less fine controls. It's new iPhone time so I guess it's ground breaking now.

16. Phullofphil

Posts: 1799; Member since: Feb 10, 2009

Marketing puppet. Really that’s a good one. What should a marketing firm do. Make there clients look bad. All you weirdo people just amaze Me with hate over an American company that’s successful. Because it’s not even founded. All there is is jealousy. You sound childish.

2. Valdomero

Posts: 683; Member since: Nov 13, 2012


5. AfterShock

Posts: 4146; Member since: Nov 02, 2012

Copied Samsung abilities, groundbreaking? RDF is at full strength and I imagine will be used heavily this week. It's like a war on truth.

9. NarutoKage14

Posts: 1324; Member since: Aug 31, 2016

I want to change that phrase. Let's start calling it the Reality Distortion Force. *Waves hand and says "this is not the tech you're looking for".

6. Sammy_DEVIL737

Posts: 1529; Member since: Nov 28, 2016

That’s Air gesture for you.

7. Tipus

Posts: 875; Member since: Sep 30, 2016

"Grounbreaking" :)) only from the fruit company :))

10. mixedfish

Posts: 1557; Member since: Nov 17, 2013

Yes, you can thank them for waiting on others to innovate for them.

11. strategic_developer

Posts: 1627; Member since: Jul 17, 2018

So when Samaung did na ligation, without touching the screen with features like, scrolling the beoswee window with goue wyes/head, of hovee fi gee above screen for peek feature? I recall those being blasted. But here is a copy basically and its groundbreaking?

13. cheetah2k

Posts: 2256; Member since: Jan 16, 2011

same is**t different day... and the circle of icopy continues to go round and round

17. audibot

Posts: 634; Member since: Jan 26, 2017

so you mean the same thing that came out already for handicapped people and the samsungs eye and head tracking to scroll page up and down and to forward and backwards

Latest Stories

This copy is for your personal, non-commercial use only. You can order presentation-ready copies for distribution to your colleagues, clients or customers at https://www.parsintl.com/phonearena or use the Reprints & Permissions tool that appears at the bottom of each web page. Visit https://www.parsintl.com/ for samples and additional information.