Google used machine learning to improve the portrait mode on the Pixel 3 and Pixel 3 XL

Google used machine learning to improve the portrait mode on the Pixel 3 and Pixel 3 XL
The Pixel 3 and Pixel 3 XL both have one of the best camera systems found on a smartphone today. Yet, Google makes it work with just one camera on the back of both phones. Even without a second camera on back, the phone still produces the bokeh effect in portrait mode thanks to the use of software and other processing tricks. In a blog post published today by Google, the company explains how it can predict depth on the Pixel 3 without the use of a second camera.

Last year's Pixel 2 and Pixel 2 XL used Phase Detection Autofocus (PDAF), also known as dual-pixel auto focus, along with a "traditional non-learned stereo algorithm" to take portraits on the second-gen Pixels. PDAF captures two slightly different views of the same scene and creates a parallax effect. This is used to create a depth map required to achieve the bokeh effect. And while the 2017 models take great portraits with a background blur that can be made weaker or stronger, Google wanted to improve portrait mode for the Pixel 3 and Pixel 3 XL.

Google Photo app version 6.1 or later allows Pixel users to weaken or intensify the bokeh effect on a portrait

Google Photo app version 6.1 or later allows Pixel users to weaken or intensify the bokeh effect on a portrait


While using PDAF works okay, there are factors that could lead to errors when it comes to estimating depth. To improve the depth estimation with the Pixel 3 models, Google added some new cues including the comparison of out-of-focus images in the background with sharply focused images that are closer. That is known as the defocus depth cue. Counting the number of pixels in an image of a person's face helps give an estimate to how far away that person is from the camera. That is known as a semantic cue. Google needed to use machine learning to help create an algorithm that would allow Google to combine the cues for a more accurate depth estimate. To do this, the company needed to train the neural network.

Training the network required plenty of PDAF images and high-quality depth maps. So Google created a case that fit five Pixel 3 phones at once. Using Wi-Fi, the company captured the pictures from all five cameras at the same time (or within approximately 2 milliseconds of each other). The five different viewpoints allowed Google to create parallax in five different directions, helping to create more accurate depth information.


This five Pixel 3 case allows Google to collect data to train the neural network required for more accurate depth estimates

This five Pixel 3 case allows Google to collect data to train the neural network required for more accurate depth estimates


Google continues to use the Pixel cameras to market the phones. A series of videos called the 'Unswitchables' shows various phone owners testing out the Pixel 3 to see if they will eventually switch from their current handsets. At first, most of these people say that they would never switch, but are won over by the end of each episode by the camera and some of Google's features.

Related phones

Pixel 3
  • Display 5.5" 1080 x 2160 pixels
  • Camera 12.2 MP / 8 MP front
  • Processor Qualcomm Snapdragon 845, Octa-core, 2500 MHz
  • Storage 128 GB
  • Battery 2915 mAh
Pixel 3 XL
  • Display 6.3" 1440 x 2960 pixels
  • Camera 12.2 MP / 8 MP front
  • Processor Qualcomm Snapdragon 845, Octa-core, 2500 MHz
  • Storage 128 GB
  • Battery 3430 mAh

FEATURED VIDEO

6 Comments

1. pokharkarsaga

Posts: 545; Member since: Feb 23, 2012

Wish every Android phone could run Pixel OS.No need to buy overpriced phones.

2. lyndon420

Posts: 6495; Member since: Jul 11, 2012

Google should use machine learning to revive their dead nexus line, and ditch this Pixel garbage.

3. Alex_12

Posts: 41; Member since: May 30, 2018

Ditch this Pixel garbage?? I loved Nexus 5, but Pixel is like a dream come true... Nexus had 3 advantages; software, latest cpu and price. Pixel is more expensive, but you get the best from Google... the best camera, AI, security, IP68, wireless charging, squeeze functionality, 3 years instant OS updates, display, front facing stereo speakers ... what else can we ask?!

4. mootu

Posts: 1411; Member since: Mar 16, 2017

" what else can we ask?" Maybe a bug free device and OS. A phone that isn't the ugliest device on the market. That might be a good place to start.

5. lmao2016

Posts: 75; Member since: Mar 17, 2016

Speak for yourself my pixel 3 xl does not have these bugs that everyone else is taking about and I don't think it's ugly.... So yeah I agree with Alex_12, the nexus was not really fulfilling like the pixel line is.

6. applesnapple93

Posts: 301; Member since: Jan 06, 2016

Nexus 6P was arguably better than the Pixel XL. If snapdragon didn't have trash processors the year it launched, it would've been a better phone entirely. It was much cheaper too.

Latest Stories

This copy is for your personal, non-commercial use only. You can order presentation-ready copies for distribution to your colleagues, clients or customers at https://www.parsintl.com/phonearena or use the Reprints & Permissions tool that appears at the bottom of each web page. Visit https://www.parsintl.com/ for samples and additional information.