Google Pixel 2/XL Portrait Mode is unlike any other: here is how it works

With just one camera and AI, the Pixel 2 does Portrait Mode unlike any other phone

With just one camera and AI, the Pixel 2 does Portrait Mode unlike any other phone


The Google Pixel 2/XL looks just like any other smartphone out there, but hidden inside is something powerful and innovative: artificial intelligence (AI).

Google publicly stated that it is moving from a mobile-first to an AI-first company last year, but this year it really doubled down on AI, machine learning and neural networks, and the best result of its effort are summed up in the Pixe 2/XL camera.

First, let's state the obvious: the Pixel 2/XL - unlike many other phones - does not have two cameras on its back. However, it does have Portrait Mode. How is that possible?

Clever engineering and the power of AI come together to make this possible, and it's time to explain exactly how. But first, let's take a look at the...

Google Pixel 2 camera specs:



Interestingly, while Google has improved the camera quality on the new Pixel 2/XL, it has done so while reducing the actual sensor size. The new Pixel 2/XL has a smaller, 1/2.6-inch sensor, while the original Pixel has a 1/2.3-inch sensor. Interestingly, the original Pixel with its larger camera did not have any camera bump, while the new Pixel - despite the smaller camera size - actually has one (this probably has something to do with the addition of optical image stabilization that takes extra space). Another welcome upgrade is in aperture: it's now wider, so more light can get through.

Portrait Mode explained


The same picture without (left) and with (right) Portrait Mode

The same picture without (left) and with (right) Portrait Mode


There are three key things that happen on the Pixel 2/XL to enable portrait mode with just one camera: HDR+, a segmentation mask and a depth map.

HDR+, the first step


And it all starts with HDR+. What is it? HDR alone stands for "high-dynamic range" and has been on phones for years. It works by taking multiple pictures with different exposures - a darker one, a brighter one - and combining them into one picture where the detail in both the dark and lighter areas is well preserved. HDR+ on the Pixel 2 is a whole different animal. It takes up to 10 photographs, way more than others, and does this everytime. Those pictures are underexposed to avoid getting too bright (burned) highlights, and then are all combined and aligned in one picture. Here is what the results looks like:

HDR+ off (left), HDR+ on (right)

HDR+ off (left), HDR+ on (right)


2. The power of AI


Secondly, using the HDR+ picture, the phone has to decide which part of the image should be in focus and which one should be blurred. This is where machine learning comes to help: Google has trained a neural network to recognize people, pets, flowers and the like. Google uses a convolutional neural network and this alone is worth a separate look, but let's just say that the neural network filters the image by weighing the sum of the neighbors around each pixel. It first starts by figuring out color and edges, then continues to filter until it finds faces, then facial features, and so on. The network was trained on nearly a million pictures of people (some with hats, others with sunglasses, etc). The result of this filtering is essential to the quality of the image and that is the key advantage of the Pixel 2 camera: it produces some amazing results. The result is the so called "segmentation mask" and here is how it looks like, the result is already very good.

Google Pixel 2/XL Portrait Mode is unlike any other: here is how it works

3. Innovative use of Dual Pixels


There are a few problems with the image produced after the segmentation mask: first and foremost, the applied blur is uniform throughout the image. The actual scene, however, contains objects at a different distance from the camera, and if you had used a real camera, they would have been blurred to a different degree. The tiny pastry in the front would also have been blurred by a real camera as it is too close to it.

Google needs more information to create proper blur and this is where we speak about the "Depth map". The Pixel 2 has a dual pixel camera sensor. This is not new: Samsung phones were the first to have it. What it boils down to is that on the camera sensor, each pixel is actually divided in two small sub-pixels (it's called Dual Pixel Auto Focus, or DPAF) that capture the same information from a very, very slightly different angle. This difference was used in the Galaxy S7 and later top tier Samsung phones for their ultra-fast auto-focusing. Google has found another use for that: it uses this very slight difference to determine how far away each object in the frame is from the camera.

Google Pixel 2/XL Portrait Mode is unlike any other: here is how it works

The final result


Here is the result, which is also quite amazing: the brighter parts are closer, while the darker parts are further away:

Google Pixel 2/XL Portrait Mode is unlike any other: here is how it works

Or as Google fondly puts it:


If it sounds complicated it is because it really is, some impressive engineering work went into perfecting this and Google even additionally filters out noise in an additional burst of images to get more proper results. And this is the depth mask.

Final result, after HDR+, segmentation mask and depth map

Final result, after HDR+, segmentation mask and depth map


Finally, all of these steps are combined in one single image that looks stunning. Notice how the amount of blur is correct and even the pastry up front is blurred, notice every single piece of hair properly recognized and in sharp focus. Portrait mode on the Pixel 2 works on people with frizzy hair (get that Apple?), on people holding ice cream cones, people with glasses and hats, people holding bouquets and in macro photography.

For the front camera, Google cannot use the last step: the front camera lacks a dual pixel sensor, so it only uses the first two steps, but the results also look much better than the competition that completely lacks such a feature.

Finally, here are some tips Google shares to make the most out of portrait mode and some Portrait images shot on the Pixel 2/XL:

  • Stand close enough to your subjects that their head (or head and shoulders) fill the frame.
  • For a group shot where you want everyone sharp, place them at the same distance from the camera.
  • For a more pleasing blur, put some distance between your subjects and the background.
  • Remove dark sunglasses, floppy hats, giant scarves, and crocodiles.
  • For macro shots, tap to focus to ensure that the object you care about stays sharp.

Full resolution Portrait Mode Pixel 2/XL images here



Related phones

Pixel 2
  • Display 5.0" 1080 x 1920 pixels
  • Camera 12.2 MP / 8 MP front
  • Processor Qualcomm Snapdragon 835, Octa-core, 2350 MHz
  • Storage 128 GB
  • Battery 2700 mAh
Pixel 2 XL
  • Display 6.0" 1440 x 2880 pixels
  • Camera 12.2 MP / 8 MP front
  • Processor Qualcomm Snapdragon 835, Octa-core, 2350 MHz
  • Storage 128 GB
  • Battery 3520 mAh

FEATURED VIDEO

17 Comments

1. Settings

Posts: 2942; Member since: Jul 02, 2014

All these science just to make a bokeh. I'm not upset or anything, actually I'm impressed. Does it work with two or more people? Like "groufies"?

2. Victor.H

Posts: 1026; Member since: May 27, 2011

Yes, it does! Check out the gallery at the end, there is one such picture.

4. sgodsell

Posts: 6705; Member since: Mar 16, 2013

I wonder if Google's portrait mode effects is using their new Pixel Vision Core chip inside every Pixel 2's? Especially since that chip is sticking fast at 3 trillion operations of processing power per second.

7. DnB925Art

Posts: 1167; Member since: May 23, 2013

No, according to them the chip is in the phone, but it won't be activated until Android 8.1 is pushed to the phone.

14. sgodsell

Posts: 6705; Member since: Mar 16, 2013

I believe the 8.1 update will activate the chip for developers to take advantage of it. But who is to say that the chip isn't being used internally on the Pixel 2's right now.

5. Settings

Posts: 2942; Member since: Jul 02, 2014

Yeah saw those earlier, how about a group. I know that's not the scope of portrait photography anymore but I'm impressed and I want to see the limitations of the camera. The ones that trick the camera. That I want to see.

15. worldpeace

Posts: 3091; Member since: Apr 15, 2016

Yeah, they need to do test with cross fences, bike, shrubs/ bushes, 5 peoples with different distances or other tricky shaped stuff. Even better if they compare Pixel 2 XL and note 8's bokeh

3. iamtom

Posts: 1; Member since: Oct 18, 2017

Does it get any better reception? A bigger antennae? How about landscape photography? Does it do tilt-shift mode? Thx!

6. 0kax0el0

Posts: 237; Member since: Nov 15, 2012

Thats crocodile discrimination right there!

12. worldpeace

Posts: 3091; Member since: Apr 15, 2016

Alligators still fine.. Better replace your crocodile with alligator

8. roldefol

Posts: 4744; Member since: Jan 28, 2011

PA, are you deliberately choosing the hottest girl from the photos for the headline pic?

9. nepalisherpa

Posts: 334; Member since: Jul 17, 2015

Those are some amazing pictures. I will consider Pixel next year since I have a Note 8.

10. p51d007

Posts: 697; Member since: Nov 24, 2013

Still looks artificial. Better, but still artificial Bokeh

11. trojan_horse

Posts: 5868; Member since: May 06, 2016

Artificial? Is there a natural bokeh then?

13. worldpeace

Posts: 3091; Member since: Apr 15, 2016

All smartphone's bokeh is artificial, because it's the result from post processing

16. Lauticol

Posts: 404; Member since: Jun 25, 2011

Would be nice to port this into other phones with DPAF like the S7 and S8

17. PhoneCritic

Posts: 1337; Member since: Oct 05, 2011

I have to say Google does an excellent job with their camera software and hardware. Right now Smartphone cameras have gotten so good on theses so called 'flagship" devices that it is hard to go wrong with any- I am personally picking up a Note 8 for its feature set and Camera. To me the best cameras on a Smartphone have always been Samsung since the S5 ( prior to that Apple iPhones had the best hands down) the S6 and S7 changed that until the Pixel. Samsung, later, again took the crown with the S8/ Note 8. Now Google has taken it back very impressively. However, the Note 8 is still a fantastic camera phone and I believe, since Samsung and Google are partners, it won't belong till we see this sort of image processing also make its way to the S series line since they to have the dual pixel hardware. But overall a great Job by Google and to the Pixel owners enjoy you got a hell of a device there.

Latest Stories

This copy is for your personal, non-commercial use only. You can order presentation-ready copies for distribution to your colleagues, clients or customers at https://www.parsintl.com/phonearena or use the Reprints & Permissions tool that appears at the bottom of each web page. Visit https://www.parsintl.com/ for samples and additional information.