Camera Architecture & Duo Cam

The last HTC phone I used with two rear facing cameras was the EVO 3D. Thankfully the M8 One doesn’t fall victim to the 3D fad of the early 2010s. I remember seeing Intel demos years ago (think Moorestown timeframe) that showed a smartphone with two rear cameras, each with a different lens system, in order to computationally improve image quality. Unfortunately, the M8’s Duo Cam system doesn’t go that far either.

HTC One Cameras
  HTC One (M7) HTC One (M8)
Front Camera 2.1MP 5.0MP
Front Camera - Sensor OV2722
(1.4µm, 1/5.8")
S5K5E2
(1.12µm, 1/5")
Front Camera - Focal Length 1.59mm 1.83mm
Front Camera - Max Aperture F/2.0 F/2.0
Rear Camera - Sensor ST VD6869
(2.0 µm, 1/3")
OV4688
(2.0 µm, 1/3")
Rear Camera - Focal Length 3.82mm (28mm eff) 3.82mm (28mm eff)
Rear Camera - Max Aperture F/2.0 F/2.0
Secondary Rear Camera - Sensor - OV2722
(1.4µm, 1/5.8")

The architecture is pretty simple. The primary shooter is still the 4MP UltraPixel camera from last year, without optical image stabilization (more on this later). The secondary sensor is actually the front facing camera module from the M7 (OmniVision OV2722, 1080p sensor with 1.4µm pixels).

Assuming you’re shooting in the right conditions, both sensors capture an image although only the image from the primary sensor is displayed. The second sensor captures what appears to be a wider field of view, as well as stores some depth information about the scene. This added data is stored outside of the normal address space for a JPEG file, and allegedly in a non-readable format (although my guess is with enough time you’d be able to extract the secondary image).

Images captured with Duo Cam enabled end up being around 60% larger than those shot with the rear sensor alone. Around 40% of the added file size is due to the secondary 1080p image, while the remaining 60% of the data is depth information.

HTC uses this depth information to enable a number of post processing effects that can be done on the One itself. Simply take a picture using the rear facing camera, tap edit, then effects and you’re presented with a list of all of the Duo Cam effects available on the One.

UFocus allows you to change the part of the scene in focus, either using the focus plane of the primary or secondary sensor. This isn’t completely open ended where you can selectively focus on anything in a scene. Instead, you’re basically toggling between focus captured by the primary or secondary sensor with the rest of the scene computationally put in/out of focus from what I can tell. In portraiture you end up with simulated bokeh, giving you heavily blurred backgrounds while keeping your subject in sharp focus. I was sitting across a table from Scott Wasson at a coffee shop and decided to snap a photo of him. Then I fired up UFocus, tapped on his face, and in about a second I had a processed result. I've embedded both images below, click on the buttons to toggle between the two images:


It’s an awesome effect, something that would take considerable time in post normally (not to mention time on a PC). You’ll notice the effect isn’t perfect though. Pay attention to the top of his left shoulder (on the right side of the image). There are clearly some difficulties determining boundaries between foreground and background. Scott then turned the camera on me. With my back to a scene with more depth enabling UFocus around took much longer to process, a matter of a few seconds. The result was even more impressive:


Here I’m clearly separated from the background. You can see some of the limitations of UFocus here as I’m not the only subject in focus. The table to my right is also in focus. It’s still impressive, and something that would be very difficult to do with manual editing (without multiple exposures).

UFocus is easily one of the most compelling applications for the Duo Cam. There’s no real flexibility with the feature though. Moving focus to the background is basically useless. And you can forget about using UFocus for anything other than a portrait. In most other scenes you end up with awkward focus boundaries.

Foregrounder is a pretty neat effect. Similar to UFocus you select a subject to be the focal point, but instead of blurring the rest of the scene HTC uses the depth information to apply shading effects to the rest of the scene. The result can be pretty dramatic if you’re shooting the right scene.

I came across this burrito shop in downtown San Jose, home of the Burritozilla:

The scene itself made for a great photo, but have a look at what happens when I use HTC’s Duo Cam to select the sign and apply an effect to everything physically behind it:


In the right scene the effect can be pretty cool.

Seasons is a bit cheesier although I can see it being quite popular with some folks. Here HTC uses the depth information as well as the image from the second sensor to create a panning effect through a scene while computationally adding in some element of a season (e.g. falling leaves or oversized snowflakes). The effect is best shown off in a video:

Dimension Plus uses the second sensor to enable a parallax effect on a given scene, allowing you to tilt the One in any direction to manipulate the effect. Unlike the other effects, this one can’t be easily exported to a video or other commonly used file format.

Stickers let you apply props to an image:

Copy and paste lets you quickly copy a person from one image and paste him/her into another. Even here, HTC is using the depth information captured from the secondary sensor to quickly separate the subject from the background. Typically these types of functions involve the user manually selecting the area of interest, here HTC is using the second sensor and a bunch of compute to do that for you. In practice even HTC’s Duo Cam approach requires user intervention, but not a ton if you’re fine with sharing an unpolished image. I figure many of these effects have pretty high comedic value, so something that’s a little less polished likely isn’t a problem.

Applying any of the Duo Cam effects is non destructive to the source image. You can cancel at any time, and hitting save creates a copy of the image - the original always remains intact (images with effects just get a numerical suffix e.g. _2).

There are a number of situations where Duo Cam won’t work. If you cover up the secondary sensor you’ll get a warning, but you’re still able to take shots. I’ve found the secondary sensor is pretty sensitive to things even partially occluding the image. The less obvious situations where Duo Cam disables itself automatically make sense when you think about the architecture of the setup.

Very low light and macro photography both disable Duo Cam. In the former case you’re dealing with a fairly small 1080p sensor that likely becomes useless as noise levels rise. In the latter case, there’s just nothing left for the secondary sensor to do. Burst mode also disables the second sensor, probably because all of the ISP’s bandwidth is being used for delivering throughput for the primary sensor and can’t be split between the two. Dual capture (front + primary rear sensors in tandem) disables Duo Cam, probably for the same ISP limitations. Finally, Duo Cam also won’t work if the flash fires.

I can see many of the Duo Cam effects being fun things to play around with. I originally wrote them off pretty early on, but just in writing this section alone I can see them resonating very well with some users. They are fun. The good news is that most of the Duo Cam effects can be shared as simple JPEGs or videos with anyone, the bad news is they don’t really do much to further improve image quality. And ultimately that’s one of the biggest challenges with using the M8 to further increase marketshare at the high end.

Introduction & Hardware Return of the UltraPixel
POST A COMMENT

222 Comments

View All Comments

  • jonup - Friday, March 28, 2014 - link

    My point exactly! I was not arguing that the One has the best or better camera than the other flagships. All I said is that people a bashing the phone over a camera that is good enough (ok if you wish) in the day light but it (the phone) is exceptional in any other way (save for the wasted real estate by the logo). I would give up 600mah of battery to get rid of the black stripe underneath the display. Which brings me to the other Samsung sponsored reviewers argument - on screen keys vs capacitive key. On-screen keys is the Google way. Not the iOS Samsung copy cat way. I wish there is more custom-ability in the GEL and in the OEM laucher to set height of the nav bar, the width of the buttons, and the choice of buttons. As usual my two cents. Reply
  • mkozakewich - Sunday, March 30, 2014 - link

    It doesn't take steady video, which is where it's more important. All the regular photos in this article were still shots, except for the video demonstrating the difference between the M7 and the M8. Reply
  • sephirotic - Sunday, March 30, 2014 - link

    4k recording is worse than gimmick on any 1/3" sensor. Compression efficiency on smartphones is very low and unoptimized for video, there is simply no way to retain actual 1080p motion resolution with currect processors and pixel pitch of 1/3 sensors. Z2 samples makes this obvious, Actual true resolution on Z2 is worse than a semi-professional 720p video on a 2010's GH1. I'd rather have a more efficient 1080p compression and slow motion than 4k. EVEN IF there was actualy improvement from 1080p videos on 4k recorded from smartphone, 99% of regular costumers woudn´t even have a 4k screen to display it. Using for mastering casual videos? Even less likely.
    4k video on smartphone is just plain idiotic.

    However I do agree 4mp is a little on the low side even for 1/3" sensor but no doubt pushing anything beyond 8mp is pure gimmick. it's a shame that sony kept pushing the megapixel race for so long even after the other manufactures stoped with this nonsense and now this is back on Smartphones and Video.
    I do agree removing OIS is sad drawback.
    Reply
  • CoryWeston101 - Monday, March 31, 2014 - link

    Let's get some facts straight. NO ONE NEEDS 4K recording on a smartphone or period. Less than 1 percent of the poulation has 4K TVS or monitors making 4K recording a useless gimmick. You need it that badly go by an actually 4K camera from Sony or Cannon or someone. It will do a better job. But 4K recording is a stupid useless gimmick

    They took OIS out because of the 2 camera set up. And also with a 2 camera set up YOU DON'T NEED OIS as it essentially does the same thing only better. The camera is a giant leap forward And no that's not why it is getting knocked. It is getting knocked because of the uneducated people out their like.

    You bring the Moto X up? The Moto X camera sucks. The M8 camera makes the X camera look like a childs toy.

    The M8 has one of the best cameras on a smartphone. They did not regress. They progressed. You are just to daft to understand.
    Reply
  • CalaverasGrande - Monday, March 31, 2014 - link

    4k recording on your smartphone?
    What phone DOES do that now.
    Reply
  • Anand R - Tuesday, April 1, 2014 - link

    You want 4K recording? What for? Do you have a 4K screen? Besides, even if you do, why not use an actual camera for 4K recording (which would be better in every way) rather than use a phone camera? Reply
  • purerice - Saturday, April 12, 2014 - link

    Complaining about lack of recording 4K... on a cellphone... with 16GB total storage...
    Please tell me you forgot the "/sarcasm" tag.

    Few cell phone cameras can compare to even a low end PaS camera in good light. Reduce the light level, increase the distance, add some motion, and the point&shoot wins by leaps and bounds. Olympus and Panasonic make decent all-weather cameras that fit in your pocket for 1/2 the price of a flagship cellphone.

    If the phone has a good enough camera for video calling and quick snaps, it's good enough for me. I would no more replace a real camera with a cellphone because the cellphone can remedially take photos, than I would replace a computer with a cellphone because my cellphone can remedially edit spreadsheets.
    Reply
  • doosh bag - Thursday, March 27, 2014 - link

    I totally agree. The camera is more than capable enough in the opinion of professional photographers, but these nudniks out here, these spec snobs can't seem to find anything better to do than regurgitate the same tired, flimsy rhetoric, they read from some meshuggah reviewer, again and again. Far as I can gather from the comments on this first page, none of these yentas even own a M7 or a M8. It's all third person narrative. I own the M7 and let me tell you something, that camera is boss. For example, I was driving down the interstate in the desert, around 70 mph, driving into a beautiful sunset. I held the phone up to the side window and held the shutter button for a few seconds. Got about 20 shots. In one shot there was a semi truck going by, in the opposite direction, in the farthest lane away from me, traveling roughly the same speed, about 200 feet away, in failing light. So, combined speed between the two of us, about 130 mph. I snapped these photos in portrait mode. The picture looked as if everything were perfectly still. The detail so sharp that, without zooming the photo at all or flipping it to landscape view, I can clearly see and count all the lug nuts on the front wheel of that truck. There is nothing wrong with that camera. That camera does everything it's supposed to do and it does it exceptionally well. It's every bit as good as most, and almost as good as some. Comparison after comparison has proven it to be more than adequate. Someone mentioned 4K recording. Let me explain something to you people. 4K is a gimmick. It's a joke. 4K went be commercially viable for another 5 years, if then. They don't even have proper codecs to deal with it. Still using HD codecs to render Ultra HD source code. No phone can render the 4K video it shoots. Televisions and monitors are extremely expensive. Reply
  • puremind - Saturday, March 29, 2014 - link

    I also totally agree. The M7 convinced me more in terms of shooting clean blur free, fast focused pictures. Other phones have let me down big time. MP give you sharper pictures if you are lucky... That's 30% of the time. I have regretted not being able to crop but I liked thetrade off of faster shooting. Reply
  • Death666Angel - Friday, March 28, 2014 - link

    "If you are into photography that you care so much about picture detail, you won't be using you phone to take pictures."
    That is such a stupid argument. Considering that smartphones are alround devices capable of many things, you can say that just about every feature;
    - if you care about high speed browsing, you will use a laptop/desktop
    - if you care about listening to music, you will use headphones/DAC/dedicated mobile players
    - if you care about video chat, you will use a 4k camera and a high end beamer
    That do a lot and thus nothing really spectacularly. But that's not the point here, is it?
    Reply

Log in

Don't have an account? Sign up now