Camera Architecture & Duo Cam

The last HTC phone I used with two rear facing cameras was the EVO 3D. Thankfully the M8 One doesn’t fall victim to the 3D fad of the early 2010s. I remember seeing Intel demos years ago (think Moorestown timeframe) that showed a smartphone with two rear cameras, each with a different lens system, in order to computationally improve image quality. Unfortunately, the M8’s Duo Cam system doesn’t go that far either.

HTC One Cameras
  HTC One (M7) HTC One (M8)
Front Camera 2.1MP 5.0MP
Front Camera - Sensor OV2722
(1.4µm, 1/5.8")
S5K5E2
(1.12µm, 1/5")
Front Camera - Focal Length 1.59mm 1.83mm
Front Camera - Max Aperture F/2.0 F/2.0
Rear Camera - Sensor ST VD6869
(2.0 µm, 1/3")
OV4688
(2.0 µm, 1/3")
Rear Camera - Focal Length 3.82mm (28mm eff) 3.82mm (28mm eff)
Rear Camera - Max Aperture F/2.0 F/2.0
Secondary Rear Camera - Sensor - OV2722
(1.4µm, 1/5.8")

The architecture is pretty simple. The primary shooter is still the 4MP UltraPixel camera from last year, without optical image stabilization (more on this later). The secondary sensor is actually the front facing camera module from the M7 (OmniVision OV2722, 1080p sensor with 1.4µm pixels).

Assuming you’re shooting in the right conditions, both sensors capture an image although only the image from the primary sensor is displayed. The second sensor captures what appears to be a wider field of view, as well as stores some depth information about the scene. This added data is stored outside of the normal address space for a JPEG file, and allegedly in a non-readable format (although my guess is with enough time you’d be able to extract the secondary image).

Images captured with Duo Cam enabled end up being around 60% larger than those shot with the rear sensor alone. Around 40% of the added file size is due to the secondary 1080p image, while the remaining 60% of the data is depth information.

HTC uses this depth information to enable a number of post processing effects that can be done on the One itself. Simply take a picture using the rear facing camera, tap edit, then effects and you’re presented with a list of all of the Duo Cam effects available on the One.

UFocus allows you to change the part of the scene in focus, either using the focus plane of the primary or secondary sensor. This isn’t completely open ended where you can selectively focus on anything in a scene. Instead, you’re basically toggling between focus captured by the primary or secondary sensor with the rest of the scene computationally put in/out of focus from what I can tell. In portraiture you end up with simulated bokeh, giving you heavily blurred backgrounds while keeping your subject in sharp focus. I was sitting across a table from Scott Wasson at a coffee shop and decided to snap a photo of him. Then I fired up UFocus, tapped on his face, and in about a second I had a processed result. I've embedded both images below, click on the buttons to toggle between the two images:


It’s an awesome effect, something that would take considerable time in post normally (not to mention time on a PC). You’ll notice the effect isn’t perfect though. Pay attention to the top of his left shoulder (on the right side of the image). There are clearly some difficulties determining boundaries between foreground and background. Scott then turned the camera on me. With my back to a scene with more depth enabling UFocus around took much longer to process, a matter of a few seconds. The result was even more impressive:


Here I’m clearly separated from the background. You can see some of the limitations of UFocus here as I’m not the only subject in focus. The table to my right is also in focus. It’s still impressive, and something that would be very difficult to do with manual editing (without multiple exposures).

UFocus is easily one of the most compelling applications for the Duo Cam. There’s no real flexibility with the feature though. Moving focus to the background is basically useless. And you can forget about using UFocus for anything other than a portrait. In most other scenes you end up with awkward focus boundaries.

Foregrounder is a pretty neat effect. Similar to UFocus you select a subject to be the focal point, but instead of blurring the rest of the scene HTC uses the depth information to apply shading effects to the rest of the scene. The result can be pretty dramatic if you’re shooting the right scene.

I came across this burrito shop in downtown San Jose, home of the Burritozilla:

The scene itself made for a great photo, but have a look at what happens when I use HTC’s Duo Cam to select the sign and apply an effect to everything physically behind it:


In the right scene the effect can be pretty cool.

Seasons is a bit cheesier although I can see it being quite popular with some folks. Here HTC uses the depth information as well as the image from the second sensor to create a panning effect through a scene while computationally adding in some element of a season (e.g. falling leaves or oversized snowflakes). The effect is best shown off in a video:

Dimension Plus uses the second sensor to enable a parallax effect on a given scene, allowing you to tilt the One in any direction to manipulate the effect. Unlike the other effects, this one can’t be easily exported to a video or other commonly used file format.

Stickers let you apply props to an image:

Copy and paste lets you quickly copy a person from one image and paste him/her into another. Even here, HTC is using the depth information captured from the secondary sensor to quickly separate the subject from the background. Typically these types of functions involve the user manually selecting the area of interest, here HTC is using the second sensor and a bunch of compute to do that for you. In practice even HTC’s Duo Cam approach requires user intervention, but not a ton if you’re fine with sharing an unpolished image. I figure many of these effects have pretty high comedic value, so something that’s a little less polished likely isn’t a problem.

Applying any of the Duo Cam effects is non destructive to the source image. You can cancel at any time, and hitting save creates a copy of the image - the original always remains intact (images with effects just get a numerical suffix e.g. _2).

There are a number of situations where Duo Cam won’t work. If you cover up the secondary sensor you’ll get a warning, but you’re still able to take shots. I’ve found the secondary sensor is pretty sensitive to things even partially occluding the image. The less obvious situations where Duo Cam disables itself automatically make sense when you think about the architecture of the setup.

Very low light and macro photography both disable Duo Cam. In the former case you’re dealing with a fairly small 1080p sensor that likely becomes useless as noise levels rise. In the latter case, there’s just nothing left for the secondary sensor to do. Burst mode also disables the second sensor, probably because all of the ISP’s bandwidth is being used for delivering throughput for the primary sensor and can’t be split between the two. Dual capture (front + primary rear sensors in tandem) disables Duo Cam, probably for the same ISP limitations. Finally, Duo Cam also won’t work if the flash fires.

I can see many of the Duo Cam effects being fun things to play around with. I originally wrote them off pretty early on, but just in writing this section alone I can see them resonating very well with some users. They are fun. The good news is that most of the Duo Cam effects can be shared as simple JPEGs or videos with anyone, the bad news is they don’t really do much to further improve image quality. And ultimately that’s one of the biggest challenges with using the M8 to further increase marketshare at the high end.

Introduction & Hardware Return of the UltraPixel
POST A COMMENT

222 Comments

View All Comments

  • berantle - Friday, March 28, 2014 - link

    @Death666Angel, it is not a stupid argument. It is a rational reasonable argument.

    Fundamentally, it is a phone first and whatever else second. Yes, it is capable of other things but it cannot fully replace what other purpose-specific devices do better. Taking into account the size, form factor and the practical realities of smartphones, it is actually pretty foolish to expect that these fully replace the proper devices for it.

    The main point about the camera as stated ad nauseum is the camera's resolution is sufficient for what many people use the photos taken from it - i.e. post onto social websites and blogs, and view them on their 1080p (1920x1080) Full HD TV. The 4MP resolution of the HTC One (both M7 and M8) camera equates to 2688 x 1520 images at 16:9 ratio. This size provides more than enough detail for 1080p Full HD TVs that are very common nowadays. Posting on Facebook, Google+, Instagram, Snapchat, etc, the image sizes shown on screen rarely, if ever, go as large as 1920x1080.

    Good photos are rarely just about how detailed it shots. It's about picture composition/framing, quality of lenses and proper focusing, and a basic understanding and proper use of photography technicalities. If you're a crap photographer, having a higher number of pixels only means you're likely to be taking crap photos that have more detail.

    That all said, personally, 4MP (2688x1520) pictures are at the lower end of my requirements. It would be nice to have a sensor of more 2-micrometer pixels than 4MP. As it is, the 4MP picture resolution of the HTC One camera is adequate for my needs.
    Reply
  • Thermalzeal - Friday, March 28, 2014 - link

    I don't think it's a broken record. While photo's look similar on a phone's display, they certainly differ in quality when viewed on a higher DPI display. You are right that there is a small audience that appreciates this, but this is a high end product to top off the premium tier of HTC phones. I like the concept of the dual camera, but it would have been nice to get at least a 12MP sensor since they are not really cost prohibitive.

    Also in the end, you can always down sample a photo, but in the future where bazillionK resolution is the standard, these photo's will look like your mom and dad's black and whites.
    Reply
  • evonitzer - Saturday, March 29, 2014 - link

    "I like the concept of the dual camera, but it would have been nice to get at least a 12MP sensor since they are not really cost prohibitive."

    But at what cost in performance? Low light performance will suffer. I do not understand people's difficulty in this minor complexity. The resolution is good enough, and enables greater performance in low light. Daylight shots will lack resolution. We do not need to boil it down any further to "this camera is crap" or "this camera is great". This camera does exactly what it is designed to do and it is up to shoppers to decide if it is right for them.
    Reply
  • dimsum888 - Saturday, March 29, 2014 - link

    compared to other flagship phones it doesn't perform well. at its price point it is expected all features are comparable to other high end phones. Reply
  • jospoortvliet - Sunday, March 30, 2014 - link

    No, compared to other phones it does worse at still pictures with outside day light and better at moving picks and at night. So there is a choice: if you want to take pics inside orat night or of moving objects, the HTC one is best. If you take pics of static objects in good lightning conditions, buy another phone. I know why I prefer my HTC, which makes the better pics in 90% of relevant (for me) circumstances but ymmv. Reply
  • notposting - Monday, March 31, 2014 - link

    Or get one of the Lumia's which take better pics both outdoors, indoors, good light, no light, have OIS, have the resolution to allow cropping, some allow for saving a copy in RAW.

    Between the Note 3, S4/S5, iPhone 5S, Lumia 92x/1020, LG G2, and the Sony lineup, there is basically little reason to choose the HTC if the camera is at all important to you.
    Reply
  • augustofretes - Monday, March 31, 2014 - link

    The resolution isn't good enough. The detail capture is obscenely poor. People zoom-in their images to see their friends and stuffs in the picture you know? While I agree that 12MP is not really necessary, 8MP + OIS are way a better choice than the nonsense of 4MP + secondary sensor for gimmicky procedures. Reply
  • Blown503 - Monday, March 31, 2014 - link

    5 megapixels is an 8x10 picture on paper, 8 megapixels is a 16x20 picture on printed paper. The are printed @ 300 dpi. Reply
  • Chaser - Thursday, April 10, 2014 - link

    I've owned the HTC One M7 and if its one area I was disappointed in was its versatility as a point and shoot. Yes the low light level is very good but with larger scale more distance "scenic" outdoor photos there is a difference with the high pixel phone cams. So yes, while a photo of a flower pot on a table may not look "noticeably different" the quality of even smartphone cams vary. I know because I go through flagship smartphones at least once a year sometimes sooner and I use their cameras frequently as I have no desire to carry around an extra camera.

    Let me make another distinction. If you do take a photo and later desire to crop and resize that photo (and post it on social media for example) the higher pixel cameras won't lose detail like the lower pixels cameras such as my former HTC One would. So enough with the "broken record" about pixels not mattering. I can assure you they do. Even smart phone camera image quality and versatility has improved significantly since two years ago. I sold my HTC One because I was disappointed with it's overall camera quality. I am much happier with my new LG G2's higher pixel camera.
    Reply
  • perpetualdark - Wednesday, April 2, 2014 - link

    While I agree that the HTC One camera is good enough and even better than other phones in certain situations (ie low light), the megapixel argument does hold water. Unless every picture you take is at standard zoom, the number of pixels in a smartphone is very important. This is NOT a camera with a physical zoom lens on it, so zoom is purely a factor of megapixels. The more pixels you have, the more you can zoom in.

    Keep in mind, every other camera on smartphones has a smaller lens aperture though, so while you can zoom way in with higher megapixels, you have to have longer exposure to get the same amount of light to the sensor to get the same picture quality, and again the lower the ambient light, the worse the picture. In well lit scenes that are fairly static, an 8 or 16 megapixel camera will be able to zoom way in and get a great picture that is still high resolution even on a 1080p television or monitor. The 4mp might be better in low light and might take great pictures at no zoom, but it cannot touch the zoom capabilities of the higher megapixel cameras. Not without a physical zoom lense, and you aren't going to find that in a smartphone.

    Bottom line is that if you like to take a lot of pictures where you need to zoom in, and don't want to always be carrying around a professional grade camera, this isn't the phone for you. And for crying out loud, if you are going to make an argument, at least make a valid one..
    Reply

Log in

Don't have an account? Sign up now