The HTC One (M8) Review
by Anand Lal Shimpi & Joshua Ho on March 26, 2014 7:00 PM EST- Posted in
- Smartphones
- HTC
- Mobile
- HTC One
Subtle Cheating: New Benchmark Optimizations
We’ve been tracking the state of benchmark optimizations among Android OEMs for a while now. It wasn’t too long ago that we published a piece calling out nearly all Android OEMs for optimizing for benchmarks. The optimizations are pretty crude. Upon detecting a whitelisted benchmark by APK identifier alone, most OEMs would enable a performance mode that would:
1) Plug in all CPU cores
2) Run all cores at max frequency
3) Raise thermal limits to eliminate/reduce throttling
Defeating the optimization was just as simple: thorough renaming of the benchmark and all internal application IDs. For the past several months we’ve been working with benchmark, silicon and handset vendors to curb the behavior. Although we found the optimizations to have minimal impact on our test results, it’s still a messy practice that isn’t worth doing.
We’ve seen early (encouraging) indications that some vendors have reconsidered their position on benchmark optimizations. Unfortunately HTC isn’t quite there yet.
The M8’s Android 4.4.2 build includes a new, more subtle form of benchmark optimization that we hadn’t seen in previous devices. Benchmarks are still detected according to their application identifier, but instead of hot plugging in all CPU cores and driving them to max frequencies, everything appears to be normal at launch.
Here’s the state of the CPU cores after launching the Play Store version of any optimized benchmark:
Everything looks just fine. But look at what happens if we monitor CPU frequency over time on the Play Store and a special renamed version of 3DMark:
Average CPU frequency is about 15% higher while running the Play Store version of 3DMark. I still need to run some thermal analysis on the device but I don’t think HTC is raising thermal limits. Instead what appears to be happening is HTC is simply more aggressively tuning the governor response to performance demands, allowing for higher frequencies. Note that the frequency response latency is now so low that I couldn't even grab the 300MHz screenshot above in the Play Store version of 3DMark. As soon as the device detected a button press it would ramp up to 1.7GHz.
The impact on performance goes hand in hand with the increase in average clock speed. I measured performance during 3DMark’s Physics test (which is CPU bound). The difference was about 15%.
I also tracked GPU clock speed over time. Thankfully the optimization seems limited to CPU frequencies alone:
The list of optimization targets has also expanded since we last looked at HTC. The latest versions of GFXBench, BaseMark X and BaseMark OS II are now included in the benchmark whitelist.
HTC made one small concession - it’s allowing users the ability to run their device in this high performance mode at all times. Under developer tools (tap on the build number 5 times in Settings > About > Software information > More), you’ll find an option to enable high performance CPU mode. Checking that box will put your device in the same mode that’s enabled when a whitelisted benchmark is detected.
 |
 |
I do appreciate that HTC is exposing the optimization control, the only thing missing is the ability to toggle the benchmark optimization off (not to mention that I’d prefer if it was disabled to begin with). I fear that HTC’s justification in all of this is that everyone else is doing it so why opt out. The reality seems to be trending the other direction however. We’ll have to see what Samsung does with the Galaxy S 5, but I have a feeling that HTC is going to end up on the wrong side of history with this move. All of our benchmarks are already immune to the optimization, so it’s really a matter of sacrificing integrity for no real gain. There’s nothing more to say other than I’m disappointed.
Facebook
Twitter
RSS
222 Comments
View All Comments
berantle - Friday, March 28, 2014 - link
@Death666Angel, it is not a stupid argument. It is a rational reasonable argument.Fundamentally, it is a phone first and whatever else second. Yes, it is capable of other things but it cannot fully replace what other purpose-specific devices do better. Taking into account the size, form factor and the practical realities of smartphones, it is actually pretty foolish to expect that these fully replace the proper devices for it.
The main point about the camera as stated ad nauseum is the camera's resolution is sufficient for what many people use the photos taken from it - i.e. post onto social websites and blogs, and view them on their 1080p (1920x1080) Full HD TV. The 4MP resolution of the HTC One (both M7 and M8) camera equates to 2688 x 1520 images at 16:9 ratio. This size provides more than enough detail for 1080p Full HD TVs that are very common nowadays. Posting on Facebook, Google+, Instagram, Snapchat, etc, the image sizes shown on screen rarely, if ever, go as large as 1920x1080.
Good photos are rarely just about how detailed it shots. It's about picture composition/framing, quality of lenses and proper focusing, and a basic understanding and proper use of photography technicalities. If you're a crap photographer, having a higher number of pixels only means you're likely to be taking crap photos that have more detail.
That all said, personally, 4MP (2688x1520) pictures are at the lower end of my requirements. It would be nice to have a sensor of more 2-micrometer pixels than 4MP. As it is, the 4MP picture resolution of the HTC One camera is adequate for my needs.
Thermalzeal - Friday, March 28, 2014 - link
I don't think it's a broken record. While photo's look similar on a phone's display, they certainly differ in quality when viewed on a higher DPI display. You are right that there is a small audience that appreciates this, but this is a high end product to top off the premium tier of HTC phones. I like the concept of the dual camera, but it would have been nice to get at least a 12MP sensor since they are not really cost prohibitive.Also in the end, you can always down sample a photo, but in the future where bazillionK resolution is the standard, these photo's will look like your mom and dad's black and whites.
evonitzer - Saturday, March 29, 2014 - link
"I like the concept of the dual camera, but it would have been nice to get at least a 12MP sensor since they are not really cost prohibitive."But at what cost in performance? Low light performance will suffer. I do not understand people's difficulty in this minor complexity. The resolution is good enough, and enables greater performance in low light. Daylight shots will lack resolution. We do not need to boil it down any further to "this camera is crap" or "this camera is great". This camera does exactly what it is designed to do and it is up to shoppers to decide if it is right for them.
dimsum888 - Saturday, March 29, 2014 - link
compared to other flagship phones it doesn't perform well. at its price point it is expected all features are comparable to other high end phones.jospoortvliet - Sunday, March 30, 2014 - link
No, compared to other phones it does worse at still pictures with outside day light and better at moving picks and at night. So there is a choice: if you want to take pics inside orat night or of moving objects, the HTC one is best. If you take pics of static objects in good lightning conditions, buy another phone. I know why I prefer my HTC, which makes the better pics in 90% of relevant (for me) circumstances but ymmv.notposting - Monday, March 31, 2014 - link
Or get one of the Lumia's which take better pics both outdoors, indoors, good light, no light, have OIS, have the resolution to allow cropping, some allow for saving a copy in RAW.Between the Note 3, S4/S5, iPhone 5S, Lumia 92x/1020, LG G2, and the Sony lineup, there is basically little reason to choose the HTC if the camera is at all important to you.
augustofretes - Monday, March 31, 2014 - link
The resolution isn't good enough. The detail capture is obscenely poor. People zoom-in their images to see their friends and stuffs in the picture you know? While I agree that 12MP is not really necessary, 8MP + OIS are way a better choice than the nonsense of 4MP + secondary sensor for gimmicky procedures.Blown503 - Monday, March 31, 2014 - link
5 megapixels is an 8x10 picture on paper, 8 megapixels is a 16x20 picture on printed paper. The are printed @ 300 dpi.Chaser - Thursday, April 10, 2014 - link
I've owned the HTC One M7 and if its one area I was disappointed in was its versatility as a point and shoot. Yes the low light level is very good but with larger scale more distance "scenic" outdoor photos there is a difference with the high pixel phone cams. So yes, while a photo of a flower pot on a table may not look "noticeably different" the quality of even smartphone cams vary. I know because I go through flagship smartphones at least once a year sometimes sooner and I use their cameras frequently as I have no desire to carry around an extra camera.Let me make another distinction. If you do take a photo and later desire to crop and resize that photo (and post it on social media for example) the higher pixel cameras won't lose detail like the lower pixels cameras such as my former HTC One would. So enough with the "broken record" about pixels not mattering. I can assure you they do. Even smart phone camera image quality and versatility has improved significantly since two years ago. I sold my HTC One because I was disappointed with it's overall camera quality. I am much happier with my new LG G2's higher pixel camera.
perpetualdark - Wednesday, April 2, 2014 - link
While I agree that the HTC One camera is good enough and even better than other phones in certain situations (ie low light), the megapixel argument does hold water. Unless every picture you take is at standard zoom, the number of pixels in a smartphone is very important. This is NOT a camera with a physical zoom lens on it, so zoom is purely a factor of megapixels. The more pixels you have, the more you can zoom in.Keep in mind, every other camera on smartphones has a smaller lens aperture though, so while you can zoom way in with higher megapixels, you have to have longer exposure to get the same amount of light to the sensor to get the same picture quality, and again the lower the ambient light, the worse the picture. In well lit scenes that are fairly static, an 8 or 16 megapixel camera will be able to zoom way in and get a great picture that is still high resolution even on a 1080p television or monitor. The 4mp might be better in low light and might take great pictures at no zoom, but it cannot touch the zoom capabilities of the higher megapixel cameras. Not without a physical zoom lense, and you aren't going to find that in a smartphone.
Bottom line is that if you like to take a lot of pictures where you need to zoom in, and don't want to always be carrying around a professional grade camera, this isn't the phone for you. And for crying out loud, if you are going to make an argument, at least make a valid one..