The HTC One (M8) Review
by Anand Lal Shimpi & Joshua Ho on March 26, 2014 7:00 PM EST- Posted in
- Smartphones
- HTC
- Mobile
- HTC One
Subtle Cheating: New Benchmark Optimizations
We’ve been tracking the state of benchmark optimizations among Android OEMs for a while now. It wasn’t too long ago that we published a piece calling out nearly all Android OEMs for optimizing for benchmarks. The optimizations are pretty crude. Upon detecting a whitelisted benchmark by APK identifier alone, most OEMs would enable a performance mode that would:
1) Plug in all CPU cores
2) Run all cores at max frequency
3) Raise thermal limits to eliminate/reduce throttling
Defeating the optimization was just as simple: thorough renaming of the benchmark and all internal application IDs. For the past several months we’ve been working with benchmark, silicon and handset vendors to curb the behavior. Although we found the optimizations to have minimal impact on our test results, it’s still a messy practice that isn’t worth doing.
We’ve seen early (encouraging) indications that some vendors have reconsidered their position on benchmark optimizations. Unfortunately HTC isn’t quite there yet.
The M8’s Android 4.4.2 build includes a new, more subtle form of benchmark optimization that we hadn’t seen in previous devices. Benchmarks are still detected according to their application identifier, but instead of hot plugging in all CPU cores and driving them to max frequencies, everything appears to be normal at launch.
Here’s the state of the CPU cores after launching the Play Store version of any optimized benchmark:
Everything looks just fine. But look at what happens if we monitor CPU frequency over time on the Play Store and a special renamed version of 3DMark:
Average CPU frequency is about 15% higher while running the Play Store version of 3DMark. I still need to run some thermal analysis on the device but I don’t think HTC is raising thermal limits. Instead what appears to be happening is HTC is simply more aggressively tuning the governor response to performance demands, allowing for higher frequencies. Note that the frequency response latency is now so low that I couldn't even grab the 300MHz screenshot above in the Play Store version of 3DMark. As soon as the device detected a button press it would ramp up to 1.7GHz.
The impact on performance goes hand in hand with the increase in average clock speed. I measured performance during 3DMark’s Physics test (which is CPU bound). The difference was about 15%.
I also tracked GPU clock speed over time. Thankfully the optimization seems limited to CPU frequencies alone:
The list of optimization targets has also expanded since we last looked at HTC. The latest versions of GFXBench, BaseMark X and BaseMark OS II are now included in the benchmark whitelist.
HTC made one small concession - it’s allowing users the ability to run their device in this high performance mode at all times. Under developer tools (tap on the build number 5 times in Settings > About > Software information > More), you’ll find an option to enable high performance CPU mode. Checking that box will put your device in the same mode that’s enabled when a whitelisted benchmark is detected.
 |
 |
I do appreciate that HTC is exposing the optimization control, the only thing missing is the ability to toggle the benchmark optimization off (not to mention that I’d prefer if it was disabled to begin with). I fear that HTC’s justification in all of this is that everyone else is doing it so why opt out. The reality seems to be trending the other direction however. We’ll have to see what Samsung does with the Galaxy S 5, but I have a feeling that HTC is going to end up on the wrong side of history with this move. All of our benchmarks are already immune to the optimization, so it’s really a matter of sacrificing integrity for no real gain. There’s nothing more to say other than I’m disappointed.
Facebook
Twitter
RSS
222 Comments
View All Comments
HangFire - Thursday, March 27, 2014 - link
The problem with creating an "absolute best" (in all respects) phone is time-to-market; by the time you get all the top-of-the-line components sorted out, it's obsolete in some way, a competitor just bested one aspect or another.deskjob - Thursday, March 27, 2014 - link
I agree that the M8's camera is less than ideal. I personally would've taken OIS over the second sensor. If I had my pick - a 8MP @ 2um with OIS, and they can throw in the second sensor if there's room.That being said, I think you have to keep in mind, a big chunk of photos taken with phones are strictly shared digitally, and most likely via social networking sites. I have never printed out a photo taken with my phone. I make sure to have a "real" camera, whether a decent pocket or DSLR, for those situations where photo quality really matters.
Of course it all boils down to each person's unique priorities. And again, I agree they should have gone up to a 8MP main shooter @ 2um. The current 4MP doesn't not leave much spatial resolution for down-sampling.
HangFire - Friday, March 28, 2014 - link
Thinking about it, 8MP @ 2uM might require more focal length and glass then the phone's profile would allow, and I don't want to carry one of those big ugly phone/cam hybrids like Samsung and Sony have made. Nokia has shown us how it's done. I don't care exactly what the MegalPixel count or sensor size is. I just want a cam that takes a really good pic within 1/4 second of being triggered, on a flagship phone.itsthesun - Thursday, March 27, 2014 - link
OK guys whatever about this phone... Where is Briankevith - Thursday, March 27, 2014 - link
Great review!One question: Is the camera lens still covered with a soft plastic, that will scratch very easiy?
Taracta - Thursday, March 27, 2014 - link
Anand,How about a running (side) list of the cellular bands that carriers support, not just what the phones support, in each of your reviews?
We would like to see the bands that the carriers uses and whether they support carrier aggregation (CA) where applicable and compare that to what the phones actually support and whether all feature of the carriers are implemented or just some. There could also be times where carriers add additional bands which could make a difference in buying decisions.
jk1 - Thursday, March 27, 2014 - link
people have different priorities in choosing a device. for me, i care about call quality both incoming and outgoing [not addressed], sound quality as a music player to headphones- choice of dac, fidelity of output [not addressed]. and i'd prefer a replaceable battery and the ability to use a micro sd card. [specified in the review]. then comes input options: ocr for documents or for handwriting for a note-type device.the review seemed to be mostly about the camera and - essentially- the phone's ability to play games. and of course, it's ability to make android users look almost as fashionable as apple users.
when i care about pictures i use a real camera- it produces better output than any phone camera. i don't play games. and i don't think of my phone as a fashion statement. thus the review left out the content that would have been most valuable to me.
JacksonSparks - Thursday, March 27, 2014 - link
This is a depressing comment. In summary, booooo.evonitzer - Saturday, March 29, 2014 - link
God I wish there was upvoting on Anandtech comments. +1I leave my phone in my pocket, get dirt in my pocket, and then have to meticulously clean my phone off, none of which was addressed in this review. Why do I even go on living?
sevenmack - Wednesday, April 2, 2014 - link
I understand your point, jk1. From where I sit, camera and build quality are both important. But not so important that other matters -- including call quality, and sound quality as a music player -- are rendered unimportant. A top-notch review should cover all of those issues, and this one doesn't; I would also say the same for reviews of the M8 by Anandtech's competing sites.Based on my own handling of the M8, the phone actually does well on all those counts. The call quality is actually quite solid, and the sound quality is more than good enough for audiophiles; like the M7, the M8 can also play lossless audio such as FLAC and does so well. The BoomSound speakers do the job and this generation's version is louder than that on the M7. But put in your Audio-Technica ATH-M50x headsets and the sound is sublime, as good as you can get on a smartphone.
The camera? Not perfect, but it is also better than most reviewers will admit; the big problem has more to do with the inability of many users to do something as simple as adjust settings than with pixel counts. If pro shutterbugs such as Colby Brown think the M8 is good enough for their work, then the rest of us should stop carping. And the phone is definitely nice to hold; even though it is longer than the M7, it is still easy to handle with one hand. You don't need to worry much about getting it dirty or scratched.