The HTC One (M8) Review
by Anand Lal Shimpi & Joshua Ho on March 26, 2014 7:00 PM EST- Posted in
- Smartphones
- HTC
- Mobile
- HTC One
Subtle Cheating: New Benchmark Optimizations
We’ve been tracking the state of benchmark optimizations among Android OEMs for a while now. It wasn’t too long ago that we published a piece calling out nearly all Android OEMs for optimizing for benchmarks. The optimizations are pretty crude. Upon detecting a whitelisted benchmark by APK identifier alone, most OEMs would enable a performance mode that would:
1) Plug in all CPU cores
2) Run all cores at max frequency
3) Raise thermal limits to eliminate/reduce throttling
Defeating the optimization was just as simple: thorough renaming of the benchmark and all internal application IDs. For the past several months we’ve been working with benchmark, silicon and handset vendors to curb the behavior. Although we found the optimizations to have minimal impact on our test results, it’s still a messy practice that isn’t worth doing.
We’ve seen early (encouraging) indications that some vendors have reconsidered their position on benchmark optimizations. Unfortunately HTC isn’t quite there yet.
The M8’s Android 4.4.2 build includes a new, more subtle form of benchmark optimization that we hadn’t seen in previous devices. Benchmarks are still detected according to their application identifier, but instead of hot plugging in all CPU cores and driving them to max frequencies, everything appears to be normal at launch.
Here’s the state of the CPU cores after launching the Play Store version of any optimized benchmark:
Everything looks just fine. But look at what happens if we monitor CPU frequency over time on the Play Store and a special renamed version of 3DMark:
Average CPU frequency is about 15% higher while running the Play Store version of 3DMark. I still need to run some thermal analysis on the device but I don’t think HTC is raising thermal limits. Instead what appears to be happening is HTC is simply more aggressively tuning the governor response to performance demands, allowing for higher frequencies. Note that the frequency response latency is now so low that I couldn't even grab the 300MHz screenshot above in the Play Store version of 3DMark. As soon as the device detected a button press it would ramp up to 1.7GHz.
The impact on performance goes hand in hand with the increase in average clock speed. I measured performance during 3DMark’s Physics test (which is CPU bound). The difference was about 15%.
I also tracked GPU clock speed over time. Thankfully the optimization seems limited to CPU frequencies alone:
The list of optimization targets has also expanded since we last looked at HTC. The latest versions of GFXBench, BaseMark X and BaseMark OS II are now included in the benchmark whitelist.
HTC made one small concession - it’s allowing users the ability to run their device in this high performance mode at all times. Under developer tools (tap on the build number 5 times in Settings > About > Software information > More), you’ll find an option to enable high performance CPU mode. Checking that box will put your device in the same mode that’s enabled when a whitelisted benchmark is detected.
 |
 |
I do appreciate that HTC is exposing the optimization control, the only thing missing is the ability to toggle the benchmark optimization off (not to mention that I’d prefer if it was disabled to begin with). I fear that HTC’s justification in all of this is that everyone else is doing it so why opt out. The reality seems to be trending the other direction however. We’ll have to see what Samsung does with the Galaxy S 5, but I have a feeling that HTC is going to end up on the wrong side of history with this move. All of our benchmarks are already immune to the optimization, so it’s really a matter of sacrificing integrity for no real gain. There’s nothing more to say other than I’m disappointed.
Facebook
Twitter
RSS
222 Comments
View All Comments
Gam3sTr - Thursday, March 27, 2014 - link
Htc m8 in Asia will be equipped with the ac version of snapdragon 801...geniekid - Thursday, March 27, 2014 - link
They should've dropped their logo from the front since they're using on screen buttons now. That would've reduced the height close to the M7.asaini007 - Thursday, March 27, 2014 - link
They would if they could. Why does nobody realize that? There exists such a thing as physical size constraints.Death666Angel - Friday, March 28, 2014 - link
But only because of their internal space arrangement, maybe even the secondary camera. There are enough phones out there with minimal bezel that your statement is a bit weird.asaini007 - Saturday, March 29, 2014 - link
You're ignoring the boomsound speakers. Those take space. If you remove those (and add a little to make room for the front facing camera and other sensors) then the One would have relatively thin bezels. People like the boomsound and then complain about bezels. Honestly it's one or the other.LifeEngineer - Thursday, March 27, 2014 - link
@AnandI trust all your reviews and you are THE best reviewer ever!
In one word, "LG G2" or "HTC M8"?
Thanks.
Human Bass - Thursday, March 27, 2014 - link
I would go for the g2, i find a 4MP camera unacceptable.jospoortvliet - Sunday, March 30, 2014 - link
I'd go for the HTC, most pictures are made under mediocre light circumstances and the HTC will thus usually beat the crap out of the g2 in terms of image quality...pjcamp - Thursday, March 27, 2014 - link
I had such hopes for this one. Unfortunately, a solid camera is a core function of a smartphone, second only to phone calls. Often, it is the only camera we have with us. This one is clearly intended for nothing but snapshots for text messages and maybe web posting. For those of us with large displays or who, god forbid, still print the occasional picture and hang it on our walls, it is crap. The photos I saw here were somewhat desaturated colors with blown highlights and would be pixellated to death if printed at any reasonable resolution. Comedy effects are not an adequate substitute and would not be used after the novelty wears off (though you really should cultivate an actual Simon Legree mustache). On my last vacation, I forgot my usual camera so the phone was all I had with me to take pictures of the visit to Mount St. Helens. God forbid that phone was the One.It continues to baffle me why no Android manufacturer seems inclined to create an "absolute best." It might add an extra $50 to the phone, but many of us would pay that. If Samsung gave up their addiction to butt ugly polycarbonate pincushions, they'd have it. If HTC offered a version with a 13, or even 8, mp camera, they'd have it. Oddball cost cutting doesn't make any sense to me. If they expect me to plunk down $600+ for a phone, there had better be no compromises.
The One is not that phone. I need a new device, but I fear I'm going to get into the summer having my choice of a plastic Galaxy, a One with no usable camera, and a G3 with no sd card slot. Android manufacturers, pay attention! Fix any one of these problems and you will have hit it out of the park.
Blairh - Thursday, March 27, 2014 - link
I find it odd that you forgot your 'usual camera' for a vacation. Who forgets their camera in such a scenario?After handling the M8 I went ahead and purchased a Nexus 5 online. The quality of the camera appears to be fine for moments where I don't have my dedicated shooter. It's really a shame that an Android device with the camera of the Nokia 1020 does not exist. The G3 isn't coming until this fall I believe. If you must have an SD card slot and hate Samsung phones you might be SOL right now. (Or maybe the Sony Z2?)