The HTC One (M8) Review
by Anand Lal Shimpi & Joshua Ho on March 26, 2014 7:00 PM EST- Posted in
- Smartphones
- HTC
- Mobile
- HTC One
Subtle Cheating: New Benchmark Optimizations
We’ve been tracking the state of benchmark optimizations among Android OEMs for a while now. It wasn’t too long ago that we published a piece calling out nearly all Android OEMs for optimizing for benchmarks. The optimizations are pretty crude. Upon detecting a whitelisted benchmark by APK identifier alone, most OEMs would enable a performance mode that would:
1) Plug in all CPU cores
2) Run all cores at max frequency
3) Raise thermal limits to eliminate/reduce throttling
Defeating the optimization was just as simple: thorough renaming of the benchmark and all internal application IDs. For the past several months we’ve been working with benchmark, silicon and handset vendors to curb the behavior. Although we found the optimizations to have minimal impact on our test results, it’s still a messy practice that isn’t worth doing.
We’ve seen early (encouraging) indications that some vendors have reconsidered their position on benchmark optimizations. Unfortunately HTC isn’t quite there yet.
The M8’s Android 4.4.2 build includes a new, more subtle form of benchmark optimization that we hadn’t seen in previous devices. Benchmarks are still detected according to their application identifier, but instead of hot plugging in all CPU cores and driving them to max frequencies, everything appears to be normal at launch.
Here’s the state of the CPU cores after launching the Play Store version of any optimized benchmark:
Everything looks just fine. But look at what happens if we monitor CPU frequency over time on the Play Store and a special renamed version of 3DMark:
Average CPU frequency is about 15% higher while running the Play Store version of 3DMark. I still need to run some thermal analysis on the device but I don’t think HTC is raising thermal limits. Instead what appears to be happening is HTC is simply more aggressively tuning the governor response to performance demands, allowing for higher frequencies. Note that the frequency response latency is now so low that I couldn't even grab the 300MHz screenshot above in the Play Store version of 3DMark. As soon as the device detected a button press it would ramp up to 1.7GHz.
The impact on performance goes hand in hand with the increase in average clock speed. I measured performance during 3DMark’s Physics test (which is CPU bound). The difference was about 15%.
I also tracked GPU clock speed over time. Thankfully the optimization seems limited to CPU frequencies alone:
The list of optimization targets has also expanded since we last looked at HTC. The latest versions of GFXBench, BaseMark X and BaseMark OS II are now included in the benchmark whitelist.
HTC made one small concession - it’s allowing users the ability to run their device in this high performance mode at all times. Under developer tools (tap on the build number 5 times in Settings > About > Software information > More), you’ll find an option to enable high performance CPU mode. Checking that box will put your device in the same mode that’s enabled when a whitelisted benchmark is detected.
 |
 |
I do appreciate that HTC is exposing the optimization control, the only thing missing is the ability to toggle the benchmark optimization off (not to mention that I’d prefer if it was disabled to begin with). I fear that HTC’s justification in all of this is that everyone else is doing it so why opt out. The reality seems to be trending the other direction however. We’ll have to see what Samsung does with the Galaxy S 5, but I have a feeling that HTC is going to end up on the wrong side of history with this move. All of our benchmarks are already immune to the optimization, so it’s really a matter of sacrificing integrity for no real gain. There’s nothing more to say other than I’m disappointed.
Facebook
Twitter
RSS
222 Comments
View All Comments
Johnmcl7 - Sunday, March 30, 2014 - link
Because there is no such thing as an absolute phone - everyone has different preferences and the market reflects that. I much prefer Samsung's lighter and durable plastic construction which can hold up to daily use and allows a removable battery rather than HTC's heavier metal cases which if used without a case easily pick up scuffs and scratches making them look decidedly second hand within a matter of months.w2aith - Friday, March 28, 2014 - link
Fantastic phone. Good weight, great hand feel. Faster than I expected. Camera is unimpressive, but that is not what i look for in a phone. Sound quality from the speakers on the phone is the best I've heard.Death666Angel - Friday, March 28, 2014 - link
"There’s nothing more to say other than I’m disappointed."Golden! Is it wrong that I read that in my head in Jon Stewart's Jewish Mother voice? :D
Death666Angel - Friday, March 28, 2014 - link
Just finished reading your review.Feeling kinda underwhelmed by the M8. First off, I don't really care about the all metal design of a phone. I can appreciate that for all of 2 seconds before I put a plastic case on it and never see the body again unless it is for cleaning. So that advantage is lost for me. I want to like the camera and I kinda do. But 4MP is too low for me. Decrease pixel size, increase the sensor size a bit and give me 6 or 8. I don't need 13MP at 1.12 (?) um, but 4MP@2um is just not good enough, especially as a follow up to that same camera. I don't do that much low-light photography either, but I've used my SGS2 as a camera on holidays when my point and shoot was low on batteries, so outdoor pictures are very important. Losing OIS is an astonishing move and puzzles me greatly. The 801 is looking pretty cool, the mSD card helps, but those aren't unique features. For the price, the display needs to be calibrated better (although I also don't care much about that). I don't really care about the sound of a phone because my usage model does not involve playing anything without headphones plugged in.
The old M7 is still ~70 to 120€ more expensive than any of its competitors phones (Nexus 5, LG G2, SGS4, Xperia Z, not comparing the same capacities, because I am fine with 16GB on my phone), so I don't think the new M8 will be a price competitive offering. I don't see myself being impressed by the SGS5 either. Note 4 maybe? Haven't heard too much about it. But any smartphone that costs over 400€ needs to offer some serious advantages to get me to consider it instead of a Nexus.
If Nexus 5.2 gets a bigger battery, marginally faster SoC at better power characteristics and a better performing camera (1.4 - 1.7 um with 6 or 8MP image and OIS and a 4 or 5MP front facing camera), I'll be a happy boy and upgrade from my Galaxy Nexus.
tvandegr - Friday, March 28, 2014 - link
Do we know if the Developer edition or the unlocked edition support carrier aggregation? They seem to be based off of the ATT model from what I have seen other places.skylinkdave - Friday, March 28, 2014 - link
You know what's even more fun than a fiddly phone camera? A real camera that's able to do all of this in-camera without all the post.ol1bit - Saturday, March 29, 2014 - link
As always, Anandtech is spot on with the review. Good Job! Just read some of the other reviews for comparison.pdc200 - Saturday, March 29, 2014 - link
Hmmmm so what happened to reviewing things like Call Quality, Boomsound Quality and loudness? Quality of Audio from the Headphone jack?ghm3 - Saturday, March 29, 2014 - link
I've been eagerly awaiting the Sony Xperia Z2 ever since MWC, and this review doesn't change that at all, but at this is clearly my next choice. The addition of a micro SD slot makes this far more appealing to me now.Either way I'm so done with Samsung's cheap garbage loaded with shitware you can't remove without cracking the phone. Beyond the hardware with the Z2 I'm really looking forward to the mostly un-molested Android build on it. Can anyone provide insight into how HTC is in this department?
thisiskicker - Sunday, March 30, 2014 - link
Awesome It has a micro SD card expansion. This is what phones are now a days are missing.