The HTC One (M8) Review
by Anand Lal Shimpi & Joshua Ho on March 26, 2014 7:00 PM EST- Posted in
- Smartphones
- HTC
- Mobile
- HTC One
Subtle Cheating: New Benchmark Optimizations
We’ve been tracking the state of benchmark optimizations among Android OEMs for a while now. It wasn’t too long ago that we published a piece calling out nearly all Android OEMs for optimizing for benchmarks. The optimizations are pretty crude. Upon detecting a whitelisted benchmark by APK identifier alone, most OEMs would enable a performance mode that would:
1) Plug in all CPU cores
2) Run all cores at max frequency
3) Raise thermal limits to eliminate/reduce throttling
Defeating the optimization was just as simple: thorough renaming of the benchmark and all internal application IDs. For the past several months we’ve been working with benchmark, silicon and handset vendors to curb the behavior. Although we found the optimizations to have minimal impact on our test results, it’s still a messy practice that isn’t worth doing.
We’ve seen early (encouraging) indications that some vendors have reconsidered their position on benchmark optimizations. Unfortunately HTC isn’t quite there yet.
The M8’s Android 4.4.2 build includes a new, more subtle form of benchmark optimization that we hadn’t seen in previous devices. Benchmarks are still detected according to their application identifier, but instead of hot plugging in all CPU cores and driving them to max frequencies, everything appears to be normal at launch.
Here’s the state of the CPU cores after launching the Play Store version of any optimized benchmark:
Everything looks just fine. But look at what happens if we monitor CPU frequency over time on the Play Store and a special renamed version of 3DMark:
Average CPU frequency is about 15% higher while running the Play Store version of 3DMark. I still need to run some thermal analysis on the device but I don’t think HTC is raising thermal limits. Instead what appears to be happening is HTC is simply more aggressively tuning the governor response to performance demands, allowing for higher frequencies. Note that the frequency response latency is now so low that I couldn't even grab the 300MHz screenshot above in the Play Store version of 3DMark. As soon as the device detected a button press it would ramp up to 1.7GHz.
The impact on performance goes hand in hand with the increase in average clock speed. I measured performance during 3DMark’s Physics test (which is CPU bound). The difference was about 15%.
I also tracked GPU clock speed over time. Thankfully the optimization seems limited to CPU frequencies alone:
The list of optimization targets has also expanded since we last looked at HTC. The latest versions of GFXBench, BaseMark X and BaseMark OS II are now included in the benchmark whitelist.
HTC made one small concession - it’s allowing users the ability to run their device in this high performance mode at all times. Under developer tools (tap on the build number 5 times in Settings > About > Software information > More), you’ll find an option to enable high performance CPU mode. Checking that box will put your device in the same mode that’s enabled when a whitelisted benchmark is detected.
I do appreciate that HTC is exposing the optimization control, the only thing missing is the ability to toggle the benchmark optimization off (not to mention that I’d prefer if it was disabled to begin with). I fear that HTC’s justification in all of this is that everyone else is doing it so why opt out. The reality seems to be trending the other direction however. We’ll have to see what Samsung does with the Galaxy S 5, but I have a feeling that HTC is going to end up on the wrong side of history with this move. All of our benchmarks are already immune to the optimization, so it’s really a matter of sacrificing integrity for no real gain. There’s nothing more to say other than I’m disappointed.
222 Comments
View All Comments
nerdstalker - Thursday, March 27, 2014 - link
It could have been a good move by Brian if he updated his info on Twitter, LinkedIn etc, to reflect any changes (if there is) to his affiliation with AnandTech. A short "thank you, good luck and btw we left in good terms" confirmation tweet from Anand (similar to his post above) would clear most questions. Just my 2 cents. :)Anand Lal Shimpi - Thursday, March 27, 2014 - link
All in due time, privacy is an important thing :)Anand Lal Shimpi - Thursday, March 27, 2014 - link
Audio quality is coming, just didn't have the time to get the review unit out to Chris. Similarly time constrained on network connectivity, I plan on grabbing the data when I'm back in NC this weekend.yelped - Thursday, March 27, 2014 - link
Ok. Thanks for the clarification.George Carlin Jr. - Wednesday, March 26, 2014 - link
Only a 4MP camera? WTF? Why can't just ONE single phone maker make a phone with the best of everything? They always skimp or screw up at least one thing. Why is that?madwolfa - Wednesday, March 26, 2014 - link
Never needed more than 4 megapixels for sharing pics on Facebook (which is what 99% people are doing with their phones). For anything else, I have my DSLR.Blairh - Thursday, March 27, 2014 - link
Total BS.Egg - Wednesday, March 26, 2014 - link
Read the 2013 HTC One review to understand this design choice...AnnonymousCoward - Wednesday, March 26, 2014 - link
4MP made great sense a couple years ago, but now days more people are getting higher res screens. 30"ers are 4.1MP, and some new tablets and laptops are also in the territory. I think ~2x the screen res is a good amount for the sensor, so that pixel averaging works well for the final product. I think they should have gone with 6-8MP, same pixel size, and with optical stabilization.asaini007 - Thursday, March 27, 2014 - link
I feel like they could've if they would've. Perhaps such components don't exist, are far too expensive, or would not meet design constraints. They did they best with what they could I assume.And I actually think the secondary camera is kinda neat for fun effects, which will appeal to the mass market which HTC desperately needs.