The HTC One (M8) Review
by Anand Lal Shimpi & Joshua Ho on March 26, 2014 7:00 PM EST- Posted in
- Smartphones
- HTC
- Mobile
- HTC One
Subtle Cheating: New Benchmark Optimizations
We’ve been tracking the state of benchmark optimizations among Android OEMs for a while now. It wasn’t too long ago that we published a piece calling out nearly all Android OEMs for optimizing for benchmarks. The optimizations are pretty crude. Upon detecting a whitelisted benchmark by APK identifier alone, most OEMs would enable a performance mode that would:
1) Plug in all CPU cores
2) Run all cores at max frequency
3) Raise thermal limits to eliminate/reduce throttling
Defeating the optimization was just as simple: thorough renaming of the benchmark and all internal application IDs. For the past several months we’ve been working with benchmark, silicon and handset vendors to curb the behavior. Although we found the optimizations to have minimal impact on our test results, it’s still a messy practice that isn’t worth doing.
We’ve seen early (encouraging) indications that some vendors have reconsidered their position on benchmark optimizations. Unfortunately HTC isn’t quite there yet.
The M8’s Android 4.4.2 build includes a new, more subtle form of benchmark optimization that we hadn’t seen in previous devices. Benchmarks are still detected according to their application identifier, but instead of hot plugging in all CPU cores and driving them to max frequencies, everything appears to be normal at launch.
Here’s the state of the CPU cores after launching the Play Store version of any optimized benchmark:
Everything looks just fine. But look at what happens if we monitor CPU frequency over time on the Play Store and a special renamed version of 3DMark:
Average CPU frequency is about 15% higher while running the Play Store version of 3DMark. I still need to run some thermal analysis on the device but I don’t think HTC is raising thermal limits. Instead what appears to be happening is HTC is simply more aggressively tuning the governor response to performance demands, allowing for higher frequencies. Note that the frequency response latency is now so low that I couldn't even grab the 300MHz screenshot above in the Play Store version of 3DMark. As soon as the device detected a button press it would ramp up to 1.7GHz.
The impact on performance goes hand in hand with the increase in average clock speed. I measured performance during 3DMark’s Physics test (which is CPU bound). The difference was about 15%.
I also tracked GPU clock speed over time. Thankfully the optimization seems limited to CPU frequencies alone:
The list of optimization targets has also expanded since we last looked at HTC. The latest versions of GFXBench, BaseMark X and BaseMark OS II are now included in the benchmark whitelist.
HTC made one small concession - it’s allowing users the ability to run their device in this high performance mode at all times. Under developer tools (tap on the build number 5 times in Settings > About > Software information > More), you’ll find an option to enable high performance CPU mode. Checking that box will put your device in the same mode that’s enabled when a whitelisted benchmark is detected.
 |
 |
I do appreciate that HTC is exposing the optimization control, the only thing missing is the ability to toggle the benchmark optimization off (not to mention that I’d prefer if it was disabled to begin with). I fear that HTC’s justification in all of this is that everyone else is doing it so why opt out. The reality seems to be trending the other direction however. We’ll have to see what Samsung does with the Galaxy S 5, but I have a feeling that HTC is going to end up on the wrong side of history with this move. All of our benchmarks are already immune to the optimization, so it’s really a matter of sacrificing integrity for no real gain. There’s nothing more to say other than I’m disappointed.
Facebook
Twitter
RSS
222 Comments
View All Comments
hangfirew8 - Monday, March 31, 2014 - link
OK I will admit I'm not familiar with the black model. I do know something about anodization. Properly done, it penetrates into the metal and light scratches still display the same color. Deep scratches of course still show the base metal color. But I have no idea how deep HTC's anodization went, or how deep you're mate's scratches or rubbings went.Scootiep7 - Friday, March 28, 2014 - link
GAH! So close. This phone is everything I want with the exception of the camera. And the sad part for me is that the camera is so important it trumps all the other outstanding features this phone has. I love everything HTC is doing and the smartphone industry needs them to continue innovating in this direction. But until they can bump the camera up to the S4/iP5s level, I just can't make the switch.HTC, why must you torment me so!
alain2 - Friday, March 28, 2014 - link
Too much useless talk about the camera ! Since last year m7 all comments is about the camera ! But why ! My question is : how many users do actually know how to use the camera ? Even me I don t know how to properly use it. ! So what's the point of all this rubbish comments about the use of ultra pixel camera ! Aside from that HTC proved again that they are the best ! In term of quality and reliability and innovations they are on top !archa1c - Friday, March 28, 2014 - link
Regarding your UFocus comment:"Here I’m clearly separated from the background. You can see some of the limitations of UFocus here as I’m not the only subject in focus."
Technically, if that table was at the same depth/distance from the sensor as you were, it should be in focus. UFocus isn't going for a subject-specific focus effect. It is applying a depth-of-field effect.
With that said, it appears to be very unreliable, and I don't think it is truly able to capture accurate depth information. I've already seen many sample images in good conditions where part of the in-focus subject has been blurred out of focus with the background.
Demi9OD - Friday, March 28, 2014 - link
Are there any new cell phones with manufacturer car docks? My Atrix HD comes up for upgrade soon, but these universal docks with no built in charging are a pain. Need more real solutions like the one I currently have.http://www.amazon.com/Motorola-Vehicle-Navigation-...
liteon163 - Friday, March 28, 2014 - link
I understand it has front facing speakers, but you can't tell me there's no way to reduce the bottom bezel, even if it increases thickness. Integrate the "htc" logo into the bottom speaker grille and get rid of the black stripe. If this increases thickness, add a larger battery, too.kg2128 - Friday, March 28, 2014 - link
wow so much HTC praise on the first page, I was kind of expecting it after last year's extremely biased review but not to this extent. The One still has the best design but does nothing else that sets it apart, and in one respect stayed the same/got worse (camera) and this review is glowing with praise. So disappointing...tytung - Friday, March 28, 2014 - link
It's not just Anandtech. Virtually all of the critics on the internet praised HTC One 2013 and it was named the best phone of the year by GSM.hangfirew8 - Saturday, March 29, 2014 - link
Not just that but Anandtech INVENTED an award for the M7. Maybe "the best design" is more important than "does nothing else". Maybe all those reviewers were really, really tired of seeing the same things from the other players. Maybe this thing called "design" really matters?NicoleMWilson - Friday, March 28, 2014 - link
It continues to baffle me why no Android manufacturer seems inclined to create an "absolute best." It might add an extra $50 to the phone, but many of us would pay that. If Samsung gave up their addiction to butt ugly polycarbonate pincushions, they'd have it. If HTC offered a version with a 13, or even 8, mp camera, they'd have it. Oddball cost cutting doesn't make any sense to me. If they expect me to plunk down $600+ for a phone, there had better be no compromises. http://num.to/2688-3989-2477