Subtle Cheating: New Benchmark Optimizations

We’ve been tracking the state of benchmark optimizations among Android OEMs for a while now. It wasn’t too long ago that we published a piece calling out nearly all Android OEMs for optimizing for benchmarks. The optimizations are pretty crude. Upon detecting a whitelisted benchmark by APK identifier alone, most OEMs would enable a performance mode that would:

1) Plug in all CPU cores
2) Run all cores at max frequency
3) Raise thermal limits to eliminate/reduce throttling

Defeating the optimization was just as simple: thorough renaming of the benchmark and all internal application IDs. For the past several months we’ve been working with benchmark, silicon and handset vendors to curb the behavior. Although we found the optimizations to have minimal impact on our test results, it’s still a messy practice that isn’t worth doing.

We’ve seen early (encouraging) indications that some vendors have reconsidered their position on benchmark optimizations. Unfortunately HTC isn’t quite there yet.

The M8’s Android 4.4.2 build includes a new, more subtle form of benchmark optimization that we hadn’t seen in previous devices. Benchmarks are still detected according to their application identifier, but instead of hot plugging in all CPU cores and driving them to max frequencies, everything appears to be normal at launch.

Here’s the state of the CPU cores after launching the Play Store version of any optimized benchmark:

Everything looks just fine. But look at what happens if we monitor CPU frequency over time on the Play Store and a special renamed version of 3DMark:

Average CPU frequency is about 15% higher while running the Play Store version of 3DMark. I still need to run some thermal analysis on the device but I don’t think HTC is raising thermal limits. Instead what appears to be happening is HTC is simply more aggressively tuning the governor response to performance demands, allowing for higher frequencies. Note that the frequency response latency is now so low that I couldn't even grab the 300MHz screenshot above in the Play Store version of 3DMark. As soon as the device detected a button press it would ramp up to 1.7GHz.

The impact on performance goes hand in hand with the increase in average clock speed. I measured performance during 3DMark’s Physics test (which is CPU bound). The difference was about 15%.

I also tracked GPU clock speed over time. Thankfully the optimization seems limited to CPU frequencies alone:

The list of optimization targets has also expanded since we last looked at HTC. The latest versions of GFXBench, BaseMark X and BaseMark OS II are now included in the benchmark whitelist.

HTC made one small concession - it’s allowing users the ability to run their device in this high performance mode at all times. Under developer tools (tap on the build number 5 times in Settings > About > Software information > More), you’ll find an option to enable high performance CPU mode. Checking that box will put your device in the same mode that’s enabled when a whitelisted benchmark is detected.

I do appreciate that HTC is exposing the optimization control, the only thing missing is the ability to toggle the benchmark optimization off (not to mention that I’d prefer if it was disabled to begin with). I fear that HTC’s justification in all of this is that everyone else is doing it so why opt out. The reality seems to be trending the other direction however. We’ll have to see what Samsung does with the Galaxy S 5, but I have a feeling that HTC is going to end up on the wrong side of history with this move. All of our benchmarks are already immune to the optimization, so it’s really a matter of sacrificing integrity for no real gain. There’s nothing more to say other than I’m disappointed.

Sense 6.0, Motion Launch & Sensor Hub Snapdragon 801 Performance
POST A COMMENT

222 Comments

View All Comments

  • Grooveriding - Wednesday, March 26, 2014 - link

    The best just got better!

    Only on page 2 so far, but it's looking excellent.
    Reply
  • dylan522p - Wednesday, March 26, 2014 - link

    While it's got flaws, all around best phone IMO. One weird thing is that even though HTC didn't advertise it, the One has passed water proofness tests in a sink for 2 hours.. Reply
  • fokka - Wednesday, March 26, 2014 - link

    i almost couldn't watch it... air slowly escaping out of the headphone port, like a fish slowly drowning. (wait a minute...) Reply
  • blanarahul - Wednesday, March 26, 2014 - link

    Disappointed in the camera department. The camera is the single most important factor for me when buying a smartphone, so the One is a no go. Reply
  • blanarahul - Wednesday, March 26, 2014 - link

    Btw, I am referring to the rear camera.

    HTC did a really awesome job by putting the power button on the top on a phone this tall.

    They should have gone with capacitive buttons IMO. Atleast you wouldn't have to stare at the logo.

    The S5 is the only remaining good phone for me (I have bad experiences with Sony's quality control, so Z2 is a no go). I just have to find a good back cover for it.
    Reply
  • Cptn_Slo - Saturday, March 29, 2014 - link

    actually, I think HTC is one of the few smart phone manufacturers to invest in specs that will actually make a difference rather than to impress idiots. Reply
  • shaolin95 - Tuesday, April 01, 2014 - link

    Or fool idiots that think they are too smart... Reply
  • theduckofdeath - Tuesday, April 01, 2014 - link

    The reality is, HTC cuts back on the internals simply to make up for the cost of vanity. Calling that "specs that makes sense" is a poor excuse, I think. :) Reply
  • noel_newell - Friday, October 03, 2014 - link

    I agree, HTC One M8 is one fantastic phone and most sources seem to agree (my favorite source on the topic is http://www.consumertop.com/best-phone-guide/ ). The people that prefer Samsung or iPhone really haven't tried the M8. When my friends try out mine they are amazed. Reply
  • jonup - Thursday, March 27, 2014 - link

    Can we turn off the broken record!? Listen, consumer does not need 8+MP camera other than for bragging rights. I bet you we can show a daylight picture taken on a One and S5 to 95% of the people on the street on a ~5" display and they will not tell which is higher resolution. Then we can do the same with a low light picture. You see what I did there!? ;) All I hear about every review out there is "oh the camera is only 4MP", especially the once paid off by Samsung. And all the people like you repeat it all day long. I have someone with an HTC M7 in the office along with iPnone 5s and my Nokia 925 and N5. Side by side shots with all 4 of them look pleasing to the eye. If you are into photography that you care so much about picture detail, you won't be using you phone to take pictures. You can overanalyze the picture quality but It's a mute, geeky point that makes for great marketing. When in reality, the way most of us use their phone camera, the HTC setup is the best camera setup out there. (obviously exaggerating a bit but you get the point) At the end of the day as Anand said it will end up compressed on social media and most likely will be displayed on a phone. Reply

Log in

Don't have an account? Sign up now