Subtle Cheating: New Benchmark Optimizations

We’ve been tracking the state of benchmark optimizations among Android OEMs for a while now. It wasn’t too long ago that we published a piece calling out nearly all Android OEMs for optimizing for benchmarks. The optimizations are pretty crude. Upon detecting a whitelisted benchmark by APK identifier alone, most OEMs would enable a performance mode that would:

1) Plug in all CPU cores
2) Run all cores at max frequency
3) Raise thermal limits to eliminate/reduce throttling

Defeating the optimization was just as simple: thorough renaming of the benchmark and all internal application IDs. For the past several months we’ve been working with benchmark, silicon and handset vendors to curb the behavior. Although we found the optimizations to have minimal impact on our test results, it’s still a messy practice that isn’t worth doing.

We’ve seen early (encouraging) indications that some vendors have reconsidered their position on benchmark optimizations. Unfortunately HTC isn’t quite there yet.

The M8’s Android 4.4.2 build includes a new, more subtle form of benchmark optimization that we hadn’t seen in previous devices. Benchmarks are still detected according to their application identifier, but instead of hot plugging in all CPU cores and driving them to max frequencies, everything appears to be normal at launch.

Here’s the state of the CPU cores after launching the Play Store version of any optimized benchmark:

Everything looks just fine. But look at what happens if we monitor CPU frequency over time on the Play Store and a special renamed version of 3DMark:

Average CPU frequency is about 15% higher while running the Play Store version of 3DMark. I still need to run some thermal analysis on the device but I don’t think HTC is raising thermal limits. Instead what appears to be happening is HTC is simply more aggressively tuning the governor response to performance demands, allowing for higher frequencies. Note that the frequency response latency is now so low that I couldn't even grab the 300MHz screenshot above in the Play Store version of 3DMark. As soon as the device detected a button press it would ramp up to 1.7GHz.

The impact on performance goes hand in hand with the increase in average clock speed. I measured performance during 3DMark’s Physics test (which is CPU bound). The difference was about 15%.

I also tracked GPU clock speed over time. Thankfully the optimization seems limited to CPU frequencies alone:

The list of optimization targets has also expanded since we last looked at HTC. The latest versions of GFXBench, BaseMark X and BaseMark OS II are now included in the benchmark whitelist.

HTC made one small concession - it’s allowing users the ability to run their device in this high performance mode at all times. Under developer tools (tap on the build number 5 times in Settings > About > Software information > More), you’ll find an option to enable high performance CPU mode. Checking that box will put your device in the same mode that’s enabled when a whitelisted benchmark is detected.

I do appreciate that HTC is exposing the optimization control, the only thing missing is the ability to toggle the benchmark optimization off (not to mention that I’d prefer if it was disabled to begin with). I fear that HTC’s justification in all of this is that everyone else is doing it so why opt out. The reality seems to be trending the other direction however. We’ll have to see what Samsung does with the Galaxy S 5, but I have a feeling that HTC is going to end up on the wrong side of history with this move. All of our benchmarks are already immune to the optimization, so it’s really a matter of sacrificing integrity for no real gain. There’s nothing more to say other than I’m disappointed.

Sense 6.0, Motion Launch & Sensor Hub Snapdragon 801 Performance
POST A COMMENT

222 Comments

View All Comments

  • Jeffrey Bosboom - Wednesday, March 26, 2014 - link

    I guess I should clarify: that means headphones, not speakers. (I share an office.) Reply
  • jk1 - Thursday, March 27, 2014 - link

    i'd like to second this. i know htc is making a big deal out of its double camera, but we get endless details about the camera, special effects, humorous overlays and on and on. but we get nothing in any of these reviews about the quality of the dac for headphone use, or the quality of the sound of an incoming voice call or how you sound making a call using the phone. Reply
  • djvita - Thursday, March 27, 2014 - link

    from gsmarena:

    "The HTC One (M8) audio quality is by far the best we've seen a mobile device produce so far - and that's including tablets. Even its impressive predecessor pales in comparison with the latest HTC flagship, which is not only the loudest around, but also delivers perfectly clean output.
    The HTC One (M8) did perfectly in the active external amplifier part of the test, posting great scores all over the field. In addition it had volume levels higher than every other mobile device on the market right now.
    More impressively, there's virtually no degradation when you plug in a pair of headphones. The stereo crosstalk rises so little that it remains better than what some smartphones deliver without headphones. The rest of the readings remain perfect too, while the volume is as high as they come. What do you know - dropping the Beats logo actually led to even more spectacular performance by the HTC flagship."

    http://www.gsmarena.com/htc_one_m8-review-1062p7.p...
    Reply
  • dylan522p - Wednesday, March 26, 2014 - link

    The One was great for audio after you disabled beats EQ, so I'd expect the same. Reply
  • CoryWeston101 - Monday, March 31, 2014 - link

    Amazing. Best audio experience headphones or no headphones on a smartphone. Reply
  • RaistlinZ - Wednesday, March 26, 2014 - link

    The lack of OIS really sucks. The difference is just painful to watch. Reply
  • fokka - Wednesday, March 26, 2014 - link

    but at least we can add bad looking "bokeh" to our shots...

    really, going for the second camera, which adds cost, requires space and is just plain ugly to look at, all while dropping an important feature for true photography has been a big mistake, if you ask me.
    Reply
  • CoryWeston101 - Monday, March 31, 2014 - link

    There is not difference. The duo camera does OIS but in a better way. With the second camera it doesn't need OIS because the second camera acts in the same way but better. Reply
  • Samuel Lord - Thursday, April 3, 2014 - link

    @CoryWeston101:
    A nice theory, utterly untrue with the M8. No, it needs OIS for decent video but HTC punted, end of story.
    Reply
  • Fergy - Wednesday, March 26, 2014 - link

    I love my HTC One. It feels great. Has great sound. A great screen. Good battery life.
    What I wanted improved: better camera and smaller. If they could just make the 4.7inch screen borderless and integrate the HTC bar buttons in the screen it would be the perfect size for me.

    Instead it is larger in every way and even heavier! And the camera is the same...

    The HTC One m8 is still the phone I would recommend to people but for me I have to keep on the lookout for a phone I can use with one hand.
    Reply

Log in

Don't have an account? Sign up now