Subtle Cheating: New Benchmark Optimizations

We’ve been tracking the state of benchmark optimizations among Android OEMs for a while now. It wasn’t too long ago that we published a piece calling out nearly all Android OEMs for optimizing for benchmarks. The optimizations are pretty crude. Upon detecting a whitelisted benchmark by APK identifier alone, most OEMs would enable a performance mode that would:

1) Plug in all CPU cores
2) Run all cores at max frequency
3) Raise thermal limits to eliminate/reduce throttling

Defeating the optimization was just as simple: thorough renaming of the benchmark and all internal application IDs. For the past several months we’ve been working with benchmark, silicon and handset vendors to curb the behavior. Although we found the optimizations to have minimal impact on our test results, it’s still a messy practice that isn’t worth doing.

We’ve seen early (encouraging) indications that some vendors have reconsidered their position on benchmark optimizations. Unfortunately HTC isn’t quite there yet.

The M8’s Android 4.4.2 build includes a new, more subtle form of benchmark optimization that we hadn’t seen in previous devices. Benchmarks are still detected according to their application identifier, but instead of hot plugging in all CPU cores and driving them to max frequencies, everything appears to be normal at launch.

Here’s the state of the CPU cores after launching the Play Store version of any optimized benchmark:

Everything looks just fine. But look at what happens if we monitor CPU frequency over time on the Play Store and a special renamed version of 3DMark:

Average CPU frequency is about 15% higher while running the Play Store version of 3DMark. I still need to run some thermal analysis on the device but I don’t think HTC is raising thermal limits. Instead what appears to be happening is HTC is simply more aggressively tuning the governor response to performance demands, allowing for higher frequencies. Note that the frequency response latency is now so low that I couldn't even grab the 300MHz screenshot above in the Play Store version of 3DMark. As soon as the device detected a button press it would ramp up to 1.7GHz.

The impact on performance goes hand in hand with the increase in average clock speed. I measured performance during 3DMark’s Physics test (which is CPU bound). The difference was about 15%.

I also tracked GPU clock speed over time. Thankfully the optimization seems limited to CPU frequencies alone:

The list of optimization targets has also expanded since we last looked at HTC. The latest versions of GFXBench, BaseMark X and BaseMark OS II are now included in the benchmark whitelist.

HTC made one small concession - it’s allowing users the ability to run their device in this high performance mode at all times. Under developer tools (tap on the build number 5 times in Settings > About > Software information > More), you’ll find an option to enable high performance CPU mode. Checking that box will put your device in the same mode that’s enabled when a whitelisted benchmark is detected.

I do appreciate that HTC is exposing the optimization control, the only thing missing is the ability to toggle the benchmark optimization off (not to mention that I’d prefer if it was disabled to begin with). I fear that HTC’s justification in all of this is that everyone else is doing it so why opt out. The reality seems to be trending the other direction however. We’ll have to see what Samsung does with the Galaxy S 5, but I have a feeling that HTC is going to end up on the wrong side of history with this move. All of our benchmarks are already immune to the optimization, so it’s really a matter of sacrificing integrity for no real gain. There’s nothing more to say other than I’m disappointed.

Sense 6.0, Motion Launch & Sensor Hub Snapdragon 801 Performance
POST A COMMENT

222 Comments

View All Comments

  • tipoo - Wednesday, March 26, 2014 - link

    I'm dying at the mustachioed Anand and the punk hat. Reply
  • bj_murphy - Thursday, March 27, 2014 - link

    Yeah me too, that was my favourite part of the review, hands down! Reply
  • tzhu07 - Thursday, March 27, 2014 - link

    HTC has a real-time 3D model of the M8. Pretty cool.
    https://skfb.ly/yvMN
    Reply
  • skingtech - Tuesday, April 15, 2014 - link

    This article helps convince me to go Samsung Galaxy S5 for sure. The M8 has a nice feature set - better built in sound and nicer look but, Samsung has better display, better camera, and comparable if not faster everything else (I don't know about you but, I put a case on my phone - the look of the device should be the LAST thing focus'd on for crying out loud. Reply
  • dylan522p - Wednesday, March 26, 2014 - link

    I think those selfies may be the greatest thing ever! Reply
  • KPOM - Wednesday, March 26, 2014 - link

    Nice review, as usual. I wonder if 5" is where phone manufacturers finally "stop" in terms of the "standard" phone size. Above that, and a small tablet paired with a 4-5" phone really is a better solution.

    If I were to get an Android phone, it would be an HTC. They should lobby hard to get a Nexus phone. With Google subsidies that might be what kicks their sales efforts into gear. Otherwise, I see the new HTC One in much the same light as the old HTC One: the best Android phone out there in terms of build quality and style, and one of the best in terms of performance, but a distant also-ran behind Samsung in terms of sales.
    Reply
  • fokka - Wednesday, March 26, 2014 - link

    i don't think we reached the end of increasing screen sizes yet. the one just hit 5", in a relatively big body though, the s5 jumped to 5,1", also by increasing body size. the z2 is at 5,2" and the only phones seemingly at least trying to offset the big screen with small bezels are the g2 and note3.

    the next note will be 5,9-6" and the next round of flagships will be in the 5,5" ballpark.

    we can only hope for devices like the z1 compact to make top notch hardware available in more pocketable form factors.
    Reply
  • Honest Accounting - Wednesday, March 26, 2014 - link

    Moto X: optimal form factor and minimal bezels.
    I think we'll end up with 3 'standard' sizes. 4.7", 5.5", and 6.3"
    Reply
  • piroroadkill - Thursday, March 27, 2014 - link

    Yeah, Moto X is the right size.

    2.6mm shorter and 2.6mm narrower than my Motorola DROID RAZR MAXX HD. (Although the Moto X is thicker with a lot less battery!).

    I still think if I had to pick any phone available, I'd stick with the one I have. The new DROID MAXX is closest to what I'd want, but it has no microSD slot or HDMI out, which the HD has both of.
    Reply
  • Jeffrey Bosboom - Wednesday, March 26, 2014 - link

    Can we get some remarks about audio quality? I use my phone for two things: taking pictures of whiteboards and listening to music. Given that my old EVO 3D's camera (and heavy JPEG compression) takes acceptable pictures, I think the ultrapixel camera will suffice. So if it also offers decent audio quality, I'll probably get one. Reply

Log in

Don't have an account? Sign up now