Subtle Cheating: New Benchmark Optimizations

We’ve been tracking the state of benchmark optimizations among Android OEMs for a while now. It wasn’t too long ago that we published a piece calling out nearly all Android OEMs for optimizing for benchmarks. The optimizations are pretty crude. Upon detecting a whitelisted benchmark by APK identifier alone, most OEMs would enable a performance mode that would:

1) Plug in all CPU cores
2) Run all cores at max frequency
3) Raise thermal limits to eliminate/reduce throttling

Defeating the optimization was just as simple: thorough renaming of the benchmark and all internal application IDs. For the past several months we’ve been working with benchmark, silicon and handset vendors to curb the behavior. Although we found the optimizations to have minimal impact on our test results, it’s still a messy practice that isn’t worth doing.

We’ve seen early (encouraging) indications that some vendors have reconsidered their position on benchmark optimizations. Unfortunately HTC isn’t quite there yet.

The M8’s Android 4.4.2 build includes a new, more subtle form of benchmark optimization that we hadn’t seen in previous devices. Benchmarks are still detected according to their application identifier, but instead of hot plugging in all CPU cores and driving them to max frequencies, everything appears to be normal at launch.

Here’s the state of the CPU cores after launching the Play Store version of any optimized benchmark:

Everything looks just fine. But look at what happens if we monitor CPU frequency over time on the Play Store and a special renamed version of 3DMark:

Average CPU frequency is about 15% higher while running the Play Store version of 3DMark. I still need to run some thermal analysis on the device but I don’t think HTC is raising thermal limits. Instead what appears to be happening is HTC is simply more aggressively tuning the governor response to performance demands, allowing for higher frequencies. Note that the frequency response latency is now so low that I couldn't even grab the 300MHz screenshot above in the Play Store version of 3DMark. As soon as the device detected a button press it would ramp up to 1.7GHz.

The impact on performance goes hand in hand with the increase in average clock speed. I measured performance during 3DMark’s Physics test (which is CPU bound). The difference was about 15%.

I also tracked GPU clock speed over time. Thankfully the optimization seems limited to CPU frequencies alone:

The list of optimization targets has also expanded since we last looked at HTC. The latest versions of GFXBench, BaseMark X and BaseMark OS II are now included in the benchmark whitelist.

HTC made one small concession - it’s allowing users the ability to run their device in this high performance mode at all times. Under developer tools (tap on the build number 5 times in Settings > About > Software information > More), you’ll find an option to enable high performance CPU mode. Checking that box will put your device in the same mode that’s enabled when a whitelisted benchmark is detected.

I do appreciate that HTC is exposing the optimization control, the only thing missing is the ability to toggle the benchmark optimization off (not to mention that I’d prefer if it was disabled to begin with). I fear that HTC’s justification in all of this is that everyone else is doing it so why opt out. The reality seems to be trending the other direction however. We’ll have to see what Samsung does with the Galaxy S 5, but I have a feeling that HTC is going to end up on the wrong side of history with this move. All of our benchmarks are already immune to the optimization, so it’s really a matter of sacrificing integrity for no real gain. There’s nothing more to say other than I’m disappointed.

Sense 6.0, Motion Launch & Sensor Hub Snapdragon 801 Performance
POST A COMMENT

222 Comments

View All Comments

  • ShieTar - Thursday, March 27, 2014 - link

    I think if you compare a 2MP and a 8+MP picture by looking at it on a 2MP-5" screen, about 100% of the people will not be able to tell which one has the better resolution.

    But I agree with your general comment, people complaining endlessly about the photographing qualities of a device that can't be connected to a good objective anyways is just weird.
    Reply
  • Braumin - Thursday, March 27, 2014 - link

    Sorry I don't agree with you. 4MP is lots for twitter and facebook, but you can't crop the image at all. Plus 4MP doesn't allow 4k recording.

    Also they took out the OIS. The camera is fine in the One, but it's a regression from last year's model. That's why it's getting knocked. People had issues with the camera last year, and now they've actually gone back a step rather than address the issues.

    Don't forget you're not talking about a Lumia 520 here, or even a Moto X. This is a full flagship phone, with the price to match. They needed to put a larger physical sensor in this year. If they had done 6-8 MP with the "ultrapixels" that would be one of the best camera's on a phone. Instead they regressed. That's never what you want to see in a flagship and they've been rightfully called out on it.
    Reply
  • darkich - Thursday, March 27, 2014 - link

    Don't you ever, like, zoom in the photos?
    For, you know, seeing the finer detail?
    Reply
  • sevenmack - Thursday, March 27, 2014 - link

    Depends. While I do a lot of cropping of my photos, I usually use Photoshop for that purpose. Does the job better than most mobile apps (including those that come with the phone itself). But for the average person, cropping is a rare event. In fact, you would be lucky to talk to someone who regularly crops smartphone camera photos for detail purposes.

    So Jonup does have a point. Doesn't mean that the M8 suits your needs; to each their own on that one. But for most folks -- including many professional photographers such as Mahmoud Mfinanga of EmmazedPhotog.com, and Colby Brown -- the M8 (and the M7 before it) more than does the job.
    Reply
  • Scootiep7 - Friday, March 28, 2014 - link

    You do realize that most of the actual, professional photography work that Mahmoud Mfinanga and Colby Brown do uses either film or high end DSLR's cameras right? I mean it's literally on their websites. And no, the M8 and M7 would not cut it for them. Reply
  • doosh bag - Friday, March 28, 2014 - link

    They didn't say they used the phone camera, they simply endorsed it. Have it a thumbs up, if you will. Reply
  • CoryWeston101 - Monday, March 31, 2014 - link

    I know a few that would disagree. And believe the M8 and M7 have great camera's for smartphones. Reply
  • sephirotic - Sunday, March 30, 2014 - link

    Finer detail, on 16mpx 1/3" sensor? Good joke, sir. Reply
  • CoryWeston101 - Monday, March 31, 2014 - link

    No. That's stupid. Reply
  • jond11 - Thursday, March 27, 2014 - link

    Maybe you should do better research. The camera improved from last year's model even with them removing ois. Not one tech site yet said picture quality is worse on the M8 compared to the M7. Actually, anyone who reviews the 2 phones all agree it takes better pictures. It's funny how the HTC ONE 2014 performed better than any phone on the plantet, but it wasn't worthy because it had a 4 ultra pixel camera. Yet the Note2, Note3, and S4 don't perform half as smoothly as the ONE and was considered great phones. So Iguess iit's ok now to make flagship phones buggy and filled with lag as long as it carries a 8 megapixel camera or better, lol. Reply

Log in

Don't have an account? Sign up now