Futuremark Begins Delisting "Cheating" Mobile Devices from 3DMark Database
by Anand Lal Shimpi on November 26, 2013 9:37 PM EST- Posted in
- Smartphones
- Mobile
- Tablets
Over the summer, following a tip from @AndreiF7, we documented an interesting behavior on the Exynos 5 Octa versions of Samsung's Galaxy S 4. Upon detecting certain benchmarks the device would plug in all cores and increase/remove thermal limits, the latter enabling it to reach higher GPU frequencies than would otherwise be available in normal games. In our investigation we pointed out that other devices appeared to be doing something similar on the CPU front, while avoiding increasing thermal limits. Since then we've been updating a table in our reviews that keeps track of device behavior in various benchmarks.
It turns out there's a core group of benchmarks that seems to always trigger this special performance mode. Among them are AnTuTu and, interestingly enough, Vellamo. Other tests like 3DMark or GFXBench appear to be optimized for, but on a far less frequent basis. As Brian discovered in his review of HTC's One max, the list of optimization/cheating targets seems to grow with subsequent software updates.
In response to OEMs effectively gaming benchmarks, we're finally seeing benchmark vendors take a public stand on all of this. Futuremark is the first to do something about it. Futuremark now flags and delists devices caught cheating from its online benchmark comparison tool. The only devices that are delisted at this point are the HTC One mini, HTC One (One max remains in the score list), Samsung Galaxy Note 3 and Samsung Galaxy Tab 10.1 (2014 Edition). In the case of the Samsung devices, both Exynos 5 Octa and Qualcomm based versions are delisted.
Obviously this does nothing to stop users from running the benchmark, but it does publicly reprimand those guilty of gaming 3DMark scores. Delisted devices are sent to the bottom of the 3DMark Device Channel and the Best Mobile Devices list. I'm personally very pleased to see Futuremark's decision on this and I hope other benchmark vendors follow suit. Honestly I think the best approach would be for the benchmark vendors to toss up a warning splash screen on devices that auto-detect the app and adjust behavior accordingly. That's going to be one of the best routes to end-user education of what's going on.
Ultimately, I'd love to see the device OEMs simply drop the silliness and treat benchmarks like any other application (alternatively, exposing a global toggle for their benchmark/performance mode would be an interesting compromise). We're continuing to put pressure on device makers, but the benchmark vendors doing the same will surely help.
Source: Futuremark
Facebook
Twitter
RSS
46 Comments
View All Comments
tipoo - Wednesday, November 27, 2013 - link
Turbo boost isn't the same thing as benchmark boosting. Turbo boost works on all applications and benefits them. Benchmark boosting ONLY clocks up when certain apps are detected, it doesn't run like that in games or anything else stressing the system.xype - Wednesday, November 27, 2013 - link
Yes, obviously Apple _made_ the Android handset makers cheat at the benchmarks so they could culminate the effort in delisting their devices in the Futuremark benchmark listings. It all makes sense now!ddriver, I know you’re probably incapable of holding one thought long enough for this, but please find me a benchmark or specification that Apple deliberately mislead their customers with in in the past few years. Battery life, GPU performance, CPU performance. Show me where they cheated in order to look better.
What these Android vendors did is a _deliberate_ effort to game the benchmarks. That’s not a mistake or an oversight. That’s not something that Apple had any part in. And that’s what’s upsetting people. Even as an Apple customer I can admit that a lot of Android devices are better in certain areas than what Apple is producing. And that’s exactly the reason why I find such attempts to game the benchmarks retarded, because it screams "We actually don’t think our product is good enough to stand on its own merits, we need to game the system!", and that’s pathetic, doubly so because it’s absolutely not needed.
steven75 - Wednesday, November 27, 2013 - link
I've seen some stretches in my time, but this one takes the top of the list.Samsung and HTC's decision to cheat at specific benchmarks was Apple's fault all along!
dylan522p - Wednesday, November 27, 2013 - link
Huh? You do realize that Qualomm and every other ARM vendor has boost as well, they just advertise the boost clock as the nominal because they are somewhat shady in that regard.mark3785 - Thursday, November 28, 2013 - link
I'm actually stunned that someone could be so delusional that he could conjure up an Apple/Futuremark plot to discredit Samsung and HTC. Kudos on the fertile imagination!KoolAidMan1 - Friday, November 29, 2013 - link
Except that's not how normal CPU throttling works. The offending companies put flags to boost clock speed specifically for those benchmarks. If something else demands similar performance boosts, it doesn't do it. Intel and Apple don't cheat in this way, clock speed is regulated consistently.This is cheating, plain and simple. The baseless accusation at Apple in the end also makes your bias very very clear.
Notmyusualid - Thursday, December 5, 2013 - link
^^ +1I'm more surprised some here are defending the cheating...
Drumsticks - Tuesday, November 26, 2013 - link
Good job people :DWinterCharm - Tuesday, November 26, 2013 - link
Good. cheating on benchmarks only serves to harm consumers who are trying to make an educated decision on which devices to buy.Tehk17 - Wednesday, November 27, 2013 - link
Go check out the 3DMark reviews in the Play Store. Grab a bag of popcorn while you're at it. Pretty funny stuff in the review section.