Futuremark Begins Delisting "Cheating" Mobile Devices from 3DMark Database
by Anand Lal Shimpi on November 26, 2013 9:37 PM EST- Posted in
- Smartphones
- Mobile
- Tablets
Over the summer, following a tip from @AndreiF7, we documented an interesting behavior on the Exynos 5 Octa versions of Samsung's Galaxy S 4. Upon detecting certain benchmarks the device would plug in all cores and increase/remove thermal limits, the latter enabling it to reach higher GPU frequencies than would otherwise be available in normal games. In our investigation we pointed out that other devices appeared to be doing something similar on the CPU front, while avoiding increasing thermal limits. Since then we've been updating a table in our reviews that keeps track of device behavior in various benchmarks.
It turns out there's a core group of benchmarks that seems to always trigger this special performance mode. Among them are AnTuTu and, interestingly enough, Vellamo. Other tests like 3DMark or GFXBench appear to be optimized for, but on a far less frequent basis. As Brian discovered in his review of HTC's One max, the list of optimization/cheating targets seems to grow with subsequent software updates.
In response to OEMs effectively gaming benchmarks, we're finally seeing benchmark vendors take a public stand on all of this. Futuremark is the first to do something about it. Futuremark now flags and delists devices caught cheating from its online benchmark comparison tool. The only devices that are delisted at this point are the HTC One mini, HTC One (One max remains in the score list), Samsung Galaxy Note 3 and Samsung Galaxy Tab 10.1 (2014 Edition). In the case of the Samsung devices, both Exynos 5 Octa and Qualcomm based versions are delisted.
Obviously this does nothing to stop users from running the benchmark, but it does publicly reprimand those guilty of gaming 3DMark scores. Delisted devices are sent to the bottom of the 3DMark Device Channel and the Best Mobile Devices list. I'm personally very pleased to see Futuremark's decision on this and I hope other benchmark vendors follow suit. Honestly I think the best approach would be for the benchmark vendors to toss up a warning splash screen on devices that auto-detect the app and adjust behavior accordingly. That's going to be one of the best routes to end-user education of what's going on.
Ultimately, I'd love to see the device OEMs simply drop the silliness and treat benchmarks like any other application (alternatively, exposing a global toggle for their benchmark/performance mode would be an interesting compromise). We're continuing to put pressure on device makers, but the benchmark vendors doing the same will surely help.
Source: Futuremark
Facebook
Twitter
RSS
46 Comments
View All Comments
Ma Deuce - Sunday, December 1, 2013 - link
"Considering expectation of benchmarking apps, its primary objective of gadget is expected to run at its maximum capacity to provide accurate readings"Unfortunately when Samsung and HTC game the system, instead of accurate readings we get false highs instead.
" so why it is needed to delist a device which specifically runs CPU & GPU at it full clocked speed isn't that limit is tested by benchmarking concept???"
When they cheat the benchmark they are effectively making it a less useful tool for for judging real world performance. I couldn't care less how fast Samsung can make their phone run. I want to know how fast it runs for me instead. The same with benchmarks, if a benchmark only shows me how fast Samsung can run apps on their phone, it's worthless to me.
It's good to see Futuremark take a stand and it makes their product more credible for sure.
Ichinisan - Wednesday, December 4, 2013 - link
The point is, cheating makes the benchmark run faster than any other application is allowed to run. What good is that score if your game or other performance-hungry application is never allowed to run at that speed/duration? They decrease thermal restrictions temporarily to get that score. Running the processor at that speed while playing a game would melt your device. The score is automatically invalid because you can't compare real-world performance between 2 different devices...so what's the point?Bravo, Futuremark. Cheaters will never stop if we don't call them out.
meacupla - Wednesday, December 4, 2013 - link
wow, I can't believe this many people don't understand the difference between turbo boost and cheating on benchmarks.The difference is simple, people.
Turbo boost: overclocking within thermal envelope
Cheating: overclocking above thermal envelope
I think we all know what happens when we run a stress testing program on an overclock higher than what the heatsink can handle, right?
Notmyusualid - Thursday, December 5, 2013 - link
Full respect to Futuremark.Notmyusualid - Thursday, December 5, 2013 - link
And I should add, I'm rocking a HTC One.gtjg66 - Saturday, December 7, 2013 - link
I disagree with the the delisting of these devices, when you take a look at the records in 3dmark the highest benchmark scores are achieved with LN2 cooling on the PC. Unless I'm wrong that isn't real world performance in my book, unless pouring LN2 down the throat of a CPU/GPU all day is real world.If a CPU/GPU can run a benchmark without crashing then that seems to be acceptable with 3dmark scores, if your going for a world record in 3dmark scores I don't think those people are going to be too concerned if they destroyed the silicon to achieve it. If that happens to any of these devices then maybe there should be a disclaimer for running the benchmark, if the device can do that without meltdown then so be it. It's not like AMD/ Nvidia haven't tried this before in their drivers to get the highest scores possible, isn't that what 3dmark is about to find out just how far the silicon can be pushed to get the highest scores possible? If Futuremark want the devices to have a score that is real world then maybe the PC benchmark scores should be invalid if overclocked on LN2.