Huawei & Honor's Recent Benchmarking Behaviour: A Cheating Headache
by Andrei Frumusanu & Ian Cutress on September 4, 2018 8:59 AM EST- Posted in
- Smartphones
- Huawei
- SoCs
- Benchmarks
- honor
- Kirin 970
Mea-Culpa: It Should Have Been Caught Earlier
Section By Andrei Frumusanu
As stated on the previous page, I had initially had seen the effects of this behaviour back in January when I was reviewing the Kirin 970 in the Mate 10. The numbers I originally obtained showed worse-than-expected performance of the Mate 10, which was being beaten by the Mate 9. When we discussed the issue with Huawei, they attributed it to a firmware bug, and pushed me a newer build which resolved the performance issues. At the time, Huawei never discussed what that 'bug' was, and I didn't push the issue as performance bugs do happen.
For the Kirin 970 SoC review, I went through my testing and published the article. Later on, in the P20 reviews, I observed the same lower performance again. As Huawei had told me before it was a firmware issue, I had also attributed the bad performance to a similar issue, and expected Huawei to 'fix' the P20 in due course.
Looking back in hindsight, it is pretty obvious there’s been some less than honest communications with Huawei. The newly detected performance issues were not actually issues – they were actually the real representation of the SoC's performance. As the results were somewhat lower, and Huawei was saying that they were highly competetive, I never would have expected these numbers as genuine.
It's worth noting here that I naturally test with our custom benchmark versions, as they enable us to get other data from the tests than just a simple FPS value. It never crossed my mind to test the public versions of the benchmarks to check for any discrepancy in behaviour. Suffice to say, this will change in our testing in the future, with numbers verified on both versions.
Analyzing the New Competitive Landscape
With all that being said, our past published results for Kirin 970 devices were mostly correct - we had used a variant of the benchmark that wasn’t detected by Huawei’s firmware. There is one exception however, as we weren't using a custom version of 3DMark at the time. I’ve now re-tested 3DMark, and updated the corresponding figures in past reviews to reflect the correct peak and sustained performance figures.
As far as I could tell in my testing, the cheating behaviour has only been introduced in this year’s devices. Phones such as the Mate 9 and P10 were not affected. If I’m to be more precise, it seems that only EMUI 8.0 and newer devices are affected. Based on our discussions with Huawei, we were told that this was purely a software implementation, which also corroborates our findings.
Here is the competitive landscape across our whole mobile GPU performance suite, with updated figures where applicable. We are also including new figures for the Honor Play, and the new introduction of the GFXBench 5.0 Aztec tests across all of our recent devices:
Overall, the graphs are very much self-explanatory. The Kirin 960 and Kirin 970 are lacking in both performance and efficiency compared almost every device in our small test here. This is something Huawei is hoping to address with the Kirin 980, and features such as GPU Turbo.
Facebook
Twitter
RSS
84 Comments
View All Comments
aebiv - Wednesday, September 5, 2018 - link
Benchmarking is incredibly important when people are planning and building systems, and by systems I mean those that actually support business. I'd argue you're right for most consumers though, as things are just "more than fast enough."That said, I'd still argue they're an important metric. Just might not be important to you.
websitetrafficchecker - Wednesday, February 5, 2020 - link
best website traffic estimator If you are looking for free website traffic checker, try https://seowebsitetraffic.net/check-website-traffi...Flunk - Tuesday, September 4, 2018 - link
This is not helping the perception that Chinese manufacturers will cheat and lie to their customers. That's already a big problem for Huawei in western markets. I think they need to fire everyone responsible for marketing their phones to western markets and hire native teams who actually understand the customer base.SetiroN - Tuesday, September 4, 2018 - link
You're confusing "the western markets" with the USA.Elsewhere people don't have such an obsession, actually they know American devices are probably more likely to provide backdoors.
shogun18 - Tuesday, September 4, 2018 - link
The Chinese culture is synonymous with cheating. It's an *aspiration* to cheat and "put one over" their competitor and has been for for thousands of years. You're not going to convince anyone to play by "modern" Western notions of "fair and honest" benchmarks. Even so-called Westerners have put their values aside for the almighty buck in droves. Not to suggest criminal enterprises like the East India Company (of Britain) had any intention of acting in any civil/honorable manner from the get go. Human beings are for the most part filthy, rotten, scoundrels and far too many of them will gladly commit atrocities in the name of profit or something else. Cheating is basic human behavior.Therefore, you should ALWAYS expect humans to be dishonest. See Intel for a brazen example.
Thanks for doing the article. But I also have to wonder, are there really more than a 100 basement-dwelling morons who give a flying F about how fast a GPU is on a stupid phone? And furthermore play games on such pointless platforms where frame rates or triangles/sec matters? Seriously? PC Graphics cards from 10 years ago are still stupid fast. Phone games are throwaways. Sure, some publishers some how manage to make a tidy sum but it could run in CGA and people would still buy them to fritter away their boredom. If every Youtube ran in 360p nobody would give a damn. Oh some would bitch and moan but they'd still watch. And that's the point.
Graphics performance of the SoC is just dick measuring and the percentage of people who consider the number in their purchasing valuation metric is basically zero.
goatfajitas - Tuesday, September 4, 2018 - link
American companies do it too.R0H1T - Tuesday, September 4, 2018 - link
I agree with most of what you've said about the Chinese culture but let's be 100% clear - "fair and honest" is not a modern "western" notion!Entropyq3 - Tuesday, September 4, 2018 - link
I care about SoC graphics performance, and would prefer valid data.I don’t live in a basement though, do I still count?
shogun18 - Tuesday, September 4, 2018 - link
No You're 1 in 10+ million consumers (out of just English-reading population) and therefore irrelevant except that you read Anand and/or Toms and such. :)skoondi - Sunday, September 9, 2018 - link
Utter rubbish