Huawei & Honor's Recent Benchmarking Behaviour: A Cheating Headache
by Andrei Frumusanu & Ian Cutress on September 4, 2018 8:59 AM EST- Posted in
- Smartphones
- Huawei
- SoCs
- Benchmarks
- honor
- Kirin 970
Mea-Culpa: It Should Have Been Caught Earlier
Section By Andrei Frumusanu
As stated on the previous page, I had initially had seen the effects of this behaviour back in January when I was reviewing the Kirin 970 in the Mate 10. The numbers I originally obtained showed worse-than-expected performance of the Mate 10, which was being beaten by the Mate 9. When we discussed the issue with Huawei, they attributed it to a firmware bug, and pushed me a newer build which resolved the performance issues. At the time, Huawei never discussed what that 'bug' was, and I didn't push the issue as performance bugs do happen.
For the Kirin 970 SoC review, I went through my testing and published the article. Later on, in the P20 reviews, I observed the same lower performance again. As Huawei had told me before it was a firmware issue, I had also attributed the bad performance to a similar issue, and expected Huawei to 'fix' the P20 in due course.
Looking back in hindsight, it is pretty obvious there’s been some less than honest communications with Huawei. The newly detected performance issues were not actually issues – they were actually the real representation of the SoC's performance. As the results were somewhat lower, and Huawei was saying that they were highly competetive, I never would have expected these numbers as genuine.
It's worth noting here that I naturally test with our custom benchmark versions, as they enable us to get other data from the tests than just a simple FPS value. It never crossed my mind to test the public versions of the benchmarks to check for any discrepancy in behaviour. Suffice to say, this will change in our testing in the future, with numbers verified on both versions.
Analyzing the New Competitive Landscape
With all that being said, our past published results for Kirin 970 devices were mostly correct - we had used a variant of the benchmark that wasn’t detected by Huawei’s firmware. There is one exception however, as we weren't using a custom version of 3DMark at the time. I’ve now re-tested 3DMark, and updated the corresponding figures in past reviews to reflect the correct peak and sustained performance figures.
As far as I could tell in my testing, the cheating behaviour has only been introduced in this year’s devices. Phones such as the Mate 9 and P10 were not affected. If I’m to be more precise, it seems that only EMUI 8.0 and newer devices are affected. Based on our discussions with Huawei, we were told that this was purely a software implementation, which also corroborates our findings.
Here is the competitive landscape across our whole mobile GPU performance suite, with updated figures where applicable. We are also including new figures for the Honor Play, and the new introduction of the GFXBench 5.0 Aztec tests across all of our recent devices:
Overall, the graphs are very much self-explanatory. The Kirin 960 and Kirin 970 are lacking in both performance and efficiency compared almost every device in our small test here. This is something Huawei is hoping to address with the Kirin 980, and features such as GPU Turbo.
Facebook
Twitter
RSS
84 Comments
View All Comments
Amandtec - Wednesday, September 5, 2018 - link
Actually, social science style studies reveal that (ballpark) 10% of people always cheat, 10% never cheat no matter what, and the 80% in the middle are a continum who are mostly swayed by perceived risks and incentives. If you are always mistrusting everyone it prevents one from building profitable relations.skoondi - Sunday, September 9, 2018 - link
Not sure about Chinese culture but you'll find more than a few people care about phone GPU performance, I can only wonder why you are on a site basically dedicated to people who are interested in such things Phones are the new computer, with a decent docking set up many people probably don't need a desktop or laptop. I've been gaming since the days of the first civilisation game and currently do a lot of gaming on my phone.techconc - Monday, September 10, 2018 - link
@shogun18 - You said:"But I also have to wonder, are there really more than a 100 basement-dwelling morons who give a flying F about how fast a GPU is on a stupid phone? And furthermore play games on such pointless platforms where frame rates or triangles/sec matters?"
It seems your comments are very much out of touch with what is happening today, even in the world of gaming.
PeachNCream - Tuesday, September 4, 2018 - link
I don't get the point in cheating on phone benchmarks or even the point of phone benchmarks in general. I also don't get the point of sticking an auto playing video about buying the right CPU on EVERY DAMNED PAGE of this article. The same video. Every time! If seeing it on the first page pissed me off and I didn't watch it there (while also being enraged at the auto starting nature of it) what on Earth would make you think it's a good idea to do it to me three more times in the same article? Did the Purch buyout somehow involve a lobotomy that removed logic and reason or was that SOP already in place beforehand?vgray35@hotmail.com - Tuesday, September 4, 2018 - link
I would use stronger words than that to really convey the egregious nature of repetitive onslaughts of their bloody (not a swear word) videos against us. Their blatant "up yours, screw you" attitude turns rage to a frothing mouth frenzy ready to spill blood (especially on Toms Hardware). What the flying F is wrong with allowing users to decide whether to hit the play button or not? Or am I really only talking to a lobotomized miscreant that should be restrained from interacting with the public. Cease and desist auto play videos! A "up yours, screw you" attitude is not a smart way to deal with the public. Toms are you listening, or are sending us all another finger image?shogun18 - Tuesday, September 4, 2018 - link
er it's 2018. How the hell have you been using the internet for the last decade without all browser plugins disabled or at the very least noScript (or similar) with or without AdBlock (or similar)?vgray35@hotmail.com - Tuesday, September 4, 2018 - link
You missed the point. Most of us are clueless.shogun18 - Tuesday, September 4, 2018 - link
No, I totally get your point and I agree (with) you have every right to be incensed with auto-play videos. However a random comment here is unlikely to get the author or site-admin's attention. And said person is doubtless unable to oppose corporate policy. When Anand's site readership takes a major nose dive (further?) and the morons in marketing (but I repeat myself) can be beaten over the head that one of the major reasons they are losing revenue and clicks is because of auto-play, THEN and ONLY then will the practice stop.There haven't been more than a minuscule handful (again, basically zero given the side of the ecosystem) that could be described as "responsible" or "good" website operators for well over 10 years. Therefore if you haven't been black-listing every feature in your browser and toggling the choices conveniently given to you, then your ire is misplaced IMO. I am an equal opportunity banner of site content. Join the club and be amazed at far more enjoyable the experience is.
vgray35@hotmail.com - Tuesday, September 4, 2018 - link
Put that way you may be right.1_rick - Tuesday, September 4, 2018 - link
Well, Purch has been pushing those annoying ads on Tom's Hardware for quite some time now, so I guess it's only to be expected they'd be forcing them on this site as well.TBH I hadn't even noticed the ads because I used my hosts file to block assets.purch.com, because my adblocker wasn't taking care of it. You could try that too.