Huawei & Honor's Recent Benchmarking Behaviour: A Cheating Headache
by Andrei Frumusanu & Ian Cutress on September 4, 2018 8:59 AM EST- Posted in
- Smartphones
- Huawei
- SoCs
- Benchmarks
- honor
- Kirin 970
Mea-Culpa: It Should Have Been Caught Earlier
Section By Andrei Frumusanu
As stated on the previous page, I had initially had seen the effects of this behaviour back in January when I was reviewing the Kirin 970 in the Mate 10. The numbers I originally obtained showed worse-than-expected performance of the Mate 10, which was being beaten by the Mate 9. When we discussed the issue with Huawei, they attributed it to a firmware bug, and pushed me a newer build which resolved the performance issues. At the time, Huawei never discussed what that 'bug' was, and I didn't push the issue as performance bugs do happen.
For the Kirin 970 SoC review, I went through my testing and published the article. Later on, in the P20 reviews, I observed the same lower performance again. As Huawei had told me before it was a firmware issue, I had also attributed the bad performance to a similar issue, and expected Huawei to 'fix' the P20 in due course.
Looking back in hindsight, it is pretty obvious there’s been some less than honest communications with Huawei. The newly detected performance issues were not actually issues – they were actually the real representation of the SoC's performance. As the results were somewhat lower, and Huawei was saying that they were highly competetive, I never would have expected these numbers as genuine.
It's worth noting here that I naturally test with our custom benchmark versions, as they enable us to get other data from the tests than just a simple FPS value. It never crossed my mind to test the public versions of the benchmarks to check for any discrepancy in behaviour. Suffice to say, this will change in our testing in the future, with numbers verified on both versions.
Analyzing the New Competitive Landscape
With all that being said, our past published results for Kirin 970 devices were mostly correct - we had used a variant of the benchmark that wasn’t detected by Huawei’s firmware. There is one exception however, as we weren't using a custom version of 3DMark at the time. I’ve now re-tested 3DMark, and updated the corresponding figures in past reviews to reflect the correct peak and sustained performance figures.
As far as I could tell in my testing, the cheating behaviour has only been introduced in this year’s devices. Phones such as the Mate 9 and P10 were not affected. If I’m to be more precise, it seems that only EMUI 8.0 and newer devices are affected. Based on our discussions with Huawei, we were told that this was purely a software implementation, which also corroborates our findings.
Here is the competitive landscape across our whole mobile GPU performance suite, with updated figures where applicable. We are also including new figures for the Honor Play, and the new introduction of the GFXBench 5.0 Aztec tests across all of our recent devices:
Overall, the graphs are very much self-explanatory. The Kirin 960 and Kirin 970 are lacking in both performance and efficiency compared almost every device in our small test here. This is something Huawei is hoping to address with the Kirin 980, and features such as GPU Turbo.
Facebook
Twitter
RSS
84 Comments
View All Comments
goatfajitas - Tuesday, September 4, 2018 - link
I think it speaks directly to the subject matter. Far too many companies, tech sites, and users hang far too much on bench-marking. Does 1% better "geekmarks" actually make a difference in usage?cfenton - Tuesday, September 4, 2018 - link
But it's not 1%, or even close to that. The OnePlus 6 is 3x faster than the P20 Pro in most of the sustained performance benchmarks.I think it's important for users to know that the Kirin 970 has a significantly weaker GPU than the S845, especially when Huawei is marketing some of these phones as 'gaming phones'.
goatfajitas - Tuesday, September 4, 2018 - link
Now you are comparing 2 different class phones (flagship OP6 vs mid-range P20) and CPU from 2 different years (2017's 970 vs 2018's S845). In most areas like vs. like is very close, and there is alot more complexity to it than just that. It's all way over-hyped and over-emphasizedR0H1T - Tuesday, September 4, 2018 - link
No he's right, current QC flagship SoC vs reigning Hisilicon Flasghip. In case you din't know SD845 is available in phones priced as low as $300 lower than P20 in fact.R0H1T - Tuesday, September 4, 2018 - link
>reply meant for goatfajitasgoatfajitas - Tuesday, September 4, 2018 - link
Fair enough, but the Kirin 980 is out in a month. It is last years product and when it came out it compared to the S835. Both on yearly cadence appx 6 months apart.Cicerone - Friday, September 7, 2018 - link
A phone price is not standing from it's SOC. Display, cameras quality, type of storage and software is accounted.cfenton - Tuesday, September 4, 2018 - link
The P20 Pro came out this year and actually cost more at launch than the OP6 (~700-800 euro vs ~500-600 euro). The P20 Pro also came out after the GS9, which has the S845. Even the P20 launched at 649 euro. If you'd prefer to compare phones that came out at the same time, then the GS9 is still over 2x faster. If you want to compare the Kirin 970 to the S835, since both came out in 2017, then the 835 is still 2x as fast.goatfajitas - Tuesday, September 4, 2018 - link
From a GPU perspective you aren't incorrect... If you are buying a phone as a gaming device you might want to look at GPU benchmarks more, but you do however seem to be overly obsessed with bench-marking... You are also not comparing like for like. The P20 is not Huawei's flagship, the Mate is. Oneplus is a crappy budget brand with massive quality issues and the price shows it. IF you want to buy one go for it.cfenton - Tuesday, September 4, 2018 - link
The P20 Pro is priced like a flagship and has their top-end SOC. Huawei just has two high-end models for some reason.Again, even if you insist on comparing only flagships from big companies, Huawei comes out looking bad. The Pixel 2 and Galaxy S8 are both significantly faster than the Mate 10.
I'm not obsessed with benchmarks, but I think they have value. We aren't talking about modern SSD benchmarks where one brand reads at 3100mbps and the other at 3200mbps. That's a small difference. This is very different. We're talking about at least double the performance.