The Mate 20 & Mate 20 Pro Review: Kirin 980 Powering Two Contrasting Devices
by Andrei Frumusanu on November 16, 2018 8:10 AM EST- Posted in
- Smartphones
- Huawei
- Mobile
- Kirin 980
- Mate 20
- Mate 20 Pro
Kirin 980 Second Generation NPU - NNAPI Tested
We’ve tested the first generation Kirin NPU back in January in our Kirin 970 review – Back then, we were quite limited in terms of benchmarking tests we were able to run, and I mostly relied on Master Lu’s AI test. This is still around, and we’ve also used it in performance testing Apple’s new A12 neural engine. Unfortunately or the Mate 20’s, the benchmark isn’t compatible yet as it seemingly doesn’t use HiSilicon’s HiAI API on the phones, and falls back to a CPU implementation for processing.
Google had finalised the NNAPI back in Android 8.1, and how most of the time these things go, we first need an API to come out before we can see applications be able to make use of exotic new features such as dedicated neural inferencing engines.
“AI-Benchmark” is a new tool developed by Andrey Ignatov from the Computer Vision Lab at ETH Zürich in Switzerland. The new benchmark application, is as far as I’m aware, one of the first to make extensive use of Android’s new NNAPI, rather than relying on each SoC vendor’s own SDK tools and APIs. This is an important distinction to AIMark, as AI-Benchmark should be better able to accurately represent the resulting NN performance as expected from an application which uses the NNAPI.
Andrey extensive documents the workloads such as the NN models used as well as what their function is, and has also published a paper on his methods and findings.
One thing to keep in mind, is that the NNAPI isn’t just some universal translation layer that is able to magically run a neural network model on an NPU, but the API as well as the SoC vendor’s underlying driver must be able to support the exposed functions and be able to run this on the IP block. The distinction here lies between models which use features that are to date not yet supported by the NNAPI, and thus have to fall back to a CPU implementation, and models which can be hardware accelerated and operate on quantized INT8 or FP16 data. There’s also models relying on FP32 data, and here again depending on the underlying driver this can be either run on the CPU or for example on the GPU.
For the time being, I’m withholding from using the app’s scores and will simply rely on individual comparisons between each test’s inference time. Another presentational difference is that we’ll go through the test results based on the targeted model acceleration type.
The first three CPU tests rely on models which have functions that are not yet supported by the NNAPI. Here what matters for the performance is just the CPU performance as well as the performance response time. The latter I mention, because the workload is transactional in its nature and we are just testing a single image inference. This means that mechanisms such as DVFS and scheduler responsiveness can have a huge impact on the results. This is best demonstrated by the fact that my custom kernel of the Exynos 9810 in the Galaxy S9 performs significantly better than the stock kernel of the same chip of the Note9 in the same above results.
Still, comparing the Huawei P20 Pro (most up to date software stack with Kirin 970) to the new Mate 20, we see some really impressive results of the latter. This both showcases the performance of the A76 cores, as well as possibly improvements in HiSilicon’s DVFS/scheduler.
Moving onto the next set of tests, these are based on 8-bit integer quantized NN models. Unfortunately for the Huawei phones, HiSilicons NNAPI drivers still doesn’t seem to expose acceleration to the hardware. Andrey had shared with me that in communications with Huawei, is that they plan to rectify this in a future version of the driver.
Effectively, these tests also don’t use the NPU on the Kirins, and it’s again a showcase of the CPU performance.
On the Qualcomm devices, we see the OnePlus 6 and Pixel 3 far ahead in performance, even compared to the same chipset Galaxy S9+. The reason for this is that both of these phones are running a new updated NNAPI driver from Qualcomm which came along with the Android 9/P BSP update. Here acceleration if facilitated through the HVX DSPs.
Moving on to the FP16 tests, here we finally see the Huawei devices make use of the NPU, and post some leading scores both on the old and new generation SoCs. Here the Kirin 980’s >2x NPU improvement finally materialises, with the Mate 20 showcasing a big lead.
I’m not sure if the other devices are running the workloads on the CPU or on the GPU, and the OnePlus 6 seems to suffer from some very odd regression in its NNAPI drivers that makes it perform an order of magnitude worse than other platforms.
Finally on the last FP32 model test, most phones should be running the workload on the CPU again. There’s a more limited improvement on the part of the Mate 20.
Overall, AI-Benchmark was at least able to validate some of Huawei’s NPU performance claims, even though that the real conclusion we should be drawing from these results is that most devices with NNAPI drivers are currently just inherently immature and still very limited in their functionality, which sadly enough again is a sad contrast compared where Apple’s CoreML ecosystem is at today.
I refer back to my conclusion from early in the year regarding the Kirin 970: I still don’t see the NPU as something that obviously beneficial to users, simply because we just don’t have the software applications available to make use of the hardware. I’m not sure to what extent Huawei uses the NPU for camera processing, but other than such first-party use-cases, NPUs currently still seems something mostly inconsequential to device experience
Facebook
Twitter
RSS
141 Comments
View All Comments
Andrei Frumusanu - Friday, November 16, 2018 - link
Nothing on the nanoSD, I don't even have any way to test it. As for storage, again I'm not posting any results in the review because the tools are broken/misleading. If it's any worth, both phones are leading in terms of Androbench performance.Chitti - Saturday, November 24, 2018 - link
Andrei, its almost 10 days.U didn't upload it yet 😓.
s.yu - Friday, November 16, 2018 - link
Samsung's body + Apple's notch, a popular opinion on the net.For the record Huawei's always "taking note" of somebody's design, they copied Sony's Omnibalance back with the P8, especially with that power button, moving on they've been closely following Samsung's design (S6,S7,S8) until the P20, in which they copied Apple, with the latest Mate20Pro, it's both, altogether.
Andrei Frumusanu - Friday, November 16, 2018 - link
I don't really agree with this (Beyond the P20's looking similar to the iPhone X), Huawei always had quite distinct designs in its phones, especially on the Mate series.s.yu - Friday, November 16, 2018 - link
Well for the P8 Huawei's own slide mentioning a "balanced design" following Sony's use of "Omnibalance" as well as the obvious resemblance in design did give them away, Huawei's Mate 9 Pro's a facelift of the S6/S7, and their subsequent so-called "Porsche Design" versions as well as the corresponding "Pro" versions all suspiciously follow Samsung's move in design philosophy from the S7 onwards. Then Apple's IPX and all...levizx - Saturday, November 17, 2018 - link
Well, that's quite a bold regarding P8, P8 looks nothing like SONY. By your logic, any phone that claims "slim" or "premium" would be clones.How else would you design a phone with 16:9 curved display with 3 years old technology, I'd like to see you try.
You have to be very stupid to think Mate 10/P20 series looks anything like S7/S8/iPhone being non-curved with front/back fingerprint readers.
So your logic is essentially - curved display: copying Samsung, non-curved display: copying Samsung, notch with fingerprint reader and triple camera: copying iPhone.
s.yu - Saturday, November 17, 2018 - link
That's some weak trolling.Let me tell you now how exactly the P8 looks like Sony's Xperia Z released earlier:
1. It's very boxy for its release date, compared to its peers only Sony of that era used such a design philosophy though Sony stuck to it for the many years to come while Huawei went on to copy Samsung the subsequent generation starting from Mate 9 Pro.
2. Sony's machine-milled power button was iconic in their Xperia Z and Z1, then Huawei came over, slightly changed the shape and stuck it onto an equally boxy body with suspiciously similarly named design philosophy.
How else would *I* design? That's not my problem, I'm not a team of engineers paid to innovate on a smartphone design after all, but HTC and Samsung both had unique solutions at the time, especially Samsung's S6 was an ingenious generational leap which pioneered the trend of glass-sandwich-metal bodies until this day.
When Huawei wasn't so obviously copying, well of course you know what happened, the unsightly Mate S, Mate 8 and Mate 9.
The Mate 9 Pro's screen was curved, together with the home button it's a straight Samsung S6E/S7E ripoff under a façade of "Porsche Design". At least get your basic facts straight before you troll.
I didn't say that the Mate 10 was a Samsung/Apple copy, you put that in my mouth. Not every Huawei is a copy, just that they copy somebody almost every generation, which also shows from their highly inconsistent, rapidly shifting design language, because they "borrow" from different opponents each year.
The P20 is a thorough IPX clone, up there with the Mi8, there's no doubt about that.
Quantumz0d - Friday, November 16, 2018 - link
Agreed.Press don't care unfortunately and that makes the marketing work. How else Huawei could gather all that. Look at Pixel 3XL its the best android phone according to press. Less features is more nowadays and you should pay more for the super cloned designs and submitting your ownership as well.
And it's China and their famous pro local IP theft game, no one can win there except their own.
levizx - Saturday, November 17, 2018 - link
What a stupid and racist troll. So in your mind Pixel XL is not a clone, but all Huawei phones are?Mate 10 Pro looks more like LG V30 than anything else, and they are only 2 months apart, no chance of copying at all.
s.yu - Saturday, November 17, 2018 - link
Look at you grasping at straws like that, the Mate 10 Pro may not be a copy, but Mate RS was a clear Samsung copy.