The Black Shark 2 Review: A Gaming Phone's Existential Crisis
by Andrei Frumusanu on September 25, 2019 8:00 AM EST- Posted in
- Mobile
- Smartphones
- Xiaomi
- Snapdragon 855
- Black Shark 2
Machine Learning Inference Performance
AIMark 3
AIMark makes use of various vendor SDKs to implement the benchmarks. This means that the end-results really aren’t a proper apples-to-apples comparison, however it represents an approach that actually will be used by some vendors in their in-house applications or even some rare third-party app.
Unfortunately for the Black Shark 2, the devices lacked the proper drivers to properly run AIMark, and the benchmark repeatedly crashed upon starting the benchmarks. We had the same issue on the OnePlus 7 Pro, pointing out to some software incompatibility.
AIBenchmark 3
AIBenchmark takes a different approach to benchmarking. Here the test uses the hardware agnostic NNAPI in order to accelerate inferencing, meaning it doesn’t use any proprietary aspects of a given hardware except for the drivers that actually enable the abstraction between software and hardware. This approach is more apples-to-apples, but also means that we can’t do cross-platform comparisons, like testing iPhones.
We’re publishing one-shot inference times. The difference here to sustained performance inference times is that these figures have more timing overhead on the part of the software stack from initialising the test to actually executing the computation.
AIBenchmark 3 - NNAPI CPU
We’re segregating the AIBenchmark scores by execution block, starting off with the regular CPU workloads that simply use TensorFlow libraries and do not attempt to run on specialized hardware blocks.
In AI Benchmark’s CPU workloads, the Black Shark 2 ends up with a bit of a odd spread of scores. In the shorter running benchmarks the phone is getting relatively average inference times, while on the longer running tests for some reason the BS2 falls behind other S855 devices. In fact it looks like the BS2 is landing as amongst the worse off S855 devices in the latter listed tests.
AIBenchmark 3 - NNAPI INT8
AIBenchmark 3 - NNAPI FP16
AIBenchmark 3 - NNAPI FP32
In the INT8, FP16 and FP32 accelerated tests which make use of acceleration blocks such as the Hexagon DSP and the GPU, we see the Black Shark 2 perform very well and in line with other Snapdragon 855 devices.
Overall, the Black Shark 2 is a good performer in the machine learning inferencing benchmarks, but like other devices, it’s not quite the very best in every regard, pointing out that the vendor could have improved upon its performance by keeping the software stack more up to date with what Qualcomm is offering, a widespread issue that I expect to persist over the next years as the ecosystem quickly evolves.
Facebook
Twitter
RSS
63 Comments
View All Comments
eastcoast_pete - Wednesday, September 25, 2019 - link
Thanks Andrei, also and especially for calling a (now major) manufacturer -Xiaomi- out for having benchmark cheating built into their software! The hardware side is just as bad - why all style, no substance? The aluminum case almost screams for use as a heat sink, especially now that phones are all closed up.Regarding gaming phones: will you have a chance to test the ASUS ROG2? Might tell us if all these "gaming" phones are all smoke and mirrors. The ROG 2 is even more expensive, so it'd better be spectacular.
Andrei Frumusanu - Wednesday, September 25, 2019 - link
We'll have a ROG2 review out next week.s.yu - Wednesday, September 25, 2019 - link
Andrei, please be sure to dig into Mate30P's slo-mo with detail, now that the 1-to-4 frame interpolation is confirmed, I'm not the only one suspecting that the 720P is also interpolated from an inferior readout that skips even more lines!isthisavailable - Thursday, September 26, 2019 - link
The Rog phone 2's size and weight look huge. Can you please include it's everyday usability in your review? I'm tempted to buy but 77cm wide and 9mm thicc phone looks a bit too much. And 240g weight!brucethemoose - Wednesday, September 25, 2019 - link
I can tell you the Razer Phone 2 is the real deal. Performance is fine, but holy heck, the screen! Im not even a phone gamer, but 120hz is like night and day, and you couldn't pay me to go back to a phone with a 60hz display.s.yu - Wednesday, September 25, 2019 - link
What do you think of 240Hz then? A phone with that was just released by Sharp.Mediocre phone by other metrics though.
brucethemoose - Wednesday, September 25, 2019 - link
Its probably pretty similar, as Razer uses the same Sharp LCD line those Aquos phones use.But on 120hz vs 240hz specifically... I don't know? I've never even laid eyes on 240hz anything. I'd guess that you need more tightly integrated VRR support from base Android and something faster than an LCD to really make use of it, but what do I know?
Also, for a $400 phone, the RP2 is suprisingly good by other metrics. Its not perfect (no 3.5mm, no OLED, 845 instead of 855), but I don't feel like I traded much off to get 120hz.
hemedans - Wednesday, September 25, 2019 - link
RoG 2 is around $500 base model with 128GB/8GB memory, probably best Gaming phone for now, waiting for Review too.s.yu - Wednesday, September 25, 2019 - link
I thought that was the Chinese "Tencent special edition" price, yet I checked and at least India gets the same price...it seems US customers get ripped off...or is only the top tier available there?$500 sounds good but 128GB really doesn't last.
Tams80 - Thursday, September 26, 2019 - link
Well, the Tencent thing is a deal between ASUS and Tencent (I'm guessing mainly subsidised by Tencent). I'm guessing India is a mix of a deal with some company and lower variation due to India having lower household incomes.