Machine Learning Inference Performance

AIMark 3

AIMark makes use of various vendor SDKs to implement the benchmarks. This means that the end-results really aren’t a proper apples-to-apples comparison, however it represents an approach that actually will be used by some vendors in their in-house applications or even some rare third-party app.

鲁大师 / Master Lu - AIMark 3 - InceptionV3 鲁大师 / Master Lu - AIMark 3 - ResNet34 鲁大师 / Master Lu - AIMark 3 - MobileNet-SSD 鲁大师 / Master Lu - AIMark 3 - DeepLabV3

Unfortunately for the Black Shark 2, the devices lacked the proper drivers to properly run AIMark, and the benchmark repeatedly crashed upon starting the benchmarks. We had the same issue on the OnePlus 7 Pro, pointing out to some software incompatibility.

AIBenchmark 3

AIBenchmark takes a different approach to benchmarking. Here the test uses the hardware agnostic NNAPI in order to accelerate inferencing, meaning it doesn’t use any proprietary aspects of a given hardware except for the drivers that actually enable the abstraction between software and hardware. This approach is more apples-to-apples, but also means that we can’t do cross-platform comparisons, like testing iPhones.

We’re publishing one-shot inference times. The difference here to sustained performance inference times is that these figures have more timing overhead on the part of the software stack from initialising the test to actually executing the computation.

AIBenchmark 3 - NNAPI CPU

We’re segregating the AIBenchmark scores by execution block, starting off with the regular CPU workloads that simply use TensorFlow libraries and do not attempt to run on specialized hardware blocks.

AIBenchmark 3 - 1 - The Life - CPU/FP AIBenchmark 3 - 2 - Zoo - CPU/FP AIBenchmark 3 - 3 - Pioneers - CPU/INT AIBenchmark 3 - 4 - Let's Play - CPU/FP AIBenchmark 3 - 7 - Ms. Universe - CPU/FP AIBenchmark 3 - 7 - Ms. Universe - CPU/INT AIBenchmark 3 - 8 - Blur iT! - CPU/FP

In AI Benchmark’s CPU workloads, the Black Shark 2 ends up with a bit of a odd spread of scores. In the shorter running benchmarks the phone is getting relatively average inference times, while on the longer running tests for some reason the BS2 falls behind other S855 devices. In fact it looks like the BS2 is landing as amongst the worse off S855 devices in the latter listed tests.

AIBenchmark 3 - NNAPI INT8

AIBenchmark 3 - 1 - The Life - INT8 AIBenchmark 3 - 2 - Zoo - Int8 AIBenchmark 3 - 3 - Pioneers - INT8 AIBenchmark 3 - 5 - Masterpiece - INT8 AIBenchmark 3 - 6 - Cartoons - INT8

AIBenchmark 3 - NNAPI FP16

AIBenchmark 3 - 1 - The Life - FP16 AIBenchmark 3 - 2 - Zoo - FP16 AIBenchmark 3 - 3 - Pioneers - FP16 AIBenchmark 3 - 5 - Masterpiece - FP16 AIBenchmark 3 - 6 - Cartoons - FP16 AIBenchmark 3 - 9 - Berlin Driving - FP16 AIBenchmark 3 - 10 - WESPE-dn - FP16

AIBenchmark 3 - NNAPI FP32

AIBenchmark 3 - 10 - WESPE-dn - FP32

In the INT8, FP16 and FP32 accelerated tests which make use of acceleration blocks such as the Hexagon DSP and the GPU, we see the Black Shark 2 perform very well and in line with other Snapdragon 855 devices.

Overall, the Black Shark 2 is a good performer in the machine learning inferencing benchmarks, but like other devices, it’s not quite the very best in every regard, pointing out that the vendor could have improved upon its performance by keeping the software stack more up to date with what Qualcomm is offering, a widespread issue that I expect to persist over the next years as the ecosystem quickly evolves.

System Performance GPU Performance - Worst of S855
POST A COMMENT

63 Comments

View All Comments

  • Jon Tseng - Wednesday, September 25, 2019 - link

    Thanks interesting. I guess that definitely answers the question as to whether the paid Qualcomm partnership had any impact on your ratings (not that it would, knowing AT, I know!). Reply
  • PaulHoule - Wednesday, September 25, 2019 - link

    What I found strange about the article is that, unlike most reviews of gaming hardware, there were not any benchmarks for games.

    Maybe that's because nobody cares how many frames per second you can render Candy Crush which is why they pack it in with Windows so Intel can claim there is a "game" you can play with the integrated graphics.

    If there aren't any games that need a gaming phone, why does a gaming phone have to exist? It's like the sound of one hand clapping. isn't it?
    Reply
  • Death666Angel - Wednesday, September 25, 2019 - link

    I mean, there area pretty intensive games for smartphones out there, think PUBG, Fortnite, racing games. I don't really know if they provide accurate/useful ingame benchmarks and I don't know if there are any FRAPS / FCAT like apps for Android. I don't think we get anything better than 3DMark and GFXBench right now. :) Everything else is non repeatable and therefore of little use. Reply
  • PeachNCream - Wednesday, September 25, 2019 - link

    Pretty sure the decision to include Candy Crush is between Microsoft and King. It isn't a decision related to Intel. After all AMD based Windows systems ship with it preloaded as well. Reply
  • crimson117 - Wednesday, September 25, 2019 - link

    I think benchmarks were omitted because "the firmware detects benchmarks and disables thermal throttling in order to get better scores." So benchmarks would be meaningless and misleading. Reply
  • s.yu - Wednesday, September 25, 2019 - link

    Generally Anandtech's benchmarks should be immune to detection, which is stated in the article. Reply
  • Andrei Frumusanu - Thursday, September 26, 2019 - link

    Yes, we bypassed detection for the published scores. Reply
  • raghusa - Wednesday, September 25, 2019 - link

    AT was always one of my go to place to get in-depth analysis and reviews. Now that AT is going with PAID partnership route, my view of AT analysis is definitely going to change, will likely look for other reputable sources .... Reply
  • Andrei Frumusanu - Thursday, September 26, 2019 - link

    The review content is still fully independently written, and the editorial team including myself have no contact with the sponsor in this regard. In any case this series will come to a closure on Monday. Reply
  • Wardrive86 - Thursday, September 26, 2019 - link

    I wish that it was not coming to an end! I very much enjoyed this series. Reply

Log in

Don't have an account? Sign up now