The Snapdragon 855 Phone Roundup: Searching for the Best Implementations
by Andrei Frumusanu on September 5, 2019 8:30 AM ESTMachine Learning Inference Performance
AIMark 3
AIMark makes use of various vendor SDKs to implement the benchmarks. This means that the end-results really aren’t a proper apples-to-apples comparison, however it represents an approach that actually will be used by some vendors in their in-house applications or even some rare third-party app.
Summarizing the four sub-tests of AIMark 3, we see a few clear outliers: OnePlus 7 Pro and Xiaomi’s Black Shark 2 lack Qualcomms driver libraries on which the benchmark relies on (or they’re broken), and the application crashes. Another outlier is the G8, here the phone also lacked the libraries for hardware acceleration, but the app at least fell back onto CPU execution of the workloads, albeit at a massive performance penalty.
We notably see that AIMark has been able to implement Samsung’s NPU SDK as we’re seeing evident hardware acceleration. Huawei’s P30 Pro also makes use of its NPU here via its proprietary SDK. Naturally, Apple’s iPhone XS uses the CoreML framework to accelerate the AI workloads.
Overall, the Snapdragon 855 devices with the latest SDKs and frameworks here seem to compete extremely well, offering extensive performance that is leading the pack across all the different sub-tests.
AIBenchmark 3
AIBenchmark takes a different approach to benchmarking. Here the test uses the hardware agnostic NNAPI in order to accelerate inferencing, meaning it doesn’t use any proprietary aspects of a given hardware except for the drivers that actually enable the abstraction between software and hardware. This approach is more apples-to-apples, but also means that we can’t do cross-platform comparisons, like testing iPhones.
We’re publishing one-shot inference times. The difference here to sustained performance inference times is that these figures have more timing overhead on the part of the software stack from initialising the test to actually executing the computation.
AIBenchmark 3 - NNAPI CPU
We’re segregating the AIBenchmark scores by execution block, starting off with the regular CPU workloads that simply use TensorFlow libraries and do not attempt to run on specialized hardware blocks.
We’re seeing largely regular results here, although some observations pop up again, such as seeing that the Black Shark 2 having a very conservative result in some subtests. The other big outlier is the OPPO Reno 10x, here we’re seeing that the phone consistently performs better than the rest of the pack. This is very interesting, particularly because the phone actually isn’t able to actually use the NNAPI acceleration in the subsequent tests we’re covering next. This means that the “plain” TensorFlow libraries the OPPO is making use of are performing better than what’s employed for the rest of the devices.
AIBenchmark 3 - NNAPI INT8
INT8 performance is dominated by the Snapdragon 855 devices, and this is thanks to the vector processing units of the Hexagon DSP. Test 3-6 stands out for some devices, and it’s likely that this is due to discrepancies in the NNAPI drivers which don’t fully accelerate things on the hardware for the S10, Mi9 and G8.
AIBenchmark 3 - NNAPI FP16
FP16 performance is again quite even across the board, with the difference coming down to DVFS and scheduler responsiveness. Here the Snapdragon 855 competes neck-in-neck with the Kirin 980, winning some tests while losing others. It’s to be remembered that the Kirin 980’s NPU doesn’t support INT8 acceleration at this time, and that’s why it shows up better in the FP16 benchmarks.
AIBenchmark 3 - NNAPI FP32
Finally, on the FP32 benchmark, Qualcomm accelerates these workloads on the GPU and is far ahead of the competition, which either have lacking GPU drivers or have to fall back to CPU acceleration.
Facebook
Twitter
RSS
47 Comments
View All Comments
tuxRoller - Friday, September 6, 2019 - link
?Qualcomm is unusually good in this area.
MrCommunistGen - Thursday, September 5, 2019 - link
Interesting overview. I like that there are a lot of explicit comparisons between devices -- which is admittedly the purpose of a piece like this.I know you already do a lot of work for reviews and there's a LOT of data with a roundup with this many devices, but I have a request:
Battery Capacity Normalized Battery Life.
It would help illustrate platform efficiency vs battery size.
StormyParis - Friday, September 6, 2019 - link
What about with a case...pse - Friday, September 6, 2019 - link
Hi, Andrei, excellent article, as usual. One question, perhaps I missed this in the article, but are the figures for the S10+ Snapdragon and Exynos the ones from the original S10 article, or have you retested it with the latest updates?Cheers!
edsib1 - Saturday, September 7, 2019 - link
It doesnt look like you are using the phones various gamespace modesMy Oppo Reno 10x Zoom scores like your, but in gamespace mode it gets scores around 25% higher.
Performance - 10564
Web Browsing - 8257
Video Edit - 6429
Writing - 13638
Photo - 22386
Data - 8118
antifocus - Sunday, September 8, 2019 - link
Very nice to see more Chinese manufacturers coverage.tygrus - Monday, September 9, 2019 - link
I would like to see more user controls to limit performance and allow better battery life. Some games hog CPU & GPU for no reason (static city/castle not fps), if I can force them to 15fps instead of 60fps then may be I can avoid the pocket warmer/fire-starter. Change peak MHz of the big cores or set an average power used over short to long term (30 seconds / 5 minutes / 1 hour). I don't want sudden power saving once battery is <20%. Better option if I can set the aim for 12 hours use at the start of the day and have the device limit power from the beginning. More RAM can enable users to keep more apps & data in memory to avoid the slower app startup cycles. I thought 3GB RAM would be great 4 years ago but that was quickly used up by larger OS updates & larger apps. An older phone had just 8GB storage which was swallowed up OS and a few apps.It's frustrating that not all the phones you mention are available in Australia & other countries. Grey imports are possible but do you risk getting a 2nd hand phone with no warranty? OnePlus models and Samsung Galaxy S10e with 8GB RAM are not available in Australia.
peevee - Thursday, September 12, 2019 - link
"Differences in system performance between devices with the same hardware chipset basically boil down to one aspect: software."You forgot Flash and RAM choice.
peevee - Thursday, September 12, 2019 - link
"Finally, in the overall score, the ZTE RedMagic 3 comes at the top alongside the Galaxy S10"In your list it is S10+. These are very different devices, and you cannot infer the performance of S10 from S10+.
peevee - Thursday, September 12, 2019 - link
GPU "performance" - test after test of OFF SCREEN "performance". Does anybody play off screen? Show the real thing, so everybody would see the real cost of those useless extra-invisible pixels, and real advantage or lack of thereof of "90fps" and "120fps" - COMBINED with the battery life effects of all these!