Qualcomm's Thoughts on Benchmarks versus End-User Experiences

While the primary purpose of our visit out to Qualcomm’s offices was for benchmarking, for this year’s event Qualcomm went one step further and invited us to take a tour of some of their labs. The tour itself was plenty interesting (more on this in a bit), however arguably the more important aspect of the tour isn’t what we saw, but why Qualcomm is giving the press lab tours after so long. A tour may sound trite, but for Qualcomm it’s a sign of bigger things.

Perhaps the most important thing to understand about Qualcomm is that while the bulk of the company’s revenue comes from chips, the majority of their actual profits come from various technology and patent licensing programs. The actual breakdown between the two halves of Qualcomm gets a bit weird when trying to best account for which half foots the bill for R&D, but at the end of the day it’s Qualcomm’s licensing business that is making them the most money.

Qualcomm Financials (FY'2016)
  Chip-Making Licensing
Revenue $15.4B $7.6B
Profit (Pretax) $1.8B $6.5B

As a result, if boardroom-level rumors are to be believed, the company is never quite sure how to balance the two businesses. Some groups want licensing and chip design separated, and others want them to stay combined. Qualcomm seems content to stick with the status quo and their $5.7B in profits for 2016, but as a consequence of this unusual split it means that the chip side of Qualcomm is always under a bit of pressure to prove itself.

The good news for the chip business is that it’s doing fairly well, especially in North America. Most high-end Android smartphones sold in the US are based on Qualcomm’s chips, Qualcomm’s modems are in some (but no longer all) of Apple’s iPhone SKUs, and even in the midrange market Qualcomm’s Snapdragon 600 series chips are in a number of popular phones. The flip side of this is that outside of the US and especially outside of the high-end, Qualcomm’s chip business is under constant pressure from both their partners and their competitors. Samsung is happy to stick with their own home-grown Exynos chips outside of North America, and other chip vendors like MediaTek nip at Qualcomm’s heels with competent midrange SoCs that are priced very aggressively. So while Qualcomm’s chip business is doing well right now, that could very well change in the future.

But what does this have to do with a press tour of their labs? One of the big initiatives for Qualcomm, as exemplified in last week’s Snapdragon branding tweak, is that the chip arm of the company wants to get away from being so frequently judged on the basis of their CPU and only their CPU. This is why Snapdragon is now a platform versus a processor, and similarly, it’s why we’re going to see Qualcomm increasingly talking up the other aspects of their SoCs in the future. Their SoCs and associated front-end hardware are more than just the Kryo CPU, and they want the world to know that.

For Qualcomm this move makes a lot of sense. CPU performance is still growing year-after-year, but as we’ve seen, those gains are plateauing some as a consequence of diminishing IPC gains and diminishing clockspeed gains. Meanwhile, although the Kryo is a customized CPU design, as we’ve seen in our benchmark data it doesn’t behave radically different from ARM’s Cortex-A72 and Cortex-A73 designs that other vendors can license. Short of a “Cyclone” moment for Qualcomm, their CPU performance is close enough to the rest of the Android pack that they can’t clearly stand out from other vendors in the way Apple can. And that means they need to promote and sell their SoCs as the sum of their parts, and not just a CPU with a bunch of extra stuff bolted on.

This also means that even though Qualcomm is one of the first vendors to ship a 10nm high-end SoC – and enjoying the performance benefits thereof – we’re also going to see the company downplay benchmarks a bit for not capturing the complete “end-user experience.”

And to the company’s credit, they aren’t wrong. CPU, GPU, and memory performance are all easy enough to test, vary a great deal among SoC designs, and are similarly easy to communicate. DSPs, ISPs, and other aspects are not nearly as easy to test, never mind communicating those advantages to a wider audience.

Still, once we get outside of CPU/GPU/memory and into other functional blocks of a SoC, things get murkier for everyone. A video decode block may as well be a static piece of hardware – as long as it does its job well, no one will notice – and a great Wi-Fi radio can be kneecapped by a bad device design. So as much as Qualcomm wants to push the broader end-user experience, they will never completely escape the fact that the core features of their SoCs will be the first to be put to the test, and that wider testing is going to be less meaningful to customers, or more practically speaking outside the budgets of technology journalists.

GPU Performance Qualcomm on Tour: Power, Camera Testing, & More
Comments Locked

128 Comments

View All Comments

  • niva - Thursday, March 23, 2017 - link

    You keep saying that it's not "real world" when earlier there were links provided that should be showing you that in the real world, today, multithread already matters, and having more real/virtual cores helps. This is all for the simplest and most used task for cell phones, web browsing, multi threading is quite useful. I'm fairly confident that if an Android manufacturer ran on hardware identical to the iPhone and outscored it across the board, you'll buy the iPhone anyways. Good for you but you're not helping in this discussion, just admit your apple fanboyism and bow out. Why do you even care about the SD 835?
  • yankeeDDL - Thursday, March 23, 2017 - link

    Let me be more specific then.
    Web browsing is a key part of mobile experience. In the Kraken, WebXPRT and JetStream the performance difference is stunning: Kraken: 2.4X faster; WebXPRT: 1.35X faster, JetStream: 2.4X faster.
    Yes, the difference is not only due to HW, but also to code optimization. Still: damn!

    In the GPU department: in GFXBench, the performance is on-par (when reported).
    There's a noticeable advantage of the S835 in 3DMark (1.4X Overall), but in Basemark we loose again by nearly 2X.

    Yes, the comparison is (a bit) apples to oranges, but one has to admit that for a brand new SoC it would make sense to expect an hands-down victory over a noticeably older phone.
  • BurntMyBacon - Thursday, March 23, 2017 - link

    @yankeeDDL: "Still: damn!"
    Agreed, Apple has a massive advantage in javascript benchmarks. It is impossible to say how much (if any) of that is due to the SoC vs the software stack, but the advantage is undeniable.

    It is not unexpected that the A10 would win in Basemark. The A10 is making use of a low level API (Metal) where the SD835 is using a high level API (OpenGL ES). Again, Apple's better software cohesion and better use of APIs benefits them here. Still, the difference is quite formidable and the SD835 actually looses to the Kirin 960 as well. It would seem that the Adreno 540 is not well suited to this workflow. Therefore, it is unlikely that use of Vulkan will suddenly propel them ahead, but the gap would be a lot smaller. By the time use of Vulkan becomes common place, A11(?) will be out, so it's really a moot point.

    The GFXBench Car Chase ES 3.1 / Metal chart title suggests it should have an Apple data point (only user of "Metal"). It'll likely show the same thing as the basemark test given the disparate APIs, but I'm still curious (though not critical without further considerations) as to why it wasn't included.

    The fact that you can't get an A10 without iOS and you can't get iOS on another companies SoC makes considerations about whether it is better than the android SoCs or not a tertiary concern and academic when compared to the overall platform experience. There are plenty of reasons not to like an iProduct. Performance isn't generally one of them.
  • tuxRoller - Friday, March 24, 2017 - link

    If you take into account Qualcomm's optimized browser, the differences relative to the iPhone 7 change to:
    (% better than the sd835)
    Kraken: 140 -> 106 webxprt: 35 -> -26 jetstream: 140 -> 92

    I'm sure they could do more, but I'd be amazed if the remaining differences in kraken & jetsteam were mostly due to software.
  • Despoiler - Wednesday, March 22, 2017 - link

    It's mostly the OS that Apple has superiority in. That's why they can use a dual core while Android phones are quad or octacore.
  • grayson_carr - Saturday, March 25, 2017 - link

    Isn't the A10 a quad core, or more correctly, a dual dual core big.LITTLE chip? Same as Snapdragon 820?
  • akdj - Friday, March 31, 2017 - link

    Yes, A10 is a quad-core big.LITTLE SoC, w/a 12-core GPU, I believe... as well, Apple on the 7+ added another GB of RAM = 3GB on an iOS phone, iPad 12.9" has 4GB -- but the iOS integration with the A10... as well, the last several generations of 'home brewed' ARM chips - and Apple's investments in silicon engineering from nVidia, Intel, AMD, Qualcomm, TI and others has paid dividends.

    That said, off Apple/AX chipsets for a second.... excellent 'first look' and factory/testing insight. That is very cool stuff!

    I think, as geeks, and 'passionate' groups of faithful mobile phone OS folks amongst our population, folks who take this stuff more seriously than the Sunday sermon... we should all take a breath and remember that it's a 'chip preview'
    Not an Android phone!

    The issues with using the same chip on every device running Android --and every OEM 'skinning' their handset is a huge contributor to the varying performances; real world or objective bench tests. Like Windows as an OS on the desk and lap over the years, we've ALL had our 'Vista' moments. I'm an OS X/macOS user specifically because of 'vista and a curiosity about OS X in 2006 --- but again, I digress...

    Qualcomm has built a chip able to be put in to every flagship other than iOS this year and 'compete' just fine. In the performance metrics all above are bickering about. But as an iPhone 7+ owner/lover (it's an excellent phone) -- my appreciation for the 835 goes well beyond its parity or near ...or exceeding metrics of CPU and GPU, they've built a gigabit LTE modem (who cares if you won't 'get that' - it's still gonna haul ass!) - incredible image processing and 'encryption/protection' with its iris scanning and biometric uses ...as well as the smaller node, the AIO model with all,parts of the 'brain' build in house --- IMHO, it's a 50-50 tie between chip engineering but I'm bias as an ambidextrous user since '07/'08 (iOS/Android) - I have always had one of each, the family is iOS and since switching everyone over, my workload has decreased 95%. It's vertical and horizontal integration and aggregation with macOS is, still to me, science fiction and for the family business... a God Send.

    That said --- my S6 has an Exynos (sp?) processor, Note 4 was Qualcomm and, as I skip Android gens, my next will be a Qualcomm. I know as an iPhone 7+ owner I was delighted to learn that the model I bought has a Qualcomm modem, not the Intel;)

    Special trip for you guys. Great write up and truly amazing to me ...I'm 45, born with the 8086 processor and the progress mankind has made in such a tiny package, which is high speed connected with exponentially more power than just a decade ago... in our pocket. We all need to remember between our friendly iOS/Android 'disputes' -- the special world we enjoy today specifically BECAUSE Apple and Google/Qualcomm/SnapDragons and their host of OEMs building what just a decade ago meant 110v, plugged in No mobility, significantly slower - even wired connectivity. None of the 'Millions' of free, $1, $5 & ten buck 'programs/software' then, apps, now - available on demand! Over 30 million song libraries, endless knowledge and tools, true magic is what I think the SD835 A10 Fusion and their predecessors are/were.
    I'm old now, but not compared with the mountains I live in -- lucky enough to have spent the second ½ of my life quite literally watching these chips come to fruition ...I think it's the A10 when announced... it had/has over 3 billion transistors... and the SoC's the size of our fingernail!

    Screw arguing. It's a competitive world and WE are the beneficiaries!
  • edlee - Wednesday, March 22, 2017 - link

    the 835 has a 10% stronger gpu than a10, its just nuts that apple, not being a cpu designer at heart, can design a better cpu/soc that is years ahead of of what arm and qualcomm can produce
  • BedfordTim - Wednesday, March 22, 2017 - link

    There is a price/performance trade off with processors. Apple has chosen to make a much bigger processor which is why it is faster. Think Atom vs Core. One is slow but cheap and one is expensive and fast.
    Apple are not "years ahead". They have chosen to spend more on the processor.
  • Lord-Bryan - Wednesday, March 22, 2017 - link

    "They have chosen to spend more on the processor"
    They also had 64bit arm cores 2 years before Qualcomm released theirs, And that's is why they are year's ahead in performance and power efficiency

Log in

Don't have an account? Sign up now