Overall Analysis & Conclusion

Hopefully we've managed to cover a few of the more common use-cases that are routinely encountered in daily usage on Android and get a good idea of how applications behave. We've seen some quite expected numbers for some use-cases but also stumbled on very large surprises that weren't quite as obvious. 

There were two cases that especially stood out: Browser usage and application installation and updates. It could be argued that app updates are merely a corner-case that doesn't affect a user's experience much. After all, installing an updating apps represent only an insignificant fraction of what a user does on a device. Browser usage and web-page rendering in general however, are one of the most common and often encountered scenarios on a smartphone, and here's where we encountered the largest surprises.

When I started out this piece the goals I set out to reach was to either confirm or debunk on how useful homogeneous 8-core designs would be in the real world. The fact that Chrome and to a lesser extent Samsung's stock browser were able to consistently load up to 6-8 concurrent processes while loading a page suddenly gives a lot of credence to these 8-core designs that we would have otherwise not thought of being able to fully use their designed CPU configurations. In terms of pure computational load, web-page rendering remains as one of the heaviest tasks on a smartphone so it's very encouraging to see that today's web rendering engines are able to make good use of parallelization to spread the load between the available CPU cores.

It's hard to summarize the vast amount data of the last 16 pages in an orderly and correct manner. After all we are talking about extremely varying use-cases and time-scales for each scenario. While averaging the metrics over the course of a scenario might seem a good idea at first, one has to keep in mind that this wouldn't be able to properly represent cases where load peaks for smaller durations. It's these small computational bursts which are most of the time the cause for "lags" and frame-drops. So to better represent these bottle-necks which determine the user-visible cases of application speed and performance, we rather use the 90th percentile of the CPU run-queue depths:

90th Percentile Run-Queue Depth Averages
  Little Cluster Big Cluster Little + Big
S-Browser - AnandTech Article 2.27 2.19 3.87
S-Browser - AnandTech FP 3.12 1.25 4.15
Chrome - AnandTech FP 5.69 1.84 7.10
Chrome - BBC Frontpage 5.00 2.00 6.22
Hangouts Launch 2.77 2.11 4.01
Hangouts Writing A Message 2.80 0.05 2.57
Reddit Sync Launch 1.84 1.11 2.38
Reddit Sync Scrolling 0.95 1.03 1.46
Play Store Open & Scroll 2.87 0.78 3.45
Play Store App Updates 3.73 5.42 8.51
Camera: Launch 1.45 2.73 2.98
Camera: Still Snapshot 4.12 0.87 4.59
Camera: Video Recording 5.17 2.04 5.42
Real Racing 3 Launch 2.16 1.33 3.26
Real Racing 3 Playing 2.09 0.89 2.96
Modern Combat 5 Playing 2.09 0.73 2.68

I was wary of creating this table as it can be easily misinterpreted: Because run-queue depth averages are not directly representative of the amount of concurrent threads in a given scenario, we lose information when aggregating them for a given cluster or the whole system. This for example happens on the big cluster on the AT article load scenario where the 90th percentile of the aggregate rq-depth reaches 2.19 while in reality this figure is composed of 4 medium-high threads. Readers should thus keep in mind the actual detailed graphs of the preceding pages when reading the table.

While not directly the goal of the article, the collected data also serves as a perfect case-study for heterogeneous big.LITTLE SoCs. We've long seen discussions concerning what the "ideal" big.LITTLE configuration would be. There's several angles to this: the most optimal little and big cluster core counts, and whether we're aiming for performance or power efficiency in each case. In terms of low- to medium-performance threads, we've had several cases where 4 little cores weren't enough. Web page rendering in Chrome in particular seems to be the killer use-case where actually having two clusters of highly efficient cores makes sense.

On the high-performance "big" cluster side, the discussion topic is more about whether 2 or 4 core designs make more sense. I think the decision here is not about performance but rather about power efficiency. A 2-core big-cluster design would provide more than enough performance for most use-cases, but as we've seen throughout our testing during interactive use it's more common than not to have 2+ threads placed on the big cluster. So while a 2-core design could handle bursts where ~3-4 threads are placed onto the big cluster, the CPUs would need to scale up higher in frequency to provide the same performance compared to a wider 4-core design. And scaling up higher in frequency has a quadratically detrimental effect on power efficiency as we need higher operating voltages. At the end of the day I think the 4 big core designs are not only the better performing ones but also the more efficient ones. 

This puts one particular vendor in quite of an interesting position: MediaTek. Even if one wouldn't be able to fully saturate a cluster one can still derive power efficiency advantages due to the fact that two small clusters would be able to operate at separate frequencies and thus efficiency points. I've encountered enough scenarios that would in theory fit the Helio X20's tri-cluster design that I'm starting to think that such a design would actually be a very smart choice for current Android devices.

What about more traditional SoC configurations? As mentioned earlier symmetric 8-core designs such as MediaTek's Helio X10 would, contrary to one's expectations, be seemingly able to take advantage of their higher core counts. So while it would be preferable to have higher performance cores such as Cortex A57's or A72's, one has to keep in mind the target market of these architectures are limited to higher-end SoCs. The 8 little-core designs are mostly targeted at the entry- and mid-level where adding a second Cortex A53 cluster can be very cheap way of still providing benefits in every-day usages, particularly in web-browsing.

What is clear though albeit there are corner-cases, is that the vast majority of applications do seem to be optimal for quad-core SoCs. This is why traditional 4-core and 4.4 big.LITTLE designs still appear to make the most sense in terms providing a balanced configuration and making most use of the hardware at hand. For big.LITTLE, even if there were no use-cases where all cores are concurrently used, it's not a big deal as what we are aiming for in heterogeneous systems is power efficiency gains.

This is also the point of the discussion where the debate of the potential detrimental effect of having more cores comes into play: The fact that a SoC has more cores does not automatically mean it uses more power. As demonstrated in the data, modern power management is advanced enough to make extensive use of fine-grained power-gated idle states, thus eliminating any overhead there might be of simply having more physical cores on the silicon. If there are cases (And as we've seen, there are!) which make use of more cores then this should be seen purely as an added bonus and icing on the cake. 

What about narrow CPU-core number design philosophies? Would such designs make sense on Android? This is probably another question that our readers will ask themselves when looking at the data. Apple and recently Nvidia with their Denver architecture both choose to keep going the route of employing large 2-core designs that are strong in their single-threaded performance but fall behind in terms of multi-threaded performance.

While for Apple it can be argued that we're dealing with a very different operating system and it is likely iOS applications are less threaded than their Android counter-parts. But there are cases where this doesn't need to be necessarily hold true: For example browser rendering engines, as demonstrated, can be multi-threaded if adapted to do so. Native high-end games which already make use of multiple threads are also unlikely to differ in their threading logic between the platforms.

While such narrow CPU-core designs would have higher performance at a given frequency - it is not a direct indicator of the actual performance/W efficiency that a single thread would have on these chipsets. We still haven't had a chance to make a proper apples-to-apples comparison for these architectures so we're limited to theorycrafting with the data we currently have available to us:

What we see in the use-case analysis is that the amount of use-cases where an application is visibly limited due to single-threaded performance seems be very limited. In fact, a large amount of the analyzed scenarios our test-device with Cortex A57 cores would rarely need to ramp up to their full frequency beyond short bursts (Thermal throttling was not a factor in any of the tests). On the other hand, scenarios were we'd find 3-4 high load threads seem not to be that particularly hard to find, and actually appear to be an a pretty common occurence. For mobile, the choice seems to be obvious due to the power curve implications. In scenarios where we're not talking about having loads so small that it becomes not worthwhile to spend the energy to bring a secondary core out of its idle state, one could generalize that if one is able to spread the load over multiple CPUs, it will always preferable and more efficient to do so. 

In the end what we should take away from this analysis is that Android devices can make much better use of multi-threading than initially expected. There's very solid evidence that not only are 4.4 big.LITTLE designs validated, but we also find practical benefits of using 8-core "little" designs over similar single-cluster 4-core SoCs. For the foreseeable future it seems that vendors who rely on ARM's CPU designs will be well served with a continued use of 4.4 b.L designs. Only MediaTek seems to fall out of the norm here with its upcoming X20 SoC, which I'm definitely looking forward to see as to how it behaves in the real-world. We'll also see some vendors revert back to quad-core designs in their custom architectures - while we've yet to get a better picture of how these will behave in terms of performance and power, I think that 4 cores will be a quite reasonable target and sweet-spot for vendors to aim for.

Games: Modern Combat 5 Playing


View All Comments

  • Samus - Wednesday, September 2, 2015 - link

    They are both clearly android fans and haven't ever given anything else a chance. The fact they ignore Apple has consistently had superior single threaded performance in their SOC's years and this has translated to better UX just goes to show that Android targeting multithreaded performance is a solution looking for a problem. There are so many underlying issues to address first, specifically making efficient use the Linux scheduler and perhaps setting a compatibility list for hardware instead of saying just make anything and we'll find a way to run on it no matter how crappy it runs. Reply
  • tuxRoller - Wednesday, September 2, 2015 - link

    Apple had not consistently had better performance per core. That's fairly recent (since cyclone, iirc). There are myriad issues at play.
    In the end, the market is best served by an open option, like Android, and customers choosing what works best for them and letting the rest fade away.
  • name99 - Wednesday, September 2, 2015 - link

    "Apple had not consistently had better performance per core. That's fairly recent (since cyclone, iirc). "
    Since Swift. That's iPhone 5, 5S, 6 (2012, 2013, 2014) and likely to be 6S and 2015 at least.
    Even the late-stage pre-Apple cores were substantially above average (in part because of Apple's custom SoC). The 4S was above the competition at the time:

    Most people would consider "consistent enough" for "long enough" to make the statement reasonable.
  • lopri - Wednesday, September 2, 2015 - link

    And it is not like Apple don't resort to moar-cores. When they run into walls, they also have no choice but to take whatever routes that are available. Listening to some of the zealous Apple fans, one would mistake that iPhones have been rocking on a single-core all these years.

    They have moved to dual-cores on the phones, and 3-cores on tablets. Moar-cores on iDevices are only a matter of time. Those specialized ASICs with fancy names apple give ("Motion Processor" for one) are also a concession made by Apple that there are cases where big cores are not always the best route to take when efficiency matters.
  • Buk Lau - Wednesday, September 2, 2015 - link

    "They are both clearly android fans and haven't ever given anything else a chance."
    uhh my first smart device ever is a 2nd gen iPod touch...

    So just because I proved you wrong, I have to be an Android fanboy? You said you tried all these Android phones "every week" and have "shit experiences." Again, you didn't bring up any names or so. What phones have you even tried? Who's being a fanboy here and can only provide claims without backing them up with facts?

    I don't understand why you are arguing about this superior ST performance when it's irrelevant to this article. What this article simply proves is that Android does make use of extra threads and you get a benefit in power efficiency due to running MT thread, nothing about performance. In fact in most scenarios shown in the test most of the little cores are even saturated which means the workload isn't heavy at all.

    "Apple has consistently had superior single threaded performance in their SOC's years and this has translated to better UX"
    any evidence that leads to this conclusion? also like tuxRoller said Apple only have IPC advantages in recent years with Cyclone series.

    "There are so many underlying issues to address first, specifically making efficient use the Linux scheduler and perhaps setting a compatibility list for hardware instead of saying just make anything and we'll find a way to run on it no matter how crappy it runs."
    Where did you get the concept of make anything and find a way to work? All OEMs and SoC manufacturers optimize for Android just like how they optimize for Windows in desktop. Like I said before, SoC manufacturers have to provide driver update every time there's a HAL change in Android. How well they can do to optimize is up to themselves but the fact is that they do have to make their hardware compatible for Android
  • Kutark - Wednesday, September 2, 2015 - link

    Did i suddenly log onto the pcgamer forums? The instant someone expresses any level of dismay or concern for an apple product, or says they have good experiences with android phones, it automatically means they're a nutswinging fanboy? Reply
  • Buk Lau - Wednesday, September 2, 2015 - link

    You can argue whether Apple is intentional or not but the end result is that 4S users are getting more sluggish experiences with their 4S after updated to iOS 8 Reply
  • tuxRoller - Wednesday, September 2, 2015 - link

    Linux isn't great about niceness There's a few ways to fix this. One is to use cgroups ,(which Android uses). This works pretty well but I'd still subject, ultimately, to the scheduler. The other way is to run the rt kernel. That obeys priorities nicely (heh), but would be a bear to wrestle into Android and you'd lose some power efficiency. Also the rendering framework of Android may have some issues. Reply
  • darkich - Friday, September 4, 2015 - link

    Im calling not only BS, but a truckload of it.

    Just so full of ignorance and prejudice that it's probably not worth a thorough reply..if you do want one though, let me know and you will be served.
  • brianpauler - Saturday, July 17, 2021 - link

    This is a very valuable change of Reddit. If you are a regular user of Reddit, you probably know this Reddit video downloader: Reply

Log in

Don't have an account? Sign up now