Since we first learned that the A9 SoC in Apple’s iPhone 6s lineup is dual sourced - that is that it's being made by two different vendors with two distinct manufacturing processes - one major question has remained in the process of reviewing these two phones. The main issue under question here is whether the TSMC A9 or Samsung A9 have any difference in performance and power consumption. If there is a difference, the question then becomes whether the difference is significant.

In an atypical move for the normally tight lipped manufacturer, Apple issued a statement this afternoon in response to these questions and some rudamentary end-user benchmarking showing that there may be a difference:

With the Apple-designed A9 chip in your iPhone 6s or iPhone 6s Plus, you are getting the most advanced smartphone chip in the world. Every chip we ship meets Apple's highest standards for providing incredible performance and deliver great battery life, regardless of iPhone 6s capacity, color, or model.

Certain manufactured lab tests which run the processors with a continuous heavy workload until the battery depletes are not representative of real-world usage, since they spend an unrealistic amount of time at the highest CPU performance state. It's a misleading way to measure real-world battery life. Our testing and customer data show the actual battery life of the iPhone 6s and iPhone 6s Plus, even taking into account variable component differences, vary within just 2-3% of each other.

It interesting to see this response as Apple normally doesn’t comment on anything like this, which in turn is likely a good indicator of how seriously Apple is taking any concerns. However, this statement is also of interest because it's revealing in terms of what internal data Apple has collected on the issue. Apple has in recent years been one of the better companies in accurately promoting the battery life of their products, and that kind of accuracy comes not only from taking a conservative (safe) stance in marketing, but also collecting massive amounts of data to understand their products and their capabilities.

The Test

To get to the meat of matters then, let's talk about battery life, tests, chips, and statistics. In terms of the testing that has seemingly spurred on this Apple response, it’s likely that the “manufactured lab tests” in the statement refer directly to Primate Labs’ GeekBench battery life benchmark. The GeekBench test runs parts of the GeekBench CPU benchmark in a loop, making sure to do a fixed amount of work per time interval while idling the rest of the time, and using the score result as a modifier for the runtime score. This makes the GeekBench battery life benchmark primarily a SoC/CPU/Memory benchmark, and that in turn has repercussions for interpreting the data.

In the case of our own web browsing battery life test, for example, this is a test that attempts to simulate light reading of web pages, meaning that the SoC is only working hard a fraction of the time. The vast majority of the time the SoC is idling, making the display the biggest power consumer; and this is especially the case on these latest generations of high performance flagship smartphones. A heavy test on the other hand would be a test that keeps a sustained and significant load on the CPU and GPU, which shifts the power consumption of a test from the display to the SoC.

Battery Life: Light vs. Heavy
An Example of the Wide Gulf Between Light and Heavy Battery Life Testing

Due to the nature of its use of fixed size workloads, the GeekBench battery life benchmark lies somewhere in between a heavy load and a light load (Primate Labs states it's around 30% on the 6s). And this is notable because if this is the case, it means that GeekBench is in fact highlighting the difference in power consumption between the TSMC and Samsung A9s. However as Apple points out in their statement, a sustained workload is not necessarily representative of what real world usage is like, with the real world having a burst of of different types of workloads. This doesn't mean GeekBench doesn't return valuable data, however it means we're looking at a slice of a bigger picture. Ultimately if there is a difference between the TSMC and Samsung A9s, then it means that GeekBench is likely to be exacerbating the difference versus what a real world mixed use case test would see.

The Chips

As for the data itself, due to the fact that GeekBench is a heavier workload, it means that there are a number of factors that could explain why battery life in this test shows such a large difference, and not all of these factors are easy to account for. With the chips themselves, it could be that the Samsung A9 variant is simply reaching higher average temperatures due to its smaller die size (same heat over a smaller area), which then accordingly affects power draw due to the nature of semiconductor physics (increased leakage). It doesn’t have to be at the point where the workload is causing thermal throttling, but even a sustained load for significant periods of time could be enough to cause this effect, as the CPU will reach higher temperatures even if the phone is cool to the touch.


An example of the temperature versus power consumption principle on an Intel Core i7-2600K. Image Credit: AT Forums User "Idontcare"

Given the nature of chip manufacturing, it’s also hard to say one way or another whether an individual chip and phone pair will have better or worse battery life than another chip and phone pair. This is a pretty complicated subject, but it basically boils down to the difficulty of injecting exactly a certain number of ions into a small part of the silicon wafer or depositing a layer of insulator that meets an exact thickness. This results in chip manufacturing quality being distribution based - a wafer will come out of production with individual chips of varying quality, with some chips operating at lower voltages or lower leakages than others, and other chips being altogether defective. This is colloquially known as the "silicon lottery."

It's then from these chips that a customer (e.g. Apple) needs to made tradeoffs between how "poor" of a chip they are willing to accept and how many chips per wafer they'd like to be able to use from each wafer (the yield). In doing so, a customer will set minimum tolerances, certain parameters that a chip needs to meet to be qualified. However as these are minimums, it means that a customer will also receive chips that exceed these minimums, as we can see in our completely fictitious chart below.


A completely fictitious processor quality distribution example. Chips in the white area are used, outlying chips in red areas are rejected

In the case of someone like Intel, they will bin these passing chips as different products (Core i3/i5/i7) and different clockspeeds to charge the most for the best chips. However since Apple currently only uses at most two bins of chips (those suitable for 6s and those suitable for 6s Plus), this means there is a wider variation in the chips used in each phone. As a result, even if there isn't a true and consistent difference between TSMC and Samsung for A9 SoCs, you could easily have a pair of phones where due to the silicon lottery there is a notable - though not extreme - difference in power consumption.

This variation is an expected part of chip manufacturing, and while Apple could disqualify more chips and thereby reduce the yield, they will always have a certain degree of variation. The key here is to set rigorous minimums, advertise a phone based on those minimums (e.g. battery life), and should a customer end up with a phone, then they have won the silicon lottery in this case.

The Statistics

So how does one compare phones when there's a natural variance in chip quality? Ideally it would be done just like Apple does, testing a large number of phones (chips) for power consumption and battery life to determine the distribution. Otherwise if we only test one TSMC A9 and one Samsung A9, we don't know where in the distribution each A9 lies, and consequently whether each phone is a representative "average" sample or not.

Of course this is easier said than done, as the greater the accuracy desired the greater the number of iPhones required, a bill that at $650/$750 a unit adds up quickly. This makes it incredibly impractical for any one group short of a large corporate competitive analysis team to get enough samples, especially since at the press level Apple only distributes one phone of each type (for a total of 2) to each press organization. In lieu of that, typically the best one can do is look at a small number of samples, which offers some data to account for variance, but not much.

This brings us back to where we started with GeekBench. As part of the GeekBench benchmark, results are uploaded to Primate Labs' servers, where they are available for browsing. I've had a couple of people ask whether these results are a collective whole - in essence crowdsourced benchmarking - can answer anything, and the answer to that is a "yes, but" kind of scenario.

The big problem right now is that GeekBench doesn't know what model processor is being tested, so there's no way to sort out TSMC versus Samsung. Update: And as if right on cue, the same day we publish this GeekBench 3.4 makes it through the iOS app store approval process, adding the ability for GeekBench to tell which processor is in a phone.

Even if there was, we'd get to matters of testing rigor. How bright is the screen on each phone? Are there any background tasks running? Is it in airplane mode or spending power talking on WiFi/cellular? With Geekbench running at on average just a 30% duty cycle, these are all potential power consumers that can significantly impact the resulting battery life. In turn, these are all things that are accounted for in formal testing, but they cannot reliably be accounted for in benchmarks run by the wider public. This as a result adds even more variance to the equation, which makes individual or even small groups of results potentially very inaccurate.

The Conclusion

Wrapping things up then, where do we stand? The short answer is that all we know is that we don't know. What we know is that there isn't enough information currently out there to accurately determine whether the TSMC or Samsung A9 SoC has better power consumption, and more importantly just how large any difference might be. 1-on-1 comparisons under controlled conditions can provide us with some insight in to how the TSMC and Samsung A9s compare, but due to the natural variation in chip quality, it's possible to end up testing two atypical phones and never know it.

To that end I suspect that Apple's statement is not all that far off. They are of course one of the few parties able to actually analyze a large number of phones, and perhaps more to the point, having a wide variation in battery life on phones - even if every phone meets the minimum specifications - is not a great thing for Apple. It can cause buyers to start hunting down phones with "golden" A9s, and make other buyers feel like they've been swindled by not receiving an A9 with as low the power consumption as someone else. To be clear there will always be some variance and this is normal and expected, but if Apple has done their homework they should have it well understood and reasonably narrow. The big risk to Apple is that dual sourcing A9s in this fashion makes that task all the harder, which is one of the reasons why SoCs are rarely dual sourced.

As for AnandTech, we'll continue digging into the matter. Unfortunately all of the iPhones we've received and purchased so far have used TSMC A9s - it's a silicon lottery, after all - but whether there is a real and consistent difference between the TSMC and Samsung A9s is a very interesting question and one we're still looking to ultimately be able to address.

POST A COMMENT

147 Comments

View All Comments

  • pablo906 - Sunday, October 18, 2015 - link

    And now we understand the hate, he's europoor and can only afford counterfeit gear or cheap as dirt abacus with an antenna style phones. It's ok man you can immigrate somewhere where you don't have to use smoke signals to communicate with the other villages. I hear in Romania they're up to carrier pidgeons maybe check them out? Reply
  • Hembreeder - Thursday, October 15, 2015 - link

    Given your exceedingly dumb comments, I assume you use Apple products. And if you don't, how do you know they are so bad? Reply
  • dmacfour - Monday, October 19, 2015 - link

    "Given your exceedingly dumb comments, I assume you use Apple products."

    You speak just like the unhygienic and socially inept people we employ as developers.
    Reply
  • pablo906 - Sunday, October 18, 2015 - link

    You speak in absolutes like apple doesn't make anything good. It appears you aren't an owner but seems to be a complete expert in their products. I'm an expert on how terrible Lambos are, I mean I don't own one but anyone with two brain cells to rub together can realize that there are the worst cars ever. /s Reply
  • Shiitaki - Monday, October 19, 2015 - link

    Huh?! I have a 22nm Haswell cpu in my iMac, with 32GB DDR3L ram, PCI-E SSD, 780M GTX graphics with 4GB, and a sRGB 27" screen. Yes, some of us Apple fanboys do know exactly what is in the computer. Some of use have actually tried building imitation Macs using PC hardware. It turns out there is a very good reason why Apple charges what they do. They don't use the mediocre crap that other companies do, and which people are constantly comparing to Apple. Turns out if you do build a PC with the same specs, it costs a lot! In fact if you are going 5K, there is nothing in the PC space even close in price, Dell sells a monitor for almost as much as the entire computer.

    I use FreeBSD, Linux, Windows, and OS X! OS X is easily the nicest OS to use, and the one I prefer. And after 3 years I finally bought the real thing, a 27" iMac with all of the goodies. And it sits on my desk, with my previous PC tower on the floor on the other side of me. And it is remarkable how silent, beautiful, fast, and tiny the iMac is in comparison. Both i7s, PC is Ivy, iMac is Haswell. Both have lots of ram, SSDs.

    You see, it is the enjoyment of the anticipation of having a date with a model, and then not finding out she is a vegetarian. That is the secret to Apple's fan base. Satisfaction, or lack of the disappointment PCs bestow as a reward for going cheap.

    I had to wait forever, for my Build to Order to arrive. Yes, most of the stock configurations are terrible. The longest wait was apparently for the NSA to install back doors in California for about a week. Never seen anything take that long to ship from California to Oregon. But then the day arrived, adults rarely get that kid on Christmas morning feeling. I singed for it, brought the odd shaped box in, and started the unboxing. Apple's attention to detail is amazing, I still use the box to transport the computer. In 5 minutes I had the machine booted, and already logged in to iCloud. My contacts, notes, and documents already available. Even my gmail account was configured automatically for me! My network shares were already available, no BS configuration needed like Windows. I could just click on them in Finder! And then Apple asks ME if I want to ACCEPT the software licenses included with the computer! Ten minutes in to the very pleasant ordeal, I am bragging up the computer to one of my friends by making a phone call from the new iMac, being relayed through my iPhone. You'll notice I did not mention reinstalling the OS to get rid of garbage software, no restarts to install updates, it was already updated at the factory.

    Let's contrast that with the best PC i ever bought, a 3K dollar laptop in 2007. I opened the box, connected the power adapter, started the machine. And up came Windows Vista desktop. And what did I do? Stared at the screen, checked control panel, surfed the internet, and then restarted to install updates. The thing is that Windows doesn't do much out of the box, and that is if you are fortunate enough for there to not be any garbage installed. Yes, I've used Win8.1, it is a technological wonder in all the places you can't see, but the user experience hasn't changes. I don't know about Windows 10 since Microsoft refuses the update to Win8.1 No errors, just doesn't work. And I'm used to that. That is how Microsoft rolls. I tried forcing with the ISO, no joy.

    Imagine, you are working on several things at once, and the power goes out. Why do people love Apple? Because when you restart, all of the apps open back up, and I am back where I was like nothing happened. This ability has been part of OS X for a long time. In fact, Microsoft even added key support this in NTFS and SMB! Leveraging this technology, my iMac installed updates at night and I only know because of a notification when I sit down in the morning. Thus, updates are much better on Mac. Drivers? Yes, OS X has them, I don't do anything with them. Microsoft could curate drivers from the less than 2 dozen chip manufacturers, but why bother right? WPA, I've never typed in a serial, or called Apple to install the OS. Even when I have done so on a PC. In fact reinstalling OS X on a PC is so much easier than a installing Windows. Microsoft really needs to do better, they are doing terrible. App store, there is not a lot of stuff in the Mac App store like there is in the iOS store, but when you do buy something from the store, it works on ALL of your computers. Not just 1. The myth that Apple is more expensive is false. It is only expensive for the first desktop, but you save more the more Macs you have. You also save money over time, since updates and newer versions of the OS are included in the initial buy in. There is a lot of savings in tech support and hardware issues as well. Reinstalling is so much easier if necessary, and TimeMachine backups are far, far, far ahead of Microsoft's backup solution.

    To sum it up, yes Apple is expensive. But I believe life should be made better with technology, instead of life being wasted making technology work.
    Reply
  • Alexvrb - Monday, October 12, 2015 - link

    They're good devices, they're just too expensive for what you get. Especially considering the 16GB model is still the base model. Only Apple can get away with that. It would be OK if it had mSD... but it doesn't. Reply
  • ex2bot - Saturday, October 10, 2015 - link

    Wait, what?? Did you just …

    (Scratches head. Shakes head bemusedly. Taps "Submit Comment
    Reply
  • marmalito - Sunday, October 11, 2015 - link

    "it is just a big meh."

    There are many, including experts who work for Anandtech, Ars Technica, and other reputable computer technology-focused organizations, who would argue otherwise. Many even consistently review new iOS devices and Mac computers as being among the best in their class.

    Viewed objectively, the fact that the Apple-designed A9 is a power-sipping dual-core, nearly desktop class 64-bit SOC that can elegantly keep pace and even outperform Samsung's Galaxy 8 core, battery-chugging chip, while managing memory with double the efficiency is nothing short of amazing.

    Have you seen the benchmarks? Nobody would have guessed Apple would be on the cusp of rivaling Intel for mobile processors in CPU and power performance, much less doing it inside a PHONE or tablet.

    But let's just go back to the Apple of pre-profit times, like before their first sell-out vanity product, the iPod, and also iMacs... MacBooks, iPhones, iPads, etc...?
    Reply
  • ddriver - Sunday, October 11, 2015 - link

    Yes, I've seen the benchmarks, good single threaded performance, overall chip performance is nothing to brag about.

    But let me tell you this - you should definitely trust people, getting paid to praise apple and measure performance using JS benchmarks which do not even use the same JS implementation, because that's such an expert and objective comparison. Because it is not like the IT media is sucking up to apple, it is not like the US "justice" system fighting the competition for them. Nothing like that whatsoever.

    But so what? I don't argue apple arm chips have very good single threaded performance, but so what? What good does it do for you? Is there a tangible benefit from it? Does it improve your life, or your productivity? If so, how exactly? What are you doing with this CPU horsepower? I mean people went into space with megahertz CPUs with kilobytes of memory, today we have multi core gigahertz CPUs with gigabytes of memory in our pockets, and what do we do with it?

    That's the Worst part - today the actual product is not what the consumer purchases, but the consumer himself. Sure, this applies to all device manufacturers, but it is apple who began that terrible trend, where people use less and less of the technology and are used more and more through it, turning powerful technology in toys which actually make people stupider and wasting their lives in pointless, vanity feeding activities.
    Reply
  • artificialintel - Sunday, October 11, 2015 - link

    So you're basically saying Anandtech has no integrity. Super. Why are you even here? Reply

Log in

Don't have an account? Sign up now