Who Controls the User Experience? AMD’s Carrizo Thoroughly Tested
by Ian Cutress on February 4, 2016 8:00 AM ESTBenchmark Overview
For our testing we had each of the laptops at the same time for the best part of a workweek, alongside meetings with AMD to discuss the microarchitecture and platform positioning. Each system was purged into a fresh OS state, and then we applied a high profile performance state for the benchmarking in the air-conditioned lab.
The benchmarks fall into several areas:
Short Form CPU
Our short form testing script uses a straight run through of a mixture of known apps or workloads, and requires about four hours.
| CPU Short Form Benchmarks | |
| Three Dimensional Particle Movement (3DPM) | 3DPM is a self-penned benchmark, derived from my academic research years looking at particle movement parallelism. The coding for this tool was rough, but emulates the real world in being non-CompSci trained code for a scientific endeavor. The code is unoptimized, but the test uses OpenMP to move particles around a field using one of six 3D movement algorithms in turn, each of which is found in the academic literature. This test is performed in single thread and multithreaded workloads, and uses purely floating point numbers. The code was written in Visual Studio 2008 in Release mode with all optimizations (including fast math and -Ox) enabled. We take the average of six runs in each instance. |
| WinRAR 5.01 | WinRAR is a compression based software to reduce file size at the expense of CPU cycles. We use the version that has been a stable part of our benchmark database through 2015, and run the default settings on a 1.52GB directory containing over 2800 files representing a small website with around thirty half-minute videos. We take the average of several runs in this instance. |
| POV-Ray 3.7 beta | POV-Ray is a common ray-tracing tool used to generate realistic looking scenes. We've used POV-Ray in its various guises over the years as a good benchmark for performance, as well as a tool on the march to ray-tracing limited immersive environments. We use the built-in multithreaded benchmark. |
| HandBrake | HandBrake is a freeware video conversion tool. We use the tool in to process two different videos - first a 'low quality' two hour video at 640x388 resolution to x264, then a 'high quality' ten minute video at 4320x3840. The low quality video scales at lower performance hardware, whereas the buffers required for high-quality can stretch even the biggest processors. At current, this is a CPU only test. |
| 7-Zip | 7-Zip is a freeware compression/decompression tool that is widely deployed across the world. We run the included benchmark tool using a 50MB library and take the average of a set of fixed-time results. |
Web and Synthetic
The web tests are a usual mix of Octane/Kraken with WebXPRT in the mix. Synthetic CPU testing relates to our long term data under CineBench and x264.
| Web and Synthetic Benchmarks | |
| Google Octane 2.0 | Lots of factors go into web development, including the tools used and the browser those tools play in. One of the common and widely used benchmarks to judge performance is Google Octane, now in version 2.0. To quote: 'The updated Octane 2.0 benchmark includes four new tests to measure new aspects of JavaScript performance, including garbage collection / compiler latency and asm.js-style JavaScript performance.' |
| Mozilla Kraken 1.1 | Kraken is a similar tool to Google, focusing on web tools and processing power. Kraken's tools include searching algorithms, audio processing, image filtering, flexible database parsing and cryptographic routines. |
| WebXPRT 2013/2015 | WebXPRT aims to be a souped up version of Octane and Kraken, using these tools in real time to display data in photograph enhancement, sorting, stock options, local storage manipulation, graphical enterfaces and even filtering algorithms on scientific datasets. We run the 2013 and 2015 versions of the benchmark. |
| Cinebench | Cinebench is a widely known benchmarking tool for measuring performance relative to MAXON's animation software Cinema 4D. Cinebench has been optimized over a decade and focuses on purely CPU horsepower, meaning if there is a discrepancy in pure throughput characteristics, Cinebench is likely to show that discrepancy. Arguably other software doesn't make use of all the tools available, so the real world relevance might purely be academic, but given our large database of data for Cinebench it seems difficult to ignore a small five minute test. We run the modern version 15 in this test, as well as the older 11.5 due to our back data. |
| x264 HD 3.0 | Similarly, the x264 HD 3.0 package we use here is also kept for historic regressional data. The latest version is 5.0.1, and encodes a 1080p video clip into a high quality x264 file. Version 3.0 only performs the same test on a 720p file, and in most circumstances hits its limit on high end processors, but still works well for mainstream and low-end. Also, this version only takes a few minutes, whereas the latest can take over 90 minutes to run. |
Professional and OpenCL
Our professional tests involve a synthetic workload (PCMark), a 2D to 3D image and model conversion tool used by archivists and modelers (Agisoft in CPU only and OpenCL mode) as well as Linux Bench. Unfortunately Linux Bench only seemed to work on a pair of systems.
| Professional and OpenCL Benchmarks | |
| PCMark08 | PCMark08, developed by Futuremark, is a simple press play and run benchmarking tool designed to probe how well systems cope with a variety of standard tasks that a professional user might encounter. This includes video conferencing with multiple streams, image/file manipulation, video processing, 3D modelling and other tools. In this case we take the three main benchmark sets, Creative, Home and Work, and run them in OpenCL mode which aims to take advantage of OpenCL accelerated hardware. For fun we also put in the PCMark08 Storage workset. |
| Agisoft Photoscan | Photoscan is professional software that takes a series of 2D images (as little as 50, usually 250+) and 'performs calculations' to determine where the pictures were taken and if it can create a 3D model and textures of what the images are of. This model can then be exported to other software for touch-ups or implementation in physics engines/games or, as the reader that directed me to it, national archiving. The tool has four phases, one of which can be OpenCL accelerated, while the other three are a mix of single thread and variable thread workloads. We ran the tool in CPU only and OpenCL modes. |
| Linux Bench | Linux Bench is a collection of Linux based benchmarks compiled together by ServeTheHome. The idea for this is to have some non-windows based tools that are easy enough to run with a USB key, an internet connection and three lines of code in a terminal. The tests in Linux Bench include standard synthetic compute, compression, matrix manipulation, database tools and key-value storage. |
Gaming (3DMark, Rocket League)
Due to timing we were only able to run a couple of gaming tests, namely parts of the 3DMark suite and our Rocket League test.
| Gaming Benchmarks | |
| 3DMark | 3DMark is Futuremark's premium software, developed to tax systems at various different performance levels. The software contains several benchmarks as a result, with some focusing more on smartphone use all the way up to 4K, quad-SLI systems with as many in-game and post processing effects as you can throw at it. The base test, Ice Storm, is actually a good indicator of GPU scaling performance, but we also test Cloud Gate, Sky Diver and Fire Strike to get a measure of all of our systems. |
| Rocket League | Hilariously simple pick-up-and-play games are great fun. I'm a massive fan of the Katamari franchise for that reason — passing start on a controller and rolling around, picking up things to get bigger, is extremely simple. Rocket League combines the elements of pick-up-and-play, allowing users to jump into a game with other people (or bots) to play football with cars with zero rules. The title is built on Unreal Engine 3 and it allows users to run the game on super-low-end systems while still taxing the big ones. |
Power and Performance Testing
A portion of our benchmarks were profiled for performance – namely their effect on CPU temperature, frequency and usage – which we will go in to detail over. Both of the HP Elitebooks, the Kaveri and Carrizo units, were also hooked up to a Watts Up PRO monitor for a full shakedown of power consumption on some of the more popular tests. We will go into these in detail.
Thermal Effects
As we have seen in previous laptop benchmarking scenarios, the design of the chassis is an important part in understanding how a processor will react to a workload. Some units have their skin temperature limit set unbearably high in order to get the best performance, whereas others are more restrictive. Carrizo promotes the expansion of both of these facets for either better performance or thermals, so we tested it with a FLIR thermal camera during Rocket League on all five systems as well as some internal recording scripts during a few benchmarks.
A Side Note worth Remembering
One intriguing thing to mention in our testing was background processes. Nominally all efforts are done to minimize these (disable WiFi when not needed, disable updates), however when a system comes preinstalled with Intel McAfee anti-virus, it can be an exercise to remove it. Yes, that’s right – for some odd reason, some of the OEMs systems had Intel McAfee pre-installed. I assume it is because the OEM gets a small kickback for including it on their OS image, therefore either increasing margins or reducing the price of the system. McAfee AV is an example of a simple piece of software that can provide a negative user experience – checking of updates when you least expect it, performing mini-scans of everything coming in and out of an I/O port, and for the systems that have mechanical hard drives with single channel memory, it can be the difference between casually watching a film to having to apologize for why a video is dropping frames. Needless to say, it was obliterated.
The other issue is actually a default windows problem. Whenever certain I/O devices are plugged in/removed, or even at random times, the system will call Windows Defender to start probing files and memory in use. The issue here is multiple – it eats up a thread with mostly integer/string work reducing available resources for the user, but on occasion will bring disk drive utilization to 100%, causing access delays when the user is in the middle of something. While Defender can be a critical part of a safer environment, it boggles my mind that it comes on so freely and robs a poorly configured system of its user experience. It also drains battery life as well. This is a disconnect between software developers writing code suitable for the resources available, OEMs for deciding what hardware would be good for a particular price point and believing users are satisfied with such a user experience, and the hardware manufacturers for not circling back round to test the most relevant use cases. It ends up being a negative loop where no-one works with each other, which benefits no-one (more on this later).
Consequently, for our testing I also turned down Windows Defender's activity/sensitivity on all of the test laptops. My personal (insert subjective experience mode) way of ‘delaying’ Windows Defender is to go to Task Scheduler, go to Microsoft > Windows > Windows Defender and on each of the four options change the conditions to:
- Enable ‘Start the task only if the computer is on AC power’
- Enable ‘Stop if the computer switches to battery power’
- Enable ‘Start the task only if the computer is idle for X minutes’
- Enable ‘Stop if the computer ceases to be idle’
How the system determines a true in-OS idle state is somewhat difficult, as some software will have idle periods before called (e.g. watching an online video) so having it come in after 30 seconds idle isn’t usually beneficial, so I (personally) set it for 10 minutes on lower end systems where responsiveness matters.
The System I Didn’t Get to Benchmark: The Dell Inspiron 3656
As part of my meeting schedule, I was offered an explanation as to what goes behind the scenes in retail marketing from one of the senior account managers. We took a trip to the local Best Buy and I was talked through how most areas of the store are, for lack of a better term, rented out by the retailer to the companies who have strict rules to follow. This applies to store-in-stores, end-caps, focused aisles and even the location within the store can affect the price. It made sense, but we came across this following AMD system:
This is the Dell Inspiron 3656 which, for lack of a better description, is Carrizo in a desktop form factor. I asked if I could peek inside, but for some reason no-one in Best Buy had a screwdriver (as if)! But inside is a mobile focused Carrizo CPU, presumably in 35W mode, with sufficient cooling as well as a discrete Radeon R9 360 graphics card in a PCIe x16 slot. Combine in some other factors such as a 2TB HDD, 16GB of DDR3L-1600 SO-DIMMs and you are good to go.
The 3656, as it turns out, can come with three different AMD Carrizo processors (FX-8800P, A10-8700P, A8-8600P) in a thermally unrestrained environment, which would arguably give the best possible scores. The two things I couldn’t confirm were related to the DRAM. I would have liked to know if the design is a true dual channel design for Carrizo only, or if it shares pin compatibility with Carrizo-L which would limit it to single channel only. Also the memory speed – if it is in 35W mode, this would mean the system could engage DDR3-2133 if it uses appropriate SO-DIMM modules. However, the specifications sheet only mentions DDR3L, which is limited to DDR3L-1600. In a desktop like this, the difference between DDR3L and DDR3 would be minor, and the higher speed memory would help benefit (unless it was Carrizo-L focused).
Facebook
Twitter
RSS
175 Comments
View All Comments
ncsaephanh - Monday, February 8, 2016 - link
Can you guys do a podcast on this article? Would love to hear you guys discuss it and also answer questions/comments on the article.ET - Monday, February 8, 2016 - link
Nice to see a Carrizo article finally, although it's rather disappointing, for example because only single channel was tested.You talked about solutions, here's how I see what AMD and publications like Anandtech need to do (I'm using Carrizo as an example, but it's a lesson for the future):
AMD: When Carrizo is available in a laptop, send one to Anandtech. Immediately. If you have a prototype before that, send that. We want to learn about the chip as quickly as possible, not have to wait months looking for nuggets of information on the web.
Anandtech: Benchmark the hell out of the laptop. If there's single channel with a dual channel option, show a comparative benchmark, but concentrate on dual. We're enthusiasts, we'll install a second DIMM to get better performance. For benchmarks, basic system performance and a plethora of games, and comparison to Intel, plus battery life. Deep dives are nice, but I'd rather have a quick overview of what the system is suitable for, and what kind of gaming it can achieve.
AMD: Desktop first! I know that laptops are where the money is, but desktop is where the enthusiasts are, and if your chip is worth anything, fans and publications like Anandtech will pair it with the fastest memory, configure it with the best TDP, and see what it's really capable of. OEM limitations will not get in the way.
AMD: Fans first! That's pretty much a repeat of the previous point, but AMD, you still have fans, and they are your best customers, not the OEM's or the clueless general public. If you make something that you think is good and you let your fans learn of it and get hold of it, they will tell you what they think and they will tell others. If you leave them in the dark, they will end up losing their enthusiasm.
Anandtech: Follow up on AMD stuff. It may be hard to get the latest AMD chips if AMD isn't helping, but at least let us know you're on it. An occasional news item telling us that you've tried to get some laptops for testing or whatnot will tell us that you're on it, and hopefully shame AMD and the OEM's enough to get a move on.
Personally, I would likely have bought a Carrizo system if there was one of similar size to my old Thinkpad X120e (which I still use, even if I'm not that happy with its speed). I might have bought a Carrizo for my HTPC if I could and I knew it provided decent enough performance.
sofocle10000 - Monday, February 8, 2016 - link
I just signed in to state that Asus had nice business/multimedia notebooks (I used N60DP/N56DP and I actually use an N551ZU - all based on AMD), and although my actual N551ZU is only based on the top of the line Kaveri, it is an exceptional machine for normal use/light gaming...Customers play a big part in the AMD problem, but if there were more incentives (take my current N551ZU, which is a great notebook for ~750-850 $, and if configured with an SSD, you could hardly tell it apart most of the time from the Intel i5H/i7QH + GTX 950M variants), not only a great price, but a better build quality, display, sound system the the market average, some of them would actually pay more attention to the AMD.
The OEM's should have a more defined bottom line for the AMD notebooks - were dual channel memory and a better display, a hybrid SSHD or a SSD are a must, especially for the models in the upper part of the price range 400-700 $...
dragosmp - Monday, February 8, 2016 - link
@Ian - great article, really a good example of investigative journalism. I'm happy this kind of articles are being revived, but being a reader of Tom's I see where this may be coming from.As the "guy that says what laptop/phone to buy" to my family and friends I have to say your findings and conclusions speak to me very clearly - AMD has a system-problem, not so much a CPU-problem (though some may argue differently). AMD chips are fed into cheap looking/feeling PCs with far too many corners cut, but this is how under 700$ market looks like. Could AMD's OEMs sell a 600$ 13" PC to compete with the CoreM UX305? I think not, simply because AMD's CPUs (who consume more) need thicker chassis with stronger cooling and a beefier battery and that costs money - so there's less available for the UX; even if the OEM accepted lower margins on the AMD PC, or AMD to sell the CPU at bargain prices, that design compared to the UX305 would be thicker and likely noisier.
If Zen is good, I could see it in a Mac as Apple has a history of doing good software. Or AMD should build their own surface line and set an example of what can be done.
Gunbuster - Monday, February 8, 2016 - link
People buy the cheapest $300 laptop they can get or something premium. Who are they targeting with these mid-rangers?farmergann - Tuesday, February 9, 2016 - link
Wife uses her Y700 for school and a few hours of photo editing every week. Exactly what she wanted. This article did a worthless job of representing the actual Y700 w/fx8800p you can pick up at Best Buy for $665-830. Everything is fantastic about it save for the TB HDD which I immediately replaced with a Samsung 850 Pro I had laying around.Somehow, this "investigative" nonsense missed the fact the U.S. Y700 has a superb little IPS screen with Freesync to go along with a surprisingly (truly) good sound system and -despite the author's claim- dual channel ram. Just for grins I've played BF3 and a few other games - none of which had issues. Great low/mid-range laptop with plenty of chops.
every1hasaids - Tuesday, February 9, 2016 - link
Nope, the US model is absolute garbage. They skimped on the VRMs and the laptop subsequently throttles in moderately intensive CPU tasks. Example, try running Cities: Skylines with a decent sized city and tell me that it doesn't stutter after about 20 seconds of play and every 5 seconds or so after that. The stutters which coincide with the CPU being utilized near 100% and the frequency dropping per resource monitor and Afterburner all the way down to 1.6ghz... Also I don't know what you're talking about with the Freesync capability, I could not get it to work after reading elsewhere that it may be possible.The main issue with a product like the Y700 is that the intel variant is only a couple hundred bucks more and you get a genuine quad core with HT, dual channel DDR4-2133 and comparable discreet graphics. Oh, and it has no trouble with voltage supply. Not to mention that the m.2 interface is PCI-E as opposed to SATA on the AMD model. It just doesn't make sense to purchase a far inferior product for only $200 less at the price point these models occupy.
farmergann - Thursday, February 11, 2016 - link
Cities: Skylines? LOL, that's about as rich as whining about Starcraft 2 performance on an FX Octacore - what were you expecting exactly? For people not looking to shove a laughably CPU bound title down a 35W laptop's throat, the FX8800p with user installed SSD is a far better choice, sorry guy.Peichen - Monday, February 8, 2016 - link
Wow, that's wasting a lot of time and words reviewing a product no one will buy. AMD needs to exist to keep the cheap Intel stuff dirt cheap but I don't feel anyone should waste time reviewing AMD CPU products. 10 years of marketing hype and under-delivery means AMD is actually slower than ever compares with Intel.I bought 2 AMD CPU over the last 6/7 years and frankly I wish I spend more buying Intel because I wouldn't have to spend time and money as often upgrading the CPU.
Danvelopment - Monday, February 8, 2016 - link
The way I see it, AMD needs to stop comparing themselves with themselves and needs to compare themselves with the competition. People don't understand the improvements if they aren't involved with the predecessor.They produce a reasonable product that performs at 60-80% of the competition at 50% of the price.
Good designs are produced for the competition, that could fundamentally have their parts, and they're losing on the design front.
And strangely, for similar products the AMD machines are the same cost, even though the difference is the chip (at halfish the price).
Can they not work to develop an easier transition method for OEM's to produce this-or-that designs that allow end users to pick AMD or Intel during the selection process. Tier them like Dell does for the various Intel processors but have them consistently show up as the cheapest option $100 off a $500 laptop is a decent drop and if the chip and PCB is $150 cheaper to produce the OEM still wins).
Differentiating the product creates too many variables people don't understand, and creates the issue above, CPU brand aversion on entire product stacks with no common ground.
I'd say take a long, hard look at current machines, and develop a method of getting their chips into them as an option, without OEMs designing a product from the ground up.
I'd certainly consider AMD if I could just select it as an option that knocks $100 off on the low cost tier laptop in my workplace.