The Mac Pro Review (Late 2013)
by Anand Lal Shimpi on December 31, 2013 3:18 PM ESTThe PCIe Layout
Ask anyone at Apple why they need Ivy Bridge EP vs. a conventional desktop Haswell for the Mac Pro and you’ll get two responses: core count and PCIe lanes. The first one is obvious. Haswell tops out at 4 cores today. Even though each of those cores is faster than what you get with an Ivy Bridge EP, for applications that can spawn more than 4 CPU intensive threads you’re better off taking the IPC/single threaded hit and going with an older architecture that supports more cores. The second point is a connectivity argument.
Here’s what a conventional desktop Haswell platform looks like in terms of PCIe lanes:
You’ve got a total of 16 PCIe 3.0 lanes that branch off the CPU, and then (at most) another 8 PCIe 2.0 lanes hanging off of the Platform Controller Hub (PCH). In a dual-GPU configuration those 16 PCIe 3.0 lanes are typically divided into an 8 + 8 configuration. The 8 remaining lanes are typically more than enough for networking and extra storage controllers.
Ivy Bridge E/EP on the other hand doubles the total number of PCIe lanes compared to Intel’s standard desktop platform:
Here the CPU has a total of 40 PCIe 3.0 lanes. That’s enough for each GPU in a dual-GPU setup to get a full 16 lanes, and to have another 8 left over for high-bandwidth use. The PCH also has another 8 PCIe 2.0 lanes, just like in the conventional desktop case.
I wanted to figure out how these PCIe lanes were used by the Mac Pro, so I set out to map everything out as best as I could without taking apart the system (alas, Apple tends to frown upon that sort of behavior when it comes to review samples). Here’s what I was able to come up with. Let’s start off of the PCH:
Here each Gigabit Ethernet port gets a dedicated PCIe 2.0 x1 lane, the same goes for the 802.11ac controller. All Mac Pros ship with a PCIe x4 SSD, and those four lanes also come off the PCH. That leaves a single PCIe lane unaccounted for in the Mac Pro. Here we really get to see how much of a mess Intel’s workstation chipset lineup is: the C600/X79 PCH doesn’t natively support USB 3.0. That’s right, it’s nearly 2014 and Intel is shipping a flagship platform without USB 3.0 support. The 8th PCIe lane off of the PCH is used by a Fresco Logic USB 3.0 controller. I believe it’s the FL1100, which is a PCIe 2.0 to 4-port USB 3.0 controller. A single PCIe 2.0 lane offers a maximum of 500MB/s of bandwidth in either direction (1GB/s aggregate), which is enough for the real world max transfer rates over USB 3.0. Do keep this limitation in mind if you’re thinking about populating all four USB 3.0 ports with high-speed storage with the intent of building a low-cost Thunderbolt alternative. You’ll be bound by the performance of a single PCIe 2.0 lane.
That takes care of the PCH, now let’s see what happens off of the CPU:
Of the 40 PCIe 3.0 lanes, 32 are already occupied by the two AMD FirePro GPUs. Having a full x16 interface to the GPUs isn’t really necessary for gaming performance, but if you want to treat each GPU as a first class citizen then this is the way to go. That leaves us with 8 PCIe 3.0 lanes left.
The Mac Pro has a total of six Thunderbolt 2 ports, each pair is driven by a single Thunderbolt 2 controller. Each Thunderbolt 2 controller accepts four PCIe 2.0 lanes as an input and delivers that bandwidth to any Thunderbolt devices downstream. If you do the math you’ll see we have a bit of a problem: 3 TB2 controllers x 4 PCIe 2.0 lanes per controller = 12 PCIe 2.0 lanes, but we only have 8 lanes left to allocate in the system.
I assumed there had to be a PCIe switch sharing the 8 PCIe input lanes among the Thunderbolt 2 controllers, but I needed proof. Our Senior GPU Editor, Ryan Smith, did some digging into the Mac Pro’s enumerated PCIe devices and discovered a very familiar vendor id: 10B5, the id used by PLX Technology. PLX is a well known PCIe bridge/switch manufacturer. The part used in the Mac Pro (PEX 8723) is of course not listed on PLX’s website, but it’s pretty close to another one that PLX is presently shipping: the PEX 8724. The 8724 is a 24-lane PCIe 3.0 switch. It can take 4 or 8 PCIe 3.0 lanes as an input and share that bandwidth among up to 16 (20 in the case of a x4 input) downstream PCIe lanes. Normally that would create a bandwidth bottleneck but remember that Thunderbolt 2 is still based on PCIe 2.0. The switch provides roughly 15GB/s of bandwidth to the CPU and 3 x 5GB/s of bandwidth to the Thunderbolt 2 controllers.
Literally any of the 6 Thunderbolt 2 ports on the back of the Mac Pro will give you access to the 8 remaining PCIe 3.0 lanes living off of the CPU. It’s pretty impressive when you think about it, external access to a high-speed interface located on the CPU die itself.
The part I haven’t quite figured out yet is how Apple handles DisplayPort functionality. All six Thunderbolt 2 ports are capable of outputting to a display, which means that there’s either a path from the FirePro to each Thunderbolt 2 controller or the PEX 8723 switch also handles DisplayPort switching. It doesn’t really matter from an end user perspective as you can plug a monitor into any port and have it work, it’s more of me wanting to know how it all works.
267 Comments
View All Comments
Chirpie - Thursday, January 2, 2014 - link
Uh, remind me again which low-power HTPC can run 16 4K video streams at once? Beyond that, why on earth would you buy this computer as a HTPC? The graphic cards would be a waste. If you're gonna bash, I demand some effort.Liquidmark - Friday, January 3, 2014 - link
I don't see you admitting you have a problem with haterism.Wolfpup - Thursday, January 2, 2014 - link
The problem with saying Apple's pricing isn't out of touch with reality is that you can't only compare this to high end workstation's from other companies...this is Apple's only desktop-ish device. These things 10 years ago used to start in the mid $1000-2000 range, and with inflation that would be cheaper still. They were STILL expensive, but at least not absurdly so.Yeah, Xeons, etc. cost a lot, but Apple doesn't provide options for people who want a high end notebook or desktop for normal use...this is the closest they get, and it's at least 2x as expensive as it should be for it's base unit (even with the Apple tax).
OreoCookie - Thursday, January 2, 2014 - link
I don't understand this comment: Apple does cover this *price range* with the Mac mini and the iMac. The 27" iMac sports up to 32 GB RAM, a decent graphics card and 4 fast cores. And since these machines come with Thunderbolt, you can expand them with the same ultrafast peripherals that also attach to a Mac Pro. The only thing that Apple does not offer to you is the product that you want for the price that you want (the xMac, a traditional tower system).lilo777 - Thursday, January 2, 2014 - link
iMac is not a classic desktop. It's a A-I-O computer with its inherent disadvantages (i.e. CPUs,, GPUs etc. usually getting obsolete much faster than the monitor)OreoCookie - Thursday, January 2, 2014 - link
I understand what the iMac is. It is nevertheless a desktop computer that covers the price segment between $1200 and $3500 in Apple's line-up. Compared to 10, 15 years ago, the demographics have changed: people have migrated to mobile computers for the most part, and the demographic who still use desktops are often quite happy using iMacs (e. g. have a look at The Verge's review of the Mac Pro where the video editors admit to using iMacs and Mac Pros, for instance).Certainly, if you want or need a traditional headless computer, Apple simply does not serve your needs. But looking ahead, Broadwell CPUs will be soldered to the mainboards. Most people will rely on the integrated graphics (which become increasingly powerful).
Regular Reader - Friday, January 3, 2014 - link
How often do you replace a CPU or graphics card? If you're a serious gamer, then Macs have never been the right machine for you and never will be. For people like me, the 27" iMac is perfect because we don't need a classic desktop. There's little reason to need a true desktop machine these days. AIO is the way to go. So much easier, you can get most of the power, they're quieter...the advantages far outweigh the negatives.wallysb01 - Friday, January 3, 2014 - link
The iMac is not quieter than decent desktop PC. Maybe you’ve just been around absurdly loud computers? For the $1500-$2000 you pay for an iMac, you should be able to buy a pretty much silent PC with as much or more power than the iMac. Oh, and you get your choice of monitor or you can keep your old one that you still like just fine.Chirpie - Friday, January 3, 2014 - link
I dunno man, usually the graphic card alone is enough to make it louder than an iMac. At least, until you're willing to start mucking with the RPMs through various mods/software hacks.Regular Reader - Friday, January 3, 2014 - link
27" iMac. There's no need for much more. You can upgrade everything but the CPU.I used to be the DIY PC build type. I got sick of wires everywhere, intermittent cooling issues, and just generally having a desktop full of crap. I've had a 4-core i7 27" iMac for nearly 4 years now, haven't looked back, and it is more than enough to run OSX and Windows in parallel, even only having 8 GB of PC1333 RAM. And with a firmware update, my old 27" can support up to 32 GB. I have Thunderbolt even, along with FW800. If you need external SATA, OWC makes a component to do that (though you have to send your machine to them to get it installed). I'd happily buy another if I needed to do even more serious work than I do.