The Mac Pro Review (Late 2013)by Anand Lal Shimpi on December 31, 2013 3:18 PM EST
The PCIe Layout
Ask anyone at Apple why they need Ivy Bridge EP vs. a conventional desktop Haswell for the Mac Pro and you’ll get two responses: core count and PCIe lanes. The first one is obvious. Haswell tops out at 4 cores today. Even though each of those cores is faster than what you get with an Ivy Bridge EP, for applications that can spawn more than 4 CPU intensive threads you’re better off taking the IPC/single threaded hit and going with an older architecture that supports more cores. The second point is a connectivity argument.
Here’s what a conventional desktop Haswell platform looks like in terms of PCIe lanes:
You’ve got a total of 16 PCIe 3.0 lanes that branch off the CPU, and then (at most) another 8 PCIe 2.0 lanes hanging off of the Platform Controller Hub (PCH). In a dual-GPU configuration those 16 PCIe 3.0 lanes are typically divided into an 8 + 8 configuration. The 8 remaining lanes are typically more than enough for networking and extra storage controllers.
Ivy Bridge E/EP on the other hand doubles the total number of PCIe lanes compared to Intel’s standard desktop platform:
Here the CPU has a total of 40 PCIe 3.0 lanes. That’s enough for each GPU in a dual-GPU setup to get a full 16 lanes, and to have another 8 left over for high-bandwidth use. The PCH also has another 8 PCIe 2.0 lanes, just like in the conventional desktop case.
I wanted to figure out how these PCIe lanes were used by the Mac Pro, so I set out to map everything out as best as I could without taking apart the system (alas, Apple tends to frown upon that sort of behavior when it comes to review samples). Here’s what I was able to come up with. Let’s start off of the PCH:
Here each Gigabit Ethernet port gets a dedicated PCIe 2.0 x1 lane, the same goes for the 802.11ac controller. All Mac Pros ship with a PCIe x4 SSD, and those four lanes also come off the PCH. That leaves a single PCIe lane unaccounted for in the Mac Pro. Here we really get to see how much of a mess Intel’s workstation chipset lineup is: the C600/X79 PCH doesn’t natively support USB 3.0. That’s right, it’s nearly 2014 and Intel is shipping a flagship platform without USB 3.0 support. The 8th PCIe lane off of the PCH is used by a Fresco Logic USB 3.0 controller. I believe it’s the FL1100, which is a PCIe 2.0 to 4-port USB 3.0 controller. A single PCIe 2.0 lane offers a maximum of 500MB/s of bandwidth in either direction (1GB/s aggregate), which is enough for the real world max transfer rates over USB 3.0. Do keep this limitation in mind if you’re thinking about populating all four USB 3.0 ports with high-speed storage with the intent of building a low-cost Thunderbolt alternative. You’ll be bound by the performance of a single PCIe 2.0 lane.
That takes care of the PCH, now let’s see what happens off of the CPU:
Of the 40 PCIe 3.0 lanes, 32 are already occupied by the two AMD FirePro GPUs. Having a full x16 interface to the GPUs isn’t really necessary for gaming performance, but if you want to treat each GPU as a first class citizen then this is the way to go. That leaves us with 8 PCIe 3.0 lanes left.
The Mac Pro has a total of six Thunderbolt 2 ports, each pair is driven by a single Thunderbolt 2 controller. Each Thunderbolt 2 controller accepts four PCIe 2.0 lanes as an input and delivers that bandwidth to any Thunderbolt devices downstream. If you do the math you’ll see we have a bit of a problem: 3 TB2 controllers x 4 PCIe 2.0 lanes per controller = 12 PCIe 2.0 lanes, but we only have 8 lanes left to allocate in the system.
I assumed there had to be a PCIe switch sharing the 8 PCIe input lanes among the Thunderbolt 2 controllers, but I needed proof. Our Senior GPU Editor, Ryan Smith, did some digging into the Mac Pro’s enumerated PCIe devices and discovered a very familiar vendor id: 10B5, the id used by PLX Technology. PLX is a well known PCIe bridge/switch manufacturer. The part used in the Mac Pro (PEX 8723) is of course not listed on PLX’s website, but it’s pretty close to another one that PLX is presently shipping: the PEX 8724. The 8724 is a 24-lane PCIe 3.0 switch. It can take 4 or 8 PCIe 3.0 lanes as an input and share that bandwidth among up to 16 (20 in the case of a x4 input) downstream PCIe lanes. Normally that would create a bandwidth bottleneck but remember that Thunderbolt 2 is still based on PCIe 2.0. The switch provides roughly 15GB/s of bandwidth to the CPU and 3 x 5GB/s of bandwidth to the Thunderbolt 2 controllers.
Literally any of the 6 Thunderbolt 2 ports on the back of the Mac Pro will give you access to the 8 remaining PCIe 3.0 lanes living off of the CPU. It’s pretty impressive when you think about it, external access to a high-speed interface located on the CPU die itself.
The part I haven’t quite figured out yet is how Apple handles DisplayPort functionality. All six Thunderbolt 2 ports are capable of outputting to a display, which means that there’s either a path from the FirePro to each Thunderbolt 2 controller or the PEX 8723 switch also handles DisplayPort switching. It doesn’t really matter from an end user perspective as you can plug a monitor into any port and have it work, it’s more of me wanting to know how it all works.
Post Your CommentPlease log in or sign up to comment.
View All Comments
LorneKwe - Friday, January 3, 2014 - linkHow about we replace those Quadro 6000's with Radeon 7970s which are what the Mac Pro has inside. You can make any price comparison look like shit when you erroneously drop a few thousand dollars worth of GPUs into the build.
Dual 6GB 7970s cost $1,400.
Using that, we can build out a 24C Dual Xeon 2697v2 workstation, with 64GB of RAM, dual 512GB 840 Pro SSDs in RAID0, H100i cooling on each CPU, and it would come to around $10,000. Same price as the Mac Pro's kitted out, 12C config, double the CPU power, and a good chunk more GPU power as they aren't underclocked.
You also have the option of going with 4x 7970s if you chose to, and if you forgo to ASUS Z9PE board for a proper workstation board, you can go up to 512GB of RAM instead of being capped at 64GB. If you want dual Quadro K5000s instead of 7970s, raise the PC's price by around $1600, but understand that these will outperform Apple's D700 pair by an enormous margin.
If you're looking for portability, or absolutely require OS X for what you do; choose the Mac Pro. If you're looking for computer power that won't throttle down when faced with tough workloads; have a full-fledged workstation built for you and get much more bang for your buck.
stingerman - Sunday, January 5, 2014 - linkSorry Dude, this is a Pro Workstation, that mean workstation class GPUs. 7970 is a great CPU but there is a reason it doesn't go into Workstations...
madwolfa - Sunday, January 5, 2014 - linkAnd what is it? Drivers? D700s in Mac Pro don't even have ECC memory enabled in them.
wheelhot - Monday, January 6, 2014 - linkyes, the drivers provided (and I believe you can only test this in Windows?) will determine if it's actual workstation class GPUs or just Radeons with the name FirePro slapped on it. I'm seriously hoping it's supplied with actual workstation GPU driver, as it'll greatly benefit the software I use.
LorneKwe - Tuesday, January 7, 2014 - linkI'd agree with you if we didn't have solid evidence that Apple's D700 is a repurposed 7970; lack of ECC memory being the best clue, and pricing being a great clue as well.
scarhead - Saturday, January 4, 2014 - linkOne Xeon E5-2697 processor costs $2,614. Two costs $5,228. I doubt your $3,712 figure for complete system is accurate.
p51mustang6 - Thursday, January 2, 2014 - linkApparently any comment disagreeing with the author gets deleted.
chaos215bar2 - Friday, January 3, 2014 - linkClearly not. Perhaps your comment was rude, trolling, and / or being argumentative without actually adding anything to the discussion?
Johan Niklasson - Friday, January 3, 2014 - linkGreat review - thanks. I just wanted to add that the new Mac Pro design and its finish does matter more than most people understand. The ones who work on graphic and video applications usually are artists with some sense of beauty and aesthetics. I am sure that the much more pleasant looking Mac Pro is a welcome addition to such people's workspaces.
Also the smaller form factor and the silence will make it hard to resist!
james-bond - Friday, January 3, 2014 - linkThanks for the great review. I wonder if some of the USB ports should have been on the other side of the chassis. Seems like the power cord and monitors are things that are rarely unplugged and are usually on the back side of current case designs. Having to reach around to plug and unplug a flash drive seems inconvenient.
Please keep and eye out for an upgrade to Logic Pro (Apple's audio pro app). A would love for you to benchmark this in the same way you did Final Cut. I doubt Apple will be able to harness the second GPU for compute in audio applications do to latency in doing real time audio monitoring. Audio has taken a back seat to video apps in this version of the Mac Pro it seems.