The PCIe Layout

Ask anyone at Apple why they need Ivy Bridge EP vs. a conventional desktop Haswell for the Mac Pro and you’ll get two responses: core count and PCIe lanes. The first one is obvious. Haswell tops out at 4 cores today. Even though each of those cores is faster than what you get with an Ivy Bridge EP, for applications that can spawn more than 4 CPU intensive threads you’re better off taking the IPC/single threaded hit and going with an older architecture that supports more cores. The second point is a connectivity argument.

Here’s what a conventional desktop Haswell platform looks like in terms of PCIe lanes:

You’ve got a total of 16 PCIe 3.0 lanes that branch off the CPU, and then (at most) another 8 PCIe 2.0 lanes hanging off of the Platform Controller Hub (PCH). In a dual-GPU configuration those 16 PCIe 3.0 lanes are typically divided into an 8 + 8 configuration. The 8 remaining lanes are typically more than enough for networking and extra storage controllers.

Ivy Bridge E/EP on the other hand doubles the total number of PCIe lanes compared to Intel’s standard desktop platform:

Here the CPU has a total of 40 PCIe 3.0 lanes. That’s enough for each GPU in a dual-GPU setup to get a full 16 lanes, and to have another 8 left over for high-bandwidth use. The PCH also has another 8 PCIe 2.0 lanes, just like in the conventional desktop case.

I wanted to figure out how these PCIe lanes were used by the Mac Pro, so I set out to map everything out as best as I could without taking apart the system (alas, Apple tends to frown upon that sort of behavior when it comes to review samples). Here’s what I was able to come up with. Let’s start off of the PCH:

Here each Gigabit Ethernet port gets a dedicated PCIe 2.0 x1 lane, the same goes for the 802.11ac controller. All Mac Pros ship with a PCIe x4 SSD, and those four lanes also come off the PCH. That leaves a single PCIe lane unaccounted for in the Mac Pro. Here we really get to see how much of a mess Intel’s workstation chipset lineup is: the C600/X79 PCH doesn’t natively support USB 3.0. That’s right, it’s nearly 2014 and Intel is shipping a flagship platform without USB 3.0 support. The 8th PCIe lane off of the PCH is used by a Fresco Logic USB 3.0 controller. I believe it’s the FL1100, which is a PCIe 2.0 to 4-port USB 3.0 controller. A single PCIe 2.0 lane offers a maximum of 500MB/s of bandwidth in either direction (1GB/s aggregate), which is enough for the real world max transfer rates over USB 3.0. Do keep this limitation in mind if you’re thinking about populating all four USB 3.0 ports with high-speed storage with the intent of building a low-cost Thunderbolt alternative. You’ll be bound by the performance of a single PCIe 2.0 lane.

That takes care of the PCH, now let’s see what happens off of the CPU:

Of the 40 PCIe 3.0 lanes, 32 are already occupied by the two AMD FirePro GPUs. Having a full x16 interface to the GPUs isn’t really necessary for gaming performance, but if you want to treat each GPU as a first class citizen then this is the way to go. That leaves us with 8 PCIe 3.0 lanes left.

The Mac Pro has a total of six Thunderbolt 2 ports, each pair is driven by a single Thunderbolt 2 controller. Each Thunderbolt 2 controller accepts four PCIe 2.0 lanes as an input and delivers that bandwidth to any Thunderbolt devices downstream. If you do the math you’ll see we have a bit of a problem: 3 TB2 controllers x 4 PCIe 2.0 lanes per controller = 12 PCIe 2.0 lanes, but we only have 8 lanes left to allocate in the system.

I assumed there had to be a PCIe switch sharing the 8 PCIe input lanes among the Thunderbolt 2 controllers, but I needed proof. Our Senior GPU Editor, Ryan Smith, did some digging into the Mac Pro’s enumerated PCIe devices and discovered a very familiar vendor id: 10B5, the id used by PLX Technology. PLX is a well known PCIe bridge/switch manufacturer. The part used in the Mac Pro (PEX 8723) is of course not listed on PLX’s website, but it’s pretty close to another one that PLX is presently shipping: the PEX 8724. The 8724 is a 24-lane PCIe 3.0 switch. It can take 4 or 8 PCIe 3.0 lanes as an input and share that bandwidth among up to 16 (20 in the case of a x4 input) downstream PCIe lanes. Normally that would create a bandwidth bottleneck but remember that Thunderbolt 2 is still based on PCIe 2.0. The switch provides roughly 15GB/s of bandwidth to the CPU and 3 x 5GB/s of bandwidth to the Thunderbolt 2 controllers.

Literally any of the 6 Thunderbolt 2 ports on the back of the Mac Pro will give you access to the 8 remaining PCIe 3.0 lanes living off of the CPU. It’s pretty impressive when you think about it, external access to a high-speed interface located on the CPU die itself.

The part I haven’t quite figured out yet is how Apple handles DisplayPort functionality. All six Thunderbolt 2 ports are capable of outputting to a display, which means that there’s either a path from the FirePro to each Thunderbolt 2 controller or the PEX 8723 switch also handles DisplayPort switching. It doesn’t really matter from an end user perspective as you can plug a monitor into any port and have it work, it’s more of me wanting to know how it all works.

Mac Pro vs. Consumer Macs GPU Choices
Comments Locked

267 Comments

View All Comments

  • estern53 - Monday, January 6, 2014 - link

    Now all we need Apple to do is make a prosumer version of the Mac Pro for under 2k for the rest of us.
  • wheelhot - Monday, January 6, 2014 - link

    They already did, it's called the iMac, or MacMini
  • tipoo - Monday, January 6, 2014 - link

    A Mini redesigned as a mini version of this might be cool, especially with a discreet GPU.
  • affinityseattle - Wednesday, January 8, 2014 - link

    The LR test is a bit off. LR is not great at exporting. As a pro, I've found the trick is to stack export processes. The more cores you have, the more it can handle and utilize the CPU. So, the iMac i5 might be faster on a single export, but the Mac Pro should spank it if you start dividing the export up. Also, if you have a 1000-image export, the imac and mbp will overheat and reduce the CPU power (TLD). For a pro machine, these types of usage are relevant.
  • GRAFiZ - Wednesday, January 8, 2014 - link

    It's an impressive product... but, as with most Apple designs, form is first, function is second. The fact that it has to scale CPU speed to reduce heat says all I need to know. Obviously the Apple fanbois will argue "THATS A THEORETICAL SITUATION!!!" but who cares? The fact is I'm buying brand new parts at the highest premium on the market... thermal throttling should NEVER EVER be necessary.

    Bottom line, like all Apple products, it's impressive... but, you can do better for less money elsewhere.
  • DotFab - Wednesday, January 8, 2014 - link

    You've read it all wrong!

    The work load that put the MP under slower run was purely artificial.
    It's not anything actual programs run.
    The conclusion to draw is that the global thermal dissipation is great!

    You missed the point of the test.
  • lukarak - Thursday, January 9, 2014 - link

    You clearly don't care, but people who will actually use it, will care that it won't throttle down when they load it.
    Only haters care about a situation that will never take place. As they can, in their frustration and insignificance, gloat about something.
  • GRAFiZ - Thursday, January 9, 2014 - link

    No hater here... I just think it's poor design that a brand new product, costing as much as TEN THOUSAND DOLLARS can not handle a theoretical max load without overheating.

    I can build a dual octa-core Xeon E5v2 system for far less that could process the same simulated work load without any thermal of processor speed restrictions at all.

    But, I guess if you want the smallest little desktop made, it's really your only choice. I just find it funny that when you spend as much as a brand new car might cost on a desktop computer, that any such limitations would need to be accepted at all.
  • wordsofpeace - Friday, January 10, 2014 - link

    If Apple had made it 10mm wider and maybe 20mm taller, the extra thermal capacity could have allowed more headroom. But no, it had to be 9.9" x 6.6" and 11lbs. It's almost as if the marketing dept. decided on the most wow factor specs and poor old engineering had to come up with a solution.
    Don't get me wrong, I'd love one on my desk, but I too don't understand Apple's addiction to form over function.
  • tsk2 - Thursday, February 20, 2014 - link

    I share your view. I enjoy my mac pro 2008 (packed with all the stuff I need) and a nice cinema display. Sure, both are big, but they both look nice and I don't feel limited. I have tried small nice looking boxes in the past and my experience has always been that it is a lot of effort to expand, add cables, and still get that uncluttered feel. A bigger box, I can live with, but this solution, albeit "initially" good looking, is too short term. I wish Apple would notice that there are users who fall in our category..

Log in

Don't have an account? Sign up now