The PCIe Layout

Ask anyone at Apple why they need Ivy Bridge EP vs. a conventional desktop Haswell for the Mac Pro and you’ll get two responses: core count and PCIe lanes. The first one is obvious. Haswell tops out at 4 cores today. Even though each of those cores is faster than what you get with an Ivy Bridge EP, for applications that can spawn more than 4 CPU intensive threads you’re better off taking the IPC/single threaded hit and going with an older architecture that supports more cores. The second point is a connectivity argument.

Here’s what a conventional desktop Haswell platform looks like in terms of PCIe lanes:

You’ve got a total of 16 PCIe 3.0 lanes that branch off the CPU, and then (at most) another 8 PCIe 2.0 lanes hanging off of the Platform Controller Hub (PCH). In a dual-GPU configuration those 16 PCIe 3.0 lanes are typically divided into an 8 + 8 configuration. The 8 remaining lanes are typically more than enough for networking and extra storage controllers.

Ivy Bridge E/EP on the other hand doubles the total number of PCIe lanes compared to Intel’s standard desktop platform:

Here the CPU has a total of 40 PCIe 3.0 lanes. That’s enough for each GPU in a dual-GPU setup to get a full 16 lanes, and to have another 8 left over for high-bandwidth use. The PCH also has another 8 PCIe 2.0 lanes, just like in the conventional desktop case.

I wanted to figure out how these PCIe lanes were used by the Mac Pro, so I set out to map everything out as best as I could without taking apart the system (alas, Apple tends to frown upon that sort of behavior when it comes to review samples). Here’s what I was able to come up with. Let’s start off of the PCH:

Here each Gigabit Ethernet port gets a dedicated PCIe 2.0 x1 lane, the same goes for the 802.11ac controller. All Mac Pros ship with a PCIe x4 SSD, and those four lanes also come off the PCH. That leaves a single PCIe lane unaccounted for in the Mac Pro. Here we really get to see how much of a mess Intel’s workstation chipset lineup is: the C600/X79 PCH doesn’t natively support USB 3.0. That’s right, it’s nearly 2014 and Intel is shipping a flagship platform without USB 3.0 support. The 8th PCIe lane off of the PCH is used by a Fresco Logic USB 3.0 controller. I believe it’s the FL1100, which is a PCIe 2.0 to 4-port USB 3.0 controller. A single PCIe 2.0 lane offers a maximum of 500MB/s of bandwidth in either direction (1GB/s aggregate), which is enough for the real world max transfer rates over USB 3.0. Do keep this limitation in mind if you’re thinking about populating all four USB 3.0 ports with high-speed storage with the intent of building a low-cost Thunderbolt alternative. You’ll be bound by the performance of a single PCIe 2.0 lane.

That takes care of the PCH, now let’s see what happens off of the CPU:

Of the 40 PCIe 3.0 lanes, 32 are already occupied by the two AMD FirePro GPUs. Having a full x16 interface to the GPUs isn’t really necessary for gaming performance, but if you want to treat each GPU as a first class citizen then this is the way to go. That leaves us with 8 PCIe 3.0 lanes left.

The Mac Pro has a total of six Thunderbolt 2 ports, each pair is driven by a single Thunderbolt 2 controller. Each Thunderbolt 2 controller accepts four PCIe 2.0 lanes as an input and delivers that bandwidth to any Thunderbolt devices downstream. If you do the math you’ll see we have a bit of a problem: 3 TB2 controllers x 4 PCIe 2.0 lanes per controller = 12 PCIe 2.0 lanes, but we only have 8 lanes left to allocate in the system.

I assumed there had to be a PCIe switch sharing the 8 PCIe input lanes among the Thunderbolt 2 controllers, but I needed proof. Our Senior GPU Editor, Ryan Smith, did some digging into the Mac Pro’s enumerated PCIe devices and discovered a very familiar vendor id: 10B5, the id used by PLX Technology. PLX is a well known PCIe bridge/switch manufacturer. The part used in the Mac Pro (PEX 8723) is of course not listed on PLX’s website, but it’s pretty close to another one that PLX is presently shipping: the PEX 8724. The 8724 is a 24-lane PCIe 3.0 switch. It can take 4 or 8 PCIe 3.0 lanes as an input and share that bandwidth among up to 16 (20 in the case of a x4 input) downstream PCIe lanes. Normally that would create a bandwidth bottleneck but remember that Thunderbolt 2 is still based on PCIe 2.0. The switch provides roughly 15GB/s of bandwidth to the CPU and 3 x 5GB/s of bandwidth to the Thunderbolt 2 controllers.

Literally any of the 6 Thunderbolt 2 ports on the back of the Mac Pro will give you access to the 8 remaining PCIe 3.0 lanes living off of the CPU. It’s pretty impressive when you think about it, external access to a high-speed interface located on the CPU die itself.

The part I haven’t quite figured out yet is how Apple handles DisplayPort functionality. All six Thunderbolt 2 ports are capable of outputting to a display, which means that there’s either a path from the FirePro to each Thunderbolt 2 controller or the PEX 8723 switch also handles DisplayPort switching. It doesn’t really matter from an end user perspective as you can plug a monitor into any port and have it work, it’s more of me wanting to know how it all works.

Mac Pro vs. Consumer Macs GPU Choices
POST A COMMENT

269 Comments

View All Comments

  • Anand Lal Shimpi - Tuesday, December 31, 2013 - link

    Behind the multithreaded curve, the two are almost identical :) Reply
  • Calista - Tuesday, December 31, 2013 - link

    It seem to follow the multithreaded graph perfectly, a tiny bit of a blue graph can be seen in the upper right corner. So it's actually hidden by the second graph. Reply
  • japtor - Tuesday, December 31, 2013 - link

    One thing I keep forgetting to ask since it hasn't been mentioned anywhere, does AirPlay display spanning/mirroring work? I figured it used QSV on the other Macs which this machine doesn't have, so just curious if they just left it out completely. Reply
  • skiboysteve - Tuesday, December 31, 2013 - link

    I wonder if they changed it to AMD hardware encode... Reply
  • Calista - Tuesday, December 31, 2013 - link

    So not really a proper upgrade for anyone owning a Mac Pro from the last few years unless Thunderbolt, faster GPU or a small form factor is needed.

    Anyway, it's an impressive package and it's clear Apple have brought with them a lot of the knowledge they have gained over the years building laptops.
    Reply
  • Lonyo - Tuesday, December 31, 2013 - link

    Indeed. As was the question with the "super thin" iMac... what's the point?

    It's all very well having a super small computer blah blah, but in this instance, for this type of machine and end user, what's the real benefit?
    The cost has gone up for a base model, performance per dollar has gone down compared to the previous one, there's no ability to upgrade GPUs.
    As soon as you start plugging in Thunderbolt devices, there goes your "sleek looks" etc. Plus it's more expensive to get a Thunderbolt HDD/etc than just stick one inside the case, further increasing costs.

    Yes, it looks nice, and from an engineering standpoint it's very well done, but... is it really the right product for the market?
    Reply
  • Calista - Wednesday, January 01, 2014 - link

    For anyone not planning on bringing the computer with them from time to time it's certainly not a very practical design. Desktop-space is often more highly valued than floorspace, and lack of upgrade paths are obviously a con.

    But for those with a need to bring a powerful computer with them on a set or similar it's a much more practical solution as compared to the previous design. I think Apple was quite aware what they were doing. A complete field setup with a 27" monitor, the Mac Pro, cables, keyboard and mouse is less than 20 kg. Much more than a laptop for sure, but still a fairly acceptable weight.
    Reply
  • Lonyo - Wednesday, January 01, 2014 - link

    Wouldn't it make more sense to have TWO designs then? A Mac Pro for people who need portability, and a Mac Pro for standard single location users...?

    I mean, I know Apple tends to be all about deciding what the consumer wants for them and removing choice as much as possible, but sometimes that's not the best way.
    Reply
  • akdj - Wednesday, January 01, 2014 - link

    Why doesn't this model fit that mold? For the stay at home/office/studio...one can easily AND reasonably tie thunderbolt storage together in a very acceptable and aesthetic way. Whether it be a drive enclosure...set of enclosures, TB docks that are now available adding more USB 3/HDMI/audio/et al I/O....who needs a huge box for slow internal 3.5" HDDs anymore? These PCIe SSDs tear the 2.5" models apart. Inside the 'old' style MP, a 'new' GPU on X16 takes up two slots! Sure doesn't leave much room for your MIDI, PCIe SSD or external pro sound card!
    I'm amazed at how few 'get it' here anymore. Especially after such an exhaustive review. I'm a bit biased as I make my mortgage and have for 22 years doing audio and video production. From hauling reel to reels, vinyl, film and racks of around and lighting gear to rMBPs, iPads (now with fill 64 channel wifi front of house control with Mackie) and this new Pro....I've shaved thousands of pounds from load ins and outs. Same in the camera realm. Working the last seven years with Discovery and it's subsidiaries in Alaska, I can't put into words what this machine means to us. And it's ability to pay itself off many times over just in the course of a year. Exciting times. Hopefully an evolution Win OEMs will consider as well. Shouldn't be any moving parts any longer. Wait and space are ALWAYS an issue. As is the price of power....the advantage of speed, and software developers following suit to unload computational crunching to the GPU
    Reply
  • nunomoreira10 - Tuesday, December 31, 2013 - link

    Now there just need to be a dual cpu single gpu offer to please everybody. Reply

Log in

Don't have an account? Sign up now