PCI Express 3.0: More Bandwidth For Compute

It may seem like it’s still fairly new, but PCI Express 2 is actually a relatively old addition to motherboards and video cards. AMD first added support for it with the Radeon HD 3870 back in 2008 so it’s been nearly 4 years since video cards made the jump. At the same time PCI Express 3.0 has been in the works for some time now and although it hasn’t been 4 years it feels like it has been much longer. PCIe 3.0 motherboards only finally became available last month with the launch of the Sandy Bridge-E platform and now the first PCIe 3.0 video cards are becoming available with Tahiti.

But at first glance it may not seem like PCIe 3.0 is all that important. Additional PCIe bandwidth has proven to be generally unnecessary when it comes to gaming, as single-GPU cards typically only benefit by a couple percent (if at all) when moving from PCIe 2.1 x8 to x16. There will of course come a time where games need more PCIe bandwidth, but right now PCIe 2.1 x16 (8GB/sec) handles the task with room to spare.

So why is PCIe 3.0 important then? It’s not the games, it’s the computing. GPUs have a great deal of internal memory bandwidth (264GB/sec; more with cache) but shuffling data between the GPU and the CPU is a high latency, heavily bottlenecked process that tops out at 8GB/sec under PCIe 2.1. And since GPUs are still specialized devices that excel at parallel code execution, a lot of workloads exist that will need to constantly move data between the GPU and the CPU to maximize parallel and serial code execution. As it stands today GPUs are really only best suited for workloads that involve sending work to the GPU and keeping it there; heterogeneous computing is a luxury there isn’t bandwidth for.

The long term solution of course is to bring the CPU and the GPU together, which is what Fusion does. CPU/GPU bandwidth just in Llano is over 20GB/sec, and latency is greatly reduced due to the CPU and GPU being on the same die. But this doesn’t preclude the fact that AMD also wants to bring some of these same benefits to discrete GPUs, which is where PCI e 3.0 comes in.

With PCIe 3.0 transport bandwidth is again being doubled, from 500MB/sec per lane bidirectional to 1GB/sec per lane bidirectional, which for an x16 device means doubling the available bandwidth from 8GB/sec to 16GB/sec. This is accomplished by increasing the frequency of the underlying bus itself from 5 GT/sec to 8 GT/sec, while decreasing overhead from 20% (8b/10b encoding) to 1% through the use of a highly efficient 128b/130b encoding scheme. Meanwhile latency doesn’t change – it’s largely a product of physics and physical distances – but merely doubling the bandwidth can greatly improve performance for bandwidth-hungry compute applications.

As with any other specialized change like this the benefit is going to heavily depend on the application being used, however AMD is confident that there are applications that will completely saturate PCIe 3.0 (and thensome), and it’s easy to imagine why.

Even among our limited selection compute benchmarks we found something that directly benefitted from PCIe 3.0. AESEncryptDecrypt, a sample application from AMD’s APP SDK, demonstrates AES encryption performance by running it on square image files.  Throwing it a large 8K x 8K image not only creates a lot of work for the GPU, but a lot of PCIe traffic too. In our case simply enabling PCIe 3.0 improved performance by 9%, from 324ms down to 297ms.

Ultimately having more bandwidth is not only going to improve compute performance for AMD, but will give the company a critical edge over NVIDIA for the time being. Kepler will no doubt ship with PCIe 3.0, but that’s months down the line. In the meantime users and organizations with high bandwidth compute workloads have Tahiti.

Video & Movies: The Video Codec Engine, UVD3, & Steady Video 2.0 Managing Idle Power: Introducing ZeroCore Power
Comments Locked

292 Comments

View All Comments

  • Zingam - Thursday, December 22, 2011 - link

    I think this card is a kinda fail. Well, maybe it is a driver issue and they'll up the performance 20-25% in the future but it is still not fast enough for such huge jump - 2 nodes down!!!
    It smell like a graphics Bulldozer for AMD. Good ideas on paper but in practice something doesn't work quite right. Raw performance is all that counts (of course raw performance/$).
    If NVIDIA does better than usual this time. AMD might be in trouble. Well, will wait and see.
    Hopefully they'll be able to release improved CPUs and GPUs soon because this generation does not seem to be very impressive.

    I've expected at least triple performance over the previous generation. Maybe the drivers are not that well optimized yet. After all it is a huge architecture change.

    I don't really care that much about that GPU generation but I'm worried that they won't be able to put something impressively new in the next generation of consoles. I really hope that we are not stuck with obsolete CPU/GPU combination for the next 7-8 years again.

    Anyway: massively parallel computing sounds tasty!
  • B3an - Thursday, December 22, 2011 - link

    You dont seem to understand that all them extra transistors are mostly there for computing. Thats mostly what this was designed for. Not specifically for gaming performance. Computing is where this card will offer massive increases over the previous AMD generation.
    Look at Nvidia's Fermi, that had way more transistors than the previous generation but wasn't that much faster than AMD's cards at the time. Because again all the extra transistors were mainly for computing.

    And come on LOL, expecting over triple the performance?? That has never happened once with any GPU release.
  • SlyNine - Friday, December 23, 2011 - link

    The 9700pro was up to 4x faster then the 4600 in certian situations. So yes it has happened.
  • tzhu07 - Thursday, December 22, 2011 - link

    LOL, triple the performance?

    Do you also have a standard of dating only Victoria's Secret models?
  • eanazag - Thursday, December 22, 2011 - link

    I have a 3870 which I got in early 2007. It still does well for the main games I play: Dawn of War 2 and Starcraft 2 (25 fps has been fine for me here with settings mostly maxed). I have eyeing a new card. I like the power usage and thermals here. I am not spending $500+ though. I am thinking they are using that price to compensate for the mediocre yields they getting on 28nm, but either way the numbers look justified. I will be look for the best card between $150-$250, maybe $300. I am counting on this cards price coming down, but I doubt it will hit under $400-350 next year.

    No matter what this looks like a successful soft launch of a video card. For me, anything smokes what I have in performance but not so much on power usage. I'd really not mind the extra noise as the heat is better than my 3870.

    I'm in the single card strategy camp.

    Monitor is a single 42" 1920x1200 60 Hz.
    Intel Core i5 760 at stock clocks. My first Intel since the P3 days.

    Great article.
  • Death666Angel - Thursday, December 22, 2011 - link

    Can someone explain the different heights in the die-size comparison picture? Does that reflect processing-changes? I'm lost. :D Otherwise, good review. I don't see the HD7970 in Bench, am I blind or is it just missing.
  • Ryan Smith - Thursday, December 22, 2011 - link

    The Y axis is the die size. The higher a GPU the bigger it is (relative to the other GPUs from that company).
  • Death666Angel - Friday, December 23, 2011 - link

    Thanks! I thought the actual sizes were the sizes and the y-axis meant something else. Makes sense though how you did it! :-)
  • MonkeyPaw - Thursday, December 22, 2011 - link

    As a former owner of the 3870, mine had the short-lived GDDR4. That old card has a place in my nerd heart, as it played Bioshock wonderfully.
  • Peichen - Thursday, December 22, 2011 - link

    The improvement is simply not as impressive as I was led to believed. Rumor has it that a single 7970 would have the power of a 6990. In fact, if you crunch the numbers, it would be at least 50% faster than 6970 which should put it close to 6990. (63.25% increase in transistors, 40.37% in TFLOP and 50% increase in memory bandwidth.)

    What we got is a Fermi 1st gen with the price to match. Remember, this is not a half-node improvement in manufacturing process, it is a full-node and we waited two years for this.

    In any case, I am just ranting because I am waiting for something to replace my current card before GTA 5 came out. Nvidia's GK104 in Q1 2012 should be interesting. Rumored to be slightly faster than GTX 580 (slower than 7970) but much cheaper. We'll see.

Log in

Don't have an account? Sign up now