PCI Express 3.0: More Bandwidth For Compute

It may seem like it’s still fairly new, but PCI Express 2 is actually a relatively old addition to motherboards and video cards. AMD first added support for it with the Radeon HD 3870 back in 2008 so it’s been nearly 4 years since video cards made the jump. At the same time PCI Express 3.0 has been in the works for some time now and although it hasn’t been 4 years it feels like it has been much longer. PCIe 3.0 motherboards only finally became available last month with the launch of the Sandy Bridge-E platform and now the first PCIe 3.0 video cards are becoming available with Tahiti.

But at first glance it may not seem like PCIe 3.0 is all that important. Additional PCIe bandwidth has proven to be generally unnecessary when it comes to gaming, as single-GPU cards typically only benefit by a couple percent (if at all) when moving from PCIe 2.1 x8 to x16. There will of course come a time where games need more PCIe bandwidth, but right now PCIe 2.1 x16 (8GB/sec) handles the task with room to spare.

So why is PCIe 3.0 important then? It’s not the games, it’s the computing. GPUs have a great deal of internal memory bandwidth (264GB/sec; more with cache) but shuffling data between the GPU and the CPU is a high latency, heavily bottlenecked process that tops out at 8GB/sec under PCIe 2.1. And since GPUs are still specialized devices that excel at parallel code execution, a lot of workloads exist that will need to constantly move data between the GPU and the CPU to maximize parallel and serial code execution. As it stands today GPUs are really only best suited for workloads that involve sending work to the GPU and keeping it there; heterogeneous computing is a luxury there isn’t bandwidth for.

The long term solution of course is to bring the CPU and the GPU together, which is what Fusion does. CPU/GPU bandwidth just in Llano is over 20GB/sec, and latency is greatly reduced due to the CPU and GPU being on the same die. But this doesn’t preclude the fact that AMD also wants to bring some of these same benefits to discrete GPUs, which is where PCI e 3.0 comes in.

With PCIe 3.0 transport bandwidth is again being doubled, from 500MB/sec per lane bidirectional to 1GB/sec per lane bidirectional, which for an x16 device means doubling the available bandwidth from 8GB/sec to 16GB/sec. This is accomplished by increasing the frequency of the underlying bus itself from 5 GT/sec to 8 GT/sec, while decreasing overhead from 20% (8b/10b encoding) to 1% through the use of a highly efficient 128b/130b encoding scheme. Meanwhile latency doesn’t change – it’s largely a product of physics and physical distances – but merely doubling the bandwidth can greatly improve performance for bandwidth-hungry compute applications.

As with any other specialized change like this the benefit is going to heavily depend on the application being used, however AMD is confident that there are applications that will completely saturate PCIe 3.0 (and thensome), and it’s easy to imagine why.

Even among our limited selection compute benchmarks we found something that directly benefitted from PCIe 3.0. AESEncryptDecrypt, a sample application from AMD’s APP SDK, demonstrates AES encryption performance by running it on square image files.  Throwing it a large 8K x 8K image not only creates a lot of work for the GPU, but a lot of PCIe traffic too. In our case simply enabling PCIe 3.0 improved performance by 9%, from 324ms down to 297ms.

Ultimately having more bandwidth is not only going to improve compute performance for AMD, but will give the company a critical edge over NVIDIA for the time being. Kepler will no doubt ship with PCIe 3.0, but that’s months down the line. In the meantime users and organizations with high bandwidth compute workloads have Tahiti.

Video & Movies: The Video Codec Engine, UVD3, & Steady Video 2.0 Managing Idle Power: Introducing ZeroCore Power
Comments Locked

292 Comments

View All Comments

  • Ryan Smith - Thursday, December 22, 2011 - link

    Since 1920x1200 has already been commented on elsewhere I'm just going to jump right to your comment on minimum FPS.

    I completely agree, and we're trying to add it where it makes sense. A lot of benchmarks are wildly inconsistent about their minimum FPS, largely thanks to the fact that minimum FPS is an instantaneous data point. When your values vary by 20%+ per run (as minimums often do), even averaging repeated trials isn't nearly accurate enough to present meaningful results.
  • CeriseCogburn - Thursday, March 8, 2012 - link

    HardOCP shows long in game fps per second charts that show dips and bottom outs are more than one momentary lapse and often are extended time periods of lowest frame rate runs, so I have to respectfully disagree.
    Perhaps the fault is fraps can show you a single instance of lowest frame rate number, and hence it's the analysis that utterly fails - given the time constraints that were made obvious, it is also clear that the extra work it would take for an easily reasoned and reasonable result that is actually of worthy accuracy is not in the cards here.
  • thunderising - Thursday, December 22, 2011 - link

    Okay. This card has left me thrilled, but wanting for more. Why?

    Well, for example, every reviewer has hit the CCC Core and Memory Max Limits, which turns into a healthy 10-12% performance boost, all for 10W.

    What, legit reviews got it to 1165MHz core and 6550Mhz memory for a 21-24% increase in performance. Now that's HUGE!

    I think AMD could have gone for something like this with the final clocks, to squeeze out every last bit of performance from this amazing card:

    Core - 1050 MHz
    Memory - 1500 MHz (6000MHz QDR)

    This was not only easily achievable, but would have placed this card at a 8-10% increase in performance all for a mere <10W rise in Load Power.

    Hoping for AIBs like Sapphire to show their magic! HD7970 Toxic, MmmmmmM...

    Otherwise, fantastic card I say.
  • Death666Angel - Friday, December 23, 2011 - link

    Maybe they'll do a 4870/4890 thing again? Launch the HD7970 and HD7970X2 and then launch a HD7990 with higher clocks later to counter nVidia.... Who knows. :-)
  • Mishera - Sunday, December 25, 2011 - link

    They've been doing it for quite some time now. Their plan has been to release a chip balancing die size, performance, and cost. Then later to compete on high end release a dual-chip card. Anand wrote on this a while ago with the rv770 story (http://www.anandtech.com/show/2679).

    Even looking at the picture of chip sizes, the 7970 is still a reasonable size. And this really was a brilliant move as though Nvidia has half the marketshare and does make a lot of money from their cards, their design philosophy has been hurting them a lot from a business standpoint.

    On a side note, Amd really made a great choice by choosing to wait until now to push for general computing. Though that probably means more people to support development and drivers, which means more hiring which is the opposite way Amd has been going. It will be interesting to see how this dichotomy will develop in the future. But right now kudos to Amd.
  • CeriseCogburn - Thursday, March 8, 2012 - link

    Does that mean amd is abandoning gamers as we heard the scream whilst Nvidia was doing thus ?
    I don't quite get it - now what nvidia did that hurt them, is praise worthy since amd did it, finally.
    Forgive me as I scoff at the immense dichotomy...
    "Perfect ripeness at the perfect time" - sorry not buying it....
  • privatosan - Thursday, December 22, 2011 - link

    PRT is a nice feature, but there is an failure in the article:

    'For AMD’s technology each tile will be 64KB, which for an uncompressed 32bit texture would be enough room for a 4K x 4K chunk.'

    The tile would be 128 x 128 texels; 4K x 4K would be quite big for a tile.
  • futrtrubl - Thursday, December 22, 2011 - link

    I was going to comment on that too. A 4k x 4k x 32bit (4byte) texture chunk would be around 67MB uncompressed. For a 32bit texture you could only fit a 128x128 array in a 64KB chunk. An 8bit/pixel texture could be 4k*4k
  • Stonedofmoo - Thursday, December 22, 2011 - link

    Thanks for the review. A request though...
    To the hardware sites doing these reviews, many of us in this day and age run dual monitor or more. It always frustrates in me in these reviews that we get a long write up on the power saving techniques the new cards use, and never any mention of it helps those of us running more than one display.

    For those not in the know, if you run more than one display on all the current generations the cards do NOT downclock the GPU and memory nearly as much as they do on single montor configurations. This burns quite a lot more power and obviously kicks out more heat. No site ever mentions this which is odd considering so many of us have more than one display these days.

    I would happily buy the card that finally overcomes this and actually finds a way of knocking back the clocks with multi-monitor setups. Is the new Radeon 7xxx series that card?
  • Galcobar - Thursday, December 22, 2011 - link

    It's in the article, on the page entitled "Meet the Radeon 7970."

    Ryan also replied to a similar comment by quoting the paragraph addressing multi-monitor setups and power consumption at the top of page of the comments.

    That's two mentions, and the answer to your question.

Log in

Don't have an account? Sign up now