PCI Express 3.0: More Bandwidth For Compute

It may seem like it’s still fairly new, but PCI Express 2 is actually a relatively old addition to motherboards and video cards. AMD first added support for it with the Radeon HD 3870 back in 2008 so it’s been nearly 4 years since video cards made the jump. At the same time PCI Express 3.0 has been in the works for some time now and although it hasn’t been 4 years it feels like it has been much longer. PCIe 3.0 motherboards only finally became available last month with the launch of the Sandy Bridge-E platform and now the first PCIe 3.0 video cards are becoming available with Tahiti.

But at first glance it may not seem like PCIe 3.0 is all that important. Additional PCIe bandwidth has proven to be generally unnecessary when it comes to gaming, as single-GPU cards typically only benefit by a couple percent (if at all) when moving from PCIe 2.1 x8 to x16. There will of course come a time where games need more PCIe bandwidth, but right now PCIe 2.1 x16 (8GB/sec) handles the task with room to spare.

So why is PCIe 3.0 important then? It’s not the games, it’s the computing. GPUs have a great deal of internal memory bandwidth (264GB/sec; more with cache) but shuffling data between the GPU and the CPU is a high latency, heavily bottlenecked process that tops out at 8GB/sec under PCIe 2.1. And since GPUs are still specialized devices that excel at parallel code execution, a lot of workloads exist that will need to constantly move data between the GPU and the CPU to maximize parallel and serial code execution. As it stands today GPUs are really only best suited for workloads that involve sending work to the GPU and keeping it there; heterogeneous computing is a luxury there isn’t bandwidth for.

The long term solution of course is to bring the CPU and the GPU together, which is what Fusion does. CPU/GPU bandwidth just in Llano is over 20GB/sec, and latency is greatly reduced due to the CPU and GPU being on the same die. But this doesn’t preclude the fact that AMD also wants to bring some of these same benefits to discrete GPUs, which is where PCI e 3.0 comes in.

With PCIe 3.0 transport bandwidth is again being doubled, from 500MB/sec per lane bidirectional to 1GB/sec per lane bidirectional, which for an x16 device means doubling the available bandwidth from 8GB/sec to 16GB/sec. This is accomplished by increasing the frequency of the underlying bus itself from 5 GT/sec to 8 GT/sec, while decreasing overhead from 20% (8b/10b encoding) to 1% through the use of a highly efficient 128b/130b encoding scheme. Meanwhile latency doesn’t change – it’s largely a product of physics and physical distances – but merely doubling the bandwidth can greatly improve performance for bandwidth-hungry compute applications.

As with any other specialized change like this the benefit is going to heavily depend on the application being used, however AMD is confident that there are applications that will completely saturate PCIe 3.0 (and thensome), and it’s easy to imagine why.

Even among our limited selection compute benchmarks we found something that directly benefitted from PCIe 3.0. AESEncryptDecrypt, a sample application from AMD’s APP SDK, demonstrates AES encryption performance by running it on square image files.  Throwing it a large 8K x 8K image not only creates a lot of work for the GPU, but a lot of PCIe traffic too. In our case simply enabling PCIe 3.0 improved performance by 9%, from 324ms down to 297ms.

Ultimately having more bandwidth is not only going to improve compute performance for AMD, but will give the company a critical edge over NVIDIA for the time being. Kepler will no doubt ship with PCIe 3.0, but that’s months down the line. In the meantime users and organizations with high bandwidth compute workloads have Tahiti.

Video & Movies: The Video Codec Engine, UVD3, & Steady Video 2.0 Managing Idle Power: Introducing ZeroCore Power
POST A COMMENT

291 Comments

View All Comments

  • Wreckage - Thursday, December 22, 2011 - link

    That's kind of disappointing. Reply
  • atticus14 - Thursday, December 22, 2011 - link

    oh look its that guy that was banned from the forums for being an overboard nvidia zealot. Reply
  • medi01 - Tuesday, January 03, 2012 - link

    Maybe he meant "somebody @ anandtech is again pissing on AMDs cookies"?

    I mean "oh, it's fastest and coolest single GPU card on the market, it is slightly more expensive than competitor's, but it kinda sucks since AMD didn't go "significantly cheaper than nVidia" route" is hard to call unbiased, eh?

    Kind of disappointing conclusion, indeed.
    Reply
  • ddarko - Thursday, December 22, 2011 - link

    To each their own but I think this is undeniable impressive:

    "Even with the same number of ROPs and a similar theoretical performance limit (29.6 vs 28.16), 7970 is pushing 51% more pixels than 6970 is" and

    "it’s clear that AMD’s tessellation efficiency improvements are quite real, and that with Tahiti AMD can deliver much better tessellation performance than Cayman even at virtually the same theoretical triangle throughput rate."
    Reply
  • Samus - Thursday, December 22, 2011 - link

    I prefer nVidia products, mostly because the games I play (EA/DICE Battlefield-series) are heavily sponsered by nVidia, giving them a developement-edge.

    That out of the way, nVidia has had their problems just like this card is going to experience. Remember when Fermi came out, it was a performance joke, not because it was slow, but because it used a ridiculous amount of power to do the same thing as an ATI card while costing substantially more.

    Fermi wasn't successful until second-generation products were released, most obviously the GTX460 and GT430, reasonably priced cards with quality drivers and low power consumption. But it took over a year for nVidia to release those, and it will take over a year for ATI to make this architecture shine.
    Reply
  • kyuu - Thursday, December 22, 2011 - link

    Wat? The only thing there might be an issue with is drivers. As far as power consumption goes, this should be better than Cayman. Reply
  • CeriseCogburn - Sunday, March 11, 2012 - link

    He's saying the 28mn node will have further power improvements. Take it as an amd compliment - rather you should have. Reply
  • StriderTR - Thursday, December 22, 2011 - link

    EA/Dice are just as heavily sponsored by AMD, more in fact. Not sure where your getting your information, but its .. well ... wrong. Nvidia bought the rights to advertize the game with their hardware, AMD is heavily sponsoring BF3 and related material. Example, The Controller.

    Also, the GTX 580 and HD 6970 perform within a few FPS of each other on BF3. I run dual 6970's, by buddy runs dual 580's, we are almost always within 2 FPS of one and other at any given time.

    AMD will have the new architecture "shining" in far under a year. They have been focused on it for a long time already.

    Simple bottom line, both Nvidia and AMD make world class cards these days. No matter your preference, you have cards to choose from that will rock any games on the planet for a long time to come.
    Reply
  • deaner - Thursday, December 22, 2011 - link

    Umm, yea no. Not so much with nvidia and EA/DICE Batttlefield series giving nvidia a development edge. (if it does, the results are yet to be seen)
    Facts are facts, the 5 series to our current review today, the 7970, do and again continue to edge the Nvidia lines. The AMD Catalyst performance of particular note, BF3, has been far superior.

    Reply
  • RussianSensation - Thursday, December 22, 2011 - link

    ."..most obviously the GTX460 and GT430, reasonably priced cards with quality drivers and low power consumption. But it took over a year for nVidia to release those"

    GTX470/480 launched March 26, 2010
    GTX460 launched July 12, 2010
    GT430 launched October 11, 2010

    Also, Fermi's performance at launch was not a joke. GTX470 delivered performance between HD5850 and HD5870, priced in the middle. Looking now, GTX480 ~ HD6970. So again, both of those cards did relatively well at the time. Once you consider overclocking of the 470/480, they did extremely well, both easily surprassing the 5870 in performance in overclocked states.

    Sure power consumption was high, but that's the nature of the game for highest-end GPUs.
    Reply

Log in

Don't have an account? Sign up now