PCI Express 3.0: More Bandwidth For Compute

It may seem like it’s still fairly new, but PCI Express 2 is actually a relatively old addition to motherboards and video cards. AMD first added support for it with the Radeon HD 3870 back in 2008 so it’s been nearly 4 years since video cards made the jump. At the same time PCI Express 3.0 has been in the works for some time now and although it hasn’t been 4 years it feels like it has been much longer. PCIe 3.0 motherboards only finally became available last month with the launch of the Sandy Bridge-E platform and now the first PCIe 3.0 video cards are becoming available with Tahiti.

But at first glance it may not seem like PCIe 3.0 is all that important. Additional PCIe bandwidth has proven to be generally unnecessary when it comes to gaming, as single-GPU cards typically only benefit by a couple percent (if at all) when moving from PCIe 2.1 x8 to x16. There will of course come a time where games need more PCIe bandwidth, but right now PCIe 2.1 x16 (8GB/sec) handles the task with room to spare.

So why is PCIe 3.0 important then? It’s not the games, it’s the computing. GPUs have a great deal of internal memory bandwidth (264GB/sec; more with cache) but shuffling data between the GPU and the CPU is a high latency, heavily bottlenecked process that tops out at 8GB/sec under PCIe 2.1. And since GPUs are still specialized devices that excel at parallel code execution, a lot of workloads exist that will need to constantly move data between the GPU and the CPU to maximize parallel and serial code execution. As it stands today GPUs are really only best suited for workloads that involve sending work to the GPU and keeping it there; heterogeneous computing is a luxury there isn’t bandwidth for.

The long term solution of course is to bring the CPU and the GPU together, which is what Fusion does. CPU/GPU bandwidth just in Llano is over 20GB/sec, and latency is greatly reduced due to the CPU and GPU being on the same die. But this doesn’t preclude the fact that AMD also wants to bring some of these same benefits to discrete GPUs, which is where PCI e 3.0 comes in.

With PCIe 3.0 transport bandwidth is again being doubled, from 500MB/sec per lane bidirectional to 1GB/sec per lane bidirectional, which for an x16 device means doubling the available bandwidth from 8GB/sec to 16GB/sec. This is accomplished by increasing the frequency of the underlying bus itself from 5 GT/sec to 8 GT/sec, while decreasing overhead from 20% (8b/10b encoding) to 1% through the use of a highly efficient 128b/130b encoding scheme. Meanwhile latency doesn’t change – it’s largely a product of physics and physical distances – but merely doubling the bandwidth can greatly improve performance for bandwidth-hungry compute applications.

As with any other specialized change like this the benefit is going to heavily depend on the application being used, however AMD is confident that there are applications that will completely saturate PCIe 3.0 (and thensome), and it’s easy to imagine why.

Even among our limited selection compute benchmarks we found something that directly benefitted from PCIe 3.0. AESEncryptDecrypt, a sample application from AMD’s APP SDK, demonstrates AES encryption performance by running it on square image files.  Throwing it a large 8K x 8K image not only creates a lot of work for the GPU, but a lot of PCIe traffic too. In our case simply enabling PCIe 3.0 improved performance by 9%, from 324ms down to 297ms.

Ultimately having more bandwidth is not only going to improve compute performance for AMD, but will give the company a critical edge over NVIDIA for the time being. Kepler will no doubt ship with PCIe 3.0, but that’s months down the line. In the meantime users and organizations with high bandwidth compute workloads have Tahiti.

Video & Movies: The Video Codec Engine, UVD3, & Steady Video 2.0 Managing Idle Power: Introducing ZeroCore Power
POST A COMMENT

291 Comments

View All Comments

  • Ryan Smith - Thursday, December 22, 2011 - link

    Yep. Still working on it. Hold tight Reply
  • MadMan007 - Thursday, December 22, 2011 - link

    Np, just not used to seeing incomplete articles publsihed on Anandtech that aren't clearly 'previews'...wasn't sure if you were aware of all the missing stuff. Reply
  • DoktorSleepless - Thursday, December 22, 2011 - link

    Crysis won't be defeated until we're able to play at a full 60fps with 4x super sampling. It looks ugly without the foliage AA. Reply
  • Ryan Smith - Thursday, December 22, 2011 - link

    I actually completely agree. That's even farther off than 1920 just with MSAA, but I'm looking forward to that day. Reply
  • chizow - Thursday, December 22, 2011 - link

    Honestly Crysis may be defeated once Nvidia releases its driver-level FXAA injector option. Yes, FXAA can blur textures but it also does an amazing job at reducing jaggies on both geometry and transparencies at virtually no impact on performance.

    There's leaked driver versions (R295.xx) out that allow this option now, hopefully we get them officially soon as this will be a huge boon for games like Crysis or games that don't support traditional AA modes at all (GTA4).

    Check out the results below:

    http://www.hardocp.com/image.html?image=MTMyMjQ1Mz...
    Reply
  • AnotherGuy - Thursday, December 22, 2011 - link

    If nVidia released this card tomorrow they woulda priced it easily $600... The card succeeds in almost every aspect.... except maybe noise... Reply
  • chizow - Thursday, December 22, 2011 - link

    Funny since both of Nvidia's previous flagship single-GPU cards, the GTX 480 and GTX 580, launched for $499 and were both the fastest single-GPU cards available at the time.

    I think Nvidia learned their lesson with the GTX 280, and similarly, I think AMD has learned their lesson as well with underpricing their HD 4870 and HD 5870. They've (finally) learned that in the brief period they hold the performance lead, they need to make the most of it, which is why we are seeing a $549 flagship card from them this time around.
    Reply
  • 8steve8 - Thursday, December 22, 2011 - link

    waiting for amd's 28nm 7770.

    this card is overkill in power and money.
    Reply
  • tipoo - Thursday, December 22, 2011 - link

    Same, we're not going to tax these cards at the most common resolutions until new consoles are out, such is the blessing and curse of console ports. Reply
  • CrystalBay - Thursday, December 22, 2011 - link

    Hi Ryan , All these older GPUs ie (5870 ,gtx570 ,580 ,6950 were rerun on the new hardware testbed ? If so GJ lotsa work there. Reply

Log in

Don't have an account? Sign up now