PCI Express 3.0: More Bandwidth For Compute

It may seem like it’s still fairly new, but PCI Express 2 is actually a relatively old addition to motherboards and video cards. AMD first added support for it with the Radeon HD 3870 back in 2008 so it’s been nearly 4 years since video cards made the jump. At the same time PCI Express 3.0 has been in the works for some time now and although it hasn’t been 4 years it feels like it has been much longer. PCIe 3.0 motherboards only finally became available last month with the launch of the Sandy Bridge-E platform and now the first PCIe 3.0 video cards are becoming available with Tahiti.

But at first glance it may not seem like PCIe 3.0 is all that important. Additional PCIe bandwidth has proven to be generally unnecessary when it comes to gaming, as single-GPU cards typically only benefit by a couple percent (if at all) when moving from PCIe 2.1 x8 to x16. There will of course come a time where games need more PCIe bandwidth, but right now PCIe 2.1 x16 (8GB/sec) handles the task with room to spare.

So why is PCIe 3.0 important then? It’s not the games, it’s the computing. GPUs have a great deal of internal memory bandwidth (264GB/sec; more with cache) but shuffling data between the GPU and the CPU is a high latency, heavily bottlenecked process that tops out at 8GB/sec under PCIe 2.1. And since GPUs are still specialized devices that excel at parallel code execution, a lot of workloads exist that will need to constantly move data between the GPU and the CPU to maximize parallel and serial code execution. As it stands today GPUs are really only best suited for workloads that involve sending work to the GPU and keeping it there; heterogeneous computing is a luxury there isn’t bandwidth for.

The long term solution of course is to bring the CPU and the GPU together, which is what Fusion does. CPU/GPU bandwidth just in Llano is over 20GB/sec, and latency is greatly reduced due to the CPU and GPU being on the same die. But this doesn’t preclude the fact that AMD also wants to bring some of these same benefits to discrete GPUs, which is where PCI e 3.0 comes in.

With PCIe 3.0 transport bandwidth is again being doubled, from 500MB/sec per lane bidirectional to 1GB/sec per lane bidirectional, which for an x16 device means doubling the available bandwidth from 8GB/sec to 16GB/sec. This is accomplished by increasing the frequency of the underlying bus itself from 5 GT/sec to 8 GT/sec, while decreasing overhead from 20% (8b/10b encoding) to 1% through the use of a highly efficient 128b/130b encoding scheme. Meanwhile latency doesn’t change – it’s largely a product of physics and physical distances – but merely doubling the bandwidth can greatly improve performance for bandwidth-hungry compute applications.

As with any other specialized change like this the benefit is going to heavily depend on the application being used, however AMD is confident that there are applications that will completely saturate PCIe 3.0 (and thensome), and it’s easy to imagine why.

Even among our limited selection compute benchmarks we found something that directly benefitted from PCIe 3.0. AESEncryptDecrypt, a sample application from AMD’s APP SDK, demonstrates AES encryption performance by running it on square image files.  Throwing it a large 8K x 8K image not only creates a lot of work for the GPU, but a lot of PCIe traffic too. In our case simply enabling PCIe 3.0 improved performance by 9%, from 324ms down to 297ms.

Ultimately having more bandwidth is not only going to improve compute performance for AMD, but will give the company a critical edge over NVIDIA for the time being. Kepler will no doubt ship with PCIe 3.0, but that’s months down the line. In the meantime users and organizations with high bandwidth compute workloads have Tahiti.

Video & Movies: The Video Codec Engine, UVD3, & Steady Video 2.0 Managing Idle Power: Introducing ZeroCore Power
Comments Locked

292 Comments

View All Comments

  • SlyNine - Friday, December 23, 2011 - link

    Are you nuts, the 5870 was nearly 2x as fast in DX 10/9 stuff, not to mention DX11 was way ahead of DX10. Sure the 6970 isn't a great upgrade from a 5870, but neither is the 7970.

    Questionable Premise
  • CeriseCogburn - Thursday, March 8, 2012 - link

    That happened at the end of 2006 with the G80 Roald. That means AMD and their ATI Radeon aquisition crew are five years plus late to the party.
    FIVE YEARS LATE.
    It's nice to know that what Nvidia did years ago and recently as well is now supported by more people as amd copycats the true leader.
    Good deal.
  • Hauk - Thursday, December 22, 2011 - link

    A stunningly comprehensive analysis of this new architecture. This is what sets Anandtech apart from its competition. Kudos Ryan, this is one of your best..
  • eastyy - Thursday, December 22, 2011 - link

    its funny though when it comes to new hardware you read these complicated technical jargon and lots of detailed specs about how cards do things different how much more technically complicated and in the end for me all it means is...+15fps and thats about it

    as soon as a card comes out for say 150 and the games i play become slow and jerky on my 460 then i will upgrade
  • Mockingbird - Thursday, December 22, 2011 - link

    I'd like to see some benchmarks on FX-8150 based system (990fx)
  • piroroadkill - Friday, December 23, 2011 - link

    Haha, the irony is that AMD is putting out graphics cards that would be bottlenecked HARDCORE by ANY of their CPUs, overclocked as much as you like.

    It's kind of tragic...
  • Pantsu - Friday, December 23, 2011 - link

    The performance increase was as expected, at least for me, certainly not for all those who thought this would double performance. Considering AMD had a 389mm^2 chip with Cayman, they weren't going to double the transistor count again. That would've meant the next gen after this would be Nvidia class huge ass chip. So 64% more transistors on a 365mm^2 chip. Looks like transistor density increase took a bit of a hit on 28nm, perhaps because of 384-bit bus? Still I think AMD is doing better than Nvidia when it comes to density.

    As far as the chip size is concerned, the performance is OK, but I really question whether 32 ROPs is enough on this design. Fermi has 48 ROPs and about a billion transistors less. I think AMD is losing AA performance due to such a skimpy ROP count.

    Overall the card is good regardless, but the pricing is indeed steep. I'm sure people will buy it nonetheless, but as a 365mm^2 chip with 3GB GDDR5 I feel like it should be 100$ cheaper than what it is now. I blame lack of competition. It's Nvidia's time to drop the prices. GTX 580 is simply not worth that much compared to what 6950/560Ti are going for these days. And in turn that should drop 7970/50 price.
  • nadavvadan - Friday, December 23, 2011 - link

    Am I really tired, or is:
    " 3.79TFLOPs, while its FP64 performance is ¼ that at 947MFLOPs"
    supposed to be:
    " 3.79TFLOPs, while its FP64 performance is ¼ that at 947-G-FLOPs"?

    Enjoyed the review as always.
  • Death666Angel - Friday, December 23, 2011 - link

    Now that you have changed the benchmark, would it be possible to publish a .pdf with the relevant settings of each game? I would be very interested to replicate some of the tests with my home system to better compare some results. If it is not too much work that is (and others are interested in this as well). :D
  • marc1000 - Friday, December 23, 2011 - link

    What about juniper? Could it make it's way to the 7000 series as a 7670 card? Of course, upgraded to GCN, but with same specs as current cards. I guess that at 28nm it would be possible to abandon the pci-e power requirement, making it the go-to card for oem's and low power/noise systems.

    I would not buy it because I own one now, but I'm looking forward to 7770 or 7870 and their nvidia equivalent. It looks like next year will be a great time to upgrade for who is in the middle cards market.

Log in

Don't have an account? Sign up now