That Darn Memory Bus

Among the entire GTX 600 family, the GTX 660 Ti’s one unique feature is its memory controller layout. NVIDIA built GK104 with 4 memory controllers, each 64 bits wide, giving the entire GPU a combined memory bus width of 256 bits. These memory controllers are tied into the ROPs and L2 cache, with each controller forming part of a ROP partition containing 8 ROPs (or rather 1 ROP unit capable of processing 8 operations), 128KB of L2 cache, and the memory controller. To disable any of those things means taking out a whole ROP partition, which is exactly what NVIDIA has done.

The impact on the ROPs and the L2 cache is rather straightforward – render operation throughput is reduced by 25% and there’s 25% less L2 cache to store data in – but the loss of the memory controller is a much tougher concept to deal with. This goes for both NVIDIA on the design end and for consumers on the usage end.

256 is a nice power-of-two number. For video cards with power-of-two memory bus widths, it’s very easy to equip them with a similarly power-of-two memory capacity such as 1GB, 2GB, or 4GB of memory. For various minor technical reasons (mostly the sanity of the engineers), GPU manufacturers like sticking to power-of-two memory busses. And while this is by no means a true design constraint in video card manufacturing, there are ramifications for skipping from it.

The biggest consequence of deviating from a power-of-two memory bus is that under normal circumstances this leads to a card’s memory capacity not lining up with the bulk of the cards on the market. To use the GTX 500 series as an example, NVIDIA had 1.5GB of memory on the GTX 580 at a time when the common Radeon HD 5870 had 1GB, giving NVIDIA a 512MB advantage. Later on however the common Radeon HD 6970 had 2GB of memory, leaving NVIDIA behind by 512MB. This also had one additional consequence for NVIDIA: they needed 12 memory chips where AMD needed 8, which generally inflates the bill of materials more than the price of higher speed memory in a narrower design does. This ended up not being a problem for the GTX 580 since 1.5GB was still plenty of memory for 2010/2011 and the high pricetag could easily absorb the BoM hit, but this is not always the case.

Because NVIDIA has disabled a ROP partition on GK104 in order to make the GTX 660 Ti, they’re dropping from a power-of-two 256bit bus to an off-size 192bit bus. Under normal circumstances this means that they’d need to either reduce the amount of memory on the card from 2GB to 1.5GB, or double it to 3GB. The former is undesirable for competitive reasons (AMD has 2GB cards below the 660 Ti and 3GB cards above) not to mention the fact that 1.5GB is too small for a $300 card in 2012. The latter on the other hand incurs the BoM hit as NVIDIA moves from 8 memory chips to 12 memory chips, a scenario that the lower margin GTX 660 Ti can’t as easily absorb, not to mention how silly it would be for a GTX 680 to have less memory than a GTX 660 Ti.

Rather than take the usual route NVIDIA is going to take their own 3rd route: put 2GB of memory on the GTX 660 Ti anyhow. By putting more memory on one controller than the other two – in effect breaking the symmetry of the memory banks – NVIDIA can have 2GB of memory attached to a 192bit memory bus. This is a technique that NVIDIA has had available to them for quite some time, but it’s also something they rarely pull out and only use it when necessary.

We were first introduced to this technique with the GTX 550 Ti in 2011, which had a similarly large 192bit memory bus. By using a mix of 2Gb and 1Gb modules, NVIDIA could outfit the card with 1GB of memory rather than the 1.5GB/768MB that a 192bit memory bus would typically dictate.

For the GTX 660 Ti in 2012 NVIDIA is once again going to use their asymmetrical memory technique in order to outfit the GTX 660 Ti with 2GB of memory on a 192bit bus, but they’re going to be implementing it slightly differently. Whereas the GTX 550 Ti mixed memory chip density in order to get 1GB out of 6 chips, the GTX 660 Ti will mix up the number of chips attached to each controller in order to get 2GB out of 8 chips. Specifically, there will be 4 chips instead of 2 attached to one of the memory controllers, while the other controllers will continue to have 2 chips. By doing it in this manner, this allows NVIDIA to use the same Hynix 2Gb chips they already use in the rest of the GTX 600 series, with the only high-level difference being the width of the bus connecting them.

Of course at a low-level it’s more complex than that. In a symmetrical design with an equal amount of RAM on each controller it’s rather easy to interleave memory operations across all of the controllers, which maximizes performance of the memory subsystem as a whole. However complete interleaving requires that kind of a symmetrical design, which means it’s not quite suitable for use on NVIDIA’s asymmetrical memory designs. Instead NVIDIA must start playing tricks. And when tricks are involved, there’s always a downside.

The best case scenario is always going to be that the entire 192bit bus is in use by interleaving a memory operation across all 3 controllers, giving the card 144GB/sec of memory bandwidth (192bit * 6GHz / 8). But that can only be done at up to 1.5GB of memory; the final 512MB of memory is attached to a single memory controller. This invokes the worst case scenario, where only 1 64-bit memory controller is in use and thereby reducing memory bandwidth to a much more modest 48GB/sec.

How NVIDIA spreads out memory accesses will have a great deal of impact on when we hit these scenarios. In the past we’ve tried to divine how NVIDIA is accomplishing this, but even with the compute capability of CUDA memory appears to be too far abstracted for us to test any specific theories. And because NVIDIA is continuing to label the internal details of their memory bus a competitive advantage, they’re unwilling to share the details of its operation with us. Thus we’re largely dealing with a black box here, one where poking and prodding doesn’t produce much in the way of meaningful results.

As with the GTX 550 Ti, all we can really say at this time is that the performance we get in our benchmarks is the performance we get. Our best guess remains that NVIDIA is interleaving the lower 1.5GB of address while pushing the last 512MB of address space into the larger memory bank, but we don’t have any hard data to back it up. For most users this shouldn’t be a problem (especially since GK104 is so wishy-washy at compute), but it remains that there’s always a downside to an asymmetrical memory design. With any luck one day we’ll find that downside and be able to better understand the GTX 660 Ti’s performance in the process.

The GeForce GTX 660 Ti Review Meet The EVGA GeForce GTX 660 Ti Superclocked
Comments Locked

313 Comments

View All Comments

  • claysm - Friday, August 24, 2012 - link

    I absolutely will ignore driver support for the 6 series cards. If you are using an AGP card, it's really REALLY time to upgrade.
    You are just as bad a fanboy for nVidia as any AMD guy here, moron. You are completely ignoring anything good about AMD just because it has AMD attached to it.
    I'm completely confident that if AMD had introduced adaptive v-sync and PhysX, you would still say they suck, just because they came from AMD. If you read my post, it says that 660 Ti IS more powerful than the 7870. I was just pointing out that they are closer than they seem. I have no nVidia hatred, they have a lot of cool stuff.
    And about the 660 Ti beating the 7950 at 5760x1080, look at the other three benchmarks, moron. The 7950 wins all of them, meaning BF3, Dirt 3, and Crysis 2. It only looses in Skyrim by and average of 2 FPS. Why didn't you include those games in your response.
    And when I left the games out, I said that they merely blew the average out of proportion, but that you can't leave them out because you want to. You still have to calculate them in the total. Moron.
    And for the record, I'm running a GTX 570, moron.
  • CeriseCogburn - Friday, August 24, 2012 - link

    Look, the amd crew, you, talk your crap of lies, then I correct you.
    That's why.
    Now, whatever you have that is "good by amd" go ahead and state it. Don't tell lies, don't spin, don't talk crap.
    I'm waiting...
    My guess is I'll have to correct your lies again, and your STUPID play dumb amnesia.
    The reason one game was given with 660Ti in that highest resolution winning is very obvious, isn't it, the endless your bud giradou or geradil or geritol whatever his name is was claiming that's the game he was buying the 7950 for...
    LOL
    ROFL
    MHO
    Whatever - do your worst.
  • CeriseCogburn - Saturday, August 25, 2012 - link

    " Fan noise of the card is very low in both idle and load, and temperatures are fine as well.
    Overall, MSI did an excellent job improving on the NVIDIA reference design, resulting in a significantly better card. The card's price of $330 is the same as all other GTX 660 Ti cards we reviewed today. At that price the card easily beats AMD's HD 7950 in all important criteria: performance, power, noise, heat, performance per Dollar, performance per Watt. "
    LOL
    power target 175W LOL
    " It seems that MSI has added some secret sauce, no other board partner has, to their card's BIOS. One indicator of this is that they raised the card's default power limit from 130 W to 175 W, which will certainly help in many situations. During normal gaming, we see no increased power consumption due to this change. The card essentially uses the same power as other cards, but is faster - leading to improved performance per Watt.< br />Overclocking works great as well and reaches the highest real-life performance, despite not reaching the lowest GPU clock. This is certainly an interesting development. We will, hopefully, see more board partners pick up this change. "
    Uh OH
    bad news for you amd fanboys.....
    HAHAHHAHAHAHAAAAAAAAAAAAAA
    The MSI 660Ti is uncorked from the bios !
    roflmao
  • Ambilogy - Friday, August 24, 2012 - link

    "I don't have a problem with that. 660Ti is hitting 1300+ on core and 7000+ on memory, and so you have a problem with that.
    The general idea you state, though I'M ALL FOR IT MAN!
    A FEW FPS SHOULD NOT BE THE THING YOU FOCUS ON, ESPECIALLY WHEN #1 ! ALL FOR IT ! 100% !"

    So you have not a problem with performance? good, because actually that means its a competitive card, not a omfg card. And if you want to oc a 660 you would just oc a 7950 so I don't see the omfg nvidia is so much better.

    "Thus we get down to the added features- whoops ! nVidia is about 10 ahead on that now. That settles it.
    Hello ? Can YOU accept THAT ?"

    So essentially when i ask how many people do actually 3D because you seem to think 2% is unimportant in resolution your answer is "well nvidia is 10 ahead because it has features ACCEPT BLINDLY". Not smart.

    "Nope, it's already been proven it's a misnomer. Cores are gone , fps is too, before memory can be used. In the present, a bit faster now, cranked to the max, and FAILING on both sides with CURRENT GAMES - but some fantasy future is viable ? It's already been aborted.
    You need to ACCEPT THAT FACT."

    FPS are gone and future is fantasy? amd cards still perfom, they are very gpgpu focused and they do excellent for that, and still they don't have bad gaming performance while doing it because you just buy a pre OC version or something and you get still awesome performance (very similar to your 660ti god), say to me what is not enjoyable while playing with an AMD card mr fanboy.

    And the future, well, future is gpgpu because allows big improvements to computing, yet is "fantasy". It's only non important because nvidia had good gpgpu in the past and not now?

    "Okay, so whatever that means...all I see is insane amd fanboysim - that's the PR call of the loser - MARKETING to get their failure hyped.."

    Yeah, calling fan-boy before actually noticing that nvidia told the reviewers how to review the card so it looked better, because get realist, if they include a horrible AA technique with no reason at all something is behind the table hiding you know. Haven't you noticed? theres a lot of discrepancy in 660ti's benchmarks around the web, from sites where the 660 loses to 870's radeons and where it wins to 970's, there is not a single liable review now, do you want to see the truth? buy a 660ti a 870 and a 950, and compare the 3, you will have the truth, thay they perform like they are priced and AMD cards are not shit.
  • CeriseCogburn - Saturday, August 25, 2012 - link


    Hey, I answered the guys 3 questions. I made my points. I didn't say half of what you're talking about, but who cares.
    The guy killed himself with point #1, so that's the end of it.
  • CeriseCogburn - Saturday, August 25, 2012 - link

    Oh stop the crap. nVidia is 10 features ahead, I'm not the one who talked about resolution usage, so you've got the wrong fellow there.
    3D isn't the only feature... but then you know that, but will blabber like an idiot anyway.
    Go away.
  • claysm - Saturday, August 25, 2012 - link

    "I'm not the one who talked about resolution usage". You can't fault him for mixing up his trolls. Since almost everything you and TheJian have said is complete shit it's hard to keep track of who said what.
    And if you can objectively prove that I've lied about anything, I really would like to see it. And I mean objectively, not your usual response of entirely subjective 'AMD suckz lololol' presented in almost unreadably bad grammar.
    I take that back, I won't read it anyways, since I know already know it'll be an nVidia love fest regardless of what the facts state. And I'll reiterate that I'm using an nVidia card. Moron.
  • CeriseCogburn - Saturday, August 25, 2012 - link

    Oh it is not, he showed it all to be true and so does the review man.
    Get out of your freaking goggled amd fanboy gourd.
    Look, I just realized another thing that doesn't bode well for you..
    What nVidia did here was make a very good move, and the losses of amd on the Steam Hardware Survey at the top end are going to increase....
    The amd fanboy is constantly crying about price - they're going to look at $299 with the excellent new game for free and PASS on the more expensive 7950 Russian is promoting EVEN MORE now.
    Here let me get you the little info you're now curious about. ( I hope but maybe you're just a scowling amd fanboy liar still completely uninterested because you never got 1 fact according to you LOL sad what you are it's sad)
    Aug 15th 2012 prdola0
    " Looking at Steam Survey, it is clear why AMD is so desperate. GTX680 has 0.90% share, while even the 7850 lineup has less, just 0.62%. If you look at the GTX670, it has 0.99%. The HD7970 has only 0.54%, about half of what GTX680 has, which is funny considering that the GTX680 is selling only half the time compared to HD7970. It means that GTX680 is selling 4 times faster."
    ROFL...
    No one is listening to you fools, Russian included... now it's going to GET WORSE for amd....
  • CeriseCogburn - Saturday, August 25, 2012 - link

    forgot link, sorry, page 2 comment
    http://www.anandtech.com/show/6152/amd-announces-n...

    Okay, and that stupid 7950 boost REALLY IS CRAPPY CHIPS from the low end loser harvest they had to OVER VOLT to get to their boost...

    LOL
    LOL\
    OLO
    I mean there it is man - the same JUNK amd fanboys always use to attack nVida talking about rejected chips for lower clocked down the line variants has NOW COME TRUE IN FULL BLOWN REALITY FOR AMD....~!
    HAHHAHAHAHAHAHAHHAHA
    AHHAHAHAHAHAA
    omg !
    hahahahahahhahaha
    ahhahaha
    ahaha
    Holy moly. hahahahahhahahha
  • CeriseCogburn - Thursday, August 23, 2012 - link

    plus here the 660Ti wins in 5760x1080, beating the 7950 the 7950 boost, and the 7970...

    http://www.bit-tech.net/hardware/2012/08/16/nvidia...

    Skyrim. So, throw that out too - add skyrim to your too short shortlist. Throw out your future mem whine. throw out your OC whine readied for 660Ti...

    Yep. So the argument left is " I wuv amd ! " - or the more usual " I OWS nVidia ! " ( angry face )

Log in

Don't have an account? Sign up now