The "TLB Bug" Explained

Phenom is a monolithic quad core design, each of the four cores has its own internal L2 cache and the die has a single L3 cache that all of the cores share. As programs are run, instructions and data are taken into the L2 cache, but page table entries are also cached.

Virtual memory translation is commonplace in all modern OSes, the premise is simple: each application thinks it has contiguous access to all memory locations so they don't have to worry about complex memory management. When an application attempts to load or store something in memory, the address it thinks it's accessing is simply a virtual address - the actual location in memory can be something very different.

The OS stores a table of all of these mappings from virtual to physical addresses, the CPU can cache frequently used mappings so memory accesses take place much quicker.

If the CPU didn't cache page table entries, each memory access would proceed as follows:

1) Read from a pagetable directory
2) Read a pagetable entry
3) Then read the translated address and access memory

Then there's something called a Translation Lookaside Buffer (TLB) which takes the addresses and maps them one to one, so you don't even need to perform a cache lookup - there's just a direct translation stored in the TLB. The TLB is much smaller than the cache so you can't store too many items in the TLB, but the benefit is that with a good TLB algorithm you can get good hit rates within your TLB. If the mapping isn't contained in the TLB then you have to do a lookup in cache, and if it's not there then you have to go out to main memory to figure out the actual memory address you want to access.

Page table entries eventually have to be updated, for example there are situations when the OS decides to move a set of data to another physical location so all of the virtual addresses need to be updated to reflect the new address.

When page table entries are updated the cached entries stored in a core's L2 cache also need to be updated. Page table entries are a special case in the L2, not only does the cache controller have to modify the data in the entries to reflect their new values, but it also needs to set a couple of status bits in the page table entries in order to mark that the data has been modified.

Page table entries in cache are very different than normal data. With normal data you simply modify it and your cache coherency algorithms take care of making sure everything knows that the data is modified. With page table entries the cache controller must manually mark it by setting access and dirty bits because page tables and TLBs are actually managed by the OS. The cache line has to be modified, have a couple of bits set and then put back into the cache - an exception to the standard operating procedure. And herein lies the infamous TLB erratum.

When a page table entry is modified, the core's cache controller is supposed to take the cached entry, place it in a register, modify it and then put it back in the cache. However there is a corner case whereby if the core is in the middle of making this modification and about to set the access/dirty bits and some other activity goes into the cache, hits the same index that the page table entry is stored in and if the page table entry is marked as the next thing to be evicted, then the data will be evicted from the L2 cache and put into the L3 cache in the middle of this modification process. The line that's evicted from L2 is without its access and dirty bits set and now in L3, and technically incorrect.

Now the update operation is still taking place and when it finishes setting the appropriate bits, the same page table data will be placed into the core's L2 cache again. Now the L2 and L3 cache have the same data, which shouldn't happen given AMD's exclusive cache hierarchy.

If the line in L2 gets evicted once more, it'll be sent off to the L3 and there will be a conflict creating a L3 protocol error. But the more dangerous situation is what happens if another core requests the data.

If another core requests the data it will first check for it in L3, and of course find it there, not knowing that an adjacent core also has the data in its L2. The second core will now pull the data from L3 and place it in its L2 cache but keep in mind that the data is marked as unmodified, while the first core has it marked as modified in its L2.

At this point everything is ok since the data in both L2 caches is identical, but if the second core modifies the page table data that could create a dangerous problem as you end up in a situation where two cores have different data that is expected to be the same. This could either result in a hard lock of the system or even worse, silent data corruption.



The BIOS fix

The workaround in B2 stepping Phenoms is a BIOS fix that tells the TLB it can't look in the cache for page table entries upon lookup. Obviously this drives memory latencies up significantly as it adds additional memory requests to all page table accesses.

The hardware fix implemented in B3 Phenoms is that whenever a page table entry is modified, it's evicted out of L2 and placed in L3. There's a very minor performance penalty because of this but no where near as bad as the software/BIOS TLB fix mentioned above.

AMD gave us two confirmed situations where the TLB erratum would rear its ugly head in real world usage:

1) Windows Vista 64-bit running SPEC CPU 2006
2) Xen Hypervisor running Windows XP and an unknown configuration of applications

AMD insisted that the TLB erratum was a highly random event that would not occur during normal desktop usage and we've never encountered it during our testing of Phenom. Regardless, the two scenarios listed above aren't that rare and there could be more that trigger the problem, which makes a great case for fixing the problem

Index The first B3 Stepping Phenom
POST A COMMENT

29 Comments

View All Comments

  • Griswold - Thursday, March 13, 2008 - link

    You're a tech? Is that what they call callcenter jobs at Dell nowadays?

    Take your uneducated opionion and stick it where the sun doesnt shine, please.
    Reply
  • whatthehey - Wednesday, March 12, 2008 - link

    I believe the above "you stole this" comment is just spam. So just ignore it.

    As for the x64 and SPEC CPU stuff... x64 is becoming relevant, as 2GB PCs are common and 4GB setups are increasingly popular. The problem is, you can only fully realize the use of 4GB (or more) RAM if you run a 64-bit OS. Now we just need more 64-bit apps (Photoshop, I'm looking at you!) I'm personally running a Q6600 with 2x2GB and Vista 64-bit, and have no complaints other than the lack of 64-bit applications (not games - applications). And don't even think about bringing Xbox 360 into the picture... last I checked, you couldn't run Photoshop, Office, or any other real business application on an Xbox 360.

    Regarding the TLB errata, we have at least two confirmed ways of causing this errata to rear its ugly head. Having done plenty of system testing, I can state that there are infinitely potential operations a CPU might be asked to perform. Multitasking in particular throws a wrench into the gears, because you can never test all the potential multitasking scenarios.

    With two cases where the TLB bug comes up, you can be sure there are more. They may be rare -now- but we don't know what will happen going forward. Personally, I don't want a system where any time I experience a crash, I'm left to wonder, "Was that something wrong with the OS or application, or was that just the TLB error popping up in some new way?"

    This is the AMD equivalent of the FDIV error Intel had back in the early Pentium days; unfortunately, the workaround has a far greater impact on performance. If you want to take a chance on the TLB error never affecting your system, you're welcome to do so. If that's the case, however, I'm wondering what you use your system for that makes quad-core even necessary.
    Reply
  • Darkness - Wednesday, March 12, 2008 - link

    AMDPhenom if you are going to write a review please do not cheat.

    This review was just copied and pasted from WWW.IMB.COM/UK.

    We would like GENUINE reviews written by some one who has actualy tested this range.

    You have not tested this personaly.

    Please do not say you have as well all know you have not.

    Please people the Phenom range is buggy I would not buy it untill AMD has sorted out these bugs.

    I have read up on this range on many sites and all reviews say it is buggy and no anount of patches and updates are going to solve the problems for you.
    Please wait before you decide to buy?
    Reply
  • stmok - Wednesday, March 12, 2008 - link

    Darkness if you're going to write feedback, please do not conduct fraud related activities.

    www.imb.com is a bank located in California.
    Reply
  • DigitalFreak - Wednesday, March 12, 2008 - link

    LOL. Not even a valid site. What a dipsh!t. Reply
  • nomagic - Wednesday, March 12, 2008 - link

    Is this supposed to be some kind of joke? Reply
  • vijay333 - Wednesday, March 12, 2008 - link

    dammit, who disabled the moron filter to the internet again?

    Reply
  • murphyslabrat - Wednesday, March 12, 2008 - link

    If this was Dailytech, I would soooooo rate you up! Reply
  • fitten - Wednesday, March 12, 2008 - link

    Any news on if they fixed the "Core2" problem? Reply
  • Brian23 - Wednesday, March 12, 2008 - link

    I really wish you would do a clock for clock comparison of the Phenom to a Core 2 chip. That way we could see how the TLB fix affects which CPU is the prefered one in the lower cost segment of the CPU market. Reply

Log in

Don't have an account? Sign up now