A Timely Discovery: Examining Our AMD 2nd Gen Ryzen Results
by Ian Cutress & Ryan Smith on April 25, 2018 11:15 AM EST
Last week, we published our AMD 2nd Gen Ryzen Deep Dive, covering our testing and analysis of the latest generation of processors to come out from AMD. Highlights of the new products included better cache latencies, faster memory support, an increase in IPC, an overall performance gain over the first generation products, new power management methods for turbo frequencies, and very competitive pricing.
In our review, we had a change in some of the testing. The big differences in our testing for this review was two-fold: the jump from Windows 10 Pro RS2 to Windows 10 Pro RS3, and the inclusion of the Spectre and Meltdown patches to mitigate the potential security issues. These patches are still being rolled out by motherboard manufacturers, with the latest platforms being first in that queue. For our review, we tested the new processors with the latest OS updates and microcode updates, as well as re-testing the Intel Coffee Lake processors as well. Due to time restrictions, the older Ryzen 1000-series results were used.
Due to the tight deadline of our testing and results, we pushed both our CPU and gaming tests live without as much formal analysis as we typically like to do. All the parts were competitive, however it quickly became clear that some of our results were not aligned with those from other media. Initially we were under the impression that this was as a result of the Spectre and Meltdown (or Smeltdown) updates, as we were one of the few media outlets to go back and perform retesting under the new standard.
Nonetheless, we decided to take an extensive internal audit of our testing to ensure that our results were accurate and completely reproducible. Or, failing that, understanding why our results differed. No stone was left un-turned: hardware, software, firmware, tweaks, and code. As a result of that process we believe we have found the reason for our testing being so different from the results of others, and interestingly it opened a sizable can of worms we were not expecting.
An extract from our Power testing script
What our testing identified is that the source of the issue is actually down to timers. Windows uses timers for many things, such as synchronization or ensuring linearity, and there are sets of software relating to monitoring and overclocking that require the timer with the most granularity - specifically they often require the High Precision Event Timer (HPET). HPET is very important, especially when it comes to determining if 'one second' of PC time is the equivalent to 'one second' of real-world time - the way that Windows 8 and Windows 10 implements their timing strategy, compared to Windows 7, means that in rare circumstances the system time can be liable to clock shift over time. This is often highly dependent on how the motherboard manufacturer implements certain settings. HPET is a motherboard-level timer that, as the name implies, offers a very high level of timer precision beyond what other PC timers can provide, and can mitigate this issue. This timer has been shipping in PCs for over a decade, and under normal circumstances it should not be anything but a boon to Windows.
However, it sadly appears that reality diverges from theory – sometimes extensively so – and that our CPU benchmarks for the Ryzen 2000-series review were caught in the middle. Instead of being a benefit to testing, what our investigation found is that when HPET is forced as the sole system timer, it can sometimes a hindrance to system performance, particularly gaming performance. Worse, because HPET is implemented differently on different platforms, the actual impact of enabling it isn't even consistent across vendors. Meaning that the effects of using HPET can vary from system to system, as well as the implementation.
And that brings us to the state HPET, our Ryzen 2000-series review, and CPU benchmarking in general. As we'll cover in the next few pages, HPET plays a very necessary and often very beneficial role in system timer accuracy; a role important enough that it's not desirable to completely disable HPET – and indeed in many systems this isn't even possible – all the while certain classes of software such as overclocking & monitoring software may even require it. However for a few different reasons it can also be a drain on system performance, and as a result HPET shouldn't always be used. So let's dive into the subject of hardware timers, precision, Smeltdown, and how it all came together to make a perfect storm of volatility for our Ryzen 2000-series review.
Facebook
Twitter
RSS
242 Comments
View All Comments
RafaelHerschel - Thursday, April 26, 2018 - link
It's extremely disappointing. The original benchmarks should not have been published. But since that happened, they should have been removed or at the very least, there should have been a far more obvious (and stronger) disclaimer.Even their mea culpa isn't very strong. The article is well written, but should have started with: "We made a mistake". I dislike the misleading statement that other publications did not all install the latest patches. Some publications did, others gave a good reason why they didn't.
Mistakes happen, but after days of showing incorrect benchmarks people started speculating and even now there are still people spreading misinformation based on the AnandTech article.
This is an editorial problem. I feel very ambivalent about at this. AnandTech supplied some really interesting information, but if they can't redact wrong information in timely fashion, then AnandTech is not trustworthy.
I have reached my personal conclusion, but it is with regret. A few extra lines in the original version of the review would have made all the difference.
Ryan Smith - Thursday, April 26, 2018 - link
"This is an editorial problem. I feel very ambivalent about at this. AnandTech supplied some really interesting information, but if they can't redact wrong information in timely fashion, then AnandTech is not trustworthy."Ultimately that sits with me. Ian was able to repeat the earlier numbers again and again and again. Which indicated, at least initially, that it wasn't a mere fluke with our testing. The numbers we were getting were correct for the scenario we were testing. It just wasn't the scenario we thought we were testing.
It wasn't until he found the HPET connection a couple of days later that we even began to understand what was going on. And a couple of days later still until we had enough data to confirm our hypothesis.
As far as the original review goes, while it's been updated a few times since then, once we decided to audit our results, we did post a note on several pages about it. Which I hope was plainly visible. But if not, then that is very good feedback to have.
eva02langley - Friday, April 27, 2018 - link
Your methodology was good, it is just that an important factor... that should not be a factor... is actually crippling the competition.My question now is, odes HPET should be forced in today systems? For example, for security or other stability issues?
I mean, by playing in the lab, you did discover a huge problem and Intel needs to address IMHO.
Now it is coming back to the use of HPET and the requirements behind it.
Also, is this a problem with only older games? If this is the case, changing your bench suites should be a priority.
Tchamber - Thursday, April 26, 2018 - link
It's too bad you're so upset by this. Their first results, while not representative of peak performance, are indeed valid for the way they were achieved. They are not "wrong," so much as parallel. I hope they keep the results. As this review is the first one to have this issue, I'm glad to see they discovered the cause of the performance scores in such a short time. Keep up the good work, Ian!AdditionalPylons - Thursday, April 26, 2018 - link
Fascinating!MDD1963 - Thursday, April 26, 2018 - link
All that's really missing is the admission that tinkering with the HPET settings crippled Intel performance, and gave AMD the lead which they would not have without having done so...; correct? :) (Nice attempted deflection with the 'someone at Intel told us it would not matter' spiel....!) :)BikeDude - Friday, April 27, 2018 - link
As a current Intel user, I'm more worried about what software is using this timer and how often I have been exposed to this sort of non-stellar performance.My next upgrade will take this into consideration. Best of luck to you all.
_mat - Thursday, April 26, 2018 - link
Seems like the HPET bug finally gets the right momentum. Great!I finally wrote an English article about the HPET bug. There are a lot of misconceptions going on right now about that the bug is and what it's not. It also explains what the HPET timer problem was once, why it matters again today and which platforms are affected. Sadly this shows the way Intel handles bug reports like this as well.
https://tinyurl.com/ybz8qygj
Be sure to try TimerBench as well, my Windows timer benchmark! It has already been posted here so I won't bother you again (and the download is in the article).
Maxiking - Thursday, April 26, 2018 - link
Good job, needs more attention so this Meltdown/Spectre bullshits die ASAP.The video was published on 23 Jul 2017.
AndrewJacksonZA - Thursday, April 26, 2018 - link
You raised an important, IMHO, point:Do game engines use HEPT information in their logs, their AI calculations, their gfx calculations, and whatever else they do inside?
Why this is important in my opinion, is because what will the users' experience be out of the box? They build their PC, they install Windows, they install the game, and then... do they disable or enable HEPT before they play? No, they run the darn game!
We trust you to give us review results that would typically represent what we will get. Non-technical users also trust you to give them review results that would typically represent what they will get, no fiddling around because they don't know how and aren't interested to do so. Please take this comment into account when deciding if you're going to be flipping HEPT switches with every game on both CPU brands.
______________________________
And hey, I didn't see it, but did you do any comparisons on if GPU maker makes a difference to the HEPT impact on CPU maker?
Nvidia GPU + Intel CPU
Nvidia GPU + AMD CPU
AMD GPU + Intel CPU
AMD GPU + AMD CPU
AMD + AMD APU
AMD + Intel APU
Intel + Intel APU