#CPUOverload: What is Realistic?

Truth be told, the concept of a project to benchmark almost 700-900 processors has been rattling around in my head for a few years. I actually wrote the first segment of this article way back in 2016. However, over the course of 2016 and 2017, building new testing suites has taken longer, priorities changed, and the project didn’t so much as get shelved as somewhat pushed down the order on a semi-permanent basis until there was an ideal opening. Those of you who have followed the site may have noticed my responsibilities increase over time, darting 200k miles a year around the world. It can be difficult to keep a large project buoyant without constant attention.

Between 2016 and today, we’ve still be churning though the tests on the hardware, and updating our benchmark database with as many chips as we can find, even if it wasn’t under a governed project. The most recent version of our CPU2019 Bench has 272 CPUs with data recorded on up to 246 benchmark data points for each, just to showcase perhaps what one person can do in a given year. However, the focus of Bench being a specific project wasn’t necessarily a primary target of the site. With the launch of our Bench2020 suite, with a wider variety of tests and analysis, we’re going to put this into action. That’s not to say I have more time than normal (I might have to propose what we can do about getting an intern), but with the recent pandemic keeping me on the ground, it does give a chance to take stock about what users are really after.

With #CPUOverload, the goal is to do more than before, and highlight the testing we do. This is why I’ve spent the best part of 25-30 pages talking about benchmark sustainability, usefulness, automation, and why every benchmark is relevant to some of our user base. Over the last decade, as a hardware tester providing results online for free, one obvious change in the requests from our readers has been to include specific benchmarks that target them, rather than generic ones related to their field. That’s part of what this project is, combined with testing at scale.

Users also want to find their exact CPU, and compare it to an exact CPU potential upgrade – a different model, at least in today’s naming conventions, might have different features. So getting exactly what you want to compare is always going to be better – being able to see how your Intel Core i5-2380P in that Dell OEM system you have had for 7 years compares to a newer Ryzen 7 2700E or Xeon E-2274G is all part of what makes this project exciting. That essence of scale, and trying to test as many different CPU variants as possible, is going to be a vital part of this project.

Obviously the best place to start with a project like this is two-fold: popular processors and modern processors. These get the most attention, and so covering the key parts from Coffee Lake, Kaby Lake, Ryzen and HEDT are going to be high on our list to start. The hardware that we’re also testing for review also gets a priority, so that’s why you might start seeing some Zhaoxin or Xeon/EPYC data enter Bench very soon. One funny element is that if you were to start listing what might be ‘high importance processors’, it very easily come back with a list of between 25-100 SKUs, with various i9/i7/i5/i3 and R7/R5/R3/APU as well as Intel/AMD HEDT and halo parts in there – that’s already 10 segments! Some users might want us to focus on the cheap Xeon parts coming out of China too. Obviously whatever our users want to see be tested, we want to hear about it.

As part of this project, we are also expecting to look at some retrospective performance. Future articles might include ‘how well does Ivy Bridge i5 perform today’, or given AMD and Intel’s tendency to compare five year products to each other, we are looking to do that too, in both short and longer form articles.

When I first approached AMD and Intel’s consumer processor divisions about this project, wondering how much interest there would be for it, both came back to me with positive responses. They filled in a few of my hardware gaps, but cautioned that even as internal PR teams, they won’t have access to most chips, especially the older ones. This means that as we process through the hardware, we might start reaching out to other partners in order to fill in the gaps.

Is testing 900 CPUs ultimately realistic? Based on the hardware I have today, if I had access to Narnia, I could provide data for about 350 of the CPUs. In reality, with our new suite, each CPU takes 20-30 hours to test on the CPU benchmarks, and another 10 hours for the gaming tests. Going for 50-100 CPUs/month might be a tough ask, but let’s see how we get on. We have these dozen or so CPUs in the graphs here to start.

Of course, comments are always welcome. If there’s a CPU, old or new, you want to see tested, then please drop a comment below. It will help how I arrange which test beds get priority.

Gaming Tests: Strange Brigade
POST A COMMENT

110 Comments

View All Comments

  • DiHydro - Monday, July 20, 2020 - link

    This is epic. Thank you for doing this. Reply
  • DiHydro - Monday, July 20, 2020 - link

    To add a note: I think the ~$300 CPU year-over-year performance would be an interesting metric to see. That price point seems to be pretty popular for enthusiasts, and seeing back 5-6 years how that performance has increased per dollar would be neat. Reply
  • bldr - Monday, July 20, 2020 - link

    Agree! Reply
  • close - Monday, July 20, 2020 - link

    It will be especially interesting to see those CPUs (the popular mainstream ones) tested now and compared to the numbers they got originally to see how much they lost with all the recent mitigations. Reply
  • close - Tuesday, July 21, 2020 - link

    Oh, because I forgot previously, congratulations and good luck with the endeavor! I got exhausted only by reading about the work you're going to have to do Reply
  • Fozzie - Monday, July 20, 2020 - link

    Except keep in mind that adjusted for inflation $200 in the year 2000 is worth over $300 now.

    You'd either be making a chart of the increased value over time just due to inflation or in fact the every increasing value at the $300 price point due to the reduced value of the Dollar on top of whatever performance gains occurred.
    Reply
  • biosstar - Friday, July 24, 2020 - link

    You could also use the value of a dollar in a certain year (let's say 2020) and compare the processors in the inflation adjusted equal categories. Reply
  • PeterCollier - Monday, July 20, 2020 - link

    What's the point of this Geekbench/Userbenchmark knockoff? I've never used AT's Bench tool. Especially not for smartphones, since the Bench tool is about 5 years out of date. Reply
  • BushLin - Monday, July 20, 2020 - link

    A controlled environment across all tests is reason enough. Even if I don't agree with AT policy on what speed they allow RAM to operate, it is a fair comparison. Reply
  • Byte - Monday, July 20, 2020 - link

    RAM is a really important topic. I think at this point in time, we can reasonable put almost maxed out ram for every platform. Like DDR3 can run at 2133, DDR4 we can run it at 3200 as prices are so close.
    It is like rating sports cars but all have Goodride tires on them.
    A dodge viper was a widowmaker when it came out. Today with a good set of summers like PS4S or PZero, you will have a hard time slipping even if you tried.
    Reply

Log in

Don't have an account? Sign up now