What’s in a Benchmark? This is a pertinent question that all users need to ask themselves, because if you don’t know what a benchmark actually tests and how that relates to the real world, the scores are meaningless. Today, AMD has announced that they are resigning from BAPCo over a long standing dispute over the weighting of scores within the SYSmark suite. AMD specifically references SYSmark 2012 (SM12), but there have been complaints in the past and the latest release is apparently the proverbial straw that broke the camel’s back.

You can read more about the decision on Cheif Marketing Officer (CMO) Nigel Dessau’s blog, but this announcement comes at an interesting time since BAPCo just shipped us copies of the final SM12 release. We haven’t had a chance to run the suite yet, and we’ll still have a look at the results and see how AMD and Intel platforms compare at some point, but it looks like we have a foregone conclusion: Intel will come out ahead. What we really need to examine is why Intel gets a better score.

If you’ve been reading AnandTech for any length of time, you’ll know that we place a lot more weight on real-world benchmarks rather than synthetic tests, but certain tasks can be very difficult to test in a meaningful way. How do you measure every day tasks like surfing the web in a meaningful way when most CPUs are 95% idle performing that task? When we really look at the market right now, in many cases we can conclude that just about any current computer will be fast enough for 90% of users. If you want to surf the Internet, write email, work in Office applications, watch some movies, listen to music, etc. you can do that on anything from a lowly AMD Brazos netbook to a hex-core monster system. Yes, we did leave out Atom, because there are certain areas where it falls short—specifically, certain movie formats prove to be too much for the current Atom platform, particularly if you’re looking at HD H.264 content (e.g. YouTube and Hulu).

Reading through AMD’s announcement and Nigel’s blog, it’s pretty clear what AMD is after: they want the GPU to play a more prominent role in measurements of overall system performance. On the one hand, we could say that AMD is simply trying to get benchmarks to favor their APUs, since Brazos and Llano easily surpass the Intel competition when it comes to graphics and video prowess. This would certainly be true, but then we also have to consider what users are actually doing with their PCs. SYSmark has always included a variety of tests, and certainly knowing how fast your computer is in regards to Excel performance can be useful. However, AMD claims that a disproportionate weight is given to some tests, with mention of optical character recognition and file compression activities in particular.

We don’t have the full SM12 whitepaper yet, but we can look at the list of applications that are tested, and a few things immediately stand out. There are two web browsers in the list, but both versions are now outdated. Internet Explorer 8 has been replaced by Internet Explorer 9, and Firefox 3.6 is replaced by Firefox 4.0—with Firefox 5 just around the corner. Without newer browsers, HTML5 is basically untested by SM12, and while we understand that SM12 has been in development for a while, for something calling itself 2012 to include mostly 2010 applications feels out of place. Considering IE9 and FF4 both shift to GPU-accelerated engines, AMD would certainly have benefited from the use of the latest versions. The remaining applications look reasonable, but again we have no information on weighting of scores, so we’ll have to see how the results pan out.

Ultimately, the main thing to take away from all of this is that, just like the PCMark, 3DMark, Cinebench, SunSpider, etc. benchmarks we routinely refer to, SYSmark 2012 is merely one more tool to analyze system performance. It will be interesting to see how other elements—like the presence or lack of an SSD—impact the score. In our opinion most users would benefit far more from running something like Llano with an SSD as opposed to Sandy Bridge with an HDD, so the CPU/GPU/APU are not the only factors, but it still depends on your intended use. If you’re running a server, obviously the demands placed on the system will be far different from the average home computer. Multimedia professionals that spend a lot of time in Adobe Photoshop and/or Premiere likewise have different needs.

Is AMD right? Is heterogeneous (e.g. CPU and GPU working together) computing more important now than raw CPU performance, or is SYSmark12 merely proving what we already know: Sandy Bridge is really fast? Let us know what you think, but as always remember that when you’re looking at benchmark charts, take a minute to think about what the bars actually represent. The full news release is below, but again you can find substantially more detail in Dessau’s blog.

Update: It turns out AMD is not the only party to have left the BAPCo consortium recently. We've just confirmed with NVIDIA that they have also left the BAPCo consortium. No reason was given.

Update 2: BAPCo has released a statement in return. The consortium notes that AMD approved 80% of the development milestones and that AMD was never threatened with expulsion. The full statement is attached below.

Update 3: We've finally gotten official confirmation (as rumored earlier) that VIA has also left the consortium. They have sent a short statement to SemiAccurate which we have included below. The basis of their complaints are much the same as AMD's: they don't consider SYSMark 2012 to reflect real world usage.


AMD Will Not Endorse SYSmark 2012 Benchmark

— AMD Separates from Association with Industry Group BAPCo —

SUNNYVALE, Calif. — 21, 2011 — AMD (NYSE: AMD) today announced that it will not endorse the SYSmark 2012 Benchmark (SM2012), which is published by BAPCo (Business Applications Performance Corporation). Along with the withdrawal of support, AMD has resigned from the BAPCo organization.

“Technology is evolving at an incredible pace, and customers need clear and reliable measurements to understand the expected performance and value of their systems,” said Nigel Dessau, senior vice president and Chief Marketing Officer at AMD. “AMD does not believe SM2012 achieves this objective. Hence AMD cannot endorse or support SM2012 or remain part of the BAPCo consortium.”

AMD will only endorse benchmarks based on real-world computing models and software applications, and which provide useful and relevant information. AMD believes benchmarks should be constructed to provide unbiased results and be transparent to customers making decisions based on those results. Currently, AMD is evaluating other benchmarking alternatives, including encouraging the creation of an industry consortium to establish an open benchmark to measure overall system performance.

AMD encourages anyone wanting more details about the construction and scoring methodology of the SM2012 benchmark to contact BAPCo. For more details on AMD’s decision to exit BAPCo, please read AMD’s Executive Blog authored by Nigel Dessau.


BAPCo® Reaffirms Open Development Process For SYSmark® 2012

SAN MATEO, Calif.—(BUSINESS WIRE)—Business Applications Performance Corporation (BAPCo®) is a non-profit consortium made up of many of the leaders in the high tech field, including Dell, Hewlett-Packard, Hitachi, Intel, Lenovo, Microsoft, Samsung, Seagate, Sony, Toshiba and ARCintuition. For nearly 20 years BAPCo has provided real world application based benchmarks which are used by organizations worldwide. SYSmark® 2012 is the latest release of the premiere application based performance benchmark. Applications used in SYSmark 2012 were selected based on market research and include Microsoft Office, Adobe Creative Suite, Adobe Acrobat, WinZip, Autodesk AutoCAD and 3ds Max, and others.

Advanced Micro Devices (AMD) was, until recently, a long standing member of BAPCo. We welcomed AMD’s full participation in the two year development cycle of SYSmark 2012, AMD’s leadership role in creating the development process that BAPCo uses today and in providing expert resources for developing the workload contents. Each member in BAPCo gets one vote on any proposals made by member companies. AMD voted in support of over 80% of the SYSmark 2012 development milestones, and were supported by BAPCo in 100% of the SYSmark 2012 proposals they put forward to the consortium.

BAPCo also notes for the record that, contrary to the false assertion by AMD, BAPCo never threatened AMD with expulsion from the consortium, despite previous violations of its obligations to BAPCo under the consortium member agreement.

BAPCo is disappointed that a former member of the consortium has chosen once more to violate the confidentiality agreement they signed, in an attempt to dissuade customers from using SYSmark to assess the performance of their systems. BAPCo believes the performance measured in each of the six scenarios in SYSmark 2012, which is based on the research of its membership, fairly reflects the performance that users will see when fully utilizing the included applications.


VIA's Statement About Leaving The BAPCo Consortium

VIA today confirmed reports that we have tendered our resignation to BAPCo. We strongly believe that the benchmarking applications tests developed for SYSmark 2012 and EEcoMark 2.0 do not accurately reflect real world PC usage scenarios and workloads and therefore feel we can no longer remain as a member of the organization.

We hope that the industry can adopt a much more open and transparent process for developing fair and objective benchmarks that accurately measure real world PC performance and are committed to working with companies that share our vision.

POST A COMMENT

116 Comments

View All Comments

  • cbass64 - Wednesday, June 22, 2011 - link

    I run sysmark'07 all the time and the GPU plays a HUGE role in the score. Out of 4 categories, 2 are GPU bound. You can add ram and a faster CPU and the scores hardly increase, but if you change the screen resolution or update a video driver the scores dramatically change. I would have thought AMD would like the fact that sysmark is so GPU bound...

    If 12 is anything like 07, you'd think GPU oriented companies would be drooling over it.
    Reply
  • Targon - Wednesday, June 22, 2011 - link

    I suspect that the 2012 version discounts the graphics related stuff when it comes to overall score, which WOULD make AMD walk out. Reply
  • redisnidma - Wednesday, June 22, 2011 - link

    I was really expecting more from you Jarred, but it seems to me that you're also on intel's payroll.
    How do you dare compare Llanos powerful GPU to intel's crappy graphics in anyway?

    And as much as you (and Anand) hate it, GPGPU is becoming part of modern-day apps.
    CPU performance still matters, but GPU performance matters more (if you really did your homework, you'd know what I'm talking about).

    Shame on this site.
    Reply
  • JarredWalton - Wednesday, June 22, 2011 - link

    Name one major task that every day people actually do that needs GPGPU. Yes, you can do some amazing stuff with such chips, but for my mom who only checks her email and forwards spam my way, what does she miss out on? Llano's GPU isn't that great; it's merely better than Intel's IGP. Llano is basically the equivalent of a $35 GPU with a $75 processor, but with better power characteristics. We provided a complete review that showed where Llano does well, but while the Llano GPU is three times faster than the previous HD 4250, it's only about 30-50% faster than HD 3000.

    Need some more food for thought? GPGPU performance running on an HD 5670 is more than double what Llano can provide. Anyone that cares about GPGPU is still going to need a discrete GPU. But you have to go and accuse us of "hating" GPGPU. Um, what? Where did I say I hate GPGPU? FYI, I'm running about 2200Mhash/s on ten varieties of AMD GPUs. (Yes, AMD, whom I "hate" -- because for this particular task they're about 3x to 5x faster than NVIDIA, and my overclocked i7-965 would be pulling maybe 21-22Mhash/s, or about twice as fast as an AMD E-350 that uses 1/10 the power.) You wouldn't know that, of course, because it's irrelevant; it would only be relevant for people that want to build a farm of PCs to do GPGPU computing.

    Your talk of bias is unwarranted; there's nothing in this news post that's even remotely biased. I point out that AMD has issues with SYSmark, and I point out what those issues are, and I even show how AMD has plenty of facts to back them up. And you accuse me of bias against AMD for... what? Pointing out that they're right? That SYSmark 2012 is just a benchmark that only shows a few facets of performance, and if those facets are chosen "correctly" the result can be manipulated to put one part above another? I guess it's also bias when people point out that 3DMark runs poorly on anything without a discrete GPU, and that 3DMark11 requires DX11 support.

    If SYSmark 2012 is a bad test because it puts more of an emphasis on the CPU and benefits Intel, then a gaming test that puts more of an emphasis on the GPU is equally bad. We use both types of tests for a reason, because both CPUs and GPUs (and SSDs) make a big difference in what a PC can do well and where it struggles. Yes, that's totally biased and I should be ashamed for saying such things!
    Reply
  • Regenweald - Wednesday, June 22, 2011 - link

    Jarred, you guys tested a DX11 APU in unsupported DX9 mode for your review of Llano without even mentioning that the SB competition does not support DX11 and that Llano was designed for the new API. I call that bias. I assume when Intel finally has a DX11 product, the test bench will be updated ? Reply
  • silverblue - Wednesday, June 22, 2011 - link

    I do wish people would keep comments about bias to themselves. No offence intended, but it only serves to make the complainants look silly and take more time to read through the comments.

    The fact that SB doesn't support DX11 has been said a million times before, but for completion's sake I suppose it should be mentioned.

    With a GPU like Llano's, you're not likely to be gaming at high resolutions or high detail levels with DX11. If, and when, Intel do support DX11, their past track record suggests it'll be of no use for their products either due to the lack of performance, so the comparison isn't likely to be worthwhile, and thus the point is moot. I'd really like to see DX10 and 11 performance figures for Llano but even with the fastest RAM available, is it really going to toss us a pleasant surprise? Far better than SB's HD 3000, sure, but not quite at the same level as a decent mainstream card.
    Reply
  • silverblue - Wednesday, June 22, 2011 - link

    http://www.anandtech.com/show/4448/amd-llano-deskt...

    I take it back; most of these should be DX10, even if we're talking low detail settings. DDR3-1866 might make a decent impression.
    Reply
  • Regenweald - Wednesday, June 22, 2011 - link

    I'm glad you took the time to read the review. Bumping up the API would probably bump down SB's already not so stellar performance, and that would be bad. Far less test in DX11 mode where Anandtech would have to explain to intel why a review was published with only zeroes in the intel column where numbers should be, in a case where ' lower is NOT better'.

    On the topic of the review itself, It should provide ALL relevant information to a potential customer so that they can make the most informed choice upon purchase. Failing to mention that a potentially sub 700 APU based laptop would provide a more future proof media and gaming path than a SB based laptop and be much cheaper than SB+Discreet.

    Such an omission from such an experienced review site seems motivated by external factors.
    Reply
  • silverblue - Wednesday, June 22, 2011 - link

    In terms of the desktop Llano, Anandtech haven't actually reviewed it yet. In the final review, you can bet we'll see much faster RAM and how Llano can benefit from it. We should also see better and more benchmarks, though I would hope they don't include SysMark... for obvious reasons. :P Reply
  • Justin Case - Wednesday, June 22, 2011 - link

    BAPCo and SysMark have been a joke for the past six years. No reviewer with a clue even bothers with it, and no sysadmin gives it any weight when making a decision.

    The problem isn't just that it completely ignores the GPU (which is a very important part of a desktop PC and an even more important part of devices like HTPCs, tablets, etc.), it's that all its benchmarks are based on completely unrealistic operations, carefully picked to make sure Intel wins by the biggest margin possible. For example, their Excel benchmark is based on the time it takes to load the application followed by an insane amount of sorting operations (which turned out to be the operation where Intel CPUs had the biggest advantage).

    The real mystery to me is why AMD and NVidia still had any sort of connection to BAPCo.
    Reply

Log in

Don't have an account? Sign up now