Testing Methodology, Revised

For those of you who aren't familiar or don't remember, here's a brief primer on how we used to test cases to get you up to speed.

Acoustic testing is standardized at one foot from the front of the case, using the Extech SL10 with an ambient noise floor of ~32dB. For reference, that's what my silent apartment measures with nothing running, testing acoustics in the dead of night (usually between 1am and 3am). A lot of us sit about a foot away from our computers, so this should be a fairly accurate representation of the kind of noise the case generates, and it's close enough to get noise levels that should register above ambient.

Thermal testing is run with the computer having idled at the desktop for fifteen minutes, and again with the computer running both Furmark (where applicable) and Prime95 (less one thread when a GPU is being used) for fifteen minutes. I've found that leaving one thread open in Prime95 allows the processor to heat up enough while making sure Furmark isn't CPU-limited. We're using the thermal diodes included with the hardware to keep everything standardized, and ambient testing temperature is always between 71F and 74F. Processor temperatures reported are the average of the CPU cores.

That all seems fairly reasonable, but over time subtle issues have crept in that we're taking the opportunity to correct.

For starters, we've found that while the Extech SL10 is perfectly fine for testing sound levels at about 37dBA and up, it's downright lousy for handling anything designed for silent running. The meter only has an official noise floor of 40dB, which is frankly a bit loud. To produce more accurate results, we've switched over to a beefier Extech SL130. The SL130 is rated to go as low as 30dB (basically the lowest any reasonably priced sound meter will go). In addition, I've actually moved since I started doing case reviews, and my new apartment is much quieter than the old one, resulting in an ambient noise floor well below 30dB. Unlike the SL10, the SL130 won't make "an educated guess" about sound levels below its rated floor, either. I'm continuing to test acoustics with the microphone a foot directly in front of the top of the enclosure to ensure consistent readings on that front. Anything below 30dB still rates as "near silent", but this is a big step away from 40dB.

Meanwhile, thermal testing has proven to be a bit trickier than initially anticipated. Maintaining a consistent interior temperature of an apartment (or even just one room) is easier said than done. Even a variation in ambient temperatures that I mentioned before can color results. As a result, instead of using the absolute temperatures reported from the hardware's thermal diodes during testing, I'm reporting the delta over ambient temperature. Ambient temperature is also measured at the beginning of each test cycle (after fifteen minutes of idle, and before fifteen minutes of burn-in.)

Test cycles are also being ever so slightly modified. I'm continuing to use seven threads of Prime95 to stress the processor, but GPU stress is now being handled by eVGA's OC Scanner instead of Furmark. Furmark is an odd duck that I've found to be unreliable as a GPU stress testing tool; Furmark just consumes power, but doesn't actually simulate proper GPU stress the way something like OC Scanner will. We've also seen some driver tweaks by both AMD and NVIDIA over the years designed to prevent Furmark, so it's best to use something else.

As before, I'm continuing to use the thermal diodes of the internal components rather than separate thermal sensors, and CPU temperature is reported as an average of the four cores. SSD temperature will continue to be included as a representative of how well the enclosure cools installed drives, but chipset and RAM temperatures are no longer going to be included. RAM thermals are really only relevant in extreme cases, and modern chipsets just aren't the heat generators that old dogs like the X58 were.

One new wrinkle I'm including is fan speed, though. Since the CPU and GPU fans are both thermally controlled, it may be useful to see just how hard these fans have to work in any given enclosure. These results aren't going to be strictly comparable between enclosures due to variations in ambient temperatures, but should be a reasonable starting point.

Testing Hardware (Mini-ITX), Revised Conclusion: More Reliable Comparisons
Comments Locked

28 Comments

View All Comments

  • Gnarr - Sunday, April 1, 2012 - link

    This is not a revised methodology, but a revised method. Methodology is the study of a method.
  • shin0bi272 - Monday, April 2, 2012 - link

    I would still keep furmark since its designed for stressing the cards...

    "What is FurMark?

    FurMark is a very intensive OpenGL benchmark that uses fur rendering algorithms to measure the performance of the graphics card. Fur rendering is especially adapted to overheat the GPU and that's why FurMark is also a perfect stability and stress test tool (also called GPU burner) for the graphics card."
  • sticks435 - Tuesday, April 3, 2012 - link

    Furmark doesn't come anywhere close to reality as far as temps are concerned. It's like the Linpack of GPU's. Plus like Dustin Mentioned, Nvidia and AMD have specially coded the drivers/power circuitry to step down when Furmark is detected, so it doesn't give accurate results. I actually think some of the newer work units from F@H are the best test of heat and noise as far as GPU's are concerned.
  • shin0bi272 - Tuesday, April 3, 2012 - link

    true. I was just posting whats on their site. Their stated mission is to test who can get the highest fps at the lowest temp which makes it a good candidate for a benchmark. If you go over to lanoc and read their test of the 7970 and 680 in furmark you'll see how whacked out the program's scoring is. the 680 scores higher in fps and lower in temp so you'd think it should score higher in the burn-in score point system. It doesnt. Ive never seen a benchmark of f@h come near the stress levels that furmark puts on the cards but I guess if your only looking for real world programs to test its a good one.
  • shin0bi272 - Tuesday, April 3, 2012 - link

    oh and according to legit reviews running f@h is getting harder and hard to do on the gpu.

    http://www.legitreviews.com/article/1881/15/
  • sticks435 - Tuesday, April 3, 2012 - link

    Fair enough. I don't think that it's getting harder, it's that Standford is probably swamped and hasn't had time to write new cores for the new arch. Either that or the current drivers are so terrible it causes f@h not to work. My 570 with the current work units sits at 94c with the fan speed at 75% with an ambient of 24c @850mhz. That's more stressful than any game as far as work/temp is concerned.
  • sticks435 - Tuesday, April 3, 2012 - link

    that should have been 26c ambient.
  • Knifeshade - Tuesday, April 3, 2012 - link

    I think there needs to be a 'with stock heatsink' result, and one with the Hyper 212 in your testing methodology.

    You're ultimately testing a case's capability. Having a custom cooler is misleading. "Oh those CPU temperatures look pretty good". "Wrong, it's mostly because of the custom cooler".

    Not everybody uses aftermarket cooling in their build, you know.

Log in

Don't have an account? Sign up now