Quest Benchmark Factory

Benchmark Factory for Databases is a performance and code scalability testing tool that simulates users and transactions on a database and replays a production or synthetic workload in non-production environments. This enables organizations to validate database scalability as user loads increase, application changes are made, and platform changes are implemented. Benchmark Factory is available for Oracle, SQL Server, DB2, Sybase, MySQL, and other databases via ODBC and Native connectivity.

Benchmark Factory provides many tests you can run, and has a very nice and customizable metric reporting engine. We decided to run the AS3AP test, and the Scalable Hardware CPU, Reads, and Mixed tests. Here is what Quest's help file says about these tests:

AS3AP

The AS3AP benchmark is an American National Standards Institute (ANSI) Structured Query Language (SQL) relational database benchmark. The AS3AP benchmark provides the following features:
  • Tests database processing power
  • Built-in scalability and portability that tests a broad range of database systems
  • Minimizes effort in implementing and running benchmark tests
  • Provides a uniform metric and straightforward interpretation of benchmark results
Systems tested with the AS3AP benchmark must support common data types and provide a complete relational interface with basic integrity, consistency, and recovery mechanisms. The AS3AP tests systems ranging from a single-user microcomputer Database Management System (DBMS) to a high-performance parallel or distributed database.

Scalable Hardware

The Scalable Hardware benchmark measures relational database systems. This benchmark is a subset of the AS3AP benchmark and tests the following:
  • CPU
  • Disk
  • Network
  • Any combination of the above three entities
We run three iterations of each load point, and then average the results. We also monitor deviations to ensure they are within an acceptable range. We like to see a max deviation of +/- 3%.

Benchmarking Low Voltage Test Setup
Comments Locked

27 Comments

View All Comments

  • mikepers - Friday, July 20, 2007 - link

    So, quick rough calculation. Assuming electricity costs 10 cents per KWH then I believe that with the AMD chip you're saving 80 watts x 24 hrs day x 365 / 1000 x .10 = about $70 per year. (at idle)

    One thing not addressed in this article: Assuming a large server farm, what are your infrastructure savings? How much do you save in cooling that infrastructure. How much do you save because you can buy less generating capacity for when you have power outages? Any thoughts?

    Last comment, for all the info in the article the most significant statistic for a large server farm is performance per watt. How much work can the farm do given the resources it consumes. In that respect it looks like AMD wins but not by as much as the initial difference in power consumption would have you believe.
  • bruce24 - Tuesday, July 17, 2007 - link

    You can buy 2GB and 4GB FB-DIMMS, yes they are a bit more expensive, but if you interested in power savings and an intel DP server, they are the way to do.

    I've love to see the numbers if you replaced the 8 1GB DIMMs with 4 2GB or even better 2 4GB DIMMs.
  • Justin Case - Tuesday, July 17, 2007 - link

    Then why not compare them to 2GB and 4GB registered (non-FB) DDR2 modules? Micron has been making them since 2004 (well, they announced them, at least). There must be other companies making / selling them these days (I'm pretty sure Infineon and Samsung are).

    A quick search turned up Kingston's D51272F51 module (not sure who makes the chips). Not cheap, though (almost $900).

    I think I even remember seeing some reference to an 8GB DDR2 DIMM, but it could have been just a kit (4+4).

    IIRC, FB-DIMM power consumption is roughly 80% higher than regular registered DIMMs.
  • jpeyton - Tuesday, July 17, 2007 - link

    quote:

    yes they are a bit more expensive

    2GB FB-DIMMs are feasible, since they are roughly 2X the price of 1GB modules. 4GB FB-DIMMs are roughly 11.5X more expensive than a 1GB module.

    That might lower power consumption by a noticeable amount, but nothing significant enough to erase the huge gap between platforms.

    Intel really needs to do more work on platform engineering; their gains in CPU efficiency are entirely erased (and then some) by some foolish choices (like going the FB-DIMM route).

    It may not affect the little guys, but companies with server farms or HPC take platform efficiency very seriously (which is why companies like Cray and Sun are adamant about sticking with AMD).
  • TA152H - Tuesday, July 17, 2007 - link

    "Intel really needs to do more work on platform engineering; their gains in CPU efficiency are entirely erased (and then some) by some foolish choices (like going the FB-DIMM route). "

    You're kidding, right? They made a conscious choice to use FB-DIMMS because of the benefits of it, and considered them more important than the negatives. It certainly isn't because they didn't have engineering resources for the platform, because that's exactly what they used to create the FB-DIMM platform. You might not agree with their choice of trade-offs, but using normal DDR2 or DDR3 would be comparitively easy considering they already have chipsets that support it.

    I'm mixed on FB-DIMMS. There's a lot of bad, but there's a lot of good too. I'll agree with you in one way, they do need to provide people a choice. So, in that sense, I don't think they've done a great job on their platform. FB-DIMMs are certainly the best choice in some situations, and certainly not in others. People should decide based on their situation.
  • Alyx - Tuesday, July 17, 2007 - link

    I believe intel processors still hold the edge on performance per watt. In these tests the AMD system had higher performance per watt because of the intel ram. I'd be interested in an intel system with standard ram to get a closer comparison of just the processors. Are intel processors with standard ram uncommon?

    It is illogical that a faster higher performing processor, that does more clock few clock than AMD, which also runs at lower watts would have less performance per watt than AMD.

    Interesting read, I learned a bit from it. Cheers.
  • TA152H - Tuesday, July 17, 2007 - link

    Intel servers currently support FB-DIMMs, so that is the platform that must be tested.

    Also keep in mind you can't measure processors alone for power, although most sites do this incorrectly. AMD processors simply do more, they have a memory controller. So, all things being equal, a processor with a memory controller will use more power, but at the same the chipset would use less. Of course, not everything is ever equal, but my point is, even seemingly simple comparisons like CPU to CPU aren't quite so simple.

    I have read that Intel may be backing off of FB-DIMMS, but until they do, and these folks are testing servers, it's absolutely valid data because that's what you're getting if you buy an Intel based solution. A lot of people have problems with Intel choosing this type of memory, so they might back off of it, or at least offer both. FB-DIMMS obviously have advantages too, so giving people a choice might make a lot of sense.
  • BPB - Tuesday, July 17, 2007 - link

    If energy savings is your aim, and your organization is a large one, then check out http://www.internetnews.com/storage/article.php/36...">this article from last month.



    http://www.internetnews.com/storage/article.php/36...">http://www.internetnews.com/storage/article.php/36...
  • brshoemak - Tuesday, July 17, 2007 - link

    Normally I'm not an editor but:

    quote:

    Internal storage once again comes from one WD1600YD hard drive configured in RAID 0 with the OS installed.
    [under the Test Setup configurations]

    i'm assuming it's two hard drives? or am I missing something because it's early
  • Jason Clark - Tuesday, July 17, 2007 - link

    Fixed.

Log in

Don't have an account? Sign up now