Quest Software Benchmark Factory

We mentioned that the benchmarks used previously are no longer useful, as we did not have the I/O capacity required to support them. We went looking for alternative benchmarks and stumbled upon Benchmark Factory from Quest Software. Below is a description of the product and the benchmarks that we used in this article.

Benchmark Factory for Databases is a performance and code scalability testing tool that simulates users and transactions on a database and replays a production or synthetic workload in non-production environments. This enables organizations to validate database scalability as user loads increase, application changes are made, and platform changes are implemented. Benchmark Factory is available for Oracle, SQL Server, DB2, Sybase, MySQL and other databases via ODBC and Native connectivity.

Benchmark Factory provides many tests that you can run, and has a very nice and customizable metric reporting engine. We decided to run the AS3AP test, and the Scalable Hardware CPU, and Reads tests. Here is what Quest's help file says about these tests:

AS3AP

The AS3AP benchmark is an American National Standards Institute (ANSI) Structured Query Language (SQL) relational database benchmark. The AS3AP benchmark provides the following features:

  • Tests database processing power
  • Built-in scalability and portability that tests a broad range of database systems
  • Minimizes effort in implementing and running benchmark tests
  • Provides a uniform metric and straightforward interpretation of benchmark results

Systems tested with the AS3AP benchmark must support common data types and provide a complete relational interface with basic integrity, consistency, and recovery mechanisms. The AS3AP benchmark can test systems ranging from a single-user microcomputer Database Management System (DBMS) to a high-performance parallel or distributed database.

Scalable Hardware

The Scalable Hardware benchmark measures relational database systems. This benchmark is a subset of the AS3AP benchmark and tests the following:

  • CPU
  • Disk
  • Network

It can also test any combination of the above three entities.

What's new with Shanghai? / Test Setup Idle Power and AS3AP Performance
Comments Locked

30 Comments

View All Comments

  • Jason Clark - Thursday, November 13, 2008 - link

    Hi folks, as with most all launches, there are a number of articles we do each on a specific area. Ross and I focussed on database performance under microsoft sql server. I know a VM piece is in the works, just hang on and you'll see something soon.
  • Viditor - Tuesday, December 9, 2008 - link

    "I know a VM piece is in the works, just hang on and you'll see something soon"

    Not to be picky, but it's been almost a month now. Any word on the VM piece??
  • johnsonx - Wednesday, November 19, 2008 - link

    no, you have to have it all done at once, and you have to cover every aspect we could even imagine wanting tests for, or we're going to gripe.
  • Viditor - Thursday, November 13, 2008 - link

    I am surprised that you didn't choose any VM specific benches, as that is an obvious design goal of Shanghai's...
    It would have been nice to see if they were successful.
  • duploxxx - Thursday, November 13, 2008 - link

    that's exactly what i asked, additional Vmware testing like anand did before.

    the overall vmware performance of shanghai 2,7 is already known in VMmark, it totally destroys any x86 intel platform known today even the 6 core dunnington @2,66ghz. but additional tests would be nice.

    http://www.vmware.com/products/vmmark/results.html">http://www.vmware.com/products/vmmark/results.html
  • lplatypus - Thursday, November 13, 2008 - link

    Regarding the delayed introduction of HT3: could it be that these CPUs actually do support HT3, and we're only waiting for support in the chipset?
  • duploxxx - Thursday, November 13, 2008 - link

    Common anand you can do better then that. Where are mysql loads/io/3d measurements and hypervisor testing....
  • JohanAnandtech - Thursday, November 13, 2008 - link

    Please understand that:

    Anand is the CEO and founder of the complete Anandtech, inc. and writer of many desktop/mobile oriented articles.

    Jason and Ross are part of it.anandtech.com and run the Windows/Database tests.

    The hypervisor/ MySQL/Linux stuff is done by myself (Johan De Gelas) and I am still working on our Shanghai review. ESX does not like the new BIOS that makes Shanghai possible, so we won't be able to post ESX numbers before a new BIOS is ready.


  • duploxxx - Thursday, November 13, 2008 - link

    if you would just let us use the edit button :), my usage of anand, was just meant as a short word of anandtech actually, sry for that.

    didn't know you had bios issues, seems like some vendors (dell) had less issues. looking forward to those parts.

    the info like latency is really added value.
  • Proteusza - Thursday, November 13, 2008 - link

    We see some numbers for Shanghai. Any news on when the desktop part will be released/tested?

    I'm pleased it does well in performance/watt, but slightly disappointed that it doesnt beat Barcelona significantly in performance/clock.

    I wonder how well it will fair in games?

Log in

Don't have an account? Sign up now