SQL Stress Tool Benchmark

Our first benchmark was custom-written in .NET, using ADO.NET to connect to the database. The AnandTech Forums database, which is over 14GB in size at the time of the benchmark, was used as the source database. We'll dub this benchmark tool "SQL Stress Tool" for the purposes of discussing what it does. We have done some updates to the tool since we first used it; it now supports Oracle, and MySQL. We also adjusted the test time for this test and future tests to 20 minutes. The reason for this was to ensure that we used as much memory as possible for future planned 64 bit tests.


Click to enlarge.

SQL Stress allows us to specify the following: an XML based workload file for the test, how long the test should run, and how many threads it should use in which to load the database. The XML workload file contains queries that we want executed against the database, and some random ID generator queries that populate a memory resident array with ID's to be used in conjunction with our workload queries. The purpose of using random ID's is to keep the test as real-world as possible by selecting random data. This test should give us a lot of room for growth, as the workload can be whatever we want in future tests.

Example workload:


    Select1
    select count(iuserid) as usercount from ftdb_forumusers where iforumid = 1


    Select2
    select count(u.iuserid) as currusercount from ftdb_users u,ftdb_forumusers fu where fu.iforumid = 1 and u.iuserid = fu.iuserid and dtlastvisiteddate > '[q]qGetLastVisitDate[/q]'


Example Random ID Generator:


    qGetLastVisitDate
    select dtlastvisiteddate,newid() as ldate from ftdb_users where dtlastvisiteddate is not null order by ldate


The workload used for the test was based on every day use of the Forums, which are running FuseTalk. We took the most popular queries and put them in the workload. Functions, such as reading threads and messages, getting user information, inserting threads and messages, and reading private messages, were in the spotlight. Each reiteration of the test was run for 20 minutes, with the first being from a cold boot. SQL was restarted in-between each test that was run consecutively.

The importance of this test is that it is as real world as you can get; for us, the performance in this test directly influences what upgrade decisions we make for our own IT infrastructure.

Web Tests - FuseTalk .NET SQL Stress Results
Comments Locked

144 Comments

View All Comments

  • Opteron - Monday, June 20, 2005 - link

  • mikeshoup - Wednesday, June 15, 2005 - link

    I think a better option for testing compiling speed would be to pass a -j argument to make when compiling FireFox, and tell it to run as many parallel operations as the processor can take threads. IE: -j2 for a dual core or ht cpu
  • fritz64 - Thursday, May 5, 2005 - link

    I know what I will be getting after fall this year. Those numbers are impresive!
  • jvarszegi - Friday, April 29, 2005 - link

    So they're reproducible, but only in secret. And you knew, as usual, about mistakes you were making, but made them anyway to, um, make a valid comparison to something else that no one can verify. Nicely done. Whatever they're paying you, it's not enough.
  • Ross Whitehead - Thursday, April 28, 2005 - link

    Zebo -

    You are correct you can not reproduce them, but we can and have 10's of times over the last year w/ different hardware. I do not believe that because you cannot reproduce them discounts their validity but it does require you have a small amount of trust in us.

    We have detailed the interaction of the application with the database. With this description you should be able to draw conclusions as to whether it matches the profile of your applications and database servers. Keep in mind, when it comes to performance tuning the most command phrase is "it depends". This means that there are so many variables in a test, that unless all are carefully maintained the results will vary greatly. So, even if you could reproduce it I would not recommend a change to your application hardware until it was validated with your own application as the benchmark.

    The owner of the benchmark is not AMD, or Intel, or anyone remotely related to PC hardware.

    I think if you can get beyond the trust factor there is a lot to gain from the benchmarks and our tests.
  • Reginhild - Thursday, April 28, 2005 - link

    Wow, the new AMD dual cores blow away the "patched together" Intel dual cores!!

    I can't see why anyone would choose the Intel dually over AMD unless all the AMDs are sold out.

    Intel needs to get off their arse and design a true dual core chip instead of just slapping two "unconnected" processors on one chip. The fact that the processors have to communicate with each other by going outside the chip is what killed Intel in all the benchmarks.
  • Zebo - Thursday, April 28, 2005 - link

    Ross,

    How can I reproduce them when they are not available to me?

    From your article:
    " We cannot reveal the identity of the Corporation that provided us with the application because of non-disclosure agreements in place. As a result, we will not go into specifics of the application, but rather provide an overview of its database interaction so that you can grasp the profile of this application, and understand the results of the tests better (and how they relate to your database environment)."

    Then don't include them. Benchmarking tools to which no one else has access is not scientific because it can't be reproduced so that anyone with a similar setup can verify the results.

    I don't even know what they do. How are they imporatant to me? How will this translate to anything real world I need to do? How can I trust the mysterious company? Could be AMD for all I know.
  • MPE - Wednesday, April 27, 2005 - link

    #134

    How can it be the best for the buck ? Unless you are seeing benchmarks from Anand that says so how could come to the conclusion?
    At some tests the 3800+ was the worse performer while the X2 and PD where the best.

    You are extrapolating logic from air.
  • Ross Whitehead - Wednesday, April 27, 2005 - link

    #131

    "no mystery unreproducable benchmarks like Anand's database stuff."

    It is not clear what you mean by this statement. The database benchmarks are 100% reproducable and are real life apps not synthetic or academic calcs.
  • nserra - Wednesday, April 27, 2005 - link

    You are discussion price and it's not correct since intel goes from 2800 to 3200 and amd goes from 3500+ into 4000+ (i'm ignoring amd new model numbers, still based on older).

    I complete disagree the AMD model numbers, the should be = to single core, the should just had the X2.

    The TRUE X2 will be more performer than opteron, by 2% to 5%.

Log in

Don't have an account? Sign up now