Constructing a database benchmark (average load)

Our first new benchmark was custom written in .NET, using ADO.NET to connect to the database. The AnandTech Forums database, which is over 14GB in size at the time of the benchmark, was used as the source database. We'll dub this benchmark tool "SQL Loader" for the purposes of discussing what it does.

SQL Loader allows us to specify the following: an XML based workload file for the test, how long the test should run, and how many threads it should use with which to load the database. The XML workload file contains queries that we want executed against the database, and some random ID generator queries that populate a memory resident array with ID's to be used in conjunction with our workload queries. The purpose of using random ID's is to keep the test as real-world as possible by selecting random data. This test should give us a lot of room for growth, as the workload can be whatever we want in future tests.

Example workload:

< workload>

< !--- A SAMPLE WORKLOAD QUERY THAT RETURNS ALL THE FIELDS FROM THE PRIVATEMESSAGES TABLE RANDOMLY --->

<query>

<code>select * from privatemessages where imessageid = @pmessageid</code>

<type>read</type>

<randkey>pmessageid</randkey>

</query>

<!--- RANDOM ID GENERATOR FOR SELECTING RANDOM PRIVATE MESSAGES --->

<randomid>

<rcode>select imessageid,newid() as pmsgid from privatemessages order by pmsgid</rcode>

<name>pmessageid</name>

</randomid>

< /workload>


A screenshot of the SQL Loader

Test Information

The workload used for the test was based on every day use of the Forums, which are running FuseTalk. We took the most popular queries and put them in the workload. Functions, such as reading threads and messages, getting user information, inserting threads and messages, and reading private messages, were in the spotlight. Each iteration of the test was run for 10 minutes, with the first being from a cold boot. SQL was restarted in between each test that was run consecutively.

The importance of this test is that it is as real world as you can get; for us, the performance in this test directly influences what upgrade decisions we make for our own IT infrastructure.

Hyper Threading and The Tests AnandTech Forums Database Test Results
Comments Locked

58 Comments

View All Comments

  • Blackbrrd - Wednesday, March 3, 2004 - link

    Hmm... the site below has some info about Numa (non unified memory architecture), and it looks like the os you're using isn't Numa enabled... Is this correct? Is there any real world benefit from Numa with Opteron?

    http://www.gamepc.com/labs/view_content.asp?id=opt...
  • zarjad - Wednesday, March 3, 2004 - link

    Could you speculate which way the advantage should be going in a BI benchmark (say TPC-H type of a test)? These are long running queries with gigabytes size tables.
  • Jason Clark - Wednesday, March 3, 2004 - link

    We started playing around with a couple of mysql benchmarks a few weeks ago namely OSDB and some new multithreaded benchmarks from MySQL themselves. We're hoping to get some valid tests that produce real results in the future.

    Cheers.
  • Jason Clark - Wednesday, March 3, 2004 - link

    In fact we did some recent testing to start out 64bit linux testing and mysql 4.0.17 on suse 64 had a segmentation fault starting <WINK> known issue for mysql as well... <WINK> <WINK>
  • Jason Clark - Wednesday, March 3, 2004 - link

    Steveoc, it hardly runs like a dog. Let's not turn this into a one sided os war :) The test make sense as they are, but a 64bit article is on the books for later. We've already been playing around with Suse 64bit and some others and whether you agree or not 64bit is still immature, period full stop. Support is there but it has some maturing to do.
  • steveoc - Wednesday, March 3, 2004 - link

    All these tests show is that Opteron, running Windows, runs like a Dog. As if we couldnt predict that result already ...

    The tests will only make sense once you are running 64bit linux. In fact, Id love to see a test of Dual Xeon + Win2003 + MSSQL vs Dual Opteron + 64bit Gentoo + 64bit MySQL .. that would be very interesting indeed.

    For anyone out there claiming that '64bit software has a looong way to go', that is only true for Windows. Unix (and Linux) have been running 64bit for a long time now, and the AMD64 has very good support under Linux.
  • dweigert - Wednesday, March 3, 2004 - link

    Seeing the difference whether NUMA us used or not would be *VERY* interesting. Also comparing against other NUMA aware OS's (Linux 2.63 or better kernel, or whatever) would be a good test too.
  • hirschma - Wednesday, March 3, 2004 - link

    #25 - Seems that it is not for sale to the general public, not that I could find. If anyone knows where/how to get one, please let me know.

    I have an application that is quite expensive and is licensed by the box, no matter how many CPUs it has ;) I'm guessing that building a low-end quad would give me more throughput per $$ than a second license/second box.

    Jonathan
  • Jason Clark - Wednesday, March 3, 2004 - link

    We're also looking at some 64bit .NET benchmarks as we're real close to having a real-world application that we can hammer.
  • Jason Clark - Wednesday, March 3, 2004 - link

    An interesting article would be the effect of NUMA on enterprise level applications. GamePC did a bit of a write up on it, but it was limited to desktop and synthetic benchmarks. Would any of you be interested in seeing the effects of NUMA on and off on the sql tests?

Log in

Don't have an account? Sign up now