"Order Entry" Stress Test: Measuring Enterprise Class Performance

One complaint we've historically received about our Forums database test was that it isn't strenuous enough for some of the Enterprise customers to make a good decision based on the results.

In our infinite desire to please everyone, we worked very closely with a company that could provide us with a truly Enterprise Class SQL stress application. We cannot reveal the identity of the Corporation that provided us with the application because of non-disclosure agreements in place. As a result, we will not go into specifics of the application, but rather provide an overview of its database interaction so that you can grasp the profile of this application, and understand the results of the tests better (and how they relate to your database environment).

We will use an Order Entry system as an analogy for how this test interacts with the database. All interaction with the database is via stored procedures. The main stored procedures used during the test are:
sp_AddOrder - inserts an Order
sp_AddLineItem - inserts a Line Item for an Order
sp_UpdateOrderShippingStatus - updates an status to "Shipped"
sp_AssignOrderToLoadingDock - inserts a record to indicate from which Loading Dock the Order should be shipped
sp_AddLoadingDock - inserts a new record to define an available Loading Dock
sp_GetOrderAndLineItems - selects all information related to an Order and its Line Items
The above is only intended as an overview of the stored procedure functionality; obviously, the stored procedures perform other validation, and audit operations.

Each Order had a random number of Line Items, ranging from one to three. Also randomized was the Line Items chosen for an order, from a pool of approximately 1500 line items.

Each test was run for 10 minutes and was repeated three times. The average between the three tests was used. The number of Reads to Writes was maintained at 10 reads for every write. We debated for a long while about which ratio of reads to writes that would best serve the benchmark, and we decided that there was no correct answer. So, we went with 10.

The application was developed using C#, and all database connectivity was accomplished using ADO.NET and 20 threads - 10 for reading and 10 for inserting.

So, to ensure that IO was not the bottleneck, each test was started with an empty database and expanded to ensure that autogrow activity did not occur during the test. Additionally, a gigabit switch was used between the client and the server. During the execution of the tests, there were no applications running on the server or monitoring software. Task Manager, Profiler, and Performance Monitor were used when establishing the baseline for the test, but never during execution of the tests.

At the beginning of each platform, both the server and client workstation was rebooted to ensure a clean and consistent environment. The database was always copied to the 8-disk RAID 0 array with no other files present to ensure that file placement and fragmentation was consistent between runs. In between each of the three tests, the database was deleted, and the empty one was copied again to the clean array. SQL Server was not restarted.

AnandTech Forums Database Test Results Order Entry Stress Test Results
Comments Locked

44 Comments

View All Comments

  • karlreading - Monday, September 13, 2004 - link

    #31
    whilst i agree with what you are saying in principle, perhaps a polite email to the author(s) of any article(s) pointing out the mistake(s). Would this not be more appropriate? I personally find that when one person points out such mistakes on a public forum, another 10 people throw in 2 pence worth and before you know it everybody starts to pick holes and loses site of the actual topic - like i have now so i'll shut up!!! :)
  • Jason Clark - Monday, September 13, 2004 - link

    In regards to the motherboard questions, we used a Thunder K8W board, I'll ensure the article reflects that. We did some testing on Numa, and as of yet see any difference at all in numbers.

    Cheers
  • Jason Clark - Monday, September 13, 2004 - link

    Since the broke wind pun seems to bring about so much attention.. I've changed the wording :).

  • johnsonx - Monday, September 13, 2004 - link

    @karlreading,

    I may be wrong, but I'd like to think those who write these articles actually appreciate polite corrections to grammar & wording missteps. I mostly don't bother unless the mistake either leads one to a conclusion other than what was intended, or if the mistake is potentially embarassing as is the case for 'broke wind'. Most websites contain such constant and gross errors of this kind that they are unreadable, but AT maintains a pretty high standard as all serious publications, electronic or printed, should. In most cases where folks have pointed out a mistake in an article, the AT staff has seen fit to correct it.

    No one has suggested that the article is less valid because "slightly suspect use of wording".

    All that said, for all I know the author(s) may have intended the wording used, or may have decided they like the unintended wording even if it is a bit 'suspect'.
  • CrystalBay - Monday, September 13, 2004 - link

    Nice read JC, nice replies #6 and #10...
  • karlreading - Monday, September 13, 2004 - link

    Can i just say, i wish all you moaning gits who are ripping the guys grammer and use of words apart could just pipe down. The guys have done a sterling job with this review and if i'd done some hardcore benchmarking session then id prolly be so tierd that i would be a bit fuzzy with my write up. Your just like the stupid kids in class that have to put there hands up and shout " miss, you spelt that wrong!! ". what does it achieve, not a lot, thats what!!!
    Lets try to appreciate the article for its content and commend Jason and Ross for delivering us with a good comparitive benchmark set rather than sit there nit picking about a slightly suspect use of wording!!!
  • johnsonx - Monday, September 13, 2004 - link

    Also, if in fact the board used on the Opteron was the Thunder K8W, I'd like to ask a further question:

    Did you enable cross-processor memory interleaving? (sorry, I forget what it's called in the BIOS) In this mode, memory access is interleaved across both memory banks, effectively yielding a shared 256-bit memory bus which increases performance for UMA operating systems.

    I think Windows 2003 also supports a form of NUMA, so if you *disable* the cross-processor interleaving (and get a couple of other settings right as well), then Windows will try to keep threads and data local to the processor executing them, which should increase performance in some situations. There's a bunch more terminology to this, about how the BIOS passes a configuration table to the OS which tells it which memory is local to which processor, and what other conditions have to be met in the BIOS settings (something about background ECC scrubbing if I recall correctly).

    I have to admit that I've only setup 1 Dual-Opteron Windows 2003 Server (on a Thunder K8S Pro), and I didn't completely understand all the ramifications of the above, so I just enabled the cross-processor memory interleaving (which thus disabled any NUMA support). I figured that was probably the safe bet.

    If AT could investigate this and shed some light on it, it would be most interesting.

    Dave
  • johnsonx - Monday, September 13, 2004 - link

    @Mino and Jason Clark,

    The Tyan K8W board was specified in the article; but there are two very different K8W's. The Tiger K8W is a workstation board with a single bank of memory slots; the second CPU does ALL memory access via HT. The Thunder K8W is high-end workstation & server board with memory slots for both CPUs; the only difference between it and a 'server only' board is the Thunder K8W has an AGP 3.0 tunnel and slot where a 'server only' board like the Thunder K8S Pro does not.

    I'm guessing that the Thunder K8W was used for the test, but Mino thought only of the Tiger K8W.

    Perhaps clarification should be made?
  • johnsonx - Monday, September 13, 2004 - link

    I opened the comments page just to see if anyone had already commented on this, and I see Viditor beat me to it, but still..

    "When AMD first broke wind with the K8 announcement..."

    Yikes. I'm really sure that isn't the phrasing you were looking for. No, this is some sort of Beavis & Butthead "heheheheh he said 'wind' hehehehe" thing; AFAIK "break wind" means one thing and one thing only, and it just can't be used the way you tried to.
  • daveshel - Monday, September 13, 2004 - link

    "So, we've seen AMD compete on both the desktop and server market, but does this transgress into a victory in corporate America?"

    Trans- something, I guess.

Log in

Don't have an account? Sign up now