Original Link: http://www.anandtech.com/show/808
Behind AnandTech - APPRO's 1124 760MP Server: The Perfect Server?by Anand Lal Shimpi on August 7, 2001 8:44 PM EST
- Posted in
- IT Computing
In all honesty, most of us at AnandTech expected the Dual Xeon platform to outperform the Dual Athlon MP in our database server tests from our 760MP review. Although database server environments do benefit from more and faster CPUs, if you ignore the memory and I/O on a database server regardless of how fast your CPUs are, you’re going to have a poorly performing solution. In database serving environments, especially ones as data intensive as the AnandTech Forums Database which is currently over 3GB in size, moving data around quickly is key to high server performance. The Intel Xeon’s i860 chipset provided a 400MHz data bus from the CPU to the Memory Controller Hub (MCH), offering 3.2GB/s of bandwidth to the dual processors, and the MCH also offered a dual channel RDRAM memory controller offering an equal 3.2GB/s of bandwidth to the memory. To make the platform even more attractive, the i860’s MCH includes a small cache to improve performance as well.
In spite of all of this, when placed in the very environment of our AnandTech Forums Database Server, a Dual Intel Xeon 1.7GHz system was outperformed by an identical Dual Athlon MP 1.2GHz setup. And thus came the following conclusion to our 760MP review back in June:
"If you can't tell by now, we're very impressed with the AMD 760MP chipset. In fact, one of the reasons we spent so long on this comparison is because the outcome of our results would influence what platform we used in our next database server for the AnandTech Forums. The choice is simple; the AMD 760MP is the DP workstation and server platform to have. It's reliable, it's high performing and it's very flexible; everything you'd expect from an Intel based server solution, except that it's from AMD instead."
Fast forwarding to the present day, we like to stand behind our recommendations at AnandTech and since then two of our in-lab systems have been moved to 760MP yet we still hadn't upgraded our "aging" Forums Database Server.
The word aging appears in quotes because our Forums Database Server has not been up in our Pittsburgh Colocation Facility more than 6 months; we upgraded it in January of this year. Since then, the number of registered users on the AnandTech Forums has increased tremendously; today the Forums are close to 70,000 registered users of which, 600 – 700 are logged in at once, and 2,000 more are browsing at any given time.
The database itself has close to 300,000 threads in which over 3 million messages are posted. Behind closed doors are 1.9 million private messages that are exchanged among the members. The database itself, as mentioned before, is over 3GB in size - how big is the largest file on your hard drive?
While the I/O performance of our Forums Database Server was up to par, it was clear that the bottleneck of the server was becoming the CPUs. Our Dual Pentium III 866s were pegged at 90%+ CPU utilization, across both CPUs. Think about the daily tasks you run on your computer, for the most of us, it's very rare that we can completely max out two CPUs doing something other than RC5 or heavy mathematical/rendering calculations. It was clear that the Database Server required an upgrade, and we had already found the best candidate for the upgrade – the real trick was in the approach.
When we first decided upon the 760MP platform to use as our Database Server solution we realized that there were going to be a few hurdles that we'd come across. First of all, at the time and to this day there is still only one 760MP motherboard available on the market. That motherboard is the Tyan Thunder K7 which we featured in our review of the 760MP platform. Now there's nothing wrong with the Thunder K7, in fact there is quite a bit that's right about the Thunder K7 but it does have its quirks.
First of all, the board requires the use of a custom power supply. According to AMD, at the time they were designing the reference 760MP motherboard Intel's spec for the 24-pin power connector featured on their Dual Xeon board had not been finalized so AMD went with whatever data they did have. The final spec ended up having two pins of the connector flipped, rendering AMD's implementation incompatible with Intel's. Although this didn't make much sense to us because the Xeon and the i860 platform did launch before the 760MP, we're just presenting AMD's side of the story. AMD has since told us that they will be moving to Intel's specification. Unfortunately this means that everyone that purchased the very expensive custom power supplies for their Thunder K7 boards will never be able to use them with another motherboard again.
Another issue with this custom power supply is the incredible power ratings on the unit. The NMB unit we tested with originally had a 460W rating, but the only reason for the 460W requirement was because of the fact that Tyan wanted to achieve AGP Pro110 compliance on their motherboard. The AGP Pro110 spec states that you must be able to deliver 110W of power to the AGP Pro slot which tailors to a highly specific market since very few AGP Pro cards require that much power. For those users that don't require the AGP Pro110 compliance, the added power is almost completely wasted.
Tyan has fixed this issue with the upcoming lower cost version of their Thunder K7 that will only feature a regular AGP slot and will thus work with regular 20-pin ATX power supplies. Other 760MP motherboard manufacturers will produce boards that work with regular ATX power supplies as well.
Because of the limited power supply options, we were almost forced to go with a larger 4U or 5U chassis so we could use a full sized power supply.
The second limitation the board imposed on us was the requirement for Registered ECC DDR SDRAM. Since there are only four DIMM slots on the Thunder K7 itself (a limitation of the chipset), we had to use the highest density DIMM modules we could find. Although it's quite easy to come by 256MB Registered ECC DDR SDRAM, that would only leave us with 1GB of memory in the server, a downgrade from our previous 1.5GB setup; we'd have to get 512MB modules.
It turns out that there are only two modules that work particularly well with the Thunder K7, and only one of those are available in 512MB parts. Samsung 256MB modules and Corsair 512MB modules have been tested and work well with the Thunder K7 in a fully loaded environment running in very stressful conditions. The web's favorite, Crucial, didn't have any 512MB Registered ECC modules either so there was no taking advantage of their very aggressive prices.
We instead put in a call to Memman and Corsair and managed to secure 4 x 512MB Registered ECC DDR266 modules for the server, a 33% increase in memory size – those of you that have had experience with heavily loaded database servers will know that more memory can always be of use.
Finally cooling was a major issue. Our Dual Pentium III 866s together produced less heat than most of today's higher clocked single Athlon processors, but migrating to a dual Athlon platform would definitely require some heavy cooling.
On the other side of the country…
Meanwhile, VA Linux Systems, was working very closely with AMD and Tyan to produce a 1U 760MP server based on the Thunder K7 motherboard. Requiring an experienced member of the industry to help manufacture the servers, VA Linux teamed up with APPRO International, a 10 year old rackmount server manufacturer based out of Milpitas, California. Just before the product was ready, VA Linux announced that they were pulling out of the hardware market before the end of their fiscal year, which ended on July 28, 2001. This left APPRO with the entire burden (not necessarily a bad thing) of completing, producing and promoting this 1U 760MP server.
We already knew how well the 760MP platform performed; the real question was how well designed would APPRO's 1124 Athlon MP Server be?
The APPRO 1124
The first thing you notice about the APPRO 1124 is its incredible depth; the 1U chassis is 17.1" wide and with its integrated sliding rails it can fit in a 19" rack just like any other rackmountable case. The chassis is over 27" deep which is a bit unusual for 1U cases but taking a peak under the hood explains what all the room is for.
The externals of the chassis are both attractive in functionality and in form. The front panel is home to the bare essentials; power and reset buttons, power, disk and warning lights are all commonplace on a server. It is clear that the 1124's chassis was custom tailored to the Tyan Thunder K7, made evident by the two network activity lights that report activity on the dual on-board 3Com NICs of the motherboard.
Just below the row of lights and switches are the four hot swappable drive bays. The 1124 features a SCA backplane that connects directly to the Tyan's on-board Adaptec Ultra160 SCSI. For those of you that aren't familiar with the SCA SCSI specification, SCA is a connector specification that stands for Single Connector Attachment. As the name implies, SCA devices only require a single connector that handles the interface, power and all jumper settings and connections. Whereas most Ultra160 SCSI drives feature 68-pin interface connectors and require a 4-pin power connector as well as jumpers to be set, SCA drives use a single 80-pin connector to handle all of that. Using a special controller board, these drives become "hot swappable" meaning that they can be added and removed without shutting down the machine, which is very important in a server environment to minimize downtime in the event of a drive failure.
While most 1U cases would stop at the four drive bays, the custom made 1124 chassis includes a slim 24X CD-ROM drive much like what you'd find in a thin and light notebook. By no means is a CD-ROM drive required in a server, but this is definitely an appreciated convenience.
We mentioned that the 1124's chassis was attractive in its functionality as well as its form; we've already discussed the latter as the case is very sleek and has a very convenient integrated CD-ROM drive, but what about its functionality?
The case cover is made up of three panels, two of which require no tools to remove. The first of these panels is at the rear of the chassis and is secured to the case via two thumbscrews. This is the largest panel and covers the Tyan Thunder K7 motherboard as well as the power supply. Removing this panel reveals some of the eloquence of the APPRO design that is visible from the custom power supply. The 1124's power supply is a 600W unit that extends over half of the length of the chassis. Obviously the reason for the powerful unit is to accommodate the maximum of four hard drives and power hungry Athlon MP CPUs. One thing APPRO didn't take into account is that in the 1U chassis there is no way to take advantage of an AGP Pro110 card, much less an AGP card itself meaning that factoring in the 110W AGP power requirement was unnecessary. APPRO is aware of this and they will be modifying the power supply accordingly in the future.
There is only one usable PCI slot due to height constraints as the PCI slot is adapted using a riser to accept horizontally mounted cards. Luckily the fact that the motherboard features dual on-board Ethernet controllers, on-board video and on-board Ultra160 SCSI means that you'll probably not need more than one PCI slot. In our case, the sole 64-bit/33MHz PCI slot was used for our RAID adapter.
The Tyan Thunder K7 occupies the majority of the room beneath this first panel, and the board itself is relatively unchanged from when we first looked at it back in June. The only difference, and a very important one at that, is that the AMD 762 North Bridge now features a heatsink atop its heat spreader. As we noted in our original 760MP review, the 762 North Bridge became extremely hot during normal operation and without a heatsink to help dissipate the heat it'd be tough to keep the platform running stably in such a cramped area.
Cooling is a major issue for this platform and in a 1U chassis that is less than 2" high, the issue is even more serious. There isn't enough headroom for a regular sized heatsink, much less one with a fan mounted on it. Because of this, the 1124 also implements a custom made cooling solution to not only circulate air through the extremely small case, but to also cool the dual Athlon MP processors.
What’s behind door number two?
The second panel that covers the 1124 is also removable without any tools. You simply depress two buttons allowing the panel to slide back a bit, then it is simply tilted and lifted off. Beneath the panel lay four blowers that provide the cooling necessary to keep case temperatures at a minimum.
The way the four blowers work is simple; they are mounted slightly off of the bottom of the chassis, so they can pull in air from the area below and behind them. As luck would have it, behind the blowers are the four hot swappable drive bays with ventilation holes at the front of the case. The blowers take in cool air from the outside, and in the process they manage to cool the four drive bays. The blowers are pointed at the back of the case, towards the motherboard and the CPUs.
One of the blowers is placed in front of an opening to the motherboard compartment in the case; this blower provides air circulation in the motherboard compartment.
The other three blowers are blocked off from the motherboard compartment and are all blowing into a channel adjacent to the power supply. This channel takes the quickly moving air and uses it to cool the power supply, as well as cool the CPUs via two ducts that have openings next to the CPUs.
Combined with the large surface area heatsinks on the processors, the cool air is sent parallel to the heatsink fins thus removing the heat from the CPUs. In order to prevent the heat from simply rising and not being pushed away by the stream of air, the heatsinks are covered with cardboard jackets to help channel the air.
In the case of the CPU closest to the 762 North Bridge, the air flowing over the bottom surface of the heatsink is used to cool the 762’s heatsink as well. Finally the air is exhausted out the rear of the chassis.
How effective is this cooling setup? Although quite loud, it does its job very well. Our Forums Database Server makes use of four 10,000RPM Quantum Atlas 10K Ultra2 SCSI hard drives operating in a RAID 10 configuration. All four of the drives are constantly being read from or written to, and if you’ve ever had any experience with 10,000RPM SCSI drives, after even 30 minutes of use inside a case you definitely don’t want to handle one. Removing the drives after being under heavy use for quite a while revealed that the surface of the drives was barely warm to the touch; the APPRO cooling setup had done its job well. The heat being exhausted from the case was quite warm, putting the burden on the server cabinet to have good cooling to get rid of the hot air.
And finally, door number three
The last panel that covers the 1124’s chassis is the only one that’s screwed in place, simply because there is usually no reason you’ll need to gain access to what’s lies beneath it.
The slimline 24X CD-ROM drive is below this panel, as well as the controller board that interfaces with the Thunder K7 to bring the lights and switches to the front of the case; the board also interfaces with the hot swap backplane.
A floppy power connector and floppy cable are also located under this compartment should you need to hook up a floppy drive. Again, not necessary, but definitely very convenient.
The Perfect Server?
If you’ve read up to this point you’ll realize that every single problem we anticipated having with the 760MP platform has been addressed by APPRO with the 1124.
The performance of the server is incredible, but we must accredit that to AMD and Tyan for putting together such an incredible combination. The biggest limitation we encountered was the maximum of 4 memory banks on the motherboard; unfortunately this limitation won’t be addressed until platforms supporting AMD’s upcoming 64-bit x86-64 Hammer solution arrives next year.
The size of the 1124 makes it very attractive for use in datacenter environments where the more you can fit into a rack, the cheaper your average costs get. The 1U form factor of the 1124’s chassis is made possible in part by the highly integrated nature of the Thunder K7 motherboard.
The dual on-board 3Com Ethernet controllers are perfect for servers since they allow you to maintain a public connection to the Internet and a private LAN connection as well. In our situation, the private LAN connection is perfect since we want all of our web servers to communicate with the database server across our backend 100Mbps full-duplex private LAN instead of going through the external network. The on-board ATI Rage XL video isn't useful for 3D acceleration, but in a server environment where most administration is done remotely, high end graphics isn't too important. And although we didn't make use of it, the on-board Adaptec Ultra160 SCSI controller is useful if you're not going to be doing any I/O intensive serving. In our case, we took advantage of the single PCI slot and installed an AMI MegaRAID controller to drive our RAID 10 array.
We must give credit to APPRO (as well as VA Linux) for designing the case that truly carries a bigger bite than its bark would let off. When moving it into the datacenter during last week's server upgrade no one had an idea just how powerful this tiny system was. Looks can be deceiving and with the APPRO 1124 you'd hardly guess that enclosed in the 1U chassis is the most powerful dual processor x86 platform money can buy.
AMD provided the platform to do it, Tyan supplied the motherboard, and APPRO added their elegant touch to make this truly the perfect server; it shouldn't be much of a surprise as to exactly why we chose it to run our most demanding database serving application.
To all of our faithful readers and AnandTech Community members, we hope you enjoy the speed.
Behind AnandTech - The Server Pictures
Other than the addition of this new database server, nothing much has changed in the datacenter. If you're interested in seeing other behind the scenes images of what our servers look like, have a glance at our three other Behind AnandTech articles listed below:
Behind AnandTech - 2001 Server Upgrade - 4/17/2001
Behind AnandTech - Upgrading the Server Farm - 1/16/2001
Below you can see both of our racks with doors wide open. The black cases in the rightmost rack are mostly AnandTech Forums web servers. The white cases in the center rack are AnandTech web/mail servers, and the empty space between them is where one of the larger 5U cases was removed and replaced with the very small 1U APPRO server. You can barely see it in comparison to the larger 4U and 5U racks.
Just for comparison purposes, if all of our servers were moved to APPRO's 1U platform we'd take up no more than half of a rack instead of the one and a half racks we currently use. It's too bad no Athlon motherboards other than the Tyan Thunder K7 will work in a 1U chassis, not to mention that finding 1U Athlon compliant power supplies is very difficult. If all of our web servers needed the power of the 760MP it'd be a different situation entirely.
Here is a better gauge of exactly how much smaller this 1U box is in spite of its performance advantage over our much larger solutions. That's me behind the rack about to take off the rear panel to install the 2GB of DDR SDRAM that arrived at the datacenter just in time to make it in before we left.
Another day, another server upgrade. At this point, our DB upgrade options are limited but in the future we'll be considering Intel's Xeon MP processor (4-way), faster Athlon MP solutions, as well as AMD's Hammer.
With the addition of APPRO's 1124 we can no longer say that all of the servers were assembled by AnandTech, but because of how great of a server it is we'll let that one slide. Other than the 1124, all of the servers running AnandTech were assembled in-house. The servers are connected on a private network courtesy of one Ethernet card in each system, as well as a 24-port 100Mbit SMC switch.
None of this would have been possible had it not been for our excellent host, Elite Internet Communications and their colocation datacenter with Stargate. If you're looking for a host, Elite is the best we have ever had and for those of you that have been long time visitors of AT you'll know that we've seen them all.
These companies helped us tremendously in this most recent server upgrade and we owe them our thanks:
We'd also like to thank the following companies for helping us with our servers in the past:
AMD - http://www.amd.com/
Azzo - http://www.azzo.com/
Corsair - http://www.corsairmicro.com/
Intel - http://www.intel.com/
Linux Virtual Server Project - http://www.linuxvirtualserver.org/
Memman - http://www.memman.com/
MSI - http://www.msi.com.tw/
Mushkin - http://www.mushkin.com/
NewEgg - http://www.newegg.com/
PogoLinux - http://www.pogolinux.com/
SMC Networks - http://www.smc.com/
TC Computers - http://www.tccomputers.com/
Tyan Computers - http://www.tyan.com/
And, of course, a huge thanks to the readers that make AnandTech what it is on a daily basis. Thanks guys.