Chipsets (con't)

We have mentioned this in the past, but Opteron severely differs from Xeon due to the HyperTransport links from processor to processor, and that the memory controller is on the processor die. With each processor sharing a 6.4GB/s link to two other processors, and indirectly their memory banks, a four-way Opteron configuration does not get bottlenecked on a memory controller or Northbridge.

Most high end dual and quad Opteron solutions use two AMD 8131 PCI-X tunnels to control IO off the processors, but they go a step further by daisy-chaining a third 8131 tunnel to the secondary tunnel (which is why the V40z can utilize four 64-bit PCI-X 133MHz slots and still have enough IO for the integrated controllers). The HyperTransport specification details that five devices can be within an HT chain, so having two 8131 PCI-X tunnels daisy-chained is clearly within spec. A brief block diagram of our V40z can be found below.

Click to enlarge.

There is enough headroom on primary PCI-X tunnel for two more 100MHz PCI-X channels; and a third and slowest channel runs in tandem with the gigabit Ethernet and SCSI controller. Browsing through some online documentation revealed that the seventh, 66MHz PCI-X card shares bus resources with the LSI and Broadcom controllers, which means inserting a 33MHz PCI expansion card in the bus will reduce the operating bus of the SCSI and gigabit Ethernet controllers. Considering the 800MB/s headroom on that particular bus, it would seem like a poor choice to install a high bandwidth PCI device in the 66MHz interface anyway. Likewise, using a 66MHz and a 100MHz PCI-X device in tandem on the primary 8131 tunnel will result in both buses slowing to 66MHz.

AMD’s 8111 I/O Hub is linked off the primary 8131 PCI-X tunnel, and from there, most of the basic system functions are controlled including the XGI graphics adaptor. Even though XGI hasn’t been particularly popular on the desktop, Trident’s penetration into the server market solidified XGI’s server market share. The majority of the features on the AMD 8111 remain disabled, like AC’97 audio and the integrated 10/100 Ethernet controller.

Two Broadcom BCM703 chips control the external gigabit Ethernet for the server, but there are also two 10/100 out-of-band Ethernet ports that we will go into more depth later. Winbond provides the rest of the basic functionality of the machine not handled by the AMD 8111 I/O hub. LSI’s 53C1020 Ultra320 SCSI adaptor provides the V40z with the onboard SCSI.

Click to enlarge.

If the soul of our V40z is the Opteron 850, then the temporal lobe would be the Motorola MPC855T Service Processor. The Motorola MPC855T is a particularly interesting chip that we saved an entire page for more detail.

Detailing the Chipsets System Management: Another Linux Success


View All Comments

  • dougSF30 - Tuesday, February 22, 2005 - link

    Typo page 1: "In January, Sun sent us a V40z demonstration unit that was complete with four Opteron 250s and 8GB of PC2700."

    It should be "four Opteron 850's"

    Also, page 3: "The older 130nm "CG" stepping on Opteron 8xx only allows for PC2700 memory"

    This is not true, generally. I don't know about the v40z, but CG Opterons can use PC3200 no problem.

    See here, for example:

  • Ardan - Tuesday, February 22, 2005 - link

    I agree with #16. Fantastic work, Kristopher! I have a family member that works for a division of Lockheed-Martin and they are dealing with Sun more and more now and he enjoys your articles when I show them to him. He said a few weeks ago that he shows some of these articles to co-workers because of the thorough evaluations and it is helping them to decide whether or not to outfit their systems with Serial ATA drives (for Destroyers) and now with Opteron systems from Sun.

    Good work! ;)
  • tfranzese - Tuesday, February 22, 2005 - link

    Excellent work Kris. Reply
  • KristopherKubicki - Tuesday, February 22, 2005 - link

    Something is wrong with the DB graphs: i am fixing it now.

  • Googer - Tuesday, February 22, 2005 - link Reply
  • Googer - Tuesday, February 22, 2005 - link

    This server is aperantly made by N Reply
  • Googer - Tuesday, February 22, 2005 - link

    OOPS, this is suppost to go where the blank post is.

    To the author or whom it may concern: A bad link is located on page 3 and reads:

    two Samsung 1GB PC2700 – link to Samsung.jpg>
  • Googer - Tuesday, February 22, 2005 - link

  • Googer - Tuesday, February 22, 2005 - link

    I want Game Benchmarks too! (just for fun though) Reply
  • nourdmrolNMT1 - Tuesday, February 22, 2005 - link

    database benchmarks arent workin here either.


Log in

Don't have an account? Sign up now