SUN’s UltraSparc T1 - the Next Generation Server CPUs

Thread Machine Gun

Besides hard-to-predict branches and high memory latency, server applications on MP systems also get slowed down by high latency network communication and cache coherency (keeping all the data coherent across the different caches; read more here).

To summarize, the challenges and problems that server CPUs face are:

1. Memory latency, load to load dependencies
2. Branch misprediction
3. Cache Coherency overhead
4. Keeping Power consumption low
5. Latency of the Network subsystem

So, how did Les Kohn, Dr. Marc Trembley, Poonacha Kongetira, Kathirgamar Aingaran and other engineers at SUN attack these problems? Let us take deeper look at Niagara or the UltraSparc T1.

Memory latency is by far the worst problem, causing a typical server CPU to be idle for 75% of the time. So, this is the first problem that the SUN/Afara engineers attacked.

The 8 cores of the 64 bit T1 can process 8 instructions per cycle, each of a different thread, so you might think that it is a just a massive multi-core CPU. However, the register file of each core keeps track of 4 different active threads contexts. This means that 4 threads are "kept alive" all the time by storing the contents of the General Purpose Registers (GPR), the different status registers and the instruction pointer register (which points to the instruction that should be executed next). Each core has a register file of no less than 640 64-bit registers, 5.7 KB big. That is pretty big for a register file, but it can be accessed in 1 cycle.

Each core has only one pipeline. During every cycle, each core switches between the 4 different active threads contexts that share a pipeline. So at each clock cycle, a different thread is scheduled on the pipeline in a round robin order; to put it more violently: it is a machine gun with threads instead of bullets.

In a conventional CPU, such a switch between two threads would cause a context switch where the contents of the different registers are copied to the L1-cache, and this would result in many wasted CPU cycles when switching from one thread to another thread. However, thanks to the large register file which keeps all information in the registers and Special Thread Select Logic, a context switch doesn't require any wasted CPU cycles. The CPU can switch between the 4 active threads without any penalty, without losing a cycle. This is called Fine Grained Multi-threading or FMT.

Fig 3: The SUN T1 Pipeline. Source:SUN ^[1].

This Gatling gun of 4 shooting threads per core solves both the branch and the memory latency problem. If a thread issues a load, it takes 3 cycles to get the data from the 8 KB L1-cache. If the 4 threads are active, the thread that issued the load will not be active for 3 cycles anyway, as the other 3 threads get their one cycle. By the time that the first thread is ready to make use of the load of data, the load is finished. If the load takes longer, because it has to access the L2-cache or the memory, the thread is simply skipped, and one of the other 3 threads gets its timeslot until the load is finished.

If a branch is encountered, no branch prediction is performed: it would only waste power and transistors. No, the condition on which the branch is based is simply resolved. The CPU doesn't have to guess anymore. The pipeline is not stalled because other threads are switched in while the branch is resolved. So, instead of accelerating the little bit of compute time (10-15%) that there is, the long wait periods (memory latencies, branches) of each thread is overlapped with the compute time of 3 other threads.

Fig 4: Fine Grained Chip Multi-threading in action. Source:SUN.

The result is a win-win situation: the overall throughput increases, as 4 threads get done in a little bit more time than what it would take to perform one thread on a conventional CPU. In other words, the throughput increases significantly. Secondly, the single issue pipeline is used much more efficiently: SUN claims that the pipeline is used 70 to 85% of the time.

So, one core gets an IPC of about 0.7, which is very roughly twice as good as a big 3-way superscalar CPU with branch prediction and big OOO buffers would do, and it takes less chip logic to accomplish.