Update on Intel Sapphire Rapids in 2022: Q1 for Production, Q2 for Ramp, H1 Launch

34 Comments

View All Comments

• mode_13h - Sunday, July 4, 2021 - link

  > in my experience FP16 just isn't good enough in training scenarios.
  
  People say BF16 converges faster, but Nvidia did pretty well with FP16 for both the P100, V100, and Turing. They didn't get around to adding BF16 until Ampere.
  
  > 2. AMX will make BF16 matrix multiplications on Sapphirelake much easier compared
  > to Cooperlake since we would only need one instruction to multiply 2 matrix tiles
  
  From what I've seen, it supports only dot-product, not proper matrix-multiply. They also haven't specified the throughput or latency. Don't assume you can issue one of those every cycle.
  
  > Mathematically, you still need to read every single matrix element from RAM at least once
  
  But you can hold one of the matrices on-chip (i.e. the model, which doesn't change) and just read the data that's being forward propagated. Because models are too big to fit entirely on-chip, you break them into pieces and run a batch of data through, before loading and applying the next piece.
  
  The reason this helps is that models are typically much bigger that the data being propagated through them.
  
  And as for AMX and convolutions, it can just load the new elements for each window position. Therefore, it potentially has a much lighter impact on the cache hierarchy.
  
  > Therefore it is fundamentally a data flow and bandwidth constrained problem not a
  > FPU constrained problem
  
  Nvidia's P100 had 720 GB/s of memory bandwidth to support 21.2 fp16 TFLOPS. The V100 increased memory bandwidth to 900 GB/s to support 125 tensor fp16 TFLOPS, and you claim that was a waste???
  
  Given these numbers, an Intel Xeon should be nowhere close to bandwidth-constrained if you're smart about your data movement.
  
  As I said, a lot of AI chips have just DDR4 memory, because they're smarter about using their on-die SRAM to avoid excessive data movement.

• SystemsBuilder - Tuesday, July 6, 2021 - link

  I’m probably not very clear so I’ll try again, being even more technical and hopefully more precise this time – sorry for the long post.
  1. Intel manual (https://software.intel.com/content/www/us/en/devel... page 3-17. TMUL multiplication instruction TDPBF16PS is named dot product BUT in the context of matrix tiles that means tile x tile matrix multiplication, tile being a sub matrix. In other words TDPBF16PS multiplies 2 tile matrices of BF16 floats together and accumulate the result to a third tile – see detailed pseudo code on page 3-18, note in particular the 3 level nested for loop. Like the standard dgemm mm algorithm for multiplying 2 matrices, the full matrix multiplication is a series of tile * tile (in the literature sometimes called block matrices) multiplications going over rows and columns of tiles in the standard way until all tiles are multiplied the right way. Difference between AVX-512 FP32 (besides format) is that TDPBF16PS does a full tile x tile multiplication in 1 instruction while we need about 16 * 16 = 256 vfmadd231ps to achieve a tile x tile multiplication, producing a 16x16 FP32 float tile (ignoring the obvious size advantage BF16 have over/FP32). As both you and I pointed out what remains to be seen is the TDPBF16PS clock cycle Throughput and Latency. vfmadd231ps is a about 4-7 cycles (depending on port 0 or 5 etc.) and you can cut that in half one a CPU with 2 FMAs (high end Xeons have 2 AVX-512 FMAs). We’ll just have to wait for the TDPBF16PS Throughput and Latency numbers to do a final comparison.
  
  But this is just 1 matrix multiplication in isolation… Neural net quires a tons of serially dependent once and that is the crux.
  
  2. The mathematical nature of neural net training (not just inference that is much simpler) – example: basic 3 layer neural net, applying standard forward and backwards propagation. Restraining myself abit here (removing activation function, activation function derivatives, Grad, matrix hadamults, add, subs, scalar mults and other stuff, focusing on the core of the compute complexity: the matrix multiplications themselves).
  a) A forward pass of one sample batch matrix X0, requires W1*X0 -> W2*X1 -> W3*X2 -> X3 sequential and completely serially dependent matrix multiplication (X are the signal matrices, W are the weight matrices, number = layer matrix belong to, -> is the mathematical dependency between two ops). In addition to this you need to calculate and store activation function, activation function derivatives matrices at each layer. This means you need to stream X0 (mini batch) through the compute engine and calculate and store approximately 11-13 matrices each forward propagation pass iteration and, as mentions before, the L1$/L2$ cache is not big enough to keep all of them in cache permanently.
  b) Further when you are finally done with forward propagation you need to take the final layer output (ignoring Grad etc calc stuff again for simplicity) and run the matrix result backwards to calculate error (D): D3W3^T -> D2 -> D2W2^T -> D1 and lastly update the weights(W): D3X3^T -> W3, D2X2^T -> W2, D1X1^T -> W1. Also serially dependent and updated every backwards propagation iteration (one layer compute result is mathematically depended on the previous layer result), in total about another 10-12 matrices are computed and stored.
  c) Lastly you now have to repeat this for the next X0 mini batch until exhausted all samples in the training data set AND then you have to do this last for loop step again hundreds of times (i.e. all EPOCs).
  So the so called “training model” (activation + it’s derivative, W, error, and their deltas), is about 20-24+ matrices for a 3 layer network (depending on the exact flavor of algorithm and optimization used) freshly calculated and updated every iteration based on the next mini batch X0 and can in total take up 10s of MB up to GBs+ – just for the model. And you need them all in each iteration and no one is less important than another, because they are serially dependent of each other.
  My point with this is that math is math. Doesn’t matter how smart you do things unless you change the forward backwards propagation and stochastic gradient decent (or some variation thereof - there are many…) algorithm to something fundamentally different, you are stuck with these matrix calc serial dependencies AND the memory bandwidth bottle neck that comes with it.
  At the macro level the best you can hope for is to keep all activation X, W, Error etc matrices (plus potentially/ideally? “helper” matrices, e.g. delta W/error) in cache since they are updated and read so frequently, but X0 changes with every min batch in the inner loop so… Also W and Delta W matrices tend to be the largest in the entire net of matrices so there is that going on too…
  
  3. we stared this as comparison between intel generations Icelake/Cooperlake and Sapphire Rapids. Comparing with Nvidia's highly parallel and specialized SMID architecture (and different caching all together) with intel's general purpose CPU is is a very different discussion - and a very long one - so I'll stick to the original scope.
  
  4. there is some new news though about Sapphire Rapids that may help to explain how intel have addressed the memory bandwidth challenge to try to keep up with the TMUL. https://videocardz.com/newz/intel-sapphire-rapids-... Sapphire Rapids L1$: 48KB/core, L2$: 2MB/core, L3 cache is 3.75MB/core (75 Mbyte for 20 cores - this is a big upgrade) shared + the HBMe upgrade will help, but what is the latency and B/W going to be?

• mode_13h - Wednesday, July 7, 2021 - link

  Thanks for the link to the Intel docs, but it doesn't work for me. I hope you're right that it can do more than just dot-product.
  
  As for your example, you focused only on training and skipped convolution. When inferencing with convolution layers, the output is much smaller than the weights. So, it's possible to apply a chunk of the weights to a portion of a batch of input data, save the intermediates, and repeat. This saves you from having to reload all the weights for every inference sample. GPUs continue to gain benefits from extremely large batch sizes, up to 512 images, in some cases.

• SystemsBuilder - Wednesday, July 7, 2021 - link

  Here is the link to "Intel Architecture Instruction Set Extensions Programming Reference" talking about the Tiles and Tile Multiplications in chapter 3 with my references about TDPBF16PS above page 3-17, 3-18: https://software.intel.com/content/www/us/en/devel... if it does not work just search for Intel Architecture Instruction Set Extensions Programming Reference or use the link posted in the main article above.
  
  Yes, as I wrote earlier inference is much easier since you only need fully trained W matrices that should be stable and should be possible to keep in cache BUT that is just inference after training is done. Mathematically, inference is basically a subset of training with just one single forward propagation pass a(W1*X0) -> a(W2*X1) -> a(W3*X2) -> X3 per mini batch (if you infer in mini batch format) just calculating activation functions at each layer (a) without calculating activation derivates, Grads, etc and without the entire backwards pass and no iterations. The hard time consuming work is training! Magnitudes more complex than inference.
  In inference you can even go down in resolution from BF16 to Byte and change activation functions to something like much simpler, almost binary 1/0 across the board. And AMX has a byte Tile multiplication: TDPBUUD as well for that! You can define tiles as a signed/unsigned byte data type so now each signal is just a byte long, compressing the format even further from 16 bits to 8 bits. Inference with byte data type should get a 2x speed up from just data format compression.
  
  Convolutional neural network (CNN) is a sub class of general deep neural nets that I talked about in my 3 layer example so what I wrote above in general about training deep neural net applies to them too (broadly speaking) - CNN have very interesting properties that make them very attractive in many specific AI applications - images recognition etc.

• mode_13h - Thursday, July 8, 2021 - link

  Thanks for the updated link. That works. I'll take a little time to update my knowledge of AMX, before posting more about it.
  
  > The hard time consuming work is training!
  
  Yeah, but once you've trained a model, you can inference with it billions of times. Many accelerators specialize on either training or inference. A device needn't be equally competent at both, and inference is the bigger market (also the easier problem).

• JayNor - Tuesday, June 29, 2021 - link

  "Intel is keen to point out that some customers will use DSA, some will use Intel’s new Infrastructure Processing Unit, while some will use both"
  
  The IPU might contain its own DSA, right? My understanding is current generation of IPUs would be expected to have ethernet IO and PCIE IO, plus some ASIC or FPGA to accelerate encode, decode, encryption ...
  
  Will DSA on SPR be doing the CXL coherency DMA transfers?
  
  Will DSA DMA be used for all the GPU to GPU transfers on the Aurora nodes?

• mode_13h - Friday, July 2, 2021 - link

  > Will DSA DMA be used for all the GPU to GPU transfers on the Aurora nodes?
  
  It seems that all of the GPUs are directly connected. So, probably not, unless the GPUs have their own built-in DSAs.

• JayNor - Friday, July 2, 2021 - link

  The GPU coherency is maintained by the host processor home agent, so I suspect all the gpu to gpu data movements on a single node will need to go through the host.

• zamroni - Wednesday, June 30, 2021 - link

  Will it be epycly thread ripped again by ryzen amd?
  
  Just like avx 512, that dl boost is effectively useless for servers because the corporate customers will use gpu or tpu, which is many times more powerful, if they are seriously use ai.
  Intel should simply uses the transistor budget for caches

• EthiaW - Wednesday, June 30, 2021 - link

  Will Intel launch an independent DPU as Nvidia did? They already have the DSA.