DL Boost and New Instructions: Intel’s AI Acceleration Attack

If you like buzzwords, then the field of artificial intelligence is full of them. Unless you are deeply embedded in the use or development of the technologies in this field, it can be pretty easy to be overwhelmed by all the acronyms and how they relate to each other. This complication is multiplied out by the new technologies that hardware manufacturers develop in order to support the methods the expert’s use, especially as those methods transition from software implementations to hardware accelerated methods.

Intel’s side of the equation on this is two-fold. On one level, the company wants to accelerate how these algorithms are trained – Intel wants to be able to provide the raw horsepower and bandwidth to push millions of results to train these algorithms in order to improve accuracy and efficiency. Training on this scale typically happens at the datacenter, and Intel will happily promote its FPGAs or Neural Network Processors to help do this.

The other side of the discussion is inference – the art of taking those trained algorithms and showing them new data to make accurate predictions. The simple example of inference is showing an algorithm that has been trained to identify different objects a new image that it has never seen before. Inference, in contrast to training, happens at the device level, is less computationally strenuous, but can be optimized to a super low power hardware accelerated implementation. There can also be a tradeoff between speed/power and accuracy. Again, the most obvious example of this is a smartphone identifying that the camera is pointed at food, or a sunset, or a building, or a cat.

The two sides of artificial intelligence, training and inference, happen at very different ends of the spectrum. Training occurs at the application developer level, while inference happens very much at the user level. For products that ultimately end up in the hands of end-users, being able to accelerate inference is a key to user experience.

However, Intel has a problem here. In fact, all companies that sell mobile PC products to end-users have a problem. The concepts of interference, such as identifying an item on a camera view screen, are easy to understand in a smartphone form factor. But apply that logic to a laptop, a notebook, or a desktop, and the applicability of how AI can improve the user experience starts to get a little distorted. It is not immediately obvious in what ways that AI can improve what a user does on these devices. The smartphone, by contrast, is an easier sell.

As a result, Intel has needed to go out into the community. Its new chip, Ice Lake, is a sledgehammer, and it’s looking for a nail, no matter how small. The truth of it is that a lot of end-user software does not have artificial intelligence built into it – any ‘intelligence’ that most software has is typically a bunch of ‘if this then that’ statements which do not fall in this category.

This ‘hammer looking for a nail’ implementation was very apparent at Computex 2019. Intel showcased a few demos with its new technology, however most of them were fairly niche. Two examples of this include:

  • Photo Album sorting: searching for all ‘beach’ photos in a photo album that hasn’t been manually tagged.
  • Converting a single 2D image into a 3D model for use in 3D visualization environments

The first case might be interesting to the wider public – and indeed Apple already does something like this on macOS – however the second use case is currently aimed at business or archival use, which is a specific niche. Ultimately Intel is looking to increase the amount of available tools, even if they are niche, which can be accelerated on Ice Lake. Intel is taking all suggestions on this.

How Intel is accelerating these inference models is through its ‘DL Boost’ technology, which is essentially a form of vector acceleration.

DL Boost and AVX-512

One of the features of the Ice Lake cores is that silicon is dedicated to AVX-512 operation. This allows for 512-bit instructions and math to be executed at once. With Ice Lake, it marks the first consumer platform from Intel to support the feature, if you exclude Cannon Lake which was not widely available. To date only Intel’s Xeon Skylake or newer silicon, or Xeon Phi, has had AVX-512 as part of the design. As we saw in our review of Cannon Lake, when AVX-512 code is run on these systems, it can be very effective: a dual core system with AVX-512 outscored a 16-core system running AVX2 (AVX-256).

AVX-512 forms the basis for Intel’s DL Boost libraries, which form part of several SDKs in Intel’s software strategy, such as the OpenVINO toolkit. DL Boost takes advantage of the new core’s abilities to reform neural network calculations in the form of lower precision calculations, from FP32 to FP16 to INT8 and even down to INT2. The balance here is between accuracy, power, and computational complexity: for example, in most non-critical situations, dropping the accuracy of an inference calculation from 94% to 93% accuracy would be worth it if it reduced latency and power consumption by half. DL Boost accelerated libraries also allow for mathematical numerical transforms in neural network calculations to further increase throughput and decrease power.

Ultimately DL Boost covers the gamut from training to inference – Intel promotes its use on the server Xeon product line for training, whereas the Ice Lake consumer line of products will be focused on inference. Again, its usefulness when it comes to consumer products will initially be limited, given the lack of common AI inference workloads on modern systems. Intel promotes the AIXPRT synthetic benchmark suite as an example of performance uplift, however that focuses almost entirely on image recognition.

New Instructions

Alongside DL Boost, Intel has implemented a series of AVX-512 instructions for other use cases.

AVX-512_VBMI is one layer of Vector Byte Manipulation Instructions, which includes permutes and shifts:

  • VPERMB: 64-byte any-to-any shuffle, 3 clocks, 1 per clock
  • VPERMI2B: 128-byte any-to-any overwriting indexes, 5 clocks, 1 per 2 clocks
  • VPERMT2B: 128-byte any-to-any overwriting tables, 5 clocks, 1 per 2 clocks
  • VPMULTISHIFTQB: Base64 conversion, 3 clocks, 1 per clock

AVX-512_VBMI2 takes this a stage further, enabling expand and compress functionality

  • VPCOMPRESSB: Store sparse packed byte integer values into dense memory/register
  • VPCOMPRESSW: Store sparse packed word integer values into dense memory/register
  • VPEXPANDB: Load sparse packed byte integer values from dense memory/register
  • VPEXPANDW: Load sparse packed word integer values from dense memory/register
  • VPSHLD: Concatenate and shift packed data left logical
  • VPSHRD: Concatenate and shift packed data right logical
  • VPSHLDV: Concatenate and variable shift packed data left logical
  • VPSHRDV: Concatenate and variable shift packed data right logical

AVX-512_BITALG enables a number of highly sort-after bit algorithms:

  • VPOPCNTB: Return the number of bits set to 1 in a byte
  • VPOPCNTW: Return the number of bits set to 1 in a word
  • VPSHUFBITQMB : Shuffles bits from quadword elements using byte indexes into mask

AVX-512_IFMA are 52-bit Integer fused multiply add (FMA) instructions that behave identically to AVX-512 floating point FMA, but offering a latency of four clocks and a throughput of two per clock (for xmm/ymm, zmm is four and one). This instruction is commonly listed as helping cryptographic functionality, but also means there is now added support for arbitrary precision arithmetic.

  • VPMADD52LUQ: Packed multiply of unsigned 52-bit integers and add the low 52-bit products to qword accumulators
  • VPMADD52HUQ: Packed multiply of unsigned 52-bit integers and add the high 52-bit products to 64-bit accumulators

Alexander Yee, the developer of the hyper-optimized mathematical constant calculator y-cruncher, explained to be why IFMA helps his code when calculating constants like Pi:

The standard double-precision floating-point hardware in Intel CPUs has a very powerful multiplier that has been there since antiquity. But it couldn't be effectively tapped into because that multiplier was buried inside the floating-point unit. The SIMD integer multiply instructions only let you utilize up to 32x32 out of the 52x52 size of the double-precision multiply hardware with additional overhead needed. This inefficiency didn't go unnoticed, so people ranted about it, hence why we now have IFMA.

The main focus of research papers on this is that big number arithmetic that wants the largest integer multiplier possible. On x64 the largest multiplier was the 64 x 64 -> 128-bit scalar multiply instruction. This gives you (64*64 = 4096 bits) of work per cycle. With AVX512, the best you can do is eight 32 x 32 -> 64-bit multiply via the VPMULDQ instruction, which gets you (8 SIMD lanes * 32*32 * 2FMA = 16384 bits) of work per cycle. But in practice, it ends up being about half of that because you have the overhead of additions, shifts, and shuffles competing for the same execution ports.

With AVX512-IFMA, users can unleash the full power of the double-precision hardware. A low/high IFMA pair will get you (8 SIMD lanes * 52*52 = 21632 bits) of work. That's 21632/cycle with 2 FMAs or 10816/cycle with 1 FMA. But the fused addition and 12 "spare bits" allows the user to eliminate nearly all the overhead that is needed for the AVX512-only approach. Thus it is possible to achieve nearly the full 21632/cycle of efficiency with the right port configuration (CNL/ICL only has 1 FMA).

There's more to the IFMA arbitrary precision arithmetic than just the largest multiplier possible. RSA encryption is probably one of the only applications that will get the full benefit of the IFMA as described above. y-cruncher benefits partially. Prime95 will not benefit at all.

For the algorithms that can take advantage of it, this boils down to the following table:

IFMA Performance
  Scalar x64 AVX512-F AVX512-IFMA
Single 512b FMA 4096-bit/cycle ~4000-bit/cycle 10816-bit/cycle
Dual 512b FMA 4096-bit/cycle ~8000-bit/cycle 21632-bit/cycle

AVX-512_VAES are vector extension instructions for AES commands. Current cores can enable 128-bit per-cycle, whereas using VAES instructions can enable 4x128-bit per cycle.

  • VAESDEC: Perform one round of an AES decryption flow
  • VAESENC: Perform one round of an AES encryption flow
  • VAESDECLAST: Perform last round of an AES decryption flow
  • VAESENCLAST: Perform last round of an AES encryption flow

AVX-512_VPCLMULQDQ is a single instruction also added to Ice Lake that enables carryless multiplication of a long quadword values.

  • VPCLMULQDQ: Carry-less multiplication quadword

AVX-512+GFNI are ‘Galois Field’ instructions.

  • GF2P8AFFINEINVQB: Galois field affine transformation inverse
  • GF2P8AFFINEQB: Galois field affine transformation
  • GF2P8MULB: Galois field multiply bytes

Beyond the AVX-512 instructions, there are a number of new instructions the core, particularly pertaining to cryptography. Ice Lake-U will support the SHA instructions, used for accelerating SHA-1 and SHA-256 algorithms:

  • SHA1MSG1
  • SHA1MSG2
  • SHA256RNDS2
  • SHA256MSG1
  • SHA256MSG2

The new Ice Lake core also has a Gaussian Neural Accelerator (GNA), which is a hardware block that enables low-power wake on voice. Intel also has the Gaussian Mixture Model (GMM) inside the core, as it has since Skylake, however Intel still has not provided much (if any) information on how it works. If you’re lucky, you might find it referenced in a CPU manual that it exists, but beyond this, little information is present. It stands to reason that the GNA will get the same treatment. Our best guess is that these units assist Microsoft Cortana for low-powered wake-on voice inference algorithms; however they don’t seem to be as open to other software developers, or perhaps it is but only under an NDA with Intel.

Gen11 Graphics: Competing for 1080p Gaming Two Versions of Ice Lake, Two Different Power Targets


View All Comments

  • eastcoast_pete - Tuesday, July 30, 2019 - link

    Thanks Ian! One initial observation: that slide or picture from Dell showing processor options for their 2in1 has the i7 with 4 MB of cache; my guess is Dell needs a proofreader in their marketing department (: Reply
  • Ian Cutress - Tuesday, July 30, 2019 - link

    It's likely those cache numbers are the other way round, i.e. quad core i7 with 8MB, dual core i3 with 4MB. That would align with what we would expect. Reply
  • eastcoast_pete - Tuesday, July 30, 2019 - link

    Agree. Looks a lot like somebody at Dell didn't check after cut-and-paste. Has Dell announced expected shipping date? As a launch partner, they're likely be among the first who ship finished product. Reply
  • Ian Cutress - Tuesday, July 30, 2019 - link

    Not yet, as far as I know. Reply
  • close - Tuesday, July 30, 2019 - link

    Now we know the cause of Intel's manufacturing hell. Ian's been biting their wafers... :) Reply
  • tipoo - Tuesday, July 30, 2019 - link

    iirc Dell said NDA was lifting August 1st on the ICL XPS 13 Reply
  • FloconDeNeige - Tuesday, July 30, 2019 - link

    Page 2, the table about In-Flight Stores and Loads: The values are switched around, or else the paragraph after is wrong.
    Otherwise looks like a great article. Still reading :)
  • FloconDeNeige - Tuesday, July 30, 2019 - link

    Page 3 sorry Reply
  • DanNeely - Tuesday, July 30, 2019 - link

    I'm a bit surprised to see pen support on the Athena requirement list. Everything else seems reasonable as an evolution of mainstream designs; but pens have always been very niche something I don't see changing. Even if pen support is baked into Intels <1W LCDs; including a pen's going to drive up system costs. And if loose is just going to end up lost; if a pen holster is required that's a decent amount of internal volume in increasing thin and dense designs. Reply
  • notashill - Tuesday, July 30, 2019 - link

    There are already tons of laptops with pen support but no included pen or internal holster. Reply

Log in

Don't have an account? Sign up now