06:03PM EDT - Cerebras did the wafer scale - a single chip the size of a wafer

06:03PM EDT - Here's WSE1

06:04PM EDT - Uses standard tools like TensorFlow and pyTorch with Cerebras compiler

06:04PM EDT - CS-1 fits in standard 15U rack

06:04PM EDT - 400k cores

06:04PM EDT - (Costs a few $mil each)

06:05PM EDT - No DRAM, full on-chip SRAM

06:05PM EDT - 3D mesh network

06:06PM EDT - Allows all 400k cores to work on the same problem

06:06PM EDT - Linear perf scaling

06:07PM EDT - Cerebras graph compiler

06:07PM EDT - extract compute graph, create a graph in the WSE format, route kernals on the fabric, then create executable

06:09PM EDT - Graph matching for matrix multiply loops

06:09PM EDT - Supports hand optimized kernels

06:10PM EDT - Model-parallel and data-parallel optimization with the compiler

06:11PM EDT - Trade-off as resources vs compute for each kernel

06:12PM EDT - All kernels can be resized as needed

06:12PM EDT - All functionally identical

06:13PM EDT - Global optimization function to maximize throughput and utilization

06:14PM EDT - 3 key benefits

06:15PM EDT - Flexible parallelism

06:16PM EDT - Enough Fabric performance to connect everything at scale

06:16PM EDT - Otherwise slow across GPUs or a cluster

06:16PM EDT - Small batch size has super high utilization

06:16PM EDT - no weight sync overhead

06:19PM EDT - Core is designed for sparsity

06:20PM EDT - Intrinsic sparsity harvesting

06:20PM EDT - filters out all zeros

06:23PM EDT - ML user has full control over full range of sparse techniques

06:23PM EDT - WSE is MIMD, each core can be independent

06:24PM EDT - True variable sequence length support

06:24PM EDT - No padding required

06:24PM EDT - Higher utilization for irregular models

06:25PM EDT - Dynamic depth networks

06:26PM EDT - only need to process exact lengths of sequences

06:26PM EDT - World's most powerful AI computer

06:27PM EDT - Complete flexibility due to the size of the wafer scale engine

06:28PM EDT - Working in the lab today

06:29PM EDT - More info later this year

06:29PM EDT - Q&A team

06:29PM EDT - time*

06:31PM EDT - Q: What's the main benefit of WSE? A: Bypassing the issues with workloads that need multiple GPU/TPU/DPU. Opens up novel techniques that wouldn't run on traditional hardware at any sense of speed

06:32PM EDT - Q: How to feed the beast? A: Traditional server might be the GPU/TPU, so the next one is IO. We are a system company, our product is the full system, as we control all aspects of the system. We have a 1.2 Tb/s ethernet interconnect to feed the engine to keep up with the compute

06:34PM EDT - Q: How long does it take a compile a model over 400k units? A: It's an algorithmically complex search space problem. Annealing and heuristics bring that down - we are borrowing many ideas from the EDA industry. Our problem is simpler than billions of LEs on FPGAs, so we're in the minutes.

06:34PM EDT - That's a wrap. Next talk is 4096 RISC-V chip

POST A COMMENT

28 Comments

View All Comments

  • primary000 - Monday, February 15, 2021 - link

    Welcome to Primary Projects, A good basis ensures a healthy balance between private and work. We make sure that you enjoy going to work and can carry out your assignment with a smile.
    https://www.primairprojects.nl/
    Reply
  • hancocktony - Friday, February 19, 2021 - link

    Hi, I work in IT, I would like to recommend this site to you https://mindepcasinos.com/ie/ Reply
  • investigative111 - Sunday, February 28, 2021 - link

    The Investigative Collective is Australia’s preeminent private investigator and covert surveillance investigation group.Each private investigator firm within the Investigative Collective has a focus that extends further than any other private investigator group in Sydney.
    http://www.investigation.com.au/wp-admin
    Reply
  • peace000 - Sunday, March 7, 2021 - link


    PEACE GANG the brand started out as myself just wanting to wear unique streetwear/hip hop style clothing that was centered around something POSITIVE and not "materialistic" or "trendy". The meaning behind the name PEACE GANG simply means a group of people "gang" focused on bettering the planet they live on. Our tagline R U DOWN? means " R U DOWN with doing your part to make the world a better place"? If you’re searching Positive urban streetwear then go and checkout their website.
    https://peacegangworldwide.com/
    Reply
  • energenic111 - Tuesday, March 9, 2021 - link

    Energenic Solar system Australia is a dedicated solar company that helps homeowners and businesses get solar energy perfect for their homes & businesses. Going Solar with Energenic solar systems is a simple and quick solution to your solar needs.
    https://energenic.com.au/
    Reply
  • mcmk000 - Thursday, March 25, 2021 - link

    MCMK provides B2B tech startups with a complete outsourced marketing solution so you can get back to doing what you love - innovating!
    Thank you and Visit us more!
    https://www.mcmk.io/
    Reply
  • imperium000 - Sunday, March 28, 2021 - link

    Imperium Solar Arizona Offers clients a way to save on their utility bill, BIG TIME! Some customers have even been able to cut their summer A/C bill in half! Contact us to see if you qualify for $0 down solar!
    https://www.myimperiumco.com/
    Reply
  • ngobrolgame - Saturday, May 22, 2021 - link

    In your articles, I've read a few lines. Your skills in writing are amazing. Please do not stop sharing the standard content of this type. I also share some content that may be loved by your audience here. https://ngobrolgame.com Reply

Log in

Don't have an account? Sign up now