Star Swarm & The Test

For today’s DirectX 12 preview, Microsoft and Oxide Games have supplied us with a newer version of Oxide’s Star Swarm demo. Originally released in early 2014 as a demonstration of Oxide’s Nitrous engine and the capabilities of Mantle, Star Swarm is a massive space combat demo that is designed to push the limits of high-level APIs and demonstrate the performance advantages of low-level APIs. Due to its use of thousands of units and other effects that generate a high number of draw calls, Star Swarm can push over 100K draw calls, a massive workload that causes high-level APIs to simply crumple.

Because Star Swarm generates so many draw calls, it is essentially a best-case scenario test for low-level APIs, exploiting the fact that high-level APIs can’t effectively spread out the draw call workload over several CPU threads. As a result the performance gains from DirectX 12 in Star Swarm are going to be much greater than most (if not all) video games, but none the less it’s an effective tool to demonstrate the performance capabilities of DirectX 12 and to showcase how it is capable of better distributing work over multiple CPU threads.

It should be noted that while Star Swarm itself is a synthetic benchmark, the underlying Nitrous engine is relevant and is being used in multiple upcoming games. Stardock is using the Nitrous engine for their forthcoming Star Control game, and Oxide is using the engine for their own game, set to be announced at GDC 2015. So although Star Swarm is still a best case scenario, many of its lessons will be applicable to these future games.

As for the benchmark itself, we should also note that Star Swarm is a non-deterministic simulation. The benchmark is based on having two AI fleets fight each other, and as a result the outcome can differ from run to run. The good news is that although it’s not a deterministic benchmark, the benchmark’s RTS mode is reliable enough to keep the run-to-run variation low enough to produce reasonably consistent results. Among individual runs we’ll still see some fluctuations, while the benchmark will reliably demonstrate larger performance trends.

Star Swarm RTS Mode

The Test

For today’s preview Microsoft, NVIDIA, and AMD have provided us with the necessary WDDM 2.0 drivers to enable DirectX 12 under Windows 10. The NVIDIA driver is 349.56 and the AMD driver is 15.200. At this time we do not know when these early WDDM 2.0 drivers will be released to the public, though we would be surprised not to see them released by the time of GDC in early March.

In terms of bugs and other known issues, Microsoft has informed us that there are some known memory and performance regressions in the current WDDM 2.0 path that have since been fixed in interim builds of Windows. In particular the WDDM 2.0 path may see slightly lower performance than the WDDM 1.3 path for older drivers, and there is an issue with memory exhaustion. For this reason Microsoft has suggested that a 3GB card is required to use the Star Swarm DirectX 12 binary, although in our tests we have been able to run it on 2GB cards seemingly without issue. Meanwhile DirectX 11 deferred context support is currently broken in the combination of Star Swarm and NVIDIA's drivers, causing Star Swarm to immediately crash, so these results are with D3D 11 deferred contexts disabled.

For today’s article we are looking at a small range of cards from both AMD and NVIDIA to showcase both performance and compatibility. For NVIDIA we are looking at the GTX 980 (Maxwell 2), GTX 750 Ti (Maxwell 1), and GTX 680 (Kepler). For AMD we are looking at the R9 290X (GCN 1.1), R9 285 (GCN 1.2), and R9 260X (GCN 1.1). As we mentioned earlier support for Fermi and GCN 1.0 cards will be forthcoming in future drivers.

Meanwhile on the CPU front, to showcase the performance scaling of Direct3D we are running the bulk of our tests on our GPU testbed with 3 different settings to roughly emulate high-end Core i7 (6 cores), i5 (4 cores), and i3 (2 cores) processors. Unfortunately we cannot control for our 4960X’s L3 cache size, however that should not be a significant factor in these benchmarks.

DirectX 12 Preview CPU Configurations (i7-4960X)
Configuration Emulating
6C/12T @ 4.2GHz Overclocked Core i7
4C/4T @ 3.8GHz Core i5-4670K
2C/4T @ 3.8GHz Core i3-4370

Though not included in this preview, AMD’s recent APUs should slot between the 2 and 4 core options thanks to the design of AMD’s CPU modules.

CPU: Intel Core i7-4960X @ 4.2GHz
Motherboard: ASRock Fatal1ty X79 Professional
Power Supply: Corsair AX1200i
Hard Disk: Samsung SSD 840 EVO (750GB)
Memory: G.Skill RipjawZ DDR3-1866 4 x 8GB (9-10-9-26)
Case: NZXT Phantom 630 Windowed Edition
Monitor: Asus PQ321
Video Cards: AMD Radeon R9 290X
AMD Radeon R9 285
AMD Radeon R7 260X
NVIDIA GeForce GTX 980
NVIDIA GeForce GTX 750 Ti
NVIDIA GeForce GTX 680
Video Drivers: NVIDIA Release 349.56 Beta
AMD Catalyst 15.200 Beta
OS: Windows 10 Technical Preview 2 (Build 9926)

Finally, while we’re going to take a systematic look at DirectX 12 from both a CPU standpoint and a GPU standpoint, we may as well answer the first question on everyone’s mind: does DirectX 12 work as advertised? The short answer: a resounding yes.

Star Swarm GPU Scaling - Extreme Quality (4 Cores)

The Current State of DirectX 12 & WDDM 2.0 CPU Scaling


View All Comments

  • dakishimesan - Friday, February 06, 2015 - link

    Because DirectX 10 and WDDM 2.0 are tied at the hip, and by extension tied to Windows 10, DirectX 12 will only be available under Windows 10. Reply
  • dakishimesan - Friday, February 06, 2015 - link

    PS: great article. Reply
  • FlushedBubblyJock - Sunday, February 15, 2015 - link

    First thoughts: R9 290X dx11=8 frames mantle=46 frames TEST= TOTAL FRAUD

    Although the difference there is what AMD told us mantle would do, only in this gigantic liefest is such hilarity achieved.

    Another big industry lie-test blubbered out to the sheep at large.
  • 0ldman79 - Monday, February 16, 2015 - link

    It looks more like the people that coded that game are not very experienced and have spent far more time optimizing for future API than DX11. Reply
  • Christopher1 - Monday, February 16, 2015 - link

    Not necessarily. DX11 no matter how 'optimized' still does not get you as close 'to the metal' as Mantle does. So yes, there can be these kinds of extreme differences in FPS. Reply
  • The_Countess666 - Thursday, February 19, 2015 - link

    they are in fact very experienced. but they choose to do the things that previously DX11 bottleneck prevented them from doing in the past. Reply
  • 0ldman79 - Saturday, February 21, 2015 - link

    That makes sense, still not quite an apples to apples comparison in that situation, though using previously unavailable features on the new API tends to show the differences.

    The question still remains, will we see similar improvements on the current crop of DX11 games?

    I don't think that will be the case, though I could be wrong.

    Seems the gains are from multithreading, which is part of the DX11 or 11.1 spec.
  • RobATiOyP - Sunday, February 21, 2016 - link

    Of course you won't see such a performance increase, because games have to be designed and tuned to what the platform is capable of. The console API's have allowed games, lower level access, Mantle, DX12 & Vulkan are about removing a bottleneck caused by the assumptions in DX11 & OpenGL API's which were designed when GPUs were novel items and much evolution has occured since. Those doubting the benchmark, please say why a graphics application would not want to do more draw calls per second! Reply
  • Fishymachine - Monday, February 16, 2015 - link

    DX11 can manage up 10k draw calls, Star Storms makes 100k. Also Assasins Creed Unity makes up to 50k in case you wanted a retail game that would skyrocket in low API(there's a spot where even 2 GTX980 get 17fps) Reply
  • The_Countess666 - Thursday, February 19, 2015 - link

    this engine was spefiically written to do all the things that previously DX11 doesn't allow game developers to do. it was designed to run headlong into every bottleneck that DX11 has.

    it is in fact a great demonstrations of the weaknesses of DX11.

    the fact that nvidia gets higher framerates in dx11 then ATI is because they optimized the hell out of this game. that isn't viable (costs too much, far too time consuming) for every game and was purely done by nvidia for marketing, but all it really does is further illustrate the need for a low level API where the burden of optimizations is shifted to the game engine developers where it belongs, not the driver developers.

Log in

Don't have an account? Sign up now