Benchmarking Physics
We've had a lot of responses about the benchmarking procedures we used in our first PhysX article. We would like to clear up what we are trying to accomplish with our tests, and explain why we are doing things the way we are. Hopefully, by opening up a discussion of our approach to benchmarking, we can learn how to best serve the community with future tests of this technology.First off, average FPS is a good measure of full system performance under games. Depending on how the system responds to the game over multiple resolutions, graphics cards and CPU speeds, we can usually get a good idea of the way the different components of a system impact an applications performance.
Unfortunately, when a new and under used product (like a physics accelerator) hits the market, the sharp lack of applications that make use of the hardware present a problem to consumers attempting to evaluate the capabilities of the hardware. In the case of AGEIA's PhysX card, a sharp lack of ability to test applications running with a full compliment of physics effects in software mode really hampers our ability to draw solid conclusions.
In order to fill in the gaps in our testing, we would usually look towards synthetic benchmarks or development tools. At this point, the only synthetic benchmark we have is the boxes demo that is packaged with the AGEIA PhysX driver. The older tools, demos and benchmarks (such as 3DMark06) that use the PhysX SDK (formerly named Novodex) are not directly supported by the hardware (they would need to be patched somehow to enable support if possible).
Other, more current, demos will not run without hardware in the system (like CellFactor). The idea in these cases would be to stress the hardware as much as possible to find out what it can do. We would also like to find out how code running on the PhysX hardware compares to code running on a CPU (especially in a multiprocessor environment). Being able to control the number and type of physics objects to be handled would allow us to get a better idea of what we can expect in the future.
To fill in a couple gaps, AGEIA states that the PhysX PPU is capable of handling over 533000 convex object collisions per second and 3X as many sphere collisions per second. This is quite difficult to relate back to real world performance, but it is appears to be more work than a CPU or GPU could perform per second.
Of course, there is no replacement for actual code, and (to the end user) hardware is only as good as the software that runs on it. This is the philosophy by which we live. We are dedicated first and foremost to the enthusiast who spends his or her hard earned money on computer hardware, and there is no substitute for real world performance in evaluating the usefulness of a tool.
Using FPS to benchmark the impact of PhysX on performance is not a perfect fit, but it isn't as bad as it could be. Frames per second (in an instantaneous sense) is one divided by the time it takes to render a single frame. We call this the frametime. One divided by an average FPS is the average time it takes for a game to produce a finished frame. This takes into account the time it takes for a game to take in input, update game logic (with user input, AI, physics, event handling, script processing, etc.), and draw the frame via the GPU. Even though a single frame needs to travel the same path from start to finish, things like cueing multiple frames for rendering to the GPU (usually 3 at most) and multithreaded game programming are able to hide some of the overhead. Throw PhysX into the mix, and ideally we can offload some of this work somewhere else.
Here are some examples of how frametime can be affected by a game. These are very limited examples and don't reflect the true complexity of game programming.
CPU limited situations:
CPU: |------------ Game logic ------------||---- GPU: |---- Graphics processing ----| |----
The GPU must wait on the CPU to setup the next frame before it can start rendering. In this case, PhysX could help by reducing the CPU load and thus frametime.
Severely GPU limited situations:
CPU: |------ Game Logic ------| |--- GPU: |-------- Graphics processing --------||---
The CPU can start work on the next frame before the GPU finishes, but any work after three frames ahead must be thrown out. In the extreme case, this can cause lag between user input and the graphics being displayed. In less severe cases, it is possible to keep the CPU more heavily loaded while the frametime still depends on the GPU alone.
In either case, as is currently being done in both City of Villains and Ghost Recon Advanced Warfighter, the PhysX card can ideally be added to create additional effects without adding to frametime or CPU/GPU load. Unfortunately, the real world is not ideal, and in both of these games we see an increase in frametime for at least a couple frames. There are many reasons we could be seeing this right now, but it seems to not be as much of a problem for demos and games designed around the PPU.
In our tests of PhysX technology in the games which currently make use of the hardware, multiple resolutions and CPU speeds have been tested in order to determine how the PhysX card factors into frametime. For instance, it was very clear in our initial GRAW test that the game was CPU limited at low resolutions because the framerate dropped significantly when running on a slower processor. Likewise, at high resolutions the GPU was limiting performance because the drop in processor speed didn't affect the framerate in a very significant way. In all cases, after adding the PhysX card, we were easily able to see that frametime was most significantly limited by either the PhysX hardware itself, AGEIA driver overhead, or the PCI bus.
Ideally, the PhysX GPU will not only reduce the load on the CPU (or GPU) by unloading the processing of physics code, but will also give developer the ability to perform even more physics calculations in parallel with the CPU and GPU. This solution absolutely has the potential to be more powerful than moving physics processing to the GPU or a second core on a CPU. Not only that, but the CPU and GPU will be free to allow developers to accomplish ever increasingly complex tasks. With current generation games becoming graphics limited on the GPU (even in multi-GPU configurations), it seems counterintuitive to load it even more with physics. Certainly this could offer an increase in physics realism, but we have yet to see the cost.
Facebook
Twitter
RSS
67 Comments
View All Comments
hatsurfer - Thursday, May 18, 2006 - link
I just got my card today from Newegg. I only had about an hour to play before I left for work. I wanted to see the effects when I destroyed a building. I played through the first mission on 1600x1200 and my frames stayed a solid 30 with v-sync enabled due to playing on a large LCD. It was pretty nice to see the MANY building parts flying in every direction with the smoke effect. All in all it looked pretty cool and realistic. I am currently gaming on a EVGA 7800 GTX awaiting my 7900 GTX from EVGAs step-up program. I would like to take full advantage of my optimal resolution 1920x1200 but my frames dropped to the 20's. I can only imagine my 7900 GTX will get me my full HD resolution, which I can't sustain on a single 7800 GTX anyway.I think anyone with a high-end system isn't going to have any hangups when it comes to frame rates. Is this for low end budget gaming systems? Probably not just yet, but neither is the price tag of $300+. So, right now it's a nice little extra piece of eye candy for me to enjoy and in the end that's what is important anyway.
I hope this technology takes off and drives the number of supported titles up and if it gets incorporated into other components so be it (I really hope so too!). We'll just do another upgrade that we're all used to doing every few months anyway. Such it has always been at the cutting edge of PC'ing and such it will always be.
Mabus - Wednesday, May 17, 2006 - link
Well I wonder how long before ATI or Nvidia Buys this company and integrates the logic into it's GPU. Not only will this allow better performance but it will allow optimisations with drivers and communications and instruction exclusivity. This will be very nice smaller card, one less space used on mobo and cooler running etc etc..remember the seperate match co-pro well this tech will definatly go the same way, it is only a matter of time. And face it the current benchies show that time is what we need
to get that performance where us gamers want it.
Mabus signing off.....
Trisped - Wednesday, May 17, 2006 - link
Why didn’t you compare to stats of Asus card with those of the BFG card. Sound, power, if they are the same then say so, otherwise what is the point of reviewing two different cards for the same thing?COV is an unfair test- Max Physics Debris Count should be the same so you can see if there is a performance boost at the same level. I know that if I give my GPU 1000 more objects it is going to go much slower, with or without the PhysX card. What I want to see is running with 1500 and 422 “Debris” both with and without the card.
I would like to see the tests run with not only 2 different processors, but what about an Intel dual core and what about with different video card configurations? If you have a lower end video card should you expect to have less of an FPS impact or more?
How long before there is a PCIe card? Would there be a performance boost using PCIe? I would think that if the physics and GPU were on the same bus then there would be less latency as well as faster communication. I think the fact that the PhysX card wouldn’t have to wait for the processor to stop sending it info before it started transmitting would also be important to speed.
Spelling/Grammar:
“If spawning lots of effects on the PhysX card makes the system stutter, then it is defeating its won purpose.” Should be one, not won.
VooDooAddict - Wednesday, May 17, 2006 - link
I'm waiting to pass judgment on this tech till after I see reviews with high end Dual core CPUs and a PhysX board connected via PCI-Express.I just can't see how connection something like this via PCI was ever going to work. This isn't mainly a one way communication like an audio board during gaming. Send the information needing to be output, the sound board outputs it. My understanding is that the PhysX board also needs to communicate the calculated information back to the CPU and GPU.
Now would be a great time for Nvidia to step up with some new demos for their physics hardware acceleration via SLI. For an extra couple bills ... SLI right now is much more justifiable then this physics hardware. AND SLI is really one of those things that isn't a nessesity. With SLI based physics you could run at low physics and max frames for Multiplayer twitch games... just enable hardware physics for Single player and MMOs. I'm hoping that will give you the best of both worlds. I'll be waiting for reviews like that.
DerekWilson - Wednesday, May 17, 2006 - link
You are correct in that PhysX requires two way communication during gameplay in many cases. This hits on why I think the demos run smoother than the games out right now. In GRAW and CoV, data needs to be sent to the PhysX card, the PPU needs to do some work, and data needs to be sent back to the CPU.In the demos that use the PhysX hardware for everything, as the scene is being setup, data is loaded into the PhysX on-board memory. Interacting with the world requires much less data to be sent to the PPU per frame, as all the objects have already been created, setup and stored locally. This should significantly cut down on traffic and provide less of a performance impact while also enabling more complicated effects and physics.
DerekWilson - Wednesday, May 17, 2006 - link
just going to say again that the above is just a theorypoohbear - Wednesday, May 17, 2006 - link
yea yea yea, so how do we overclock these things? :)Tephlon - Wednesday, May 17, 2006 - link
I know this doesn't have EXACT bearing on the article, but I thought some readers would be interested.I now own one of these cards (BFG Physx PPU), and it doesn't seem as bad to me as the benchmarks really make it out to be. I installed the card and played the game with the exact settings I had previously, and didn't FEEL a difference. I havn't straight benchmarked it, though, so the numbers might very well end up similar to anandtech's findings.
Although it has no real bearing on gameplay, I have noticed more with the addition of the PPU than anyone ever credits. Surrounding cars to rock more accuratly when nearby explosions errupt, and their suspension and doors move more accurately due to bullets and explosions. Even lightpoles bend and react better than without the card. I even had one explosion that caused the 'cross bars' (not sure what to call them) on top of the lightpole to break loose and swing back and forth for a good 20 seconds until they broke free and hit the ground. It was really neat. I know its not important stuff, but I can SEE a positive difference with the card, but don't really FEEL the lost performance it's claimed to have. Often I do see the fraps meter on my G15 lcd drop dramatically for a VERY quick instant, but I don't feel this in actual gameplay. I also believe this also happened on my rig before I had the PPU. Maybe its just that I'm not a 'BEST PERFORMANCE EVER' kinda guy. I've heard guys on forums complain about their FPS dropping from 56fps to 51fps. So what. This isn't a run&gun game. My roommate plays co-op with me and fraps tells him he's running 20fps average. It doesn't look it to me, and he also doesn't feel it either. An issue with fraps? Maybe. But again my point is as this point the loss of fps (for me) doesn't equal loss of performance, so I'm pleased with the card. I'm also curious for others that own a card to throw up their comments as well.
Again, my findings aren't that SCIENTIFIC, but they're real life. I enjoy it, and REALLY hope they see enough support to get better implimentation in future games. I think improved support and tweaked drivers will convince the masses. I hope it happens soon, or it might be a lost cause.
For those interested. My rig.
Asus A8N32-SLI
Corsair XMS 2GB (2 x 1GB) TWINX2048-3500LLPRO
BFG Geforce 7900GTOC x2
BFG Ageia Physx PPU
BFG 650watt PSU
Creative Audigy 2 ZS
NEC Black 16X DVD+R
WD 36gb Raptor
WD 320gb SATA
Tephlon - Wednesday, May 17, 2006 - link
ooops. I guess I probebly have a cpu to go with that rig. tehe.AMD Athlon X2 4400+ Toledo
Toodles.
poohbear - Wednesday, May 17, 2006 - link
are u sure those physics effects are'nt already there without the physix's card? havoc states GRAW already uses the havoc engine to begin w/, physix only adds extra particles for grenades and stuff.