FCAT: The Evolution of Frame Interval Benchmarking, Part 1
by Ryan Smith on March 27, 2013 9:00 AM ESTMore To Come
While we were unable to complete our work with FCAT ahead of NVIDIA’s embargo, we wanted to provide an article that at least gives a brief overview of FCAT, as FCAT is in many ways itself part two of a process we started yesterday with our article and analysis of stuttering on AMD cards.
FCAT, we believe, is the next evolution of frame interval benchmarking. Where FRAPS' coarse nature does not suffice, FCAT provides a clear picture of what’s happening at the end of the rendering pipeline, giving us for the first time an automated, quantitative look at frame intervals, stuttering, and more. To be clear it is by no means a perfect tool, but as we have taken the time to lay out yesterday and today, compared to the beginning of the rendering pipeline, it is the end of the rendering pipeline that is more meaningful both for quantitative analysis, and ultimately for the users.
Speaking more directly however, FCAT is quite simply the frame interval analysis tool we have long wanted. It is the tool that will enable us to analyze stuttering, micro-stuttering, and more, in a manner consistent with our benchmarking methods and core beliefs in the scientific method. It’s exceedingly rare that we say this, but we haven’t been this excited by a new benchmarking tool in a very long time.
Wrapping things up, we will be following up this article next week with part 2 in our look at FCAT. In part 2 we will go into further detail about how to analyze the results FCAT generates, and what we’re finding across a range of video cards and games, both in single-GPU and multi-GPU configurations. So until then, stay tuned.
Facebook
Twitter
RSS
88 Comments
View All Comments
Spoelie - Wednesday, March 27, 2013 - link
On the previous article similar remarks were made: http://www.anandtech.com/comments/6857/amd-stutter...In essence, stuttering in the game simulation is being downplayed maybe a bit too much.
Very intuitively, suppose that in the simulation an object moves from one side of the screen to the other at a constant speed, and suppose there is a stutter in there caused by:
1) the context queue being full
2) the simulation engine being blocked by this fact
(and also assume the simulation timer is written independent of the rendering pipeline of course, otherwise fast/slow rendering would accelerate/decelerate the simulation time).
Then the simulation might simulate like this (samples over time):
[-X-X-X-------X--]
While the graphic subsystem is able to smooth out the stutter via queues, buffers, ... .
[---X---X---X---X]
Frame delivery in that instance might be smooth, however visually the object will not move at a constant speed, moving slower in the beginning and stuttering/jumping or accelerating at a later stage.
... or I misunderstood some of the materials ;)
Spoelie - Wednesday, March 27, 2013 - link
Or said a little differently, the frame intervals actually should match the simulation intervals to not have a disconnect between the simulation and what's being shown on the screen.wavetrex - Wednesday, March 27, 2013 - link
Today's multi-threaded games use a different thread for simulation.The simulation runs at a constant speed, and sets up variables, which the graphic rendering thread pick up when generating frames. It's totally irrelevant if you have 20 frames or 120 or horrible visual stuttering. The simulation itself will be smooth... but the way it's displayed/presented... that's another thing.
xdrol - Wednesday, March 27, 2013 - link
Doesn't matter if the game engine is running on another thread or not, if it expects one frame to be displayed at t+0ms, the next one at t+10ms, third at t+30ms, then an evenly moving object will be at one third it's way on the second compared to the first and third frames.If you even these frames out, and display them at t+0/t+15/t+30, then the object will be seemingly moving twice the speed between the first and second frames compared to between second-third. That is exactly shuttering.
JPForums - Thursday, March 28, 2013 - link
I also agree with you and Spoelie. Simulation step stutter is also an issue that should be covered. It would be really nice to get simulation timestamps directly from the output of the simulator and match them up to their corresponding output frames. However, this would probably require collaboration with game developers that you probably won't get. Until then, using a tool that works at the output of the renderer (like FRAPS) and can associate simulation steps with output frames would be nice.That said, there is really little that AMD or Nvidia can do to fix issues in the application other than trick it into working correctly. These results would be more useful for game developers developing new engines. Also keep in mind, simulation steps are tied loosely (through queues) to GPU's processing capabilities (unless the bottleneck happens to be before the command is dispatched to D3D). Simulation steps should be roughly equal to frame times on average. If the GPU processing were completely consistent, then the latency between the simulation step and output would be fixed and the output would appear smooth. It is variations in frame times that cause variations in simulation steps. On average, the variations of each should be roughly equal. The worse case stutter, which should be something like double the frame time variation (when simulator is compensating it the opposite direction as the frame time variation), is what we need to look out for. That said, variation in frame time is generally smaller than frame time itself. I would suggest that simulation step stuttering is a smaller problem than frame time stuttering and becomes smaller as frame times get shorter. Point of interest, Nvidia's Frame metering may actually increase simulation step stuttering.
xdrol - Wednesday, March 27, 2013 - link
I agree with this. If the frames aren't displayed at the time the game engine is expecting to, then you will have your should-be-moving-smoothly-objects being quirkly-moving-objects redered with a smooth framerate...Sabresiberian - Wednesday, March 27, 2013 - link
I get your concern, but I'm not sure it's valid. The problem here is differences in frame rates, marked jerkiness where the moving images look like they've slowed down or even stopped and then suddenly jump forward. Evening this out by metering the frames ameliorates the issue, but at the cost of overall frame rates.And, really, what's wrong with the technique, unless it were to bring frame rates down to an unacceptable level, and why would you consider it cheating somehow?
JPForums - Thursday, March 28, 2013 - link
IIRC, frame metering occurs right before output and after the feedback loop. It does nothing to smooth out simulation steps. Further, adding random delays completely unknown to the simulator actually makes simulation step stutter worse. However, I believe simulation step stutter will prove to be a smaller problem than frame time stutter (at least until frame times themselves are long enough to be an issue).JPForums - Thursday, March 28, 2013 - link
As far as I can tell it does not reveal instances of frame metering. While I do believe simulation step stutter to be a less significant problem than frame time stuttering, judicious use of frame metering may elevate its position as you are effectively adding random delays to the output that the simulator knows nothing about and therefore cannot compensate. I would encourage the reviewers here not to dismiss FRAPS completely yet. FCAT seems to be the better tool for evaluating the end of the pipeline, but until there is a better tool at the beginning of the pipeline that can see fine grained simulation step times, a coarse tool like FRAPS is still useful for revealing coarse simulation step time inconsistencies.justniz - Thursday, March 28, 2013 - link
...but if they did do frame metering (i.e.delaying faster frames) the overall frame rate would drop, which other benchmarks would pick up easily. I'm sure that ATI and nVidia still care about overall FPS too.