A new look at game benchmarking: Measuring multi-GPU micro-stuttering

[SSPrncVegeta]

Why FPS fails
As you no doubt know, nearly all video game benchmarks are based on a single unit of measure, the ubiquitous FPS, or frames per second. FPS is a nice instant summary of performance, expressed in terms that are relatively easy to understand. After all, your average geek tends to know that movies happen at 24 FPS and television at 30 FPS, and any PC gamer who has done any tuning probably has a sense of how different frame rates "feel" in action.
Of course, there are always debates over benchmarking methods, and the usual average FPS score has come under fire repeatedly over the years for being too broad a measure. We've been persuaded by those arguments, so for quite a while now, we have provided average and low FPS rates from our benchmarking runs and, when possible, graphs of frame rates over time. We think that information gives folks a better sense of gaming performance than just an average FPS number.
Still, even that approach has some obvious weaknesses. We've noticed them at times when results from our FRAPS-based testing didn't seem to square with our seat-of-the-pants experience. The fundamental problem is that, in terms of both computer time and human visual perception, one second is a very long time. Averaging results over a single second can obscure some big and important performance differences between systems.

We didn't set out to hunt down multi-GPU micro-stuttering. We just wanted to try some new methods of measuring performance, but those methods helped us identify an interesting problem. I think that means we're on the right track, but the micro-stuttering issue complicates our task quite a bit.

In fact, in a bit of a shocking revelation, Petersen told us Nvidia has "lots of hardware" in its GPUs aimed at trying to fix multi-GPU stuttering. The basic technology, known as frame metering, dynamically tracks the average interval between frames. Those frames that show up "early" are delayed slightly—in other words, the GPU doesn't flip to a new buffer immediately—in order to ensure a more even pace of frames presented for display. The lengths of those delays are adapted depending on the frame rate at any particular time. Petersen told us this frame-metering capability has been present in Nvidia's GPUs since at least the G80 generation, if not earlier. (He offered to find out exactly when it was added, but we haven't heard back yet.)

Measuring performance in frame-time allows for several unique advantages over frame rate. You can explain why performance can feel sluggish at higher framerates because of alternating quick and slow frames. You can also compare stuttering between cards.

Would you guys prefer this method of measuring performance to the standard FPS?


Source: TechReport

Also check out this follow-up article with video.

[vai0]

Very nice article micro stutter captured on graph and video.

[Czarspuppet]

Could he be looking at it the wrong way? Frames per second is a statistic that doesn't reflect standard deviation (error, distribution, whichever you prefer).

The problem with FPS is that at 300 it doesn't rule out the possibility of the machine having spent half a second on one frame, which humans perceive as 2 FPS.

[SSPrncVegeta]

Originally Posted by Czarspuppet

Could he be looking at it the wrong way? Frames per second is a statistic that doesn't reflect standard deviation (error, distribution, whichever you prefer).

The problem with FPS is that at 300 it doesn't rule out the possibility of the machine having spent half a second on one frame, which humans perceive as 2 FPS.

That's why the article argues against FPS in favor of frame time, with being able to measure variation being one of the advantages mentioned in the article.

[Sephious]

I love this. I, for one, can see micro-stuttering so I welcome this benchmarking type.

[SSPrncVegeta]

I'm testing this out right now. Info in a bit.

[K31TH3R]

FINALLY someone who has accurately captured and shown microstuttering.

Someone give these people a cookie, this is great. Frame-time measurements is THE WAY TO GO. FPS cannot show this, and this is genius.

Time to adapt to a new form of measurement, this makes loads of sense.

Good find!

[SSPrncVegeta]

Well, Batman:AA was a poor choice because I get 200fps (usually <6ms time frame). I'll do what they did in that video and restrict the framerate with vsync.

Originally Posted by K31TH3R

FINALLY someone who has accurately captured and shown microstuttering.

Someone give these people a cookie, this is great. Frame-time measurements is THE WAY TO GO. FPS cannot show this, and this is genius.

Time to adapt to a new form of measurement, this makes loads of sense.

Good find!

I love messing with statistics, so I'm enjoying myself interpreting the data.

[SSPrncVegeta]

I chose these specific part of the benchmark (near the end when the camera flies by the skull pile) before seeing the data because I could visibly detect the stuttering. And... there it is.

[vai0]

Great graph Vegeta.

Could you feel micro-stuttering? I wonder how far apart each frame must be for the average person to feel it.

[SSPrncVegeta]

The B:AA benchmark is a flyby, so it's more like watching a movie. Let me record myself playing, stop when I notice it, and look at the data.

[SSPrncVegeta]

Here's a playthrough of the first part of the first challenge in B:AA. My games always feel like somewhat jittering when one alternating time frame is significantly different from the other. This explains why.

[Czarspuppet]

Is this a SLI/XFIRE problem or do single GPU systems do it too? (perhaps in different circumstances)

[SSPrncVegeta]

Originally Posted by Czarspuppet

Is this a SLI/XFIRE problem or do single GPU systems do it too? (perhaps in different circumstances)

The problem comes from inefficiencies of Crossfire/SLI handling multiple GPUs. A single GPU will always provide its peak performance.

This image sums it up best. Vertical variance = micro-stuttering. Notice how it's significantly less on the single-GPU 6970? Even though the 6990 is performing better, it's stuttering like crazy. A consistent, somewhat longer time frame is more important than varying, faster time frame performance.

Image is from page 4 of the article.

[webtax]

it would be nice to see if there's a difference between nvidia and ati in this aspect

[SSPrncVegeta]

Someone with two Nvidia GPUs... GO!!

[xdownsetx]

Vegeta, give these leaked 11.9's a shot. I'm interested to see if anything is different, however unlikely.

http://www.guru3d.com/news/amd-catal...out-for-grabs/

[SSPrncVegeta]

Why would 11.9 fixed an issue that's been in Crossfire and SLI since their debut? I'll give them a try and post my results, but I doubt they'll do anything close to fixing the problem.

[nhan4]

The first time I used crossfire on 2 6870s(Q6600 Abit P35 system), I've encountered micro-stuttering, I was running way more than 60 FPS(using fraps) but it did not look like smooth 60 FPS. I thought it was maybe because one card was running 16x and the other was 4x.

Upgraded to sandy bridge p67 platform(8x,8x), that problem is gone.

[Josie Wales]

Great article SSPrncVegeta. Thanks for hooking us up.