EXTREME Overclocking Forums
Home | Reviews | Forums | Downloads | $ EXTREME Deals $ | RealTime Pricing | Free Magazines | Gear | Folding Stats Newsletter | Contact Us


Go Back   EXTREME Overclocking Forums > Website Related Info (No Post Increase) > Tech News & Press Releases > The Front Page Headlines

Closed Thread
 
Thread Tools
Old 09-14-2011, 03:56 PM   #1
SSPrncVegeta
Speak softly
Senior Member
 
SSPrncVegeta's Avatar
 
Join Date: Jan 2003
Posts: 7,103
iTrader: 1 / 100%
A new look at game benchmarking: Measuring multi-GPU micro-stuttering

Why FPS fails
As you no doubt know, nearly all video game benchmarks are based on a single unit of measure, the ubiquitous FPS, or frames per second. FPS is a nice instant summary of performance, expressed in terms that are relatively easy to understand. After all, your average geek tends to know that movies happen at 24 FPS and television at 30 FPS, and any PC gamer who has done any tuning probably has a sense of how different frame rates "feel" in action.
Of course, there are always debates over benchmarking methods, and the usual average FPS score has come under fire repeatedly over the years for being too broad a measure. We've been persuaded by those arguments, so for quite a while now, we have provided average and low FPS rates from our benchmarking runs and, when possible, graphs of frame rates over time. We think that information gives folks a better sense of gaming performance than just an average FPS number.
Still, even that approach has some obvious weaknesses. We've noticed them at times when results from our FRAPS-based testing didn't seem to square with our seat-of-the-pants experience. The fundamental problem is that, in terms of both computer time and human visual perception, one second is a very long time. Averaging results over a single second can obscure some big and important performance differences between systems.

We didn't set out to hunt down multi-GPU micro-stuttering. We just wanted to try some new methods of measuring performance, but those methods helped us identify an interesting problem. I think that means we're on the right track, but the micro-stuttering issue complicates our task quite a bit.

In fact, in a bit of a shocking revelation, Petersen told us Nvidia has "lots of hardware" in its GPUs aimed at trying to fix multi-GPU stuttering. The basic technology, known as frame metering, dynamically tracks the average interval between frames. Those frames that show up "early" are delayed slightly—in other words, the GPU doesn't flip to a new buffer immediately—in order to ensure a more even pace of frames presented for display. The lengths of those delays are adapted depending on the frame rate at any particular time. Petersen told us this frame-metering capability has been present in Nvidia's GPUs since at least the G80 generation, if not earlier. (He offered to find out exactly when it was added, but we haven't heard back yet.)

Measuring performance in frame-time allows for several unique advantages over frame rate. You can explain why performance can feel sluggish at higher framerates because of alternating quick and slow frames. You can also compare stuttering between cards.

Would you guys prefer this method of measuring performance to the standard FPS?


Source: TechReport

Also check out this follow-up article with video.
__________________
Intel i7 970 @ 4ghz
6x4GB DDR3 @ 1600mhz
EVGA GeForce GTX 980 Ti 6GB HYBRID @ 1290/7000
Intel 320 series 160GB
Gigabyte EX58-UD5
Thermaltake SMART 850W


Last edited by SSPrncVegeta; 09-15-2011 at 04:43 PM.
Offline  
Old 09-15-2011, 09:56 AM   #2
vai0
<><
Senior Member
 
vai0's Avatar
 
Join Date: Mar 2007
Location: Florida
Age: 24
Posts: 3,005
iTrader: 0 / 0%
Very nice article micro stutter captured on graph and video.
Offline  
Old 09-15-2011, 11:04 AM   #3
Czarspuppet
I get around
Senior Member
 
Czarspuppet's Avatar
 
Join Date: Nov 2004
Location: Earth
Age: 35
Posts: 402
iTrader: 0 / 0%
Could he be looking at it the wrong way? Frames per second is a statistic that doesn't reflect standard deviation (error, distribution, whichever you prefer).

The problem with FPS is that at 300 it doesn't rule out the possibility of the machine having spent half a second on one frame, which humans perceive as 2 FPS.
__________________
Wooot 7970 is awesome!
(Oh and now I'm poor again)

Offline  
Old 09-15-2011, 12:03 PM   #4
SSPrncVegeta
Speak softly
Senior Member
 
SSPrncVegeta's Avatar
 
Join Date: Jan 2003
Posts: 7,103
iTrader: 1 / 100%
Quote:
Originally Posted by Czarspuppet View Post
Could he be looking at it the wrong way? Frames per second is a statistic that doesn't reflect standard deviation (error, distribution, whichever you prefer).

The problem with FPS is that at 300 it doesn't rule out the possibility of the machine having spent half a second on one frame, which humans perceive as 2 FPS.
That's why the article argues against FPS in favor of frame time, with being able to measure variation being one of the advantages mentioned in the article.
__________________
Intel i7 970 @ 4ghz
6x4GB DDR3 @ 1600mhz
EVGA GeForce GTX 980 Ti 6GB HYBRID @ 1290/7000
Intel 320 series 160GB
Gigabyte EX58-UD5
Thermaltake SMART 850W

Offline  
Old 09-15-2011, 02:59 PM   #5
Sephious
"Doctor"
Senior Member
 
Sephious's Avatar
 
Join Date: Oct 2004
Location: Saskatchewan
Age: 27
Posts: 3,362
iTrader: 8 / 100%
I love this. I, for one, can see micro-stuttering so I welcome this benchmarking type.
__________________
Intel i7 3770K 4.8GHz 1.32v
MSI Z77A-GD65
Gigabyte GTX 660 Ti
Corsair Vengeance 16GB DDR3-1600 9-9-9-24
XFX 850W Black Edition
2x Samsung 830 128GB in Raid 0/Western Digital Caviar Black 2TB & Seagate 2TB in Raid 1
Coolermaster HAF 932/XSPC RX360/XSPC Raystorm
Toshiba 40" 1080p 40RV525U/LG 19" 1360x768 W1953TE
Pioneer VSX-919AH-K Receiver/Martin Logan Source/Velodyne VDR-12
V-MODA Crossfade M-80/Monster Turbine Pro Copper/Shure SRH840
Offline  
Old 09-15-2011, 05:21 PM   #6
SSPrncVegeta
Speak softly
Senior Member
 
SSPrncVegeta's Avatar
 
Join Date: Jan 2003
Posts: 7,103
iTrader: 1 / 100%
I'm testing this out right now. Info in a bit.
__________________
Intel i7 970 @ 4ghz
6x4GB DDR3 @ 1600mhz
EVGA GeForce GTX 980 Ti 6GB HYBRID @ 1290/7000
Intel 320 series 160GB
Gigabyte EX58-UD5
Thermaltake SMART 850W

Offline  
Old 09-15-2011, 06:24 PM   #7
K31TH3R
Reality in 01101011's
Senior Member
 
K31TH3R's Avatar
 
Join Date: May 2004
Location: Iowa
Age: 28
Posts: 4,492
iTrader: 24 / 100%
FINALLY someone who has accurately captured and shown microstuttering.

Someone give these people a cookie, this is great. Frame-time measurements is THE WAY TO GO. FPS cannot show this, and this is genius.

Time to adapt to a new form of measurement, this makes loads of sense.

Good find!
Offline  
Old 09-15-2011, 06:34 PM   #8
SSPrncVegeta
Speak softly
Senior Member
 
SSPrncVegeta's Avatar
 
Join Date: Jan 2003
Posts: 7,103
iTrader: 1 / 100%
Well, Batman:AA was a poor choice because I get 200fps (usually <6ms time frame). I'll do what they did in that video and restrict the framerate with vsync.

Quote:
Originally Posted by K31TH3R View Post
FINALLY someone who has accurately captured and shown microstuttering.

Someone give these people a cookie, this is great. Frame-time measurements is THE WAY TO GO. FPS cannot show this, and this is genius.

Time to adapt to a new form of measurement, this makes loads of sense.

Good find!
I love messing with statistics, so I'm enjoying myself interpreting the data.
__________________
Intel i7 970 @ 4ghz
6x4GB DDR3 @ 1600mhz
EVGA GeForce GTX 980 Ti 6GB HYBRID @ 1290/7000
Intel 320 series 160GB
Gigabyte EX58-UD5
Thermaltake SMART 850W

Offline  
Old 09-15-2011, 07:13 PM   #9
SSPrncVegeta
Speak softly
Senior Member
 
SSPrncVegeta's Avatar
 
Join Date: Jan 2003
Posts: 7,103
iTrader: 1 / 100%
I chose these specific part of the benchmark (near the end when the camera flies by the skull pile) before seeing the data because I could visibly detect the stuttering. And... there it is.

Attached Thumbnails
Click image for larger version

Name:	microstuttering.png
Views:	1064
Size:	35.7 KB
ID:	160109  
__________________
Intel i7 970 @ 4ghz
6x4GB DDR3 @ 1600mhz
EVGA GeForce GTX 980 Ti 6GB HYBRID @ 1290/7000
Intel 320 series 160GB
Gigabyte EX58-UD5
Thermaltake SMART 850W

Offline  
Old 09-15-2011, 07:23 PM   #10
vai0
<><
Senior Member
 
vai0's Avatar
 
Join Date: Mar 2007
Location: Florida
Age: 24
Posts: 3,005
iTrader: 0 / 0%
Great graph Vegeta.

Could you feel micro-stuttering? I wonder how far apart each frame must be for the average person to feel it.
Offline  
Old 09-15-2011, 07:26 PM   #11
SSPrncVegeta
Speak softly
Senior Member
 
SSPrncVegeta's Avatar
 
Join Date: Jan 2003
Posts: 7,103
iTrader: 1 / 100%
The B:AA benchmark is a flyby, so it's more like watching a movie. Let me record myself playing, stop when I notice it, and look at the data.
__________________
Intel i7 970 @ 4ghz
6x4GB DDR3 @ 1600mhz
EVGA GeForce GTX 980 Ti 6GB HYBRID @ 1290/7000
Intel 320 series 160GB
Gigabyte EX58-UD5
Thermaltake SMART 850W

Offline  
Old 09-15-2011, 07:45 PM   #12
SSPrncVegeta
Speak softly
Senior Member
 
SSPrncVegeta's Avatar
 
Join Date: Jan 2003
Posts: 7,103
iTrader: 1 / 100%
Here's a playthrough of the first part of the first challenge in B:AA. My games always feel like somewhat jittering when one alternating time frame is significantly different from the other. This explains why.

Attached Thumbnails
Click image for larger version

Name:	microstuttering2.png
Views:	1072
Size:	32.3 KB
ID:	160110  
__________________
Intel i7 970 @ 4ghz
6x4GB DDR3 @ 1600mhz
EVGA GeForce GTX 980 Ti 6GB HYBRID @ 1290/7000
Intel 320 series 160GB
Gigabyte EX58-UD5
Thermaltake SMART 850W

Offline  
Old 09-15-2011, 07:50 PM   #13
Czarspuppet
I get around
Senior Member
 
Czarspuppet's Avatar
 
Join Date: Nov 2004
Location: Earth
Age: 35
Posts: 402
iTrader: 0 / 0%
Is this a SLI/XFIRE problem or do single GPU systems do it too? (perhaps in different circumstances)
__________________
Wooot 7970 is awesome!
(Oh and now I'm poor again)

Offline  
Old 09-15-2011, 07:57 PM   #14
SSPrncVegeta
Speak softly
Senior Member
 
SSPrncVegeta's Avatar
 
Join Date: Jan 2003
Posts: 7,103
iTrader: 1 / 100%
Quote:
Originally Posted by Czarspuppet View Post
Is this a SLI/XFIRE problem or do single GPU systems do it too? (perhaps in different circumstances)
The problem comes from inefficiencies of Crossfire/SLI handling multiple GPUs. A single GPU will always provide its peak performance.

This image sums it up best. Vertical variance = micro-stuttering. Notice how it's significantly less on the single-GPU 6970? Even though the 6990 is performing better, it's stuttering like crazy. A consistent, somewhat longer time frame is more important than varying, faster time frame performance.

Image is from page 4 of the article.
Attached Thumbnails
Click image for larger version

Name:	bc2-amd.gif
Views:	546
Size:	11.0 KB
ID:	160111  
__________________
Intel i7 970 @ 4ghz
6x4GB DDR3 @ 1600mhz
EVGA GeForce GTX 980 Ti 6GB HYBRID @ 1290/7000
Intel 320 series 160GB
Gigabyte EX58-UD5
Thermaltake SMART 850W

Offline  
Old 09-26-2011, 04:40 PM   #15
webtax
Running System Stock
Forum Newbie
 
Join Date: Apr 2011
Posts: 0
it would be nice to see if there's a difference between nvidia and ati in this aspect
Offline  
Old 09-26-2011, 05:14 PM   #16
SSPrncVegeta
Speak softly
Senior Member
 
SSPrncVegeta's Avatar
 
Join Date: Jan 2003
Posts: 7,103
iTrader: 1 / 100%
Someone with two Nvidia GPUs... GO!!
__________________
Intel i7 970 @ 4ghz
6x4GB DDR3 @ 1600mhz
EVGA GeForce GTX 980 Ti 6GB HYBRID @ 1290/7000
Intel 320 series 160GB
Gigabyte EX58-UD5
Thermaltake SMART 850W

Offline  
Old 09-26-2011, 10:47 PM   #17
xdownsetx
Menior Sember
Senior Member
 
xdownsetx's Avatar
 
Join Date: Jun 2003
Location: Anaheim, CA
Age: 29
Posts: 1,366
iTrader: 1 / 100%
Vegeta, give these leaked 11.9's a shot. I'm interested to see if anything is different, however unlikely.

http://www.guru3d.com/news/amd-catal...out-for-grabs/
__________________
Offline  
Old 09-26-2011, 10:57 PM   #18
SSPrncVegeta
Speak softly
Senior Member
 
SSPrncVegeta's Avatar
 
Join Date: Jan 2003
Posts: 7,103
iTrader: 1 / 100%
Why would 11.9 fixed an issue that's been in Crossfire and SLI since their debut? I'll give them a try and post my results, but I doubt they'll do anything close to fixing the problem.
__________________
Intel i7 970 @ 4ghz
6x4GB DDR3 @ 1600mhz
EVGA GeForce GTX 980 Ti 6GB HYBRID @ 1290/7000
Intel 320 series 160GB
Gigabyte EX58-UD5
Thermaltake SMART 850W

Offline  
Old 09-27-2011, 01:26 AM   #19
nhan4
Overclocker
Senior Member
 
nhan4's Avatar
 
Join Date: Jul 2002
Location: San Francisco
Age: 29
Posts: 393
iTrader: 13 / 100%
The first time I used crossfire on 2 6870s(Q6600 Abit P35 system), I've encountered micro-stuttering, I was running way more than 60 FPS(using fraps) but it did not look like smooth 60 FPS. I thought it was maybe because one card was running 16x and the other was 4x.

Upgraded to sandy bridge p67 platform(8x,8x), that problem is gone.
__________________

Intel i5 2500k 4.5 GHZ @ 1.3v
MSI p67-gd80
8gb GSkill Ripjaws 1600mhz
2x XFX HD 6870
Corsair 750TX

Gigabyte P35-DS3L
Q6600 GO 2.4GHZ -> 3.5GHZ 1.4V
XFX 8800GS 384MB
4GB GSkill 5-5-5-15 @ 1040fsb
Corsair 750TX
================
Offline  
Old 09-27-2011, 01:36 AM   #20
Josie Wales
Extreme Overclocker
Senior Member
 
Josie Wales's Avatar
 
Join Date: Jan 2005
Location: St Paul
Posts: 2,384
iTrader: 4 / 100%
Great article SSPrncVegeta. Thanks for hooking us up.
__________________
.
BAKE IT 'TIL YOU BREAK IT!!!!!
.

.
"The PSU cowboy strikes again!!!" - AruisDante
.
Offline  
Sponsored Links:
Closed Thread

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -6. The time now is 06:57 PM.

Copyright 2000 - 2016, vBulletin Solutions, Inc.
Powered by vBulletin
Copyright 2000 - 2016, EXTREME Overclocking