GUIMark Home

 

GUIMark 2 is out now! View the new tests here.

 

Home | Detailed Analysis | Benchmark and Rendering Engine theory

GUIMark is a benchmark test suite designed to compare the rendering systems of several popular UI runtimes. In general it should be able to give designers and developers a good indication of which technologies can draw complex interfaces at a smooth rate of motion. The test mostly addresses RIA technologies like Flash, Silverlight, HTML or Java, but was designed to be easily ported to any 2D GUI environment. The basis for this project was inspired by the Bubblemark animation test, but was designed to heavily saturate the rendering pipeline and determine what kind of visual complexity is achievable in the sub-60 fps realm.

The Test

GUIMark referenceThe reference design was originally created in Flex and then ported to the technologies listed below. All results listed in the matrix as well as the detailed results page were run on the same Macbook Pro running Leopard for OS X, and running Win XP under a Boot Camp partition. Each test case was run 3 times in a new browser instance and the highest framerate observed was recorded. For the HTML test, the fastest performing browser on a each OS was used in the comparison matrix (Internet Explorer 7 and Safari 3 won for their respective OSes). All subsequent plugin based tests for the OS were tested in those browsers.

My hope is to port the benchmark to all the other untested technologies listed below and I fully welcome any optimizations or ports that readers want to contribute. I’m really curious to see if any community experts or platform engineers are able to speed up their technology of choice. Although the code is fairly simple at a glance, there are no easy optimization paths to be found (and no cheating by turning off anti-aliasing).

Results

Results for Win XP running on Macbook Pro Intel Core 2 Duo 2.33 GHz

Tech Base Version Average FPS Source
Browser HTML 28.36 / IE 7 Download
SVG - -
Canvas - -
Flash Flex 3 46.08 Download
Flash 9 - -
Java Java 5 Swing 19.37 Download
Java 6 Swing - -
Processing - -
JavaFX - -
Silverlight Silverlight 1 / Javascript 9.12 Download
Silverlight 2 Beta / C# 7.95 Download

Results for OS X 10.5 running on Macbook Pro Intel Core 2 Duo 2.33 GHz

Tech Base Version Average FPS Source
Browser HTML 18.20 / Safari 3 Download
SVG - -
Canvas - -
Flash Flex 3 8.01 Download
Flash 9 - -
Java Java 5 Swing 7.19 Download
Java 6 Swing - -
Processing - -
JavaFX - -
Silverlight Silverlight 1 / Javascript 5.25 Download
Silverlight 2 Beta / C# 5.38 Download

Findings

I’ve been surprised with the results so far between WinXP and OS X. On the same machine its very clear which vendors take more advantage of the underlying hardware. The results for the different plugin technologies aren’t too surprising since it’s regularly admitted that most companies spend their optimization time on Windows due to its larger install base. This argument doesn’t hold any water though when comparing html rendering on Safari/Mac against IE /Windows where there’s roughly a 1.6 : 1 advantage to the IE team. I can’t help but wonder if the core apis on the Mac platform are creating any unnecessary roadblocks. I’m also extremely surprised at the rendering speed that Flash is able to pull off on Windows. I developed this benchmark under OS X and after compiling the results I’m considered making the testcase more intensive since Flash is running so fast, but for now maybe the really poor Mac performance will give Adobe something to work on.

You can read more about rendering engine theory, the structure of the test case itself, and detailed analysis of the results on the sub pages within the site.

Updates

John Dowdell from Adobe brought up a valid point that plugin vendors are restricted by the browser environments they run in. This is true to an extent, but the limitations enforced on plugins don’t come into play with the GUIMark test. Browsers typically restrict the number of event loops available to a plugin which caps the framerate to around 40 – 50 fps. GUIMark doesn’t come anywhere close to hitting that limit on Mac. There are also no restrictions to the amount of cpu available to a plugin running within the browser which is why all of them peg the cpu to 100%. To illustrate the point, I created an AIR implementation of GUIMark and ran it on my Powerbook and here are the results I got. The Flash players rendering engine performs no differently outside the browser then it does inside the browser. Until plugins start bumping up against the event loop or Beam Sync restrictions, Adobe, Sun, and Microsoft don’t get a free pass for slow performance on Mac.

Reader Comments (108) Leave a Comment

  1. Jordan Snyder | May 21, 2008 at 2:55 pm | permalink

    This is awesome, Sean, and I’m really surprised too. Have you already hacked up the Processing test? I’d like to understand this a bit better; I might want to do that one :)

    Cheers

  2. hgfhgf | May 22, 2008 at 11:01 am | permalink

    take it the windows version was running in IE?

    Lets keep in mind thats fucking simplistic compared to Safari which antialiases text correctly and antialiases images, rounded corners and has true opacity compositing.

  3. DG | May 22, 2008 at 11:12 am | permalink

    I ran 2D animation & rendering tests using a particle system a month ago. Results were inline with yours. Stuck with HTML/Ajax. Performance was a factor. But open standards & ubiquity were bigger factors ;)

    The result that continues to surprise is the poor performance of Flex apps on OSX. I just can’t believe this still hasn’t been addressed yet.

    Nice work! Are you planning to create a database people can submit results to?

  4. Sean Christmann | May 22, 2008 at 11:32 am | permalink

    Just to clarify, the IE tests were run on IE 7 under Windows with ClearType enabled at the OS level. PNG opacity is correctly composited under IE 7 and since bitmap antialiasing is built into the image and not calculated by the browser, I don’t think that Safari renders any “truer” then IE. IE does have 1 problem with text clipping but I thats my fault and I need to update the CSS.

    I’ll let other more knowledgeable then me debate the differences between Windows ClearType and whatever Apples font rendering system is.

  5. Sean Christmann | May 22, 2008 at 11:49 am | permalink

    DG, if anyone wants to contribute a test or try optimizing a current one I’ll be glad to host it and run the test on my machine to post results. I wanted to make sure the fps results listed here though are all based off the same machine so it keeps the results more apples to apples.

  6. Mark | May 22, 2008 at 11:55 am | permalink

    Linux results (2.66 Core 2 Duo, radeon):

    HTML/firefox 5fps
    HTML/konqueror 17fps
    Flash 23fps
    Java6 1.5fps
    Silverlight 0 (not supported)

  7. Tarik Mahmood | May 22, 2008 at 12:13 pm | permalink

    I found your benchmark results to be very interesting, especially in light of the performance on non-Windows platforms. Am I correct in my reading that you are testing the effectiveness of the development tools to produce compiled animations for maximum framerates?

    If so, you might also consider the browser’s rendering plug-in as a bottleneck. On Linux, I have noticed far superior .swf rendering using the Flash 10 beta plugin (http://labs.adobe.com/technologies/flashplayer10/) vs the official Flash 9 plugin release, and I would expect the newer plugin would improve performance on OS X as well.

  8. George Schizas | May 22, 2008 at 12:23 pm | permalink

    Your Silverlight results seem tainted. In order to perform Silverlight 1.0 test you should uninstall Silverlight 2.0 first and install only Silverlight 1.0. Silverlight 2.0 (beta) does indeed perform the worst (in Windows Vista at least, where I tried it). But before installing SL2, the SL1 test was going at a respectable 22-26fps. After installing SL2, the same test barely made it to 10fps.

  9. Pete Brown | May 22, 2008 at 12:32 pm | permalink

    Windows Vista, Dual Core 3.0 running at 3.6. Silverlight 2 Beta 1, Flash 9. IE7

    HTML: 32.06
    Flex: 36.31
    Java/Swing: (wouldn’t load)
    Silverlight/JavaScript: 21.54
    Silverlight 2 B1: 11.6

    I’m going to look at the SL2 code and see if there’s anything funky going on in there.

    Pete

  10. Jim | May 22, 2008 at 1:22 pm | permalink

    30fps on all three – windows vista / ie7. my video card isn’t that good either.

  11. Alexandre Grenier | May 22, 2008 at 1:23 pm | permalink

    Interesting test. I was surprised at the Silverlight 2 performance so I did my bit of investigation. At first the performance on this laptop was about 8 FPS.

    First, I changed the timer from 17ms to 1 tick as this would otherwise delay the rendering of frames up to 17ms since your timer thread and the SL rendering threads are not being synchronized, creating a time gap. This pushed the performance up to about 11 FPS. Note that the proper way of creating a synched loop is by hooking up to storyboard events.

    Second, Silverlight rendering is also on a timer that can create another time gap. This is configurable through the CreateObject/CreateObjectEx framerate property, or Settings.MaxFramerate. In Application_Startup I added “this.Host.Settings.MaxFrameRate = 1000;”. This pushed the performance up to about 16 FPS.

    Double the original speed! But I am still unimpressed, so I set out to find a bottleneck.

    Finally, I found the culprit: text rendering! I removed the text from Holder.xaml, and the performance was up to about 500 FPS!

    Hopefully in B2 or RTM the text rendering will be highly optimized, and we can see the expected performance.

  12. Sean Christmann | May 22, 2008 at 1:32 pm | permalink

    Alexandre, try it again but this time hit the “Run Test” button and let me know your final average fps or send me a link to the test result. As I noted on this page http://www.craftymind.com/guimark-results/ Silverlight 1 and 2 have a bad timer object and queues up callbacks to fire twice in a row even though it will only render once. The lower the Timer, the more often it will occur and the higher the simulated fps will be. You can see it in the raw data packets when using the “Run Test” option and I explicitly filter it out in the final chart. In my tests setting a 1ms Timer versus a 17ms Timer didn’t actually increase the rendering rate, it just called the fps counter more often then it should.

  13. Alexandre Grenier | May 22, 2008 at 3:20 pm | permalink

    Hi Sean, I have made some changes to your code to remove the dispatcher timer thread in exchange for a storyboard loop:

    private Storyboard storyboard = new Storyboard();
    private void initStoryboard()
    {
    storyboard.SetValue(Storyboard.TargetNameProperty, “loop”);
    // note: must set x:Name=”cParent” in Page’s XAML root canvas
    (FindName(“cParent”) as Canvas).Resources.Add(storyboard);
    storyboard.Completed += new EventHandler(storyboard_Completed);
    storyboard.Begin();
    }

    int frametimer;
    int lastframetimer = 0;
    void storyboard_Completed(object sender, EventArgs e)
    {
    frametimer = getTimer();
    if (frametimer != lastframetimer)
    {
    processFrame();
    lastframetimer = frametimer;
    }
    storyboard.Begin();
    }

    [ .. ]

    public void UserControl_Loaded(object sender, RoutedEventArgs e){
    panImage1.DataContext = this;
    panImage2.DataContext = this;
    fpsOutput.DataContext = this;

    initStoryboard();

    //System.Windows.Threading.DispatcherTimer frameLoop = new System.Windows.Threading.DispatcherTimer();
    //frameLoop.Interval = new TimeSpan(0, 0, 0, 0, 17); // 17 Milliseconds
    //frameLoop.Interval = new TimeSpan(1);
    //frameLoop.Tick += new EventHandler(processFrame);
    //frameLoop.Start();
    }

    I also changed all the calls to getTimer() to use frametimer, just for fun :)

    So I see the repeat timer value issue now! It seems the timer values are “snapped” to frame timings. So with “if (frametimer != lastframetimer)” the FPS won’t go above 64 no matter what, even setting maxframerate to a value above 64 – or a frame each 15ms. Perhaps it’s locked to my laptop’s screen refresh rate (60hz)… or perhaps only a MS engineer in some secret facility knows the truth behind this *ahum* synch mechanism.

    So (drumroll) without the text, this is the chart I got (64.01):

    Link

    Obviously, re-adding the text still makes it slow as hell.

  14. Sean Christmann | May 22, 2008 at 3:47 pm | permalink

    Interesting. I had read that MS was discouraging the use of the Storyboard object for managing timer loops post 1.0. The textblock is a significant portion of the benchmark since most all RIAs contain text and it really stresses the layout engine. It also plays an important equalizer in balancing out the things HTML is good at, with the things the plugins are good at. Although ultimately Flash just chews through both text and graphics with ease.

  15. John Dowdell | May 22, 2008 at 3:57 pm | permalink

    “I’ve been surprised with the results so far between WinXP and OS X.”

    Try it outside of a browser, in a Projector or in AIR. The Mac browsers have been a little dictatorial in the timeslices they’ll allot to plugins.

    It’s been an issue for a few years, but not a well-known one. Chet Haase did a new analysis last month:
    http://graphics-geek.blogspot.com/2008/04/times-up.html

    jd/adobe

  16. Alexandre Grenier | May 22, 2008 at 4:05 pm | permalink

    Indeed, I am shocked by the performance of text rendering in Silverlight 2 Beta 1… especially knowing that MS has been coding very fast text renderers since the early days of Windows. I would rather use some sort of dispatcher timer than the ugly storyboard code above but it doesn’t seem like it plays well with the renderer yet.
    In short, Microsoft has some work to do, but obviously they know this hence the Beta 1 – already quite a few months old so I imagine it has evolved quite a lot! It will be interesting to try this again on Silverlight 2 final and compare the results.

  17. Sean Christmann | May 22, 2008 at 4:30 pm | permalink

    John, this first round was designed for browser based UI technologies and when I start moving my test to AIR you guys will be up against WPF and Cocoa, and I imagine that battle is going to be just as interesting :)

    I think its important that Adobe, Sun, and Microsoft use their weight to make both Mac and Win browsers into better platforms for plugins since it appears to be out of the control of the common developer. At the end of the day, if developers find they’ll get better performance out of HTML derived RIAs over plugin based ones, excuses or not, they might start moving that direction.

  18. John Dowdell | May 22, 2008 at 5:00 pm | permalink

    (Thanks Sean, but the point was that the Mac browsers themselves have been the cause of Mac/Win performance differences in the past, as shown by running outside the browser, on the desktop. Make sure to test them Cocoa apps on Windows too…! ;-)

    (I’m not sure how much influence we can exert on Apple or Mozilla, and for Microsoft we had to endure the whole Eolas thing. But competitive pressures will provide an equalizer, if Mac users see that AIR outperforms in-browser apps.)

    jd

  19. Mitch | May 23, 2008 at 11:18 am | permalink

    Oddly, I found very different results when I did a recent 3D transition test for SlideRocket in Flex. On the Mac, performance was better than on my PC.

    Here are links for the tests, one using Papervision in Flash Player 9 and the other using native 3D in Flash Player 10. The FP10 performs best in full-screen mode.

    http://www.sliderocket.com/test/fp9test.html

    http://www.sliderocket.com/test/fp10test.html

    mitch
    sliderocket

  20. Matthew Fabb | May 23, 2008 at 11:38 am | permalink

    Thanks Sean for setting up this benchmarking project.

    Could you provide what browser you used with the various plugin tests, since it’s possible that the browser is effecting the performance of the plugin? Thanks.

    Also for those who are interested, here’s more information on framerates for the Flash player on different operating systems and different browsers, right from an Adobe engineer, Tinic Uro:
    http://www.kaourantin.net/2006/05/frame-rates-in-flash-player.html

  21. Sean Christmann | May 23, 2008 at 1:28 pm | permalink

    Mitch, how do you feel about the performance of the core Slide Rocket app aside from the 3d transitions? Your app is a great example of the visual complexity that is slowly becoming the norm in RIAs, and is a primary example of why GUIMark stresses so many different visual features. When I ran your demos I had a chance to put them side by side to actually watch them. I noticed that the framerate was higher on Mac when idle, but they’d both drop to about the same level when running the transition. Visually they both had the same relative amount of stutter in the transitional period. One thing about Windows though is there was alot of image tearing occurring, whereas Mac had better image quality.

    Matthew, I just realized that I didn’t post which browsers won on the home page, I only mention it in the detailed analysis so I added that into the page above. I tested on IE 7 and Safari 3 for the respective OSes. Both you and John posted about the same issue and I know I’m going to see this alot. What Tinic is refering to is a maximum possible framerate allowed by both the browser and the OS. Since GUIMark comes nowhere near those caps, the only thing being stressed is the quality of the renderer itself. Once Mac is pushing 30 fps or higher, then it will start becoming applicable.

  22. John Jones | May 23, 2008 at 4:58 pm | permalink

    Thanks for providing such a effective benchmark !
    it really shows what vendors are up to and metrics help them speed these things up

    well done

    regards

    John Jones
    http://www.johnjones.me.uk

  23. Krzysiek85 | May 24, 2008 at 7:35 am | permalink

    My results on C2D 6600 2.4 GHz(only 3% faster than yours), FF3b5 and Ubuntu 8.04:

    HTML: 21
    Flash: 52
    Java: 18

  24. Daniel Wijk | May 24, 2008 at 9:31 am | permalink

    I have tried the HTML test on my Mac Pro on these browsers under Mac OS X:
    Safari 3.1
    Webkit nightly
    Firefox 3 RC1

    and on these browser under Windows XP SP3:
    Safari 3.1
    Firefox 3 RC1
    Internet Explorer 7

    The difference in FPS between the results on my machine is between 1-3. I would like to point out that I just reinstalled Windows XP so its not bloated with anything that slowes it down.

  25. Ryan O'Connell | May 26, 2008 at 3:22 am | permalink

    Wow, this crawls on Firefox 3 RC1. 5-6 FPS!?
    Flash/Flex – 26 FPS
    Tested on a XP SP3 Athlon 3500+.

  26. Stephen Bradley | May 27, 2008 at 4:35 am | permalink

    It’s great that you’ve put this together, I’m very impressed that you could develop this in so many different languages. I’m not convinced that the reflowing of text on each frame makes for a very real world test. I don’t think any text rendering system is going to be optimised for reflowing text that often, and I can’t think of a design that would use that capability. I think that bitmap resizing and transforming as in Mitch’s test should be included in the benchmark as this is fairly widely used in RIAs (think of all those coverflow and flipbook clones). There should also be some form of alpha channel animation element. I have found it is in these areas that you need to reach the framerate for the animation to be effective on the chosen platform and if the framerate is not high enough then the effect is lost and users get annoyed with your unresponsive interface.

  27. Scott E. King | May 27, 2008 at 10:29 am | permalink

    Just curious, did you optimize Mac OS X for performance?

    Since Mac OS X for portables uses settings that optimizes various aspects of the user experience such as “Better Performance”, “Better Battery Life”, or a cross between the two, which setting did you use when you performed these tests? Mac portables tend to emphasize battery life over performance, so you should probably optimize the OS as much as possible for performance if that’s what you care about. I imagine that XP already runs the CPU much hotter than Mac OS X to maximize performance, though equivalent settings on the XP side should be optimized for performance as well.

    Ultimately, what you may have inadvertently discovered is not that Mac OS X makes inefficient use of the hardware, but rather makes highly efficient use of the hardware to optimize user experience and not necessarily raw performance.

  28. Mitch | May 27, 2008 at 10:58 am | permalink

    So SlideRocket has both an editor for creating presentations and a player for playing them back. We’ve seen very few performance issues with the editor. The GUI is very responsive and is pretty comparable with a desktop app. Thumbnail generation is probably the worst performing aspect since we can’t do it server-side and there is no threading in the client.

    Older builds of SlideRocket definitely attempted to have more GUI eye candy (sliding drawers, etc) but it quickly became clear that that stuff was difficult to maintain good performance with.

    I’m reasonably happy with baseline playback performance but I think our biggest challenge is that SlideRocket is flexible enough that a user can bring it to it’s knees quite easily. Drop a snow particle plugin on top of an HD movie and then put a drop shadow effect on it and you are hosed.

    If anyone would like a beta account for SlideRocket, please feel free to email me at mitch@sliderocket.com

  29. Sean Christmann | May 27, 2008 at 11:29 am | permalink

    Stephen – Thanks. I think its a valid argument that the benchmark goes a bit overboard on text reflow, but since almost all websites are mostly text driven, I wanted this test to be valid for simple transitions like Digg and Slashdots commenting system, all the way up to more complicated visual system. The test stresses a good amount of matrix transforms as well as vector transforms, but theres no way I’d be able to rewrite Papervision in each different language. That kind of test is really going to have to remain a Flash thing only.

    Scott – I ran the tests with my power cord plugged in and my laptop set to ‘Better Performance’ for the power cord setting. One thing to also point out is that I kept both my Activity Monitor and Task Manager open during testing to ensure that my cpu was pegged to 100%. This was a pretty important aspect of the test for me since I didn’t want to build something that wasn’t correctly thrashing the cpu.

  30. darby | May 27, 2008 at 11:46 am | permalink

    Maybe comparing like-for-like may have been slightly more fair.

    You’re comparing Leopard with an OS that’s been around since 2001, why didn’t you compare Apple’s latest OS (Leopard), with Microsoft’s latest OS? (Vista)

    Or if you want to compare Microsoft’s most updated and speed-optimised OS (XP), then compare it against the same from Apple – i.e. Tiger, which has had 11 updates since its release.

    Leopard is an OS that’s only had a couple of updates, and also has a lot more eye-candy than Tiger, and is far from speed-optimised.

  31. Sean Christmann | May 27, 2008 at 1:12 pm | permalink

    Darby, its just a matter of what I have installed on my machine. As for Tiger, I just got one of my coworkers to run the HTML test in Safari on his Tiger install. He saw a slightly lower framerate then I got in Leopard at 17.19 fps. Here are the raw results. Personally I think XP results are more relevant right now since it owns such a large market share compared to vista.

  32. Dmitri Trembovetski | May 27, 2008 at 4:29 pm | permalink

    For the java benchmark, could you please run it on 6u10 (beta builds
    available from http://jdk6.dev.java.net ). This benchmark will benefit from the new Java2D pipeline which was added in that release, assuming you have a decent video board.

    I get pretty much the same scores as Flex on my system with 6u10 installed.

    Thanks,
    Dmitri
    Java2D Team

  33. Tilo | May 27, 2008 at 9:14 pm | permalink

    I just ran this on my iMac, under both Mac OS X 10.5.2 and VMWare Fusion with Windows XP SP3. VMWare Fusion w/ WinXP comes close to the published numbers for Windows XP, both in windowed and fullscreen mode. My OSX reaches the same sad numbers as published. It’s a shame that emulated Windows apps get more performance than apps running directly on the host system.
    Supposedly Apple has told users of the latest dev seeds for OS X 10.5.3 to report back on Flash performance in Safari, and the Mac native software Delicious Library 2 mentions drastic rendering speed improvements that will be available with OS X 10.5.3, so there is hope :-)

  34. Dave Hyatt | May 28, 2008 at 2:39 pm | permalink

    The difference between IE7 and Safari basically just boils down to image scaling. Safari as much higher quality image scaling than IE7 does, so our rendering of the scaled image is superior. However this higher quality scaling comes at the expense of speed.

    On my machine if I switch WebKit to use lower quality scaling, we jump from 21fps to 35fps. So basically that explains the difference. You might consider forcing IE to use higher quality image scaling:

    http://msdn.microsoft.com/en-us/library/ms530822(VS.85).aspx

    I’m going to look into making WebKit dynamically switch to lower quality scaling if an image changes size repeatedly and quickly (going back to high quality mode when the animation stops). That should ideally provide the best of both worlds (high quality scaling for static scaled images, and fast low quality scaling when an animation scale is happening).

  35. Sean Christmann | May 28, 2008 at 6:12 pm | permalink

    Thanks for the info Dave. I’m surprised those 4 stars are causing that much trouble since all the other images on the page are being clipped and repositioned vs scaled. I imagine Safari is limited by Beam Sync so ~35 fps is going to be the max you can hit?

  36. Dave Hyatt | May 28, 2008 at 8:04 pm | permalink

    I have some other optimizations that push the result close to 40fps on my Mac Pro (up from 21fps when I first started hacking).

    It’s worth noting that if you dropped the cross-browser code you could cut out a significant # of elements for a more optimized rendering (for example border-image on a single div instead of a bunch of table rows and cells).

  37. Kamil | May 29, 2008 at 12:42 am | permalink

    Grattz for the test …

    Why not Vista vs. OS X 10.5 ?!?!?

    MS dont sell XP now, only Vista left

  38. Dave Hyatt | May 29, 2008 at 1:12 pm | permalink

    Up to to 43fps on my machine. :) Patches should be landing soon, and then you will be able to try out a WebKit nightly and see for yourself.

  39. Chris Brind | May 30, 2008 at 3:34 pm | permalink

    Hi,

    I think this is a great post and the findings are very interesting!

    I was wondering if your tests could handle CURL as well?
    http://www.curl.com/

    It would kill two birds with one stone for me, the XP vs Mac thing as well as being able to see how CURL compares to Flash and Java in a rendering situation (their current tests have been based on JPEG encoding which doesn’t really seem relevant).

    More info:
    http://developers.curl.com/blogs/community_blog/2008/05/12/comparing-the-performance-of-curl-and-flex-3

    Thanks in advance!

    Cheers,
    Chris

  40. Dave Hyatt | May 30, 2008 at 4:14 pm | permalink

    My changes have landed if you are interested in trying out a nightly build.

    http://nightly.webkit.org/

  41. Peter | June 10, 2008 at 5:13 pm | permalink

    Hmm, The HTML test on Windows 2003 is almost twice as fast in IE7 than FF3 RC2; wasn’t expecting this…

  42. Shane Guillet | June 11, 2008 at 4:17 pm | permalink

    [Microsoft - Silverlight Performance]

    This benchmark was very interesting to investigate, as mentioned earlier our text rendering was a major bottleneck with this scenario. We’ve made several optimizations to text performance as a result (caching glyph paths, better detection of what’s outside the clipping bounds). These features will be integrated when we release the final version of Silverlight 2 and show substantial improvement with this scenario.

    Additionally our timers have been improved and now fire reliably. You will be able to use either the DispatcherTimer or StoryBoardTimer and see the same results.

  43. Sean Christmann | June 15, 2008 at 11:41 am | permalink

    Shane, good to hear, I’ve been getting such good feedback from all you major vendors that I imagine I’ll have to make a new test in the future to start breaking all your optimizations again. I look forward to testing the final plugin.

  44. Tinic Uro | June 18, 2008 at 7:04 pm | permalink

    Sean,

    We have identified the bottleneck in the Flash Player for OSX. Like in the other plugins the culprit is text rendering, in this case rendering using device text. This benchmark spends >50% in a single OSX function: ATSUGetUnjustifiedBounds. You can verify this yourself using Shark. I am working on a change which will cache the results returned by that API to where this call should completely disappear from the performance profile.

    If everything turns out well I hope to see that change in future release. Some new numbers on my Mac Pro (note that these represent numbers which can change at any time if we decide to do things differently):

    Before the change: ~8.5fps
    After the change: ~28fps in Safari (~26fps in Firefox)

    (On the same machine I get this for the HTML test: Firefox 3 RC3: ~14fps, Safari 3.1.1: ~21fps, WebKit nightly: ~42fps)

    These numbers are for the plugin. There are still issues with the plugin API on OSX. I assume other plugins will also hit a ceiling. When I use the standalone Flash Player I get this:

    Before the change: ~9fps
    After the change: ~42fps

    I should also point out the 3+ different ways you can render text in Flash. If you want to be totally fair you should embed a font and render it using the different anti-aliasing options. This is closer to what Silverlight and Java do.

    Lastly I want to mention that the Flash Player is optimized for both x86 and PPC. It always has been. But these kind of benchmarks are really useful for us to identify platform specific bottlenecks like in this case.

  45. Sean Christmann | June 19, 2008 at 11:29 am | permalink

    Nice Tinic,
    That’s great to see a 3x performance increase in the browser.

    I’m still curious why the standalone player is getting such higher performance then the browser plugin since we’re still far below the event loop cap? I just ran a quick Flex test and saw that my event loops in Firefox were capped at around 50 fps. If the standalone player can hit 42 fps shouldn’t the browser as well?

    Is AVM2 written in Carbon or Cocoa on OS X?

  46. mario | June 24, 2008 at 7:07 am | permalink

    no tweaking necessary, just run the tests noobs and stop trying to skew the numbers because you bought into the OS X hype

    on Vista x64/IE7

    32fps HTML

    what’s interesting is FF3 is slower

  47. Brian Matzon | July 3, 2008 at 11:58 am | permalink

    Why are you doing:
    timer = new Timer(17, this);
    in the Java benchmark ? – I am limited to 32 fps, and if I remove this restriction I go up to ~ 60fps.

  48. Brian Matzon | July 3, 2008 at 12:24 pm | permalink

    I was going through the Java code and there are some small things that are being done “wrong”:
    * new Date().getTime() => System.currentTimeMillis();
    * use of double precision (Java is more precise than most when using doubles – change to float).

    With these changes I got ~ 70+ fps (I changed to 10 second sampling, due to too high fluctuations)

    I then switched to the Server vm by passing -server, and my fps increased to ~ 120 (!).

    Oh the joys of benchmarking ;)

  49. mark | July 5, 2008 at 5:29 am | permalink

    ha IE7 speed, yeah great but now check how it renders the page.
    it’s complete shambles, being inaccurate of course gives u a speed advantage this isn’t down to good coding practice just lazy coding practice.

  50. YouCantCodeJava | July 31, 2008 at 6:14 pm | permalink

    Sorry, but this benchmark is just misleading – it only shows that you know how to program in Flex, but dont know how to do it in Java.

    You expect that Java is forgiven your lazy coding, but it doesnt.

    Try to optimize your class with the comments above and you will see, that Java is quiet alot faster then any other of your testers.

  51. Mis | August 11, 2008 at 3:41 am | permalink

    Does anyone know how flash player reacts to multiple processors? Looking at performance in iStat it seems to lean on one processor only.

  52. Navigateur | August 17, 2008 at 5:05 pm | permalink

    I’ve read elsewhere also numbers which suggest that Java kicks ass. Would like to see some benchmark tests of the Java applet loading time and best performance vs Flash loading time and best performance. Especially in the areas of 3D as well as 2D rendering. Found this unbelievable pure-Java 3D engine from about a DECADE ago called idx3D (an example demo is http://www.idx3d.ch/idx3d/demo6.html) vs PaperView3D for Flash. Flash will have native 3D in the next version and maybe Java will also provide native 3D. Would also like to see best-performance 2D comparison.

  53. mbien | August 31, 2008 at 4:38 pm | permalink

    please run the test with java 6 update 10 too. Comparing old versions against new isn’t fair.

  54. Uchimata | October 8, 2008 at 6:19 am | permalink

    Have difficulties to run a test on Silverlight 2 Beta 2. Anyone else?

  55. Uchimata | October 16, 2008 at 6:14 am | permalink

    I would like to check a Silverlight RTW version of GUIMark. I would appreciate an updated benchmark ;)

  56. lynn | October 17, 2008 at 10:27 am | permalink

    the info on this page is out of date…no silverlight 2 final release and no flash 10.

  57. David Sanchez | October 18, 2008 at 9:19 pm | permalink

    There is something curious about this.

    I got the same results running the test in Mac OS X 10.5.5, using version Flash 10. However, I run the thest in windows XP on Fusion 2.0 and I also got the same results. I did not get the hyper fast renderings you have here.

    It seems the problem is related to Mac OS X graphics subsystem.

    I new Silverlight 2 example update to finale version would be nice also.

  58. Bart Czernicki | October 21, 2008 at 9:09 pm | permalink

    I updated the Silverlight 2 beta code to Silverlight 2 RTW. Check out the results/demo/code here:

    http://www.silverlighthack.com/post/2008/10/21/Silveright-2-RTW-Performance-GUIMark.aspx

    Microsoft has fixed the Text Glyph rendering and Silverlight 2 RTW and the performance is pretty much the same as Flex 3.

  59. Bart Czernicki | October 22, 2008 at 3:15 pm | permalink

    I updated the Silverlight beta 1 code with Silverlight 2 RTM. There is a dramatic positive change in Silverlight 2 performance…

    http://www.silverlighthack.com/post/2008/10/21/Silveright-2-RTW-Performance-GUIMark.aspx

  60. Rohit | April 1, 2009 at 3:53 pm | permalink

    Thanks for putting together this information.

    In my tests with Silverlight code by Bart Czernicki, Silverlight 2.0 beat Flash 10 player in terms of frames (SL 43 fps, Flash 34 FPS), but CPU utilization with SL was 83% whereas Flash did not exceed 67%. Any comments on the CPU utilization difference – is it because Flash uses Graphics horsepower and Silverlight does not?

  61. Casey | April 15, 2009 at 2:13 pm | permalink

    This is very interesting. I was just running the first HTML test again on both OS/browsers and constantly moved the browser windows around while the test was being performed and the average frame rate of both browsers practically match. IE goes down to basically meet up with Safari. Safari’s frame rate doesn’t really change though. Try it for yourself. Apple must cap browser processing.

  62. MacBook Pro User | May 7, 2009 at 5:02 pm | permalink

    On my MBP OS X is faster… Even if I do serval things…
    Test is fake. For ex.: First test when downloading 2 files, have open 5 programs i have 31 fps…

  63. pgi | July 1, 2009 at 10:57 am | permalink

    whoever taught you Java ought to be jailed for ten years.

  64. MacUser | July 12, 2009 at 5:01 am | permalink

    macbook intel core2duo 2.13Ghz:

    safari 4.0:
    HTML: 55 fps 60% cpu
    Flash: 23 – 25 fps 100% cpu
    Java: 17 – 18 fps 100% cpu
    silverlight – doesn’t work.

    Firefox 3.5:
    HTML: 10 fps 100% cpu
    Flash: 24 fps 100% cpu
    Java: 10 fps 100% cpu
    silverlight – doesn’t work

    safari 4′s rendering engine is apparently quite fast.
    flash player’s performance still lags behind on the mac.
    c’mon adobe!

  65. xargs | August 31, 2009 at 1:38 pm | permalink

    Intel Core i7 2.66 GHz, 64-bit Linux, Adobe’s 64-bit Flash 10 plugin, Sun’s 64-bit Java 6 plugin, and 64-bit Firefox 3.5:

    HTML: 50 fps
    Flash: 60 fps
    Java: 38 fps

  66. rgriff | November 3, 2009 at 3:13 pm | permalink

    I cannot get this test to work reliably using IE8 on Win XP Pro or Win 7. I’ve tried in compatibility mode, and in normal mode, and the end result is the same: I see the raw data displayed in a box over the name image, instead of a chart on a new page.

    It works fine in every other browser I’ve tried, but not IE8. Sometimes, just sometimes, I can make it work by pausing and unpausing the test while it runs, but that obviously gives bad results.

    This is on a 100% stock Win XP Pro SP3 install (OEM version), with all updates and patches applied. The Win 7 install is via VMWare on a Mac, but XP Pro is running natively (still on a Mac, but booted into Windows via Boot Camp).

    I’d love to be able to get this working, as I’d like to use the results in an article I’m writing. Suggestions welcomed (I’ve tried creating a new Windows user; no difference).

    -rob.

  67. rgriff | November 3, 2009 at 3:13 pm | permalink

    Update: I’m referencing just the Flash version of the test, not any of the other modes.

    -rob.

  68. Salvop | November 17, 2009 at 11:43 am | permalink

    Now i get 50fps with Flash 10.1 with MacOS X 10.6.2 and Safari 4.0.4! Was 8fps with Flash 10!

    Great!!!!

  69. [...] Tested using guimark [...]

  70. Apple’s master plan | blixtsystems.com | April 24, 2010 at 9:21 am | permalink

    [...] hardware and software manufacturers and ensuring backwards compatibility for the Flash Player. Java and Silverlight also runs slower on the Mac, and I guess that according to Apple it is because Sun and MS are lazy as [...]

  71. Joel | August 21, 2010 at 6:02 am | permalink

    Hi Sean,
    I have recompiled your GUIMark for Silverlight 4 and it looks like performance is very similar between Flash 10.1 and Silverlight 4.

    If you don’t mind I have hosted a version of the GUIMark on my site. Let me know if you have an issue with it.
    http://www.joelypolly.com/blog-posts/silverlight-4-performance.aspx

    Cheers,
    Joel

  72. [...] doing as well as on Windows. The reasons are pretty technical, as that same blogger explains in his comment on the GUIMark site (which he refers to in the above [...]

  73. [...] moves around bitmaps on the screen capturing the current fps. The other one is Sean Christmann’s GUIMark, which simulates a common website layout and lets it scale up and down. While Gavrilov’s attempt [...]

  74. [...] doing as well as on Windows. The reasons are pretty technical, as that same blogger explains in his comment on the GUIMark site (which he refers to in the above [...]

  75. ugg classic black | November 13, 2010 at 8:58 pm | permalink

    Hey guys this blog’s RSS feed appear kind of strange on my reader. Is anyone else facing this issue?

  76. xcfgsd | December 9, 2010 at 7:28 pm | permalink

    Thank you fors sharing your admin would get ready a severely beneficial write-up I congratulate.w

  77. lionel | March 7, 2011 at 12:58 pm | permalink

    window seven 64 bit browser
    Firefox3.6.15…… : HTML 4….=> 18.47
    opera 980………..: HTML 4….=> 32.34
    safari503(webkit533): HTML 4….=> 34.63
    lunascape(tridentie): HTML 4….=> 13.53
    lunascape(gecko)….: HTML 4….=> 19.2
    lunascape(webkit137): HTML 4….=> 3.94
    flash 10….. ……: flex …. => 44.73
    jre 6.0.24……… : javaswing => 36.33
    silverlight 3…… : S1 S2 => not possible
    http://www.webkit.org/perf/sunspider/sunspider.html
    http://www.iopus.com/imacros/demo/v5/user-agent.htm

  78. Sergey | March 26, 2011 at 8:49 am | permalink

    Silverlight version doesn’t work, could you change it?

  79. [...] doing as well as on Windows. The reasons are pretty technical, as that same blogger explains in his comment on the GUIMark site (which he refers to in the above [...]

  80. [...] doing as well as on Windows. The reasons are pretty technical, as that same blogger explains in his comment on the GUIMark site (which he refers to in the above [...]

  81. [...] doing as well as on Windows. The reasons are pretty technical, as that same blogger explains in his comment on the GUIMark site (which he refers to in the above [...]

  82. [...] doing as well as on Windows. The reasons are pretty technical, as that same blogger explains in his comment on the GUIMark site (which he refers to in the above [...]

  83. Sinclair Scientific | Real Estate | July 20, 2011 at 2:38 pm | permalink

    [...] doing as well as on Windows. The reasons are pretty technical, as that same blogger explains in his comment on the GUIMark site (which he refers to in the above [...]

  84. [...] doing as well as on Windows. The reasons are pretty technical, as that same blogger explains in his comment on the GUIMark site (which he refers to in the above [...]

  85. [...] as actually as on Windows. The reasons are fair technical, as that alike blogger explains in his comment on the GUIMark site (which he refers to in the over [...]

  86. Mitchelle Bill | November 18, 2011 at 12:02 am | permalink

    Hi, I’m Mitchelle Bill,I really do appreciate wVery interesting to read this blog.. I really do appreciate what you are writtenhat you are written.. And I’m sure to be back again for the latest update of your blog. I am looking forward to see more content like this .. Keep it up and keep on writing this kind of blog ..

    If you have time you can also visit this site that I managed to surf in:
    Kidney Disease Diet,Kidney Disease Symptoms and Kidney Failure Treatment

  87. fernando morris | December 29, 2011 at 7:28 pm | permalink

    Oxycodone is used to treat moderate to severe pain. The extended-release form of this medication is for around-the-clock treatment of pain. Oxycodone is not for treating pain just after a surgery unless you were already taking oxycodone before the surgery and also we do have other like ,
    Hydromorphone (Palladdune, Dilaudid)
    Vicodin (Hydrocodone/Acetaminophen)
    Morphine (MSContin, Oramorph)
    Oxycodone (OxyContin, Roxicodone)
    Percocet (Oxycodone/Acetaminophen)
    Percodan (Oxycodone/Aspirin)
    Phenazone
    Sulfinpyrazone
    Piroxicam
    Droxicam
    Lornoxicam
    Meloxicam
    and other we do sell our product in pills and powder forms .so if you are interested do contact us at fernando.morris@rocketmail.com

  88. [...] doing as well as on Windows. The reasons are pretty technical, as that same blogger explains in his comment on the GUIMark site (which he refers to in the above [...]

  89. [...] doing as well as on Windows. The reasons are pretty technical, as that same blogger explains in his comment on the GUIMark site (which he refers to in the above [...]

  90. [...] as on Windows. The reasons are fairly technical, as that identical blogger explains in his comment on the GUIMark web site (which he refers to in the above [...]

  91. [...] doing as well as on Windows. The reasons are pretty technical, as that same blogger explains in his comment on the GUIMark site (which he refers to in the above [...]

  92. Cool Microsoft News images – Daily Brief | September 20, 2012 at 5:40 pm | permalink

    [...] as on Windows. The factors are pretty technical, as that very same blogger explains in his comment on the GUIMark web site (which he refers to in the above [...]

  93. [...] doing as well as on Windows. The reasons are pretty technical, as that same blogger explains in his comment on the GUIMark site (which he refers to in the above [...]

  94. [...] doing as well as on Windows. The reasons are pretty technical, as that same blogger explains in his comment on the GUIMark site (which he refers to in the above [...]

  95. [...] and about Windows. The reasons are very technical, because which same blogger explains inside his comment found on the GUIMark site (that he pertains to inside the above mentioned [...]

  96. Nice Cpu Fan Tool photos | December 6, 2012 at 3:41 pm | permalink

    [...] doing as well as on Windows. The reasons are pretty technical, as that same blogger explains in his comment on the GUIMark site (which he refers to in the above [...]

  97. seguros de coche para jovenes | December 24, 2012 at 11:32 pm | permalink

    Nice info, as well as comments

  98. » Flash GUIMark: 4.45 fps average | January 26, 2013 at 9:31 pm | permalink

    [...] doing as well as on Windows. The reasons are pretty technical, as that same blogger explains in his comment on the GUIMark site (which he refers to in the above [...]

  99. [...] doing as well as on Windows. The reasons are pretty technical, as that same blogger explains in his comment on the GUIMark site (which he refers to in the above [...]

  100. Sophia | December 11, 2013 at 10:59 am | permalink

    As we have all heard that Slide Rocket does not offer anymore subscription…
    I suggest you to check out a new web app called : EWC Presenter, which is a great alternative.
    Here are more details : http://www.easywebcontent.com/blog/the-sliderocket-alternative-and-replacement-is-presenter/

Leave a Comment

Submit