Flash, Flex, Html, Silverlight

Introducing GUIMark – An RIA benchmark for Flex, Silverlight, HTML and more

10 Comments

If you’re wondering why I’ve been quiet the past few weeks it’s because I’ve been devoting most of my free time to finishing off a new benchmark I’m releasing today called GUIMark. GUIMark is kinda like an Acid3 test on speed that’s geared towards RIA technologies. The goal was to figure out how to implement a reference design in different runtimes and then benchmark how smoothly that design could be animated. So far I have implementations in DHTML, Flex, Java, Silverlight 1 and Silverlight 2. All the results and and implementation details can be found under the GUIMark page.

GUIMark shares alot in common with another RIA benchmark Bubblemark. I’ve written a bit about Bubblemark and why I think an alternative is necessary, but I do believe Bubblemark and GUIMark can coexist while serving 2 different purposes. Alexey Gavrilov stated it best in that he sees Bubblemark as a sortof ‘Hello World’ launchpad into comparing different environments and I agree with him. Bubblemark is a *very* accessible test suite and its easy for any kind of developer to jump in and play around with performance techniques. GUIMark takes a different approach by trying to benchmark the types of UI elements common in our Web 2.0 world. This includes things like vector redraws, alpha transparencies, text reflow, bitmap motion, and 9 scale slicing rules. From there I just fill up the render pipeline until it becomes so over-saturated that it becomes easy to visually distinguish which rendering engines are more efficient then others. As a result, the benchmark is more complicated on a visual level and requires a bit more time then Bubblemark to understand the implementation rules. Lastly with GUIMark I’ve tried to get into some of the lower level details behind how rendering engines work and how that’s affected the creation of this project.

I’m hoping that developers and designers will be able to use this test suite to identify any pros or cons to choosing a particular environment when visual transitions are a key element of the experience. I’m also hoping these benchmarks provide a spotlight for the community that we can turn toward the runtime engineers inside Sun or Adobe or Mozilla to demand better performance.

Go to GUIMark home page

10 thoughts on “Introducing GUIMark – An RIA benchmark for Flex, Silverlight, HTML and more”

  1. Ubuntu Linux 8.04 (P4 2.8C@3.44 Radeon9600SE@1280×1024)

    FF3b5 5.46
    Flash in FF3b5 9.04
    Java in FF3b5 8.96
    Moonlight 0.6 (Silverlight 1.0) Failed to detect moonlight (FF3 issue I believe)
    Moonlight 0.6 (Silverlight 2.0 beta) Failed with a blank screen [Moonlight loaded]

    Midori (Webkit) ~19*
    Opera9.27 ~19*
    Konqueror ~7.5*

    *Flash plugin failed to load to show exact results.

    The above machine with Virtualbox running Windows XP@1024×768:
    FF2 8.4
    Flash in FF2 11.75
    Java in FF2 9.15

    Get this; after upgrading FF2 on the Virtualboxed XP to FF3RC1:
    FF3RC1 3.69
    Flash in FF3RC1 14.84
    Java in FF3RC1 10.27

    I was watching my CPU for usage through all this and repeated tests.
    Midori was by far the smoothest on the eye while Opera the worst.
    3.0GHz -> 3.4GHz manual clocking of the CPU had no noticeable impact upon results.
    Official ATI Drivers for Linux were slightly slower than the open source Radeon drivers.

  2. Thanks for the results Craig, when you say you were watching your CPU usage, did you find it was pegged at 100%?

  3. HTML on 1.7Ghz PowerMac:
    • Opera 9.5b2: 9.5fps
    • Opera 9.2: 8fps
    • WebKit Nightly: 8fps
    • Firefox 3 Nightly: 3.5fps
    • Firefox 2: 5.5fps (suprisingly)

  4. Sean, most of the tests we’re 50%-75% CPU bound and I was mainly watching for background process spikes. Had to close a few initially before I ran the tests.
    What I found interesting was my hyperthreading CPU and FF3b5 didn’t use the threads as evenly as did Flash or the best; Java which was dead even for both threads. When FF3b5 hit near 90% with the first thread, the second was only ~50% but I really noticed the image get choppy. Multicore may get even more interesting. :)

  5. Sean,

    Would you consider adding Curl to your benchmark? We’ve demonstrated that Curl is 10 times faster than Flash/Flex in JPEG compression tests which you can learn more about at the following URL:

    http://www.curl.com/company_news052808.php

    A little background. Curl was first created in 1998 and has constantly evolved since that time to become, imo, the fastest most secure RIA platform available today. It provides rich 2D and 3D graphics, both a browser-based and desktop-based runtime environment, and has been deployed by over 300 very large corporations for mission critical enterprise RIA deployments.

    Although the technology was developed at MIT (our offices are still in that area) Curl had been used mostly in Asia. It was just introduced to the united states last year.

    All the best,

    Richard Monson-Haefel
    VP of Developer Relations
    Curl, Inc.

  6. Absolutely Richard, we were looking into Curl at EUI just recently seeing if there was a project it would make sense for. I’d be interested in learning the environment but if anyone in the Curl community is able to create one I’d also post it.

Comments are closed.