Posted by Sean Christmann
Introduction
Two years ago I had an itch that needed scratching. “RIA” was the future of the web and every major company seemed to have a solution to get us there. I developed the first version of GUIMark to not only get a good understanding of the respective technologies, but also to give my clients through EffectiveUI and everyone else something to actively gauge the rendering performance of the different runtimes. After releasing it I got a good response from both the tech community as well as several platform engineers interested in resolving problems. There were however two serious flaws in the test that immediately stood out. First, the test was relying too heavily on text layout performance. I was barely engaging the the vector and bitmap side of the rendering engines. Secondly, the test was too artificial and developers have a tendency of resisting optimizing apis against unrecognizable test cases.
Evolution
Fast forward to today and the web is a different beast. Attempt to shine a positive light on a plugin technology and you will be booed off the stage. Create something fun and silly in HTML5 and you’ll have hundreds of thousands of visitors pounding down the front door of your blog to speculate on the death of Flash. It’s undeniable that a new anchor technology is taking root in the web space, and needless to say I’ve got a new itch to scratch.
GUIMark 2
Like the first GUIMark, this new benchmark is designed for one sole purpose, to burn a hole in your CPU. I still believe that by completely saturating the rendering pipeline, we can get a better idea of which technologies are best suited for running interactive content on the web. Developers tend to focus primarily on the speed of the programming language itself, when in reality, most of your cpu time is spent inside internal rendering APIs. I also firmly believe that any benchmark testing rendering performance should stick to sub 60 fps numbers. Almost all users on the web today are browsing with 60Hz LCD monitors and there’s no reason to design a test that has to throw away frame data.
While the new benchmark sticks to the original in theory, this version introduces some much needed changes. First, I’ve split GUIMark into 3 separate tests: Vector, Bitmap, and Text rendering, and I’ve attempted to make the test cases as real world as possible. Second, I’ve only implemented these tests in HTML5 and Flash. I’m not opposed to adding Silverlight and JavaFX to the benchmark, it’s just that I didn’t have the time to build them right now and something tells me a much smaller percentage of the internet crowd is interested in those results anyway. (Feel free to flame me in the comments section for that one). Lastly, I’ve added mobile versions of some of the tests, we’ve all heard inflammatory statements from certain CEOs about mobile web performance, let’s see if the numbers back that up.
Enough already, on to the results.
Test environment
All of the tests below were performed on a 15″ unibody Macbook Pro with a 2.53 GHz Intel Core 2 Duo and an NVIDIA GeForce 9400m. On the Mac side I’m running Snow Leopard with Flash player 10.0.45.2 installed. On the PC side I have Windows 7 32bit with Aero turned on and again with Flash player 10.0.45.2. For Linux I ran a Linux Mint 8 Live CD with Firefox and Flash player 10.0.45.2. Unfortunately running off the Live CD meant no access to Nvidia drivers.
Vector Charting Test
This benchmark is designed to stress the vector apis by simulating a streaming stock chart. The test makes heavy use of strokes with complex alpha fills. Originally I had added gradient fills into the mix to make sure that a good majority of vector APIs were being flexed, but there was no significant difference in the results so I pulled them out to make the visuals cleaner. While the source may appear to be heavy on the javascript side, the actual speed of code excluding canvas draw calls is less then 1 millisecond.
HTML5 | Flash 10 | |
---|---|---|
Windows 7 | ||
Internet Explorer 8.0.7600 | N/A | 30.7 |
Firefox 3.6.3 | 15.73 | 29.65 |
Chrome 4.1.249 | 6.41 | 26 |
Opera 10.53 | 24.77 | 29.9 |
Safari 4.0.5 | Safari* | 29.5 |
Avg (15.64) fps |
Avg (29.15) fps |
|
Snow Leopard | ||
Safari 4.0.5 | 4.04 | 20.55 |
Firefox 3.6.3 | 3 | 23.92 |
Chrome 5.0.342 | 2.86 | 25.48 |
Opera 10.10 | 12.22 | 15.24 |
Avg (5.53) fps |
Avg (21.29) fps |
|
Linux Mint | ||
Firefox 3.5.9 | 14.61 fps |
22.88 fps |
*Safari on Windows 7 will not animate the chart, it will only render one frame each time I press down on my mouse button.
Results are all over the place for this test. On the HTML5 side Opera delivers the best performance on both platforms. Flash on the PC is consistently high, but on OS X, Chrome takes the top spot. Linux pulls off great numbers despite running off a Live CD. HTML5 on the Mac side requires closer inspection though. When I first made the test and showed it to my co-worker John Blanco, he started ripping apart the code to find any mistakes I might have made. What he discovered was that by changing the stroke size on my lines from 2 pixels to 1 pixel, performance in Safari, Firefox and Chrome shot up to rates closer to Flash, while Opera stayed at the exact same FPS.
1 Pixel Stroke Results | Safari – 23.59 fps | Firefox – 17.43 fps | Chrome – 17.12 fps | Opera – 12.12 fps |
Flash on the Mac, as well as HTML5 and Flash on PC were largely unaffected by this change though, gaining maybe a single frame rate by changing to 1 pixel strokes. I’m not sure what to make of these findings. What kind of bug causes this and what side effects might be introduced by fixing it? Will a change allow both 1 and 2 pixel strokes to run at higher speeds, or will they both settle somewhere near Operas numbers.
Bitmap Gaming Test
The bitmap test was designed to simulate a tower defense type game. The test stresses pushing around lots of bitmap assets that animate each frame. The entire rectangle view needs to be cleared each frame to account for all the changes happening in the scene. The test supports a minimal amount of z depth ordering but not so much as to cause user scripts to take more then 1 millisecond to execute. Both environments are using anti-aliasing to scale the bitmap images.
HTML5 | Flash 10 | |
---|---|---|
Windows 7 | ||
Internet Explorer 8.0.7600 | N/A | 17.34 |
Firefox 3.6.3 | 5.78 | 17.7 |
Chrome 4.1.249 | 10.1 | 15.98 |
Opera 10.53 | 13.59 | 17.23 |
Safari 4.0.5 | Error* | 17.29 |
Avg (9.82) fps |
Avg (17.1) fps |
|
Snow Leopard | ||
Safari 4.0.5 | 11.76 | 13.21 |
Firefox 3.6.3 | 7.5 | 14.09 |
Chrome 5.0.342 | 7.4 | 19.96 |
Opera 10.10 | 5.86 | 14.53 |
Avg (8.13) fps |
Avg (15.44) fps |
|
Linux Mint | ||
Firefox 3.5.9 | 4.84 fps |
10.91 fps |
*Safari on PC again only renders one frame per mousedown event, so the results are impossible to verify.
These results are really surprising. Chrome on OS X manages to push Flash higher then even Windows based browsers. I was so surprised I ended up rebooting and running the test again just to make sure something wasn’t wrong. We’re starting to see a trend where HTML5 on average runs slower for Canvas based animations and I’ll explain why a bit further below. Linux takes a huge performance hit in this test but the percentage difference mirrors the other platforms exactly. With Nvidia drivers I’d imagine the real numbers would be closer to Mac performance.
Text Column Test
This test is designed to push the text layout and rendering engine in HTML and Flash. The test utilizes custom fonts introduced with CSS3 as well as multibyte character string. This is my least favorite test in the group because it doesn’t simulate any real world test cases, however it should provide a good estimate of how quickly a page full of text can be calculated. I call it the “iceberg” test since 80% of the hit on the CPU happens outside the renderable view. It works because although text that overflows outside the textblock doesn’t get rendered, it does have to get calculated in order to know how many lines of text can be scrolled. HTML pages do this all the time when you load a site with text below the fold.
HTML5 | Flash 10 | |
---|---|---|
Windows 7 | ||
Internet Explorer 8.0.7600 | 21.79* | 1.51 |
Firefox 3.6.3 | 24.7 | 1.5 |
Chrome 4.1.249 | 23.58* | 1.44 |
Opera 10.53 | 21.16 | 1.49 |
Safari 4.0.5 | 30* | 1.46 |
Avg (24.24) fps |
Avg (1.48) fps |
|
Snow Leopard | ||
Safari 4.0.5 | 27.26 | 16.24 |
Firefox 3.6.3 | 23.61 | 18.71 |
Chrome 5.0.342 | 26.07* | 22.85 |
Opera 10.10 | 22.72 | 15.22 |
Avg (24.91) fps |
Avg (18.25) fps |
|
Linux Mint | ||
Firefox 3.5.9 | 25.89 | 11.67 |
*Safari continues to show problems on PC. Safari reports 30 fps but it looks like it’s running at 10 fps. I’ve included the results but they’re really wrong.
*Internet Explorer renders the view, but is unable to display the custom fonts.
*Chrome on both platforms is unable to render the Jedi custom font.
I didn’t have time to investigate whether the super slow PC performance in Flash is my fault or Adobe’s, but I expect that will be uncovered soon enough. As for the general differences between HTML and Flash in the text test, this is exactly what I was expecting. HTML was built for text rendering and this is further proof that browsers do this best.
GUIMark Mobile
The Vector and Bitmap tests have been ported into miniature forms to test on mobile devices with a minimum resolution of 320×480. This is the area I imagine will see a lot of updates over the next 6 months. I’ve ordered the results by the release date of each phone tested.
HTML5 Vector | HTML5 Bitmap | Flash Vector | Flash Bitmap | |
---|---|---|---|---|
Palm Pre c/o Kevin O’Shea | 21.46 | 32.89 | ||
iPhone 3GS | 10.79 | 12.86 | ||
Motorola Droid | 8.95 | 12.59 | ||
Nokia N900 Flash 9 | 9.51 | 9.65 | 16.69 | 19.78 |
Nexus One | 15.86 | 18.83 | ||
HTC HD2 c/o Matt Emory | 10.43 | 17.59 | 29.91 | 37.62 |
Two phones running the Flash player isn’t conclusive evidence about Flash’s performance in general in the mobile space, but it does cast immediate doubts on claims that Flash is slow on ARM based smart phones. Meanwhile, if you want the best performance in HTML5 based web content, Palm Pre and Nexus One are sitting at the top of the pile. If you have results you’d like to see added to the chart, you can email results to mech {at} craftymind dot com.
What about video comparison?
I had really hoped to add a video test to this benchmark but I quickly found out there’s no reliable way to record rendering performance for video objects. As far as I can tell, HTML5 video doesn’t provide an api to catch frame dropping events, or a way to determine the playback fps. Blindly running a Timer object on the main thread didn’t seem to help either. At that point I didn’t even bother seeing what hurdles Flash had to testing playback.
Parsing the Results
I imagine half of the people reading this page will have one of two thoughts at this point, “Who cares if HTML5 is slower, I just want Flash to die” and “HTML5 is still brand new, it’s going to get a lot faster”. While I’m not interested in addressing the first point, developers should have context around the second point. There is a fundamental difference between the rendering models used in HTML5 Canvas and Flash which heavily influence the performance divide. The difference is, Canvas uses an immediate mode renderer while Flash uses a retained mode renderer.
When you write a line of javascript that draws a vector or bitmap to a Canvas the browser will immediately render that change before moving on to the next line of javascript. Since the browser has to block that line of javascript while rendering, it means the environment is most efficient when running on a single thread. Text rendering on the other hand occurs at the end of the event loop, behaving more like Flash.
In contrast, Flash commits all renderable changes to an internal store, and after the main event loop finishes processing user code it hands out rendering tasks to all available cores. As a result, Flash scales with both the speed of your processor cores and the number of cores available. Here’s an illustration to better understand.
Theoretically you might achieve twice the performance in Flash on a dual core system, but in practice there is overhead that you need to take into account like z ordering, bounds checking and re-compositing, and dividing tasks between cores is never perfect. All of this might not seem like a big deal to HTML5 developers, but the truth is the next 10 years are going to be dominated by increases to cpu core count, not single threaded execution speed. You can already see the results of this on a quad core i7 2.67 GHz processor.
Windows 7 Firefox 3.6.3 | HTML5 Bitmap Game – 6.07 fps | Flash 10 Bitmap Game – 30.1 fps |
HTML5 Canvas performance saw virtually no increase jumping to 4 cores, while Flash performance nearly doubled. Without a major shift in execution processing, Javascript based animations and interactions are going to remain stagnate over the coming years. Unfortunately I don’t see that change coming. All the talks of multithreading coming from browser vendors right now is between the browser interface and the html view, but not the HTML rendering model.
HTML5 video is largely exempt from this problem however. While the video api exposes hooks to the main thread for playback control, all rendering and sound is processed under the hood on secondary threads. As a result, media performance increases with GPU and CPU cores.
TL;DR
There is no doubt that HTML5 adoption will grow significantly in the next 2 years, and that more and more content will be targeted to SVG and Canvas implementations. But developers need to be cautious with adopting one technology or another wholesale. HTML5 may not be fast, but it is proficient at a good amount of tasks. If you need static or limited interactive content on your website, HTML5 will soon be your best option. If you need complex interactive content, you’re probably better targeting Flash. As for me, you’ll find me abusing the hell out of both technologies and posting the results to this blog.
In the meantime, if you want the best HTML5 performance on Windows, you should be using Opera right now. On the Mac side, it’s a tossup depending on the type of content you’re interacting with. If you’re looking for good Flash performance on Windows any browser will do, whereas on the Mac side Chrome is clearly outperforming everyone else.
The sources for each test should be linked within the test itself if you want to peruse the code. I tried to make sure everything could be contained to one file whenever possible and not rely on external dependencies. You can download all the fonts and tower defense assets here. If you find any errors with the results, or feel like taking a stab at testing another technology, feel free to email me at mech {at} craftymind dot com.
Flame on!
Updates
10/05/07 – As I should have assumed, quite a few people have started sending in updated tests with their own levels of optimizations. I want to be very clear on certain improvements, the purpose of these benchamarks is to stress the graphics APIs available to developers, not cancel them out. An optimization that affects both platforms equally (like caching the grid lines behind the charting test) doesn’t further the goal of exposing how efficient the two platforms are internally. If you have a unique optimization that can only be applied to one platform and not the other, please let me know and I’ll try to incorporate the change and retest.
10/05/09 – Changed some language on the rendering model for HTML5. Canvas paints immediately, standard text paints at the end of the event loop
Love these tests. Detailed and “real world.” I can’t comment specifically on the HTML5 tests (I’m not JavaScript or DOM performance expert), but a few thoughts on the Flash Gaming test:
– The way your test is setup Flash will be using sub-pixel rendering and anti-aliasing for the moving characters, while the HTML5 version will not (I’m not a DOM expert, but I don’t believe it’s doing this). You can see that if you set the Flash Player’s quality setting to “LOW”, which will disable this, you get a boost in framerate (+5fps on my MacBookPro).
– Further, most high-performance Flash games, for this particular scenario, would actually use the copyPixels() API to render directly to a bitmap (similar to drawing to Canvas) instead of having the Flash Player manage all of the display objects. This gives a bump in quality (pixel-perfect rendering for pixel-perfect art) and a significant bump in performance (as much as 10x in some of my games).
If I get time this week I may tweak your example to use copyPixels() and see what the difference is.
Hi Sean, not intending to flame here but it’s not a correct comparison you can’t compare a markup language against a plugin ( can you ? ) an interesting comparison would be canvas vs flash drawing api, html vs xhtml, but correct me if i’m wrong.
anyway Troy is correct there should be a performance boost with copyPixels() API and quality settings set to low.
And if I’m not totally wrong I’d say that canvas was “built” in 2009 and the flash drawing api exists since 2005 so if memory serves well I’d say it’s not to fair of a comparison …
Troy,
HTML5 automatically smooths bitmaps drawn to the canvas, so for the sake of fairness I’ve turned smoothing on for Flash. It’s true there are a lot of tweaks you can do in flash to get high performance bitmap rendering, and any real game would not be doing matrix transformations on the bitmaps. But the goal here is to stress the API functions that are comparable between platforms.
@Hugo
Everyone and their brother is stating how HTML will replace Flash. This is what the author is trying to convey: If you plan on trying to replace your Flash games / applications with HTML5, here’s a look at the performance differences.
Merk
Flash’s Text layout engine…would that help the Flash results w/text?
I agree with Troy, drawing the Bitmaps to a BitmapData would be the proper way to compare both things. (and Flash should be way faster that way)
Another thing that most people don’t see is that it’s not just about the performance but also having many extra features that makes flash a better tool for complex interactive stuff.. – On canvas you don’t have “hit areas” for each element, no built-in “event dispatcher”, hard to make a preloaders, hard to do animations, usually bigger file size (the tower defense example has more than TWICE THE FILE SIZE of the flash version..), etc.. (the list goes forever..)
BTW, you should check the CSS3 Animations and Transitions.. ( http://developer.apple.com/safari/library/documentation/InternetWeb/Conceptual/SafariVisualEffectsProgGuide/Animations/Animations.html ) – on the iPhone OS4 / Safari / Chrome the performance is really good. – canvas is not the only way to have some things moving on the screen
PS: I’m mainly a HTML/JS/CSS developer..
That’s very nice!!
Can you compare unity? It’s like flash, but 100x.
Ex:
http://www.youtube.com/watch?v=5E1QSrbGBY0
Google “honey defender”
or “honey cheerios unity”
.V
I looked at the code for the Text rendering text, it seems that you are not using the latest Text Layout Framework available in the Flash Player. I will try to play around with the code and see if I can send you an update.
-Om
Miller,
I may be wrong but if I did pure bitmap drawing in Flash I would lose the ability to apply transforms and I’d end up effectively single threading my rendering. CSS3 transitions are great but they’re bound to webkit which would lock out half the browsers tested.
Vic,
Don’t tempt me Unity3D vs Papervision would offer sobering results.
Everyone else regarding Flash Text Layout engine. I had incorrectly assumed that the performance enhancements in the new text engine were being applied to plain old TextField. Looks like I have to use newer APIs.
Great post!
Although, using an immediate mode renderer doesn’t mean that you have to block on draw calls. Browsers may buffer all drawing commands until JS execution is completely finished and the browser repaints. This buffer could be flushed prematurely if you called getImageData().
I tried a few optimizations in the Bitmap Gaming Test.
Cutting the sprites down to 48×48 (didn’t remove anything that wasn’t 100% transparent) improved Canvas by about 60% and Flash by about 40%.
Turning off the scaling improved Canvas by 80+% and Flash by about 25%.
Those were just on Safari on the Mac, don’t have any other convenient testing capabilities. I was trying things I thought would improve Canvas, so it’s possible other improvements would skew the other way.
Excellent comparison. You can always find fault with any synthetic benchmark, but I don’t see any deliberate bias in the implementations.
Personally, I saw much better performance for Flash than what you documented here. Here were my results:
Vector Chart Test: 59.24
Bitmap Game Test: 27.49
Text Column Test: 52.72
I’m running on a unibody MBP, 2.8GHz/4GB RAM, Chrome 5.0.375.29, Flash 10.1.53.22. The HTML 5 numbers were only slightly higher than your numbers, probably just a function of GHz.
It would also be interesting to see what kind of performance improvements you could get by re-writing the HTML 5 code using WebGL, which gives you a direct JavaScript-> OpenGL bridge. This is implemented on the WebKit nightlies, Chrome developer channel, and the Firefox 3.7 nightlies.
Damn, if Flash 10.1 is already hitting the 60 fps ceiling I’m gonna have to scale the test up by at least 3x the current lines,bitmap,text.
*Edit, Michael, after some investigation, I’m pretty sure you have your Flash setting set to low quality.
In order to achieve a staggering 17 fps for a tower defense game, you’d have to be an extraordinarily bad programmer. So how exactly did you set that test up?
vonWolfehaus et al,
Please keep in mind, the goal here is NOT to create the most efficient tower defense game, it’s to create a test that overburdens the available APIs. When you create a strongman competition, you ask the competitors to move boulders, not basketballs.
I think it’s safe to assume we’re all aware of what a benchmark is, I was just asking if you could provide the actual design of the tests, perhaps even the code so we can run our own if desired. If I learned one thing from benchmarking Flash, it’s that results vary substantially, even within the same testing environment. Bizarre, I know, but it happens. So it’s customary to post source and/or exact designs so others can compare notes.
I also wanted to make sure no one was confused by these tests, like @Troy Gilbert was–I admire his work and everything, I just think it’s quite unfair to call these types of tests “real world” (even with the quotes).
Sorry for interrogating, and I sincerely thank you for the effort you put into these tests. Cheers =)
Sean, thanks for the work on the comparisons. While many will target specific aspects of what you’ve shown, it gets everyone thinking about the big picture: replacing Flash content with HTML5 content. We begin to consider both actual and perceived performance, using multiple animated/video widgets on a given page, as well as the ease with which said content can be produced.
Your doing the right thing by putting data points around certain cases, and allowing the community to improve HTML5 to the point that it’s a suitable replacement for Flash, where appropriate.
HTML5 just isn’t there yet.
I recently experimented with using CSS3 animations (because that would eventually have to replace flash graphically) on Apple’s own iPad page.
My version uses CSS3 animations instead of scripaculous effects and only works in Chrome / Safari.
http://www.visualmedia.nl/html5/ipad/
CPU usage is around 40% with both solutions.
vonWolfehaus,
All the sources are linked directly within the examples, have a look.
Michael,
I agree, I hope some of the browser vendors are able to use examples like this to start setting goals to try and reach with HTML5
These are superb tests and your presentation is remarkably agenda free.
> the browser will immediately calculate and re-composite that change before
> moving on to the next line of javascript
This is wrong. Pretty much any modern browser will just note that something needs to be recomputed and move on. It’ll recompute things off a timer or when asked for layout information explicitly, and will re-composite off a timer.
Note also that Gecko is in fact experimenting with moving to a more retained-mode model for the actual painting, and yes including painting off the main thread.
You should probably do the comparison with where Firefox will be in the (not too distant) future i.e. using its hardware accelerated retained mode rendering framework that’s currently under development. See Robert O’Callahan’s post about it:
http://weblogs.mozillazine.org/roc/archives/2010/04/layers.html
See also Bas Schouten’s blog (linked to in the article above) about setting up a nightly build of Firefox with hardware acceleration.
In some of my test calculating time with new Date().time was sometimes even 2-3x faster than getTimer()
Thanks Sean for putting together great tests like this.
I assume that all these tests are being done with Flash Player 10 and not 10.1? Also something to keep in mind for developers while benchmarking Flash is that there are slight speed differences between the debug version and the non-debug version of the Flash Player.
Adding a WebGL test would be interesting, but not very practical as no one uses WebGL in any commercial sites, as it’s still very much in flux and none of the browsers have WebGL support turned on as default.
For most purposes, accelerated CSS transforms should fill the performance gap (currently in OSX/iphone Safari, I believe Mozilla is working on implementing them as well). However, they might not do the trick for something like GUIMark 2, and WebGL isn’t that bad of a suggestion for a more complex application like it. For most timeline-based animations (eg: Flash animations, and not applications visualizing data from an XML or realtime source) though, CSS transforms are fine (easy to code, IDEs for designers will follow suit, check out SproutCore Greenhouse for RIAs) will be accelerated in most browsers shortly, and even the IE team is doing 2d acceleration for them (why not use WebKit and tie the renderer to DX? Pride?). To test it on your MacBook though, grab a WebKit nightly build for OSX, I don’t think it’s all there in 4.0.5 .
Boris,
All of my tests with every browser so far show that creating a function with 100 complex draw calls to the canvas will perhaps take 20 milliseconds to process. I’m not sure what’s being blocked within that function to take that long, but it’s far more then “noting that something needs to be recomputed and moving on”
The main event loop is being blocked in process to render the scene, it’s not waiting to the end of the event loop to aggregate all calls to render.
why on earth did you do the linux tests on a live cd without proper display drivers? those results are pretty much irrelevant. who’s looking for performance when they’re using a live cd?
there is chrome and opera on linux too…
Yeah I want to revisit the Linux tests in a few days once I get a better triple booting strategy figured out. The current tutorials for triple booting under bootcamp were a bit daunting, and I didn’t want to exclude Linux in the off chance I screwed up the process.
The grid lines are not pixel aligned in the html5 version. Stroking with a width of 1 will stroke half a pixel on either side of the line, therefore the lines need to go through the middle of a pixel instead of a pixel edge.
This patch will fix it:
diff –git a/HTML5ChartingTest.html b/HTML5ChartingTest.html
index 5817921..3fb0d54 100644
— a/HTML5ChartingTest.html
+++ b/HTML5ChartingTest.html
@@ -42,13 +42,13 @@ function processFrame(){
var yCoord;
for(var x=0; x<=totalMinutes; x+=40){
xCoord = x*chartXSpread;
– draw.moveTo(xCoord, 0);
– draw.lineTo(xCoord, chartHeight);
+ draw.moveTo(xCoord+.5, 0);
+ draw.lineTo(xCoord+.5, chartHeight);
}
for(var y=0; y<=highestStock; y+=20){
yCoord = y*chartYSpread;
– draw.moveTo(0, yCoord);
– draw.lineTo(chartWidth, yCoord);
+ draw.moveTo(0, yCoord+.5);
+ draw.lineTo(chartWidth, yCoord+.5);
}
draw.stroke();
draw.closePath();
I imagine a large reason for the difference in performance on this demo is the anti-aliasing quality. Flash does 4×4 supersampling, whereas most browsers use much higher quality. It's pretty easy to see the difference if you take a close look.
*Really* slow computer here, and both tests (vector and bitmap) were much faster in HTML version (vector 8-9 vs 2-3, bitmap 4 vs 2-3). When I changed flash quality from High to Low, however, Flash performance doubled (obviously, vector looked worse then, but bitmap shown no visible difference).
Browser is Opera 10.52 on Win XP.
Good call Jeff, I’ll get the change in as soon as possible and hopefully rerun the tests tonight.
Update – I’ve added Jeffs changes in, however fps values haven’t significantly changed at all, some browsers lost 0.01 fps, some gained 0.02. I don’t think this change will warrant a retest across all platforms.
>The entire rectangle view needs to be cleared each frame to account
>for all the changes happening in the scene.
There is no need to clear anything if everything is overdrawn either way.
Also, if your comment form requires JavaScript but pretends it doesn’t.
> With Nvidia drivers I’d imagine the real
> numbers would be closer to Mac performance.
Don’t bet on it. Under Windows directly manipulating bitmap data in flash (using BitmapData) is normally faster than letting the flash runtime do it for you (using Bitmap and Sprite). Under many flavours of Linux the reverse is true. Flash may be multi platform but it is certainly not platform agnostic in terms of performance and optimization.
my results for Arch Linux 32-bit, nvidia drivers, Firefox 3.6.3 and FP 10.0.45.2 (Debug)
13.6, 26
16.3, 15.5
24, 15
I think the bitmap test would be the hardest hit by the debug player, i’ll switch to the release version later and see.
Firefox 3.7a4-pre (a nightly build from a few weeks ago), Windows 7, Intel i7 920, nVidia GeForce 9600GT. This build has Direct2D acceleration enabled, which is a brand-new feature.
Flash, vector test: 50FPS
HTML, vector test: 3.9FPS
Flash, bitmap test: 32FPS
HTML, bitmap test: 58FPS
Flash, text test: 2.4FPS
HTML, text test: between 40 and 55FPS
These tests really aren’t testing anything useful or relevant. The vector test is only testing how quickly the renderer can draw anti-aliased lines. Big deal – how often do you need to draw that many anti-aliased lines that quickly? The bitmap test is only testing how quickly the renderer can do bilinear filtering, which is inherently slow in software anyway. That’s why Firefox with Direct2D beats Flash on my machine.
Oh, and Flash benefits from having Direct2D enabled in Firefox as well. If I disable it, the Flash scores are cut in half…
Jeff, why on earth would you make the suggestion without actually testing it? Surely you would’ve seen that wasn’t the issue when you tested it?
Great post ! Really usefull and showing the real situation to pepole. Let’s hope that more of them will realize it
@Sean
“Everyone else regarding Flash Text Layout engine. I had incorrectly assumed that the performance enhancements in the new text engine were being applied to plain old TextField. Looks like I have to use newer APIs.”
You are almost right. To achieve this you can use Flex 4 and the new component (instead of the )
Great work, Sean. A big help to all of us working in the RIA world.
But I’d like to comment on your comment that the Text Column test “doesn’t simulate any real world test cases”. The funny thing is that this is by far the most relevant test for one of the biggest HTML/Flex/Silverlight battlegrounds: finanical applications. By far the most complex Flex application so far written is the Matrix trading app created by Morgan Stanley (at a total cost, including the back end, of nearly $100m). And the HTML app Caplin Trader, created by my company, Caplin Systems, must be one of the most complex Ajax apps ever built. Many other similar apps are being built right now by major banks.
In all cases, data is being streamed over the internet at high rates to end users (100 updates/sec per user is not unusual) and all this data is getting displayed on the screen.
So please persevere with the Text Column test — for many of us out here, it’s the most relevant test of all!
Just wanted to report a couple of strange results with Flash 10.1 r53 / Chrome 5.0.375.38 beta on Mac OSX 10.5.8 (old MB, 2Ghz Core duo, 1GB DDR2 667):
Vector Flash: 60.08fps
Bitmap Flash: 19.81fps
Text Flash: 57.77fps
For comparison, here are the HTML5 results:
Vector HTML5 2px: 2.17fps
Vector HTML5 1px: 12.69fps
Bitmap HTML5: 3.8fps
Text HTML5: 12.63fps
While HTML5 results are similar to yours, except for the old MB being slow, the 10.1 Flash results for Vector and Text are more than 2x faster.
You should still be able to try Opera 10.5x on Mac (now out in Final), and the testing versions for Linux can be run from tarballs (so they should be workable in a LiveCD environment). Please update with those numbers.
Also, I am very curious about the HTML5 numbers for Opera Mobile, the newest features are available in a hobby build for N900/N800/N810 (except JIT, but other Carakan features are present). If someone could provide Fennec+Flash numbers to compare against Opera Mobile HTML5, that would be great. The Maemo-compatible build is available from labs.opera.com, courtesy of “The Smooth Sailing Team”.
MartinodF,
Is there a chance your quality settings are on Low? I’ve seen the 10.1 numbers you’re reporting from others but I’m unable to reproduce them on my machine without setting my quality to low.
iPad (32GB + 3G) results:
2px Vector: 0.7 fps
1px Vector: 3.23 fps
Bitmap Game 1.29 fps
Text Column: 3.18 fps
Vector Mobile: 19.44 fps
Bitmap Mobile: 23.97 fps
Here an Firefox tutorial how to turn on the GPU Rendering Direct2D / DirectWrite for Windows 7 and Windows Vista SP2.
http://www.x.se/65hp
http://www.x.se/uayh
Please upgrade your Adobe Flash Player.
http://labs.adobe.com/technologies/flashplayer10/
Adobe Flash Player 10.1 release candidate 4.
I have made a quick unoptimized Java version of the Bitmap Gaming Test.
http://soupgames.net/guimark2/
On my Windows system, the Java version is *much* faster than the Flash version. I suspect because the drawing functions will be hardware accelerated. On the MacBook pro, Flash and Java were tied.
Let me know what your test results are!
Test results
————
2007 Thinkpad T60p, running Windows XP, Java 1.6.0_20:
HTML Firefox 3.6: 4.28 FPS
Flash 10 Firefox: 12.9 FPS
Java: 21.2 FPS.
2008 MacBook Pro:
HTML Firefox 3.6: 4.5 FPS
Flash 10 Firefox: 13.5 FPS
Java: 13.5 FPS
*STOP PRESS*
Made a small optimization (using a BufferStrategy in Java).
Java jumped from 21 to *215* FPS on my XP machine – this is probably the hardware acceleration that finally kicked in. No difference on Mac.
2007 Thinkpad T60p, running Windows XP, Java 1.6.0_20:
HTML Firefox: 4.28 FPS
Flash 10 Firefox: 12.9 FPS
Java: 215.2 FPS. (No difference between browsers.)
2008 MacBook Pro:
HTML Firefox 3.6: 4.5 FPS
Flash 10 Firefox: 13.5 FPS
Java: 13.5 FPS
jesper juul !could you explain!(using bufferstrategy in java)was that made in the browser?if so is there a way to activate it in the about:config(java)small bunp i dont notice but these number are a huge difference would be nice to have info on the trick you used!ty
@Carol
You can download the program source code on the web page: http://soupgames.net/guimark2/
The concept is that Java can deal with off-screen images in a few different ways. I was first just creating an image, drawing to it, and then drawing it on the screen.
Since Java 1.4 (I think), there has been a feature called a BufferStrategy that takes care of creating the offscreen image, showing it on the screen, and so on, for you. To use this, you have to change the Java program – you can’t do it in the browser configuration.
ok ty for info would have been a nice option if it speeded up java like you mention!)
this is a good tread(i was testing firefox(yesterdays nightly)
mm!i tried your modified version(java latest)(w7 64)and it stays at same speed.i sure wish what was different
@carol What results are you getting?
I have just tested on a Dell Latitude E6500 running XP:
Firefox HTML: 6.5 FPS
Firefox Flash: 19.6 FPS
Java: 36.5 FPS
(This machine has integrated graphics, so that would explain why the Java speedup is less pronounced.)
Firefox 3.7a5-pre (a nightly build from a few weeks ago), Windows 7 (64),amd x2 550,ati 5770 . This build has Direct2D acceleration enabled,and directwrite enabled.
Flash, vector test: 35.78 fps
HTML, vector test: 5.93 fps
Flash, bitmap test: 19.74 fps
HTML, bitmap test: 58.82 fps
Flash, text test: 2.64 fps
HTML , text test: 49.1 fps
vector 1 pixel : 6.12 fps
java bitmap 600 : 39.82 fps
flash quality superior !
…any chance of seeing more 10.5x numbers on that same machine? It’s out for OSX and you can run it from a tarball on Linux, please give it a whirl.
opera 10.53, Windows 7 (64),amd x2 550,ati 5770 .
Flash, vector test:41.77 fps
HTML, vector test: 28.76 fps
Flash, bitmap test: 21.5 fps
HTML, bitmap test: 16.09fps this crashed,but when opera restarted it did load the second time.
Flash, text test: 2.52 fps
HTML , text test:32.46 fps
vector 1 pixel : 28.85 fps
java bitmap 600 : 39.45 fps
as for dsx i have amd cpu dont think it work they are compatible.if there is a way i dont know it and dont have an osx copy
(sorry)
google=firefox in all but 2 test
bitmap,text (no Direct2D acceleration enabled,and no directwrite enabled for google)
but if you load app online that take advange of these,firefox is the clear winner.
and the fact it isnt subject to far futur licence issue(htm5 pay to download in apple,ms,google version) is a huge reason to love firefox(openvideo.dailymotion.com)html5 keep up the good work firefox team.
http://beta.html5test.com/
heres a nice little test a lot of browser wont like!
Thanks for this really outstanding analysis.
And I’d like to echo the thoughts of a previous commenter. I work on RIAs in the financial industry and both the vector test and the text test are 100% real world. I have an app that is basically the vector test on steroids. I tried a build-out in HTML5 of this app and in the real-world it was a non-starter.
This analysis is a really fantastic baseline for a lot of my work going forward.
Thanks again.
http://www.basschouten.com/blog1.php/2010/03/02/presenting-direct2d-hardware-acceleratio
here are some tip for html5
also if you plan to test webm you will need the proper version and add this to the link
&webm=1
as for the result yep im an amateur!i just dig a lot.
the low result you ll get in javascript are normal.since lot of error were made in those code and would be very hard to modify
(cross your finger)ecmascript 5 is official,mozilla only miss tree feature.now the only thing left is learn ecmascript ways.thats gona be hard since not many knows how to program it .might need to use the older jslint(es3)
@jesper juul! did you tweak you java test for firefox ?i retried today your test (post 47 )
Firefox 3.7a5-pre ,received an update on the 21 of may, Windows 7 (64),amd x2 550,ati 5770 . This build has Direct2D acceleration enabled,and directwrite enabled.
java bitmap 600:buffer:strategy:647.63 fps
:buffer:image :43.5 fps
i dont know what this tweak is but if i were you i would pass it on to firefox! do you see the huge difference !true its on the latest nightly but still its very good new if it can be applied to firefox itself in some way!
I’m seeing significantly better performance in Flash 10.1 (rc5) than 10.0. The quality setting defaulted to High (and I didn’t change it). Win7 64bit, IE8 7600.16385, Chrome 4.1.249.1064
Vector Charting
Chrome 10.0 – 39.9
Chrome 10.1 – 47.54
IE 10.1 – 60.06
Chrome HTML – 6.57
Bitmap Game Test
Chrome 10.0 – 31.83
Chrome 10.1 – 33.46
IE 10.1 – 38.43
Chrome HTML – 11.86
Text Column Test
Chrome 10.0 – 2.35
Chrome 10.1 – 26.92
IE 10.1 – 31.12
Chrome HTML 29.88
I’m seeing some really weird (and awesome) results with Chrome 5.0.375.53 on Mac. In the html tests, Chrome is pretty much exactly as fast as Firefox 3.7a5pre (20100523, to be exact) and somewhat slower than Safari 4.
In the Flash tests, though, Chrome in combination with 10.1 RC 5 blows the other browsers right out of the water:
GUIMark 2 – vector
FP 10.1RC5 in Safari 4.0.5: 30.61
FP 10.1RC5 in Chrome 5: 60.09
FP 10.1RC4 in Chrome 5: 54.68
GUIMark 2 – Bitmap
FP 10.1RC5 in Safari 4.0.5: 14.49
FP 10.1RC5 in Chrome 5: 16.89
FP 10.1RC4 in Chrome 5: 14.89
GUIMark 2 – Text
FP 10.1RC5 in Safari 4.0.5: 30.58
FP 10.1RC5 in Chrome 5: 59.8
FP 10.1RC4 in Chrome 5: 44.61
I’ve got no idea where this difference in performance is coming from – but it is definitely real, as the animations look much smoother in Chrome than in Safari. Also, 10.1 RC5 is a huge step forward from RC4, especially when it comes to text rendering.
i dont know what apple do to chrome but the test i get with latest flash (rc6) is 24 fps way better then the 3 fps or so i had previously here but still very short of the firefox latest nightly(window)
i dont get why firefox is so slow at vector test tho. i mean 6 fps ?i sure dont get why vector is so bad in firefox .
java bitmap 600:buffer:strategy:585 fps
buffer:image :39 fps
test of poster 47 (java version)
as you can see a huge difference.is there a way to take advantage of this in browser?good question!
I just thought I should point out that Flash Player 10.1 uses Core Animation on Mac OS X for rendering on Firefox 3.7 nightlies. I’ve heard that Safari and Chrome both have support in various stages as well, but I’m not using either of their nightlies.
You explain that the html5 option forces a refresh with every line drawn. In the Java version, the buffer strategy lets you draw to offscreen graphics first so that there’s only 1 image drawn to the screen per frame. Is no such technique possible with html5 canvas drawing?
weird it bitpmap java (fast )doesnt work this morning .worked 2 second ago!probably a java error
god knows we ll never run out of those lol!
nope firefox did something or one setting i got is interfering with java tweak because i cant speed it up now.ho well at least i got to see that it was possible ty jasper for that tweak info
dont know what specific condition it needs to works but its pretty specific from what i m seeing today!
my.opera.com/desktopteam/
Could one of you kids PLEASE throw in some OSX numbers for 10.5x or 10.60 alpha? Actually, both on the same system if possible. 10.60 claims very large increases in javascript and render speed, and has 3x higher text parsing (than chrome) on Peackeeper!
i bet you don’t know what cacheAsBitmap is. and i certainly think there is no GPU acceleration enabled on the flash movies.
i peaked into the code.
now i wish i didn’t.
Bitmap Gaming Test:
– smoothing = true
– bitmapData = this.bitmaps[frame].bitmapData;
2 kinda inefficient lines.
– what is this?
for(var i:int=0; i<monsters.length; i++){…
try this instead and you don't have to register an array, a step counter, just 1 object.
for (x=getChain(); x; x = x.next){}
you can go way part 3000 with that @ almost 100 fps. (if you caching bitmaps accordingly)
for the text test:
if you use system fonts and not embedded fonts the rendering will be thrown up by flash and there is a constant stream of visual representation from the system into flash.
this is very slow indeed, and also points out that as this not flash is rendering the text.
embed the fonts, and you should get a better result. you may use the antialiasing mode for animation for another boost of performance.
i don’t blame you for this. most people build space stations in flash but they just simply can’t use the text API. is something like the print API, people don’t even know that exists such thing in flash, yet alone the power of it.
I tested the GUIMark – Text Column Test with the new Adobe Flash 10.1.53.64 (official release) and i got an average frame rate of 24.62 fps on Firefox on Windows 7 x64. A big jump from an average of just 1.48 fps.
yep flash 10.1 is faster on text but there is a bug or something is wrong because i get shding artifact or sutch when i try to stream i does artifact i both chrome or firefox didnt do that in previous official released
Your TL;DR section was too long; I didn’t read it.
http://www.jesperjuul.net/ludologist/?p=1030 by the way there is the java version of these test avail to this linkj
i dfont know if they are accurate programe but it give a fair indication that if that if microsoft fixed their bug pertaining to the gdi-d3d issue we would gain tremendous performance (probably why ms isnt keen on fixing it(no fixing no sun java acceleration for lot of feature ,and aero as a bug that stop some java acceleration as well
last but not least flash 10.1 accl give bad image quality in lot of system so ifg you have artefact disable accleretion of flash(ya i knoqw why rave about all those acclereation if it cant be used because mainly of ms it kind of defeat the purpose of getting acceleration lol
by the way if you are a ubuntu user this page and the one i linked previously is pretty mutch useless since
these would need to be either written in opencl,opengl webgl version of java and html5 since
ms software arent in linux .might be their own version avail soon who know one thing is sure without hard ware acceleration i saw what ubuntu miss to compete with ms its a test similar to these that would show the power of the graphic .so far .im not impressed for acceleration be it in java or html5.(firefox)(ubuntu)
Latest chrome 5.0.375.86 OSX with embedded flash has HW acceleration, and boy does it show.
Flash
59.78
24.94
60.1
HTML5
3.07
7.49
16.72
These are consistent with the results reported earler on the beta chrome build. This is *not* low quality (as far as I can tell the low quality option has been removed in this version anyway.. it’s not like anyone ever used it) … it’s just screaming fast.
10.60 is now out.
for some weird reason the newer firefox minefield made error in their new programming cause it prevent html5 acceleration right now im 30% the speed i used to get in my previous post here.so there is only one thing that can have happened .acceleration dont work anymore on the latest minefield.i do hope mozilla look into this issue cause dropping from 55 fps to 17 fps is a major drop.
A quick, but *very* important note about the canvas bitmap drawing benchmark:
Always ALWAYS pass another HTMLCanvasElement to drawImage. Passing an Image is extremely inefficient (for reasons I’m not 100% clear on, but probably because it has to copy pixels from the DOM Image representation everytime). Creating canvas elements for each image in the assets array boosts the average FPS in 64bit Chrome in Linux on my laptop from 8.22 to 14.99.
this works fine in linux ubuntu!but for some odd reason stopped being vector and bitmap accelerated in html5
(firefox latest minefield)cant compare with other borwser since most arent accelerated yet.
we ll have to wait and see if mozilla find what was in 3.7.5 minefield that isnt in minefield 4 /3
bitmap html5 is back to 60 fps (yep firefox found what the acceleration issue was!(we were at 16 fps for a bunch of week truely very slow)
You should strongly consider adding a Silverlight test to your benchmark. Its adoption is now globally above 60% (source: riastats.com) and tests like those at http://www.bubblemark.com have shown that Silverlight 3.0/4.0 CLR (.NET) apps significantly outperform Flash and HTML apps. Excluding it from an RIA performance benchmark would be like excluding Usain Bolt from a 100m sprint race.
silverlight has always been the fastest but the fact other standard are more widly accepted or being accepted pretty mutch make the speed of silverlight irrelevent!
but if you want to know on the site you provided with the silverlight 3 test i got 1054 fps!
yep close to twice faster then anything i tested on this page(yep silverlight is scary fast)
but i wonder when people will start to use it more !that a very good question.so far people pretty mutch still use flash!
Tried the various tests with IE9 beta1 + Flash 10.1 64bit beta
The results are positive for IE9, but the Flash beta sometimes scored only 1/10
Vector charting
HTML 5 8,84
Flash 3,18
Bitmap Gaming Test
HTML 5 58
Flash 2,46
Text Column Test
HTML 5 50,49
Flash 10,41
Tried Jesper Juuls 1.3 version of the Bitmap Gaming Test using JRE 1.6.0.22
score 58,41 at 600 monsters
I HATE U!!!!!!!!!!!!!!!!!!
I want to point out that the immediate mode rendering of HTML 5 can be optimized at the implementaion level, where each rendering command can be queued first and then processed once all user game code has finished. This is what most current multi-thread DirectX/OpenGl/Console game engine does, and should be able to scale with multiple core.
I have some doubts about the reliability of your method of measuring FPS in HTML5. On a low-end laptop with WinXP and Firefox 3.6, your HTML5 Text test reported a result of 13 FPS, but visually the browser display was largely frozen – it only updated a few times during the test.
Did tests again with IE9 RC1 + Flash 10.1 64bit Square preview 3
The results are positive for IE9 especially Vector test , but the Flash not so much
And Jesper Juuls test now fails to load!!
Vector charting
HTML 5 before 8,84 now 25,45
Flash before 3,18 now 4,59
Bitmap Gaming Test
HTML 5 before 58 now 59,58
Flash before 2,46 now 3,73
Text Column Test
HTML 5 before 50,49 now 49,96
Flash before 10,41 now 11,61
Tried Jesper Juuls 1.3 version of the Bitmap Gaming Test using JRE 1.6.0.23
at 600 monsters before 58,41 now IE9 RC fails
Re-installing JAVA solved it… the Juul test dropped to 50,24
did you read what i posted on the juul thread do those step and you should have speed
Firefox 4 x64 + Flash 10.3 on Win 7 x64
Vector charting
HTML 5 25.24 fps
Flash 50.16 fps
Bitmap Gaming Test
HTML 5 57.14 fps
Flash 39.24 fps
Text Column Test
HTML 5 55.77 fps
Flash 27.26 fps
update!the java trick jesperjuul had publish no longer work in minefield as of this week i would have thot mozilla would try to take advange of this trick in some way in firefox but it seems to be the reverse so now silverlight is the only true fast option.and by a lot!before?silverlight was only 2 time faster then the fastest thing avail anywhere!just thot i would let you guys know about java.ho not because there isnt speed in java the trouble is ms have bug and lot of trick for it arent widelly used !(or widely unknown to most! i guess)
SeaMonkey/2.1b3 x32 + Flash 10.3 on Win 7 x32 sp1
Vector charting
HTML 5 23.58 fps
Flash 49.53 fps
Bitmap Gaming Test
HTML 5 58.84 fps
Flash 49.16 fps
Text Column Test
HTML 5 41.77 fps
Flash 29.12 fps
after a few test i can safely say ie9 is the better browser now!(http://ie.microsoft.com/testdrive/Browser/TrackingProtectionLists/Default.html)dont forget to activate active x protection (then you can disable selectily)
Great work!!!
Why didn’t you use FTE for the text column test?
Hi, I made an update for your flash text demo, using the FTE instead of standard TextField class.
Here the source + assets:
http://memmie.lenglet.name/documents/lab/FlashFTETextTest.zip
It’s use flash.text.engine classes to emulate a standard TextField to test performances. It’s not perfect, but work correctly for this demo. I’m pretty sure (but I tested only on few mac browsers) that using FTE is better than TextField.
FTE need CFF fonts (OTF). I convert your both fonts (Champignon and Star Jedi) with FontForge but I didn’t found the correct way to embed these fonts directly with Flex SDK (with font embed metatag) without getting errors. So, I compile a SWF file with Flash Pro CS5.5 and embed its symbols (fonts class) with class embed metatag.
http://www.conceivablytech.com/wp-content/uploads/2011/06/gui1.jpg
http://www.conceivablytech.com/wp-content/uploads/2011/06/gui2.jpg
http://www.conceivablytech.com/wp-content/uploads/2011/06/gui3.jpg
HTML5 browsers totally annihilates Flash now in performance, it’s not even a contest anymore, Flash is outdated, obsolete relic of the past.
Two years ago I had an itch that needed scratching. “RIA” was the future of the web and every major company seemed to have a solution to get us there. I developed the first version of GUIMark to not only get a good understanding of the respective technologies, but also to give my clients through EffectiveUI and everyone else something to actively gauge the rendering performance of the different runtimes.