Blowing up HTML5 video and mapping it into 3D space

By Sean Christmann | Posted April 20th, 2010 | Html, Javascript

I’ve been doing a bit of experimenting with the Canvas and Video tags in HTML5 lately, and found some cool features hiding in plain sight. One of those is the Canvas.drawImage() api call. Here is the description on the draft site.

3.10 Images

To draw images onto the canvas, the drawImage method can be used.
This method can be invoked with three different sets of arguments:
  • drawImage(image, dx, dy)
  • drawImage(image, dx, dy, dw, dh)
  • drawImage(image, sx, sy, sw, sh, dx, dy, dw, dh)
Each of those three can take either an HTMLImageElement, an HTMLCanvasElement, or an HTMLVideoElement for the image argument.

The api lets you take the contents of specific HTML elements and draw them into a canvas, and the 3rd element in that list is just begging to be abused. Copying video into a canvas element means opening up the ability to manipulate or process video frames at runtime. Two concepts instantly came to mind that seemed like fun to try and figure out, here they are below.

Blowing apart fragments of video

Click around the video frame to blow up that part of the video, the exploded pieces will continue to play the video inside them. After a while they retract back to their original place. One feature I didn’t have time to figure out was adding depth to the explosion, so pieces that are closest to ground zero fly up into the air as they sail outward. With full shadow effects this could look really cool.

3D Video

This demo in particular runs really well inside webkit based browsers, but not so much in Firefox. Firefox doesn’t appear to have any hardware acceleration for Ogg decoding so I had to drop the video size in half in order to run at acceptable framerates. Even still, Firefox chokes pretty badly on my Macbook Pro.

*Update* – I’ve changed the ogg video to be 640 x 360, prepare to see firefox weep

Lessons learned

There’s a couple hints I found out along the way that are good to know if you want to play around with drawing video. First, you need a bit of hackish code to get this to work effeciently and it flows like this.

[Video playing] -> [Draw Video onto Canvas 1] -> [Draw fragments of Canvas 1 onto Canvas 2]

Don’t ask me why, but copying pixel data out of a video tag is expensive, so expensive that drawing it into a temporary canvas, and then drawing pieces of that temp canvas onto a final canvas is faster then just referencing the video tag repeatedly within the same loop. That’s why you’ll see 2 Canvases in the source code for the demos. I’m sure there’s a technical reason for this duplication process, but it’s a lazy reason.

Secondly, don’t try copying individual pixels around. You can still see the remnants of my first code attempt inside the explosion demo with getPixel() and setPixel(). This turned out to be horribly slow and completely unnecessary. Canvas.drawImage() + matrix transforms on the canvas space is far more efficient then handcrafted pixel pushing. On the other hand, pixel manipulation allows you to do things like runtime chroma keying, get ready for a new wave of “clippy” style videos with full transparency popping over websites to help you out.

Lastly, I’m learning very quickly that not all browsers are created equal when it comes to performance, it’s a crapshoot when it comes to heavy video+image manipulation. Safari and Chrome work well with h.264, Firefox slogs along with Ogg Theora, and Opera is somewhere in the middle.

Reader Comments (305) Leave a Comment

  1. Juho Vepsäläinen | April 20, 2010 at 10:18 am | permalink

    That’s way cool! I guess someday we will see a canvas based video sequencer. >.<

  2. Brent Bonet | April 20, 2010 at 11:09 am | permalink

    Awesome Sean! Thanks.

  3. Guilherme | April 20, 2010 at 11:18 am | permalink

    I can’t see the video in any off my browsers.

  4. Brian Graves | April 20, 2010 at 12:07 pm | permalink

    Worked for me Windows XP + Firefox 3.6

  5. Paul Irish | April 20, 2010 at 1:28 pm | permalink

    Wow. please add the depth effect to the first demo. That is incredible.

  6. narendra | April 20, 2010 at 1:46 pm | permalink

    I liked it very much. Also, there is a project 3D video demo present at –

  7. Lech Rzedzicki | April 20, 2010 at 3:15 pm | permalink

    Runs well in Chrome Beta on OS X, at least you have alternatives to Safari that run it well…

  8. Gootie | April 20, 2010 at 6:04 pm | permalink

    It also worked well in Ubuntu + Chromium. Awesome

  9. Raptor007 | April 20, 2010 at 6:32 pm | permalink

    Worked perfectly in Firefox 3.6.3 for Mac OS X. The second demo was a little slow, but the first one ran great.

  10. Nicholas | April 20, 2010 at 6:45 pm | permalink

    Worked on Firefox 3.6.3 for Windows XP. Both versions worked great!

  11. Sean Christmann | April 20, 2010 at 7:00 pm | permalink

    Yeah I’ve learned throughout the day that Firefox on Windows isn’t too problematic, it’s just Firefox on Mac that is dog slow in the 3D demo.

  12. Adam | April 20, 2010 at 7:29 pm | permalink

    Worked flawlessly on Windows 7 with Chrome. Really cool!

  13. Lars Pallesen | April 20, 2010 at 8:46 pm | permalink

    Incredibly cool! Very well done indeed!

    What was that Adobe said about HTML5 video not being as advanced as Flash video? You just showed us HOW advanced HTML5 is!

  14. Aaron Davies | April 20, 2010 at 8:53 pm | permalink

    Works beautifully in Safari 4.0.5 on OS X 10.5.8 (even on this 1st Gen MacBook Air, notorious for having terrible video performance). I will note though that Safari froze solid for a good twenty seconds while loading each of the pages.

  15. Justin | April 20, 2010 at 9:22 pm | permalink

    Wow, this is great! Works great on Chrome (42898).

  16. Derek | April 20, 2010 at 9:24 pm | permalink

    No happiness on iPad, chrome on karmic works well.

  17. Matt M | April 20, 2010 at 9:34 pm | permalink

    Safari on Mac Pro plays both flawlessly.

  18. Macbook | April 20, 2010 at 9:48 pm | permalink

    Works well with Safari on a plain ol’ 2010 MacBook.

  19. Jeff Singer | April 20, 2010 at 10:27 pm | permalink

    No luck on the iPad. Didn’t do anything.

  20. Sean Christmann | April 20, 2010 at 10:49 pm | permalink

    I’m not sure why iPad wouldn’t work for you guys, Safari plays it perfectly.

  21. Suman | April 20, 2010 at 11:22 pm | permalink

    Awesome yaar

  22. CantGetANick | April 20, 2010 at 11:27 pm | permalink

    Awesome, Just works. Flawless.
    (tried it on FF 3.6 on rhel 5.1.)

  23. Khalid | April 20, 2010 at 11:53 pm | permalink

    Works just fine even on an Acer Aspire D250 netbook running Chrome.

  24. paul | April 21, 2010 at 12:40 am | permalink

    hey, nice demo but i think you should also give proper attribution (as required by the Creative Commons license) to the creators of the video that you are using for this demo:

  25. acabada | April 21, 2010 at 1:05 am | permalink

    Is there a way to use already HTML5 in firefox for example?

  26. aaryoo | April 21, 2010 at 1:32 am | permalink

    in Fire Fox 3.6.3 is runnig great! but in IE-8 it doesn’t work!

  27. ppjoe | April 21, 2010 at 1:44 am | permalink

    Shows nothing on my iPhone.

  28. Rahul J | April 21, 2010 at 2:56 am | permalink

    Great work Sean…

  29. gumboot | April 21, 2010 at 3:14 am | permalink

    Transcoding the original MP4 to same-resolution Theora the performance is about the same for both in Safari (via XiphQT) and Chrome. Opera plays the full-resolution Theora just fine (earliest core2 mac mini), but doesn’t seem to want anything to do with the H.264 version. Firefox may or may not have problems (I don’t have it installed) but there’s no reason to assume they relate to video decode hardware acceleration.

    Could you provide the same resolution in both formats? Right now you’re penalising Opera artificially.

  30. holy | April 21, 2010 at 4:00 am | permalink

    works on opera 10.51 on win7.

  31. Passy | April 21, 2010 at 4:02 am | permalink

    Totally awesome!

  32. PaulG | April 21, 2010 at 4:04 am | permalink

    Chrome 4.1, FF 3.6.3 , Opera 10.50 both examples worked fine, the size of video on Chrome seemed larger and the “blow up” affected a smaller amount of video though, in FF and Opera it blew virtually all of it up with one click.

    Win2003 Server SP2

    Nice job.

  33. Jeremy | April 21, 2010 at 4:23 am | permalink

    Stunning stuff. Sweet in Chrome on Vista. Both fail in IE9 Preview (“Object no found” error at the line in Init() that says “copy = copycanvas.getContext(’2d’);”). I’ve logged an issue with Microsoft.

  34. Arthur Langereis | April 21, 2010 at 4:49 am | permalink

    This technique opens up a nice venue of video sprites where you can use a single video file to store a whole bunch of animations in various sizes for in games.
    Does the HTML5 video API allow you to set a movie to a specific frame (or time?) Could be a very efficient way of storing animation data that way.

  35. Ben | April 21, 2010 at 5:04 am | permalink

    Both examples worked surprisingly well on my “old” Dual G5 PowerMac using Safari. As others had stated though, no go on my iPad (well, the second example does show a rotating shadow).

  36. Bob Peterson | April 21, 2010 at 7:23 am | permalink

    Does not work with Safari on Windows 7.

  37. Apfelmännchen | April 21, 2010 at 7:36 am | permalink

    @Arthur Langereis:
    Sure, you can!
    Have a look at e.g.. Apple’s HTMLMediaElement-class-reference to learn how.

    You’ll find the documentation for the corresponding property at:

    Be aware of the fact that you provide the time in seconds, though!

  38. Dustin Wilson | April 21, 2010 at 8:15 am | permalink

    Works excellently in Opera 10.5. I can’t tell any noticeable difference between how fast Opera handles either compared to Safari on my computer. Opera certainly blows the pieces apart faster, but that could be just because there are less of them. Unsure why the video is smaller in Opera and Firefox, though.

  39. John Poole | April 21, 2010 at 9:50 am | permalink

    Failed in Windows XP Pro SP3 + Firefox 3.6.3 (Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv: Gecko/20100401 Firefox/3.6.3 GTB6 (.NET CLR 3.5.30729))

    Are there certain plug-ins required?

  40. Brad Neuberg | April 21, 2010 at 11:20 am | permalink

    This is hot!

  41. Juriy | April 21, 2010 at 11:34 am | permalink

    3D video is awesome !

  42. Sean Christmann | April 21, 2010 at 11:41 am | permalink

    Alright Firefox and Opera users, the ogg video is now 640 x 360, enjoy.

  43. Darryl | April 21, 2010 at 11:50 am | permalink

    very cool demo.

  44. Michael | April 21, 2010 at 12:33 pm | permalink


    Ok – so I tried to replicate on my server so I could start playing with your math and start ripping it apart to try and understand it – but I cannot get the .ogv to play. I dl’ed it from you, and uploaded… do I need to re-encode?

  45. Mark | April 21, 2010 at 12:50 pm | permalink

    3D implementation of video/web in Second Life was just introduced. Others may be interested, so I thought I would share this demo with more info:

  46. Edgeman | April 21, 2010 at 12:56 pm | permalink

    This ran super (no slowdowns at all) on Firefox 3.6.3 on Win7 32bit.
    Ran like crap in my Chrome, same OS/computer.

  47. Sean Christmann | April 21, 2010 at 1:18 pm | permalink

    michael I’m not sure why it’s wouldn’t work for you, you shouldn’t have to re-encode to play it elsewhere

  48. Paul | April 21, 2010 at 1:37 pm | permalink

    Excellent example of html5 manipulation – don’t worry to much about speed at the moment, when FF uses direct2D to draw to screen you will see a vast improvement. re: iPad & iPhone – they seemed to have ‘crippled’ html audio and video in safari for these – it only works if the src is streamed. I guess they don’t want anyone making there own iTunes in html :(

  49. Martin | April 21, 2010 at 2:00 pm | permalink

    I tried this on both Firefox 3.6 and the latest install of chrome. I found that Chrome handled the spinning and shattering of the video better but Firefox did the actually playing of the video much more smoothly. Also the video looping failed on Chrome, so it would play through once, then instead of restarting it would only stop on the first frame. I was surprised, because even without reading that the author expected poor performance on Firefox, I thought Chrome would be better for html5, it seems that this is not universally true. I’m running winxp SP3 with a 3.2 Ghz P4, and 2GB of ram.

    CORRECTION: After revisiting the page in Chrome, I found it was only choppy because ,unlike Firefox, it did not wait when the network fell behind, it would stall then skip whenever the network caught up. It still does not loop properly.

  50. infocyde | April 21, 2010 at 2:03 pm | permalink

    Awesome…I know this is just a demo, but my mind is already racing with possibilities. Great post.

  51. Matthew | April 21, 2010 at 2:05 pm | permalink

    Colour me impressed.

  52. Flix | April 21, 2010 at 6:28 pm | permalink

    Is this optimized at all or just a proof-of-concept? Because it eats ~70% of one core and ~20% total CPU on my 2.4 GHz Machine.

  53. AdrianT | April 21, 2010 at 11:29 pm | permalink

    Works well for me on Ubuntu 9.1 and Windows 7 in firefox

  54. andrew | April 22, 2010 at 12:11 am | permalink

    The slow performance copying out of the video tag is likely because the video is in the YUV color space, copying it to a Canvas will require converting it to the RGB color space (Every pixel needs to be converted individually which is, of course, murder on the CPU. It can be sped up but will require SSE/Neon/Altivec/etc optimisations but only if the blocks being copied are big enough; I’m betting most browsers don’t have those anyway). Rendering directly to a canvas means you get a bulk-transfer-bonus by doing the conversion all at once rather than piecemeal, same optimisation requirement applies though.

    getPixel/setPixel are generally a bad idea for the same reason that accessing random cells out of order in an array is a bad idea, modern CPUs are designed to operate in bulk modes — do large chunks of repetitious work on a slab of memory. Peeling individual pixels apart rather than grinding the lot as a whole will be massively slower. (There may also be GPU bus overhead with pixels flying back and forth between the video card memory and the CPU which is extra-super-slow)

    BTW. I just had a horrible idea of an SVG+JavaScript with embedded HTML5 video tags as some sort of movie palette that zooms in and out on each individual clip when clicked (with grey out and enabling sound on the selected only). I suspect you could probably lock the entire browser UI with something like that.

  55. esses designs | April 22, 2010 at 1:00 am | permalink

    brilliant!!! (in Safari on an ’07 BlackBook)

    Adobe should stop being afraid of Flash dying, and focus on making HTML5 tools…

  56. [...] 5 you’ve got video in a box, that’s it. Now that’s not to say you can’t do cool stuff with HTML 5 video – but no green screen action. Then, don’t forget about DRM (I don’t care about [...]

  57. Leif Arne Storset | April 22, 2010 at 3:17 am | permalink

    andrew: Great explanation, thanks.

  58. Michael | April 22, 2010 at 9:51 am | permalink

    So weird — when I put the video on my server it doesn’t play – it just wants to download – but on the craftymind server it plays. It there an .htaccess rule or server rule I should be aware of?

  59. Jan | April 22, 2010 at 11:15 am | permalink

    Very cool!

    Works better for me with Opera 10.52 RC for Mac than with Chrome 5 and Safari 4. Perhaps worthwhile removing the “best viewed in Webkit browsers”?

  60. yiutsu | April 22, 2010 at 11:32 am | permalink

    worked for me —> google chrome + windows XP, (sp3)

  61. grimm | April 22, 2010 at 12:09 pm | permalink

    Works great in Firefox 3.5.9 on Fedora 12. :)

  62. Sean Christmann | April 22, 2010 at 12:28 pm | permalink

    Andrew, that makes a lot of sense about the YUV to RGB conversion. I was naively assuming that a bitmap is a bitmap is a bitmap, and whether you grab that from a video raster or a canvas raster shouldn’t matter. Thanks for the explanation.

  63. Zack Weinberg | April 22, 2010 at 12:30 pm | permalink

    In addition to what andrew said @54, I want to point out that while copying data from main memory *into* GPU memory is slow, copying *back* from GPU to main memory is even slower. If you do bulk operations on canvas data, the browser can (at least in theory) stream compressed video into the GPU memory and do all the manipulation there, which is relatively fast, but individual pixel manipulation has to come back to main memory, which crawls.

    I’m not sure how optimized this stuff is in the current generation of browsers, but making it go as fast as possible is on everyone’s roadmap.

  64. KT | April 22, 2010 at 2:29 pm | permalink

    Nicely done, Sean.

    Having said that, both HTML5 video demos run beautifully in Opera v10.52 RC4 for Mac at full-resolution; so the “best viewed…” line is pedestrian. Give due credit to Opera developers. It’s a smokin’ desktop/iPhone browser that makes the agnostic Web better for users & easier for developers (especially with the full Cocoa rewrite on Mac builds now).

  65. Simone Deponti | April 22, 2010 at 3:17 pm | permalink

    Whoa, surely great work.

    Btw, it worked perfectly on a Lenovo IdeaPad s10, running Fedora 12, with latest Google Chrome… so I’d say this HAS to run fine on whatever other hardware ;)

  66. Guiltouf | April 22, 2010 at 5:18 pm | permalink

    Whaou ! Impressive effect !

    Clap clap.

  67. Hayes | April 22, 2010 at 9:47 pm | permalink

    iPad does not have ogg codec, only h.264. Html5 video is a container not a specific codec. That.s what the battle is about.

  68. bruce lawson | April 23, 2010 at 7:30 am | permalink

    Top stuff, sir.

    I query your “best viewed” message, though. Opera 10.51 on my little Win laptop behaved impeccably too.

  69. emphatic | April 23, 2010 at 1:21 pm | permalink

    And you compare this to what you can do with flash? Maybe pick up a flash book. Forget all of the platform problems. (just keep glossing over them). This level of code is beyond the vast majority of web site devs and it doesn’t even work correctly. To say this is somehow a ‘solution’ for a web site build is beyond naive. Are you telling me you would get paid for this? Hilarious. If you’re a hobbyist, great. Have fun. If you are a prfessional, HTML5 is not ready for primetime.

  70. thebear | April 23, 2010 at 2:59 pm | permalink


    Are you kidding?
    “This level of code is beyond the vast majority of web site devs and it doesn’t even work correctly.”

    If you are developing websites and you can’t work out this code, you my friend are shortchanging your customers and giving the industry a bad name.

    ” If you are a prfessional, HTML5 is not ready for primetime.”
    shows you are a windows, ie centric fool.
    There is more to computing than “Microsoft”

  71. Sean Christmann | April 23, 2010 at 5:40 pm | permalink

    emphatic, you are reading more into blog post then what I’ve stated. All I’ve done is show some code I’ve been playing around with, nothing more, nothing less.

  72. daemonna | April 25, 2010 at 3:25 am | permalink

    @thebear: sorry but html5 is really not ready… it’s not working same on all browsers. despite you dont need plugin (like flash), you need ‘right’ browser. what a win? me as flex/flash developer, i dont need to worry what browser you use… get it?
    what about ogg/h246 battle??? flash video playes everywhere same… no OS or codecs problems. :P

    and if you think this is stuff of windows centric fools, you’re wrong… i use Flex4 SDK under linux and solaris using only vim, i am happy with flash, outcome is same on every platform (and i dont care about browser problems) + it’s still easier do any coding in AS3 (few lines of code) then to do programming in Ajax (many many lines of code)…. + i got multitouch, GPU (pixel bender) effects and many other features not possible or too complicated with JavaScript…

  73. esses designs | April 25, 2010 at 9:19 pm | permalink

    i wouldn’t necessarily say that HTML5 isn’t ready. i’d say the browser support isn’t ready.

  74. NetBookUser | April 26, 2010 at 6:19 am | permalink

    Works on an Atom Netbook, Win 7 and Chrome. Very nice

  75. nate | April 26, 2010 at 11:00 am | permalink

    Michael, try adding this to your .htaccess file.

    AddType audio/ogg .oga
    AddType video/ogg .ogv .ogg

  76. James Womack | April 28, 2010 at 4:22 pm | permalink

    Hello, do you mind if I used the 3d video source code on my website? Please e-mail me if you have time. Thanks very much for the demos, I did not know this was possible!

  77. Mrmowgli | April 28, 2010 at 10:12 pm | permalink

    Worked like a champ on linux/chrome with an old video card, impressive!

  78. jake | May 2, 2010 at 1:16 am | permalink

    What the hell is the use of that? I don’t see why I’d ever need to blow up my videos or make them rotate like that.

  79. RazorX | May 2, 2010 at 6:58 pm | permalink

    Uhh yeah, “Blowing up HTML5 video” was tested on the iPad and it doesn’t even work! This new video demonstrates it:

    It’s sad how dysfunctional and pathetically slow HTML5 is!

  80. Blog Tactic | May 4, 2010 at 5:31 am | permalink

    interesting demo

  81. Tito Castro | May 4, 2010 at 2:03 pm | permalink

    The effect is really awesome!! Congrats!!

    What about the control buttons? Play/Pause/Volume, etc??

    ¿Is it possible to do that over the video on HTML5?

  82. Cd | May 8, 2010 at 8:03 am | permalink

    Doesn’t work on iPad

  83. Jason | May 8, 2010 at 7:16 pm | permalink

    Pathetically slow… like every other HTML5 example

  84. Cool stuff | May 9, 2010 at 9:00 pm | permalink

    all Your posts seem very complicated, he

  85. Deluxe Blog Tips | May 11, 2010 at 4:02 am | permalink

    Cool effect. The 3D video looks great. Thanks for sharing.

  86. Bogdan | May 11, 2010 at 4:03 am | permalink

    Pleeeeeeeeease! It’s so 80′s! Flask rocks!

  87. Ries | May 11, 2010 at 7:36 am | permalink

    Cool but not all browser support HTML5 for now

  88. eco_bach | May 11, 2010 at 8:10 am | permalink

    Suggest viewing these HTML5 Flash benchmarks

  89. tianlan | May 11, 2010 at 7:30 pm | permalink

    tai li hai le! yao duo xue xi !zhen xuan!(*^__^*)

  90. Euge | May 12, 2010 at 7:41 pm | permalink

    Ok step in the right direction, but how hard is this going to be for us UI designers/developers?

    For instance how many people are still having issues writing css that all browsers can read without having to hack it up a bit. IE, 6 -7 ring alarm bells! Hell talk about throw things back a few years. I know these browsers wont live for ever but there are still people still using these redundant old browsers, half the reason why flash was introduced in the first place – runs on all browses and platforms universally.

    The turn around time for open standards such as html 5 is too slow, and not everyone adopts them the same. At the current rate it’ll be another 10 years before its up to speed with flash, by that time we would have moved on.

    Things need to speed up before we can start using it properly. If you guys are that serious about it then make it work, create tools and flash equivalent applications that help designers use html5 in all its glory.

  91. Rich | May 13, 2010 at 11:47 am | permalink

    Fast as hell on our computers! wow

  92. AAA | May 13, 2010 at 8:55 pm | permalink


  93. ???? Flash ???? Gianduia | BBER | May 15, 2010 at 5:32 pm | permalink

    [...] ?????? ?? Flash ? Silverlight ?????????????????????? HTML ????????? HTML5 ????????? Canvas????????????? HTML ??????????????????????????????????????????(DOM)??????????????????? ???????????(CSS)??????????????? ???????????????????? Flash ??????????????????????????????????? Flash ????????? [...]

  94. [...] jouer avec des vidéos dans un espace 3D il n’est pas nécessaire d’employer WebGL. Craftymind présente ses essais avec l’appel drawImage() de l’API Canvas du [...]

  95. Cmdr._Firewalker | May 19, 2010 at 11:58 am | permalink

    While it appears to be the smoothest in Chrome (comparing Opera 10.53, Firefox 3.6.3, Chrome on a Phenom II 955 in Win XP) it also looks the ugliest on Chrome as Opera and Firefox apply anti-aliasing.

    And as Firefox is clearly slower than the others even in the apparently easier blow up page and Opera lags only slightly in the 3D-Version, I declare Opera my personal winner :)

  96. thiago da silva adriano | May 29, 2010 at 5:03 pm | permalink

    Very good your post my friend !

  97. Jeff | May 31, 2010 at 7:07 pm | permalink

    Ugh, no perspective correction on the 3D?

  98. sbmcn | June 5, 2010 at 2:30 am | permalink

    html5 has many interesting things, if the html5 standard has been confirm, many things need to study too.

  99. Huang xing | June 8, 2010 at 5:33 am | permalink

    That’s way cool!~!I love it

  100. Jan Ku?a | June 14, 2010 at 12:52 am | permalink

    omg, that’s *** awesome! I can see that on a production website.