Html, Javascript

Blowing up HTML5 video and mapping it into 3D space

305 Comments

I’ve been doing a bit of experimenting with the Canvas and Video tags in HTML5 lately, and found some cool features hiding in plain sight. One of those is the Canvas.drawImage() api call. Here is the description on the draft site.

3.10 Images

To draw images onto the canvas, the drawImage method can be used.
This method can be invoked with three different sets of arguments:

    
	
  •  drawImage(image, dx, dy)
    • 
	
  •  drawImage(image, dx, dy, dw, dh)
    • 
	
  •  drawImage(image, sx, sy, sw, sh, dx, dy, dw, dh)
    • 


Each of those three can take either an HTMLImageElement, an HTMLCanvasElement, or an HTMLVideoElement for the image argument.

The api lets you take the contents of specific HTML elements and draw them into a canvas, and the 3rd element in that list is just begging to be abused. Copying video into a canvas element means opening up the ability to manipulate or process video frames at runtime. Two concepts instantly came to mind that seemed like fun to try and figure out, here they are below.

Blowing apart fragments of video

Click around the video frame to blow up that part of the video, the exploded pieces will continue to play the video inside them. After a while they retract back to their original place. One feature I didn’t have time to figure out was adding depth to the explosion, so pieces that are closest to ground zero fly up into the air as they sail outward. With full shadow effects this could look really cool.

3D Video

This demo in particular runs really well inside webkit based browsers, but not so much in Firefox. Firefox doesn’t appear to have any hardware acceleration for Ogg decoding so I had to drop the video size in half in order to run at acceptable framerates. Even still, Firefox chokes pretty badly on my Macbook Pro.

*Update* – I’ve changed the ogg video to be 640 x 360, prepare to see firefox weep

Lessons learned

There’s a couple hints I found out along the way that are good to know if you want to play around with drawing video. First, you need a bit of hackish code to get this to work effeciently and it flows like this.

[Video playing] -> [Draw Video onto Canvas 1] -> [Draw fragments of Canvas 1 onto Canvas 2]

Don’t ask me why, but copying pixel data out of a video tag is expensive, so expensive that drawing it into a temporary canvas, and then drawing pieces of that temp canvas onto a final canvas is faster then just referencing the video tag repeatedly within the same loop. That’s why you’ll see 2 Canvases in the source code for the demos. I’m sure there’s a technical reason for this duplication process, but it’s a lazy reason.

Secondly, don’t try copying individual pixels around. You can still see the remnants of my first code attempt inside the explosion demo with getPixel() and setPixel(). This turned out to be horribly slow and completely unnecessary. Canvas.drawImage() + matrix transforms on the canvas space is far more efficient then handcrafted pixel pushing. On the other hand, pixel manipulation allows you to do things like runtime chroma keying, get ready for a new wave of “clippy” style videos with full transparency popping over websites to help you out.

Lastly, I’m learning very quickly that not all browsers are created equal when it comes to performance, it’s a crapshoot when it comes to heavy video+image manipulation. Safari and Chrome work well with h.264, Firefox slogs along with Ogg Theora, and Opera is somewhere in the middle.

305 thoughts on “Blowing up HTML5 video and mapping it into 3D space”

  2. Firefox 4b5pre runs even the 3D demo at a flawless speed on an intel pentium dual core running vindows vista.

  3. Also, hardware acceleration is turned off (like is is by default) and antialiasing is used (unlike chrome which does not use antialiasing)

  11. I’m having problems with a much simpler video example.
    I have two versions of a bouncing video. In one, the video element is moved directly.
    http://faculty.purchase.edu/jeanine.meyer/html5workshop/videobounce1.html

    In the other, the video is drawn on the canvas and shapes drawn on top to serve as a mask.
    http://faculty.purchase.edu/jeanine.meyer/html5workshop/videobounce3.html

    The first works in Firefox and Chrome.
    The second works in Firefox. The video does not appear in Chrome.

    Also, neither work with a tag, and they don’t appear to work with your doctype either.

    Any help most appreciated.

  13. Would you happen to know if this technique could be used to create an interactive video for the ipad?
    We want to create a tutorial video with “hot spots” where a user can click on a section of the video an an action would take place…

  15. I’m running an ubuntu laptop I bought in 05 with 128mg of RAM. Demos went flawless in the developers version of chrome. Gecko is light years behind webkit in canvas performance, but if any box would get ugly in chrome, it’s mine. I’d say you did an amazing job of conserving resources. It ain’t your fault Firefox is underfunded compared to chrome. Brilliant proof of concept. Bookmarked your sight you crafty mind.

  23. PERO SI ESA BASURA LA HAGO IGUALITA EN FLASH 8 Y MAS RAPIDA Y MENOS NPESADA_
    CUAL ES LA VENTAJA DE ESSA RIDICULES DE HTML5_
    ]YA LOE STUVE PROBANDO Y ES UN PROBLEMA PARA TODO
    CREO QUE LOS QUE LO ESTAN APOYANDO SON GAYS TRISTES HAHAAHH

    EL FLASH NUNCA MORIRA

  34. The change time for open standards for example html 5 is simply too slow, and never everybody switches into them exactly the same. In the current rate it’ll be another ten years before its up to date with expensive, with that time we’d have managed to move on.

  37. Has anyone got these examples to work on an iphone yet? I have money waiting to pay someone if they have.

    Contact me here or at thomasseidl “at” devitek. com

  46. I love your stories really much because they are published in an understandable comprehensible. So I can study them although I come from Germany and have some problems to translate English stories.

  51. Hi! Is it possible to use a webcam or IP camera as the video source for what you are doing inside of video and canvas?????? joshmarinacci points to your exploding video in his very nice piece on (canvasdeepdive) but while he mentions webcams at the beginning of his piece, he never goes back to it. Please — can u help? just tell me it’s not possible, OR, if it can be done — how do you set up the “source” for those inputs???

    Thank you very much for any help here.

  53. Oh, sorry! I see I am putting “canvas” into brackets, so the input control is filtering it out! OK, that previous post was supposed to be “…on his very nice piece on the canvas tag (canvasdeepdive), but while…”

  60. Firefox just keeps on chugging – whether it’s 3d, or even some of the facebook games – it’s struggle city even on a brand new computer. Chrome runs the best, easily.

  63. You were asking why you need two buffers : that’s called “double duffering” and is a good pattern in all image processing. When browser sends information of the current buffer to the screen (read operation on buffer), you can already compute the next image (write operation on buffer) but on a new buffer, else you would corrupt the previous image before it has been sent to screen. Using a second buffer, like you did, is the correct way of dealing with this. (look for double-buffering or multi-buffering for more details).

    Your exploding video is quite impressive, really nice job :)

  68. Great… it is awesome to see what HTML5 is can do !!! :)
    i was wondering if i could do the same with an image instead of a video…?
    i am very new to this, any pointers would be just fine… thanks…

  73. I tried this on both Firefox 3.6 and the latest install of chrome. I found that Chrome handled the spinning and shattering of the video better but Firefox did the actually playing of the video much more smoothly. Also the video looping failed on Chrome, so it would play through once, then instead of restarting it would only stop on the first frame. I was surprised, because even without reading that the author expected poor performance on Firefox, I thought Chrome would be better for html5, it seems that this is not universally true. I’m running winxp SP3 with a 3.2 Ghz P4, and 2GB of ram.
    sprz?t si?owy

  93. For example how many individuals are still having problems composing css that all internet explorer can study without having to crack it up a bit. IE, 6 -7 band alert bells! Terrible discuss toss things back a few years. I know these internet explorer will never stay for ever but there are still individuals still using these repetitive old internet explorer, half the reason why display was presented in the first place – operates on all browses and systems globally.

Comments are closed.