I made a relatively complex png file for testing, the source file is a 230 kb Fireworks file, which is pretty close to some of the assets we have in our projects. A quick 24bit png export from fireworks produces a 6 kb png file with no visual fidelity loss. When the project is compiled to the device, the source image is compressed by xcode down to 19 kb, while the exported version actually increases in size to 7 kb. So far this supports the theory that manual compression can be up to 3 times better then letting xcode handle it for you.

Original Files in Xcode Project

Files compressed for iOS device build.

Additionally it seems that I suffered no visual loss by letting xcode recompress these png files, and decompression times for both pngs on the device took roughly 20 milliseconds each.

Honestly I think you could read these results either way, lazy developers may choose to just use source files in xcode and suffer a 3x penalty on images compression, for those that like to control the compression exactly on their files, you’ll be better off.

3.10 Images

To draw images onto the canvas, the drawImage method can be used.
This method can be invoked with three different sets of arguments:

 drawImage(image, dx, dy)
 drawImage(image, dx, dy, dw, dh)
 drawImage(image, sx, sy, sw, sh, dx, dy, dw, dh)


Each of those three can take either an HTMLImageElement, an HTMLCanvasElement, or an HTMLVideoElement for the image argument.

The api lets you take the contents of specific HTML elements and draw them into a canvas, and the 3rd element in that list is just begging to be abused. Copying video into a canvas element means opening up the ability to manipulate or process video frames at runtime. Two concepts instantly came to mind that seemed like fun to try and figure out, here they are below.

Blowing apart fragments of video

Click around the video frame to blow up that part of the video, the exploded pieces will continue to play the video inside them. After a while they retract back to their original place. One feature I didn’t have time to figure out was adding depth to the explosion, so pieces that are closest to ground zero fly up into the air as they sail outward. With full shadow effects this could look really cool.

3D Video

This demo in particular runs really well inside webkit based browsers, but not so much in Firefox. Firefox doesn’t appear to have any hardware acceleration for Ogg decoding so I had to drop the video size in half in order to run at acceptable framerates. Even still, Firefox chokes pretty badly on my Macbook Pro.

*Update* – I’ve changed the ogg video to be 640 x 360, prepare to see firefox weep

Lessons learned

There’s a couple hints I found out along the way that are good to know if you want to play around with drawing video. First, you need a bit of hackish code to get this to work effeciently and it flows like this.

[Video playing] -> [Draw Video onto Canvas 1] -> [Draw fragments of Canvas 1 onto Canvas 2]

Don’t ask me why, but copying pixel data out of a video tag is expensive, so expensive that drawing it into a temporary canvas, and then drawing pieces of that temp canvas onto a final canvas is faster then just referencing the video tag repeatedly within the same loop. That’s why you’ll see 2 Canvases in the source code for the demos. I’m sure there’s a technical reason for this duplication process, but it’s a lazy reason.

Secondly, don’t try copying individual pixels around. You can still see the remnants of my first code attempt inside the explosion demo with getPixel() and setPixel(). This turned out to be horribly slow and completely unnecessary. Canvas.drawImage() + matrix transforms on the canvas space is far more efficient then handcrafted pixel pushing. On the other hand, pixel manipulation allows you to do things like runtime chroma keying, get ready for a new wave of “clippy” style videos with full transparency popping over websites to help you out.

Lastly, I’m learning very quickly that not all browsers are created equal when it comes to performance, it’s a crapshoot when it comes to heavy video+image manipulation. Safari and Chrome work well with h.264, Firefox slogs along with Ogg Theora, and Opera is somewhere in the middle.

My session will be running on Tuesday at 1:30 PM and again on Wednesday at 5:00 PM, hope to see you all there.

Update: I forgot to mention during my presentation, but the samples within the zip archive may contain memory leaks for the sake of code brevity. If you copy and paste the code samples into your own projects, you might need to make sure you add cleanup code to the end of it.

First up an example of what we’re able to accomplish with this effect.

You can magnify any UIView as well as the children of that view, images, text and 2d vectors are captured and magnified appropriately. There are couple critical pieces of this code to pay attention to.

First up, when a touch has been recorded long enough to kick off the magnifier effect, we create a new magnifier view that is the exact size of the view we want to magnify. This is important due to the way we’re going to be copying a bitmap of the original view. Once we know the view exists we call setNeedsDisplay on it which triggers drawRect inside of it.

if(loop == nil){
	loop = [[MagnifierView alloc] initWithFrame:self.bounds];
	loop.viewref = self;
	[self addSubview:loop];
}
UITouch *touch = [touches anyObject];
loop.touchPoint = [touch locationInView:self];
[loop setNeedsDisplay];

Next, when drawRect gets called inside the magnifier view we want to make a copy of the original view first. The reason the magnifier view is teh same size as the original view is because we are rendering the full context of the original view into our new context before grabbing a bitmap of it. If the magnifying view were smaller, the rendered bitmap would also be smaller. We want to cache the final bitmap so we’re not redrawing the original view every time the user moves their finger around the view. We’ll destroy that cached view and the magnifying glass when the user lets up off the screen.

- (void)drawRect:(CGRect)rect {
	if(cachedImage == nil){
		UIGraphicsBeginImageContext(self.bounds.size);
		[self.viewref.layer renderInContext:UIGraphicsGetCurrentContext()];
		cachedImage = [UIGraphicsGetImageFromCurrentImageContext() retain];
		UIGraphicsEndImageContext();
	}

Following that we need to generate a masked view for the magnified view to sit in, since the loop is a circle we have to mask out the corners and antialias the outer perimeter. This is accomplished using 2 images, the magnifying glass itself and a mask image with appropriate grayscale levels for masking.

CGImageRef imageRef = [cachedImage CGImage];
CGImageRef maskRef = [[UIImage imageNamed:@"loopmask.png"] CGImage];
CGImageRef overlay = [[UIImage imageNamed:@"loop.png"] CGImage];
CGImageRef mask = CGImageMaskCreate(CGImageGetWidth(maskRef),
					CGImageGetHeight(maskRef),
					CGImageGetBitsPerComponent(maskRef),
					CGImageGetBitsPerPixel(maskRef),
                                        CGImageGetBytesPerRow(maskRef),
					CGImageGetDataProvider(maskRef),
					NULL,
					true);
//Create Mask
CGImageRef subImage = CGImageCreateWithImageInRect(imageRef, CGRectMake(touchPoint.x-18, touchPoint.y-18, 36, 36));
CGImageRef xMaskedImage = CGImageCreateWithMask(subImage, mask);

Lastly, we’ll draw the magnifying glass and magnfied bitmap copy of our orginal view underneath the mask and we’re done. Since the iPhone uses a different coordinate system then other languages, we have to remember to flip the view upside down before drawing it.

CGContextRef context = UIGraphicsGetCurrentContext();
CGAffineTransform xform = CGAffineTransformMake(
					1.0,  0.0,
					0.0, -1.0,
					0.0,  0.0);
CGContextConcatCTM(context, xform);
CGRect area = CGRectMake(touchPoint.x-42, -touchPoint.y, 85, 85);
CGRect area2 = CGRectMake(touchPoint.x-40, -touchPoint.y+2, 80, 80);
CGContextDrawImage(context, area2, xMaskedImage);
CGContextDrawImage(context, area, overlay);

And that’s it, now we have a modular magnifying glass that can plug in to any UIView with minimal effort. If you’re looking for a way to add interactivity underneath the magnifying glass, like moving the cursor within a textfield, that’s gonna require a bit more custom code on the control you’re dealing with, and this example doesn’t really address that.

Download example: XCode magnifier example for iPhone.

To use this, just point the app at the url of any Blizzard Downloader exe file online (I’m not sure if it’ll work with the .dmg files) and hit the ‘Extract’ button. A link will pop up below that you can click to save the torrent file. The file has no name by default so you’ll have to rename it ‘whatever.torrent’ after it finishes downloading. The resulting torrent will still connect to Blizzards tracker and you’ll download the same file you would be if you were using the Blizzard Downloader app.

This should work with all browsers other then IE 7 and below, since IE doesn’t support the data:URI scheme to create dynamic files.

View Source

Example URLs:
Diablo 3 Gameplay Downloader
Starcraft 2 Gameplay Downloader

More Info:
Blizzard Downloader Wiki

GUIMark shares alot in common with another RIA benchmark Bubblemark. I’ve written a bit about Bubblemark and why I think an alternative is necessary, but I do believe Bubblemark and GUIMark can coexist while serving 2 different purposes. Alexey Gavrilov stated it best in that he sees Bubblemark as a sortof ‘Hello World’ launchpad into comparing different environments and I agree with him. Bubblemark is a *very* accessible test suite and its easy for any kind of developer to jump in and play around with performance techniques. GUIMark takes a different approach by trying to benchmark the types of UI elements common in our Web 2.0 world. This includes things like vector redraws, alpha transparencies, text reflow, bitmap motion, and 9 scale slicing rules. From there I just fill up the render pipeline until it becomes so over-saturated that it becomes easy to visually distinguish which rendering engines are more efficient then others. As a result, the benchmark is more complicated on a visual level and requires a bit more time then Bubblemark to understand the implementation rules. Lastly with GUIMark I’ve tried to get into some of the lower level details behind how rendering engines work and how that’s affected the creation of this project.

I’m hoping that developers and designers will be able to use this test suite to identify any pros or cons to choosing a particular environment when visual transitions are a key element of the experience. I’m also hoping these benchmarks provide a spotlight for the community that we can turn toward the runtime engineers inside Sun or Adobe or Mozilla to demand better performance.

Go to GUIMark home page

After working with some code and spending time with the SDK itself, I couldn’t help but naturally compare UI development on the iPhone with GUI frameworks like Flex or Swing, heres the things that stand out so far.

1. I’m really spoiled by higher level languages. A good high level language like ECMAScript, Ruby, or Java rides the fine line of “Making things as simple as possible, but no simpler”. I’ve never felt constrained by the language features in these technologies, only by the apis exposed. Stepping *back* into Objective-C certainly provides more power and flexibility in the language, but there’s a loss in productivity for me that I just can’t shake. Some of this loss comes from Objective-C’s design itself, and some of it just comes from XCodes introspection ability. For instance, I’m not sure if I’ll ever get to the point where I can read these lines of Objective-C as effeciently as their ActionScript counterparts would probably look.

   NSString *aString = [[[NSBundle mainBundle] infoDictionary] objectForKey:@"CFBundleName"];
   UIView *contentView = [[UIView alloc] initWithFrame:[[UIScreen mainScreen] applicationFrame]];

2. Closed source UI frameworks suck. Most of what I learned about custom UI development in Flex I learned by inspecting the source code for the bundled controls. Ripping open Containers to see how layout rules are determined, or Lists to see how delegates are passed around, or the Image control to see how different display types are handled provides invaluable gems about implementing Flash apis. It has also helped me optimize the interactions of an app knowing the intentions of the developer who created the UI controls. With the iPhone SDK, you’re given documentation for the visual components, but no source to help determine how they work.
3. Core graphics and animation is really strong. Between Quartz and the OpenGL layer there’s alot of potential for getting easy access to some of the more complex visual hacks. Although I think 3D user interfaces are prone to usability issues, the iPhone is a much better device to explore them on then a standard keyboard and mouse interface.
4. Data binding, event listeners, and mxml. The Flash Player and Flex model provide features on top of the ActionScript language that arguably optimize UI needs and keep development more declarative. Cocoa development could really benefit from a ‘gui compiler’ that takes Objective-C to a higher level and bakes in features that support common ui design patterns.
5. Garbage Collection. Objective-C 2.0 provides a unique system that lets you create objects that will be automatically garbage collected, or you can continue to manually manage object allocation/deallocation yourself. The concept sounds cool, but I can imagine allowing both systems to be mixed within the same project is just begging for trouble.

So far I think that Objective-C has alot of power and some really awesome features that outclass GUI features in Flash, but compared to Flex development as a whole, I’d have to say that XCode and the included visual frameworks are not as sophisticated.