Archive for April, 2008

From AS3 to Objective-C: Flex vs iPhone development »

Recently I’ve been given the opportunity to work full time on commercial iPhone development at EffectiveUI. The most intriguing thing about the platform for me is having access to non traditional user input mechanisms. When I was playing around with Wii remote integration on the desktop, the potential was exciting, but the ubiquity was limiting. In the same way that pc game companies develop for the keyboard and mouse first and then provide hooks for joysticks after the fact, I knew that serious Wii remote integration in a desktop app was limited. Knowing that I can write software for the iPhone that always has access to multitouch and accelerometer data from the outset really allows for unique gestures as a first class citizen within an app.

After working with some code and spending time with the SDK itself, I couldn’t help but naturally compare UI development on the iPhone with GUI frameworks like Flex or Swing, heres the things that stand out so far.

  1. I’m really spoiled by higher level languages. A good high level language like ECMAScript, Ruby, or Java rides the fine line of “Making things as simple as possible, but no simpler”. I’ve never felt constrained by the language features in these technologies, only by the apis exposed. Stepping *back* into Objective-C certainly provides more power and flexibility in the language, but there’s a loss in productivity for me that I just can’t shake. Some of this loss comes from Objective-C’s design itself, and some of it just comes from XCodes introspection ability. For instance, I’m not sure if I’ll ever get to the point where I can read these lines of Objective-C as effeciently as their ActionScript counterparts would probably look. 
    NSString *aString = [[[NSBundle mainBundle] infoDictionary] objectForKey:@"CFBundleName"];
UIView *contentView = [[UIView alloc] initWithFrame:[[UIScreen mainScreen] applicationFrame]];
  2. Closed source UI frameworks suck. Most of what I learned about custom UI development in Flex I learned by inspecting the source code for the bundled controls. Ripping open Containers to see how layout rules are determined, or Lists to see how delegates are passed around, or the Image control to see how different display types are handled provides invaluable gems about implementing Flash apis. It has also helped me optimize the interactions of an app knowing the intentions of the developer who created the UI controls. With the iPhone SDK, you’re given documentation for the visual components, but no source to help determine how they work.
  3. Core graphics and animation is really strong. Between Quartz and the OpenGL layer there’s alot of potential for getting easy access to some of the more complex visual hacks. Although I think 3D user interfaces are prone to usability issues, the iPhone is a much better device to explore them on then a standard keyboard and mouse interface.
  4. Data binding, event listeners, and mxml. The Flash Player and Flex model provide features on top of the ActionScript language that arguably optimize UI needs and keep development more declarative. Cocoa development could really benefit from a ‘gui compiler’ that takes Objective-C to a higher level and bakes in features that support common ui design patterns.
  5. Garbage Collection. Objective-C 2.0 provides a unique system that lets you create objects that will be automatically garbage collected, or you can continue to manually manage object allocation/deallocation yourself. The concept sounds cool, but I can imagine allowing both systems to be mixed within the same project is just begging for trouble.

So far I think that Objective-C has alot of power and some really awesome features that outclass GUI features in Flash, but compared to Flex development as a whole, I’d have to say that XCode and the included visual frameworks are not as sophisticated.

Updated ‘Elastic Racetrack’ for Flash 9 and AVM2 »

In 2005 Ted Patrick posted a great article on the frame execution model inside the Flash Player that he dubbed the ‘elastic racetrack‘. It’s served as a great reference for me over the years to help understand how code execution and rendering were balanced within the processing of a frame. Since the introduction of Flash Player 9 and the new AVM2, I’ve noticed a few changes to the elastic racetrack model and thought I’d share them. This information is based on research into Flash player internals as well as observations I’ve made playing around with the event and rendering model, but the full model hasn’t been confirmed by Adobe engineers.

The basic premise of the original elastic racetrack is still the same. Given a specific frame rate to operate on, the Flash player will devote the first segment of the frame to execute code, and the second segment to render display objects. Either segment can grow its part of the racetrack to accommodate more processing and effectively extend the duration of the frame.

Flash Player Elastic Racetrack

What changes from the previous model is how those segments look under a microscope and how they come together to form a ‘frame’.

AVM2 is controlled by what I’m going to call the Marshal. The Marshal is responsible for carving out time slices for the Flash Player to operate on. Its important to clarify up front that these time slices are not the same thing as the framerate compiled into a swf, but we’ll see below how the player ultimately synthesizes a framerate from these slices. Running a Flex compiled swf within Firefox under Mac OS X, the Marshal appears to be carving out 19-20 millisecond slices, but this can be different between platforms and browsers based on what I’ve observed as well as Adobe employees have hinted at. This can also change depending on how the swf was compiled, see some of the comments below. For the sake of the article lets assume we’re only talking about a 20 millisecond slice to make the math easy. This means the Marshal will attempt to generate and execute no more then 50 slices each second, and it may be less depending on the elasticity of code execution or rendering. Inside each slice, 5 possible steps are processed in the following order.

  1. Player events are dispatched – This includes events dispatched by the Timer, Mouse, ENTER_FRAMEs, URLLoader, etc…
  2. User code is executed – Any code listening to events dispatched by step 1 are executed at this stage.
  3. RENDER event is dispatched – This special event is dispatched when the user calls stage.invalidate() during normal user code operation.
  4. Final user code is executed – User code listening specifically for step 3 is executed at this point.
  5. Player renders changes to the display list.

AVM2 Marshalled Slice

The Marshal executes this 20 millisecond slice over and over and decides on the fly which actions to run. The exact actions processed within a slice will ultimately derive the 2 main racetrack segments (code execution and rendering) that constitute a ‘frame’. User actions and Invalidation actions fill up the code segment track, while Render actions fill up the render segment track. Its important to note that actions will only occur at times predetermined by the Marshal, so a if you have a short running User action, the Marshal will still wait a few milliseconds before moving on to the Invalidate and Render actions.

The best way to illustrate which actions are run and how the elastic racetrack is created, is to look at how those slices are processed on a swf running at 5 fps, 25, fps, and 50 fps.

Flash Frame Marshaling Diagram

As you can see, the elastic racetrack performs different actions per frame and requires a different visual illustration depending on the framerate that the player is trying to synthesize. So for a swf running at 5 fps, each frame processed 10 User actions, 1 Invalidation action, and 1 Render action. At 25 fps, each frame processed 2 User actions, 1 Invalidation action, and 1 Render action. At 50 fps, each frame processed 1 User action, 1 Invalidation action, and 1 Render action. Whats important to note in the above chart is that some events are only available in certain slices. For instance, the Event.ENTER_FRAME event will only ever be dispatched in a slice that occurs at the start of a frame.

So what does this all mean? Theres a couple quick ideas to take away from this.

  1. Long running code execution or render segments can extend a given slice beyond 20 milliseconds. Elasticity will be applied to that particular slice and the duration of the frame may or may not be extended as a result. The Marshal may drop the number of slices that constitute a frame in order to keep the active framerate close to the compiled framerate.
  2. A swfs real framerate won’t exceed the Marshals rate defined for the player instance. You can set your compiled framerate at 120fps, but Flash will still only process 50 slices max that generate 50 render calls (more or less depending on the system config).
  3. Code can be executed more often then the compiled framerate. A swf compiled at 1 fps can execute a Timer or Mouse event in every slice, even though it will only render in the last slice. Additionally, if you choose, you can render to the screen sooner then the compiled framerate by calling updateAfterEvent() , but only within a Mouse, Timer, or Keyboard event handler. In this instance though, the Marshal will consider that the end of the frame and start a new frame on the next slice. Lastly, Flash will force an automatic render when mousing over any Sprite that has had its visual properties (x,y,width,height,etc..) changed, naturally this still occurs at the end of the slice and any prerender logic will still run.
  4. Compiling a framerate that isn’t a multiple of the total number of slices per second for your platform will cause irregular rendering as it tries to divide up the slices. If you were to compile in a framerate of 20 on a platform executing 50 slices per second, then the player has to render 2 frames every 5 slices and would follow a 3-2-3-2-3-2 slice-to-render rate.

These 4 facts are moving targets though, since for this article we’re working on a 20 millisecond slice that’s processed 50 times per second. In reality you’ll see time slices as low as 5 milliseconds or as high at 100 milliseconds and some of the math will change as a result.

If you’d like to test this model for yourself, the easiest route is to create a swf running at 1 fps and another at 100 fps both with a Timer object set on a 1 millisecond interval. Inside the Timer event handler change the x property of a display object and hook a bunch of getTimer() traces up to different player events like Mouse, EnterFrame, and Render and watch the carnage unfold in your console. The rest of the information you can’t derive from the results comes from alot of context about the player I’ve learned over the past 2 years and so isn’t as easily visible. If anyone has any information to help add to or correct the above model, please submit it in the comments.

Thanks to several readers who have clarified some of the differences between Flex and Flash as well as how the Flash API is able to change the default behaviors described above.

Why Bubblemark is a poor ui benchmark »

A few months ago someone on the Adobe boards asked why the Flex testcase in Bubblemark seemed to act so different in AIR versus in the browser. Yesterday, I saw the same question come up again and I figured I’d finally weigh in on the topic. The simple answer is that the test was created improperly, the complex answer has to do with the inherent limitations of the test itself.

First off, for those who don’t know what the Bubblemark test is, its a simple animation test case implemented in different GUI frameworks, its kinda like an Acid2 test for rendering speed. The charts should ideally give you a base number to understand how well one technology compares against another for rendering. As a GUI developer I’ve been a bit underwhelmed with the whole thing and heres why:

  1. The author doesn’t understand Flash’s rendering engine. The easiest way to illustrate how incorrectly the Flash test was designed, is to download the source and change the compiled framerate to 1 fps. Re-compile and run the test and you’ll notice the benchmark framerate running at ~50 fps. You can clearly see the balls only moving once per second, yet the test thinks its flying along. This is because the testcase makes the incorrect assumptions that changing the properties of a DisplayObject causes it to render right away. The reality is, Flash holds on to all display updates till the next render pass and applies all the latest changes at once. Changing the position of an object every 5 milliseconds is meaningless when Flash is bound by a 33 millisecond render pass (or whatever you’re framerate divided by 1000 happens to be). A correct test case would rely on an ENTER_FRAME handler to change x and y values and get rid of any Timer calls.
  2. Framerate tests above 60 fps are meaningless. Seriously, any GUI benchmark designed to test above 60 fps is bogus. In fact, a pretty simple optimization technique for Adobe or Sun would be to cap the paint requests that get forwarded to OS X or Windows, simply because the majority of computer users these days are on LCD panels which natively run at 60 fps. Some operating systems even go a step further and limit the effective framerate of paint requests it sends to the videocard (see Beam Sync on Mac). So when you see the Java test case fly up to 120 fps on Bubblemark, you can realistically only see 60 of those frames, and there might be a chance the other 60 are never even calculated by Javas layout engine.
  3. The test just moves balls around! This is my biggest beef with the benchmark because it only tests one simple aspect of the rendering engine in these technologies, which is bitmap translation. How do bitmaps moving around the screen tell you anything about the capabilities of the respective technologies? Do the JavaFX guys really think optimizing this usecase will make their technology relevant? The only thing Bubblemark will tell you is which runtimes might best handle bitmap particle emitters….thats about it. Theres a lot more that goes into both the layout engine and the rendering pipeline of these different technologies and its a shame that only the most basic aspect is being tested. The funny thing is, if you open up your task manager while running the tests, you’ll notice that several of them don’t even try to run at full speed, my CPU is sitting as low as 20% in some cases. This means the runtimes don’t even consider the test difficult enough to give it full attention and have opted for using less power over faster motion.

I don’t mean to cut down the developers responsible for Bubblemark because at least they came up with a simple way to help us all compare these different technologies, I just think its a bit misguided to put any meaning behind these numbers. When evaluating your options for a GUI framework in our flashy web 2.0 world, you need to consider how well a technology can handle object scaling, alpha transparencies, rotations, text reflow, along with basic x and y translation and dynamic redraws. Even more realistically, developers need to be aware of the limits in the 25-45 framerate region since this is where you can efficiently balance render complexity with smooth animation. I’ve uploaded a couple quick test cases in Flash, HTML, and Silverlight that I think provide a good foundation for stressing a rendering engine and hopefully I’ll get a chance to expand them more into a full test suite.