In 2005 Ted Patrick posted a great article on the frame execution model inside the Flash Player that he dubbed the ‘elastic racetrack‘. It’s served as a great reference for me over the years to help understand how code execution and rendering were balanced within the processing of a frame. Since the introduction of Flash Player 9 and the new AVM2, I’ve noticed a few changes to the elastic racetrack model and thought I’d share them. This information is based on research into Flash player internals as well as observations I’ve made playing around with the event and rendering model, but the full model hasn’t been confirmed by Adobe engineers.
The basic premise of the original elastic racetrack is still the same. Given a specific frame rate to operate on, the Flash player will devote the first segment of the frame to execute code, and the second segment to render display objects. Either segment can grow its part of the racetrack to accommodate more processing and effectively extend the duration of the frame.
What changes from the previous model is how those segments look under a microscope and how they come together to form a ‘frame’.
AVM2 is controlled by what I’m going to call the Marshal. The Marshal is responsible for carving out time slices for the Flash Player to operate on. Its important to clarify up front that these time slices are not the same thing as the framerate compiled into a swf, but we’ll see below how the player ultimately synthesizes a framerate from these slices. Running a Flex compiled swf within Firefox under Mac OS X, the Marshal appears to be carving out 19-20 millisecond slices, but this can be different between platforms and browsers based on what I’ve observed as well as Adobe employees have hinted at. This can also change depending on how the swf was compiled, see some of the comments below. For the sake of the article lets assume we’re only talking about a 20 millisecond slice to make the math easy. This means the Marshal will attempt to generate and execute no more then 50 slices each second, and it may be less depending on the elasticity of code execution or rendering. Inside each slice, 5 possible steps are processed in the following order.
- Player events are dispatched – This includes events dispatched by the Timer, Mouse, ENTER_FRAMEs, URLLoader, etc…
- User code is executed – Any code listening to events dispatched by step 1 are executed at this stage.
- RENDER event is dispatched – This special event is dispatched when the user calls stage.invalidate() during normal user code operation.
- Final user code is executed – User code listening specifically for step 3 is executed at this point.
- Player renders changes to the display list.
The Marshal executes this 20 millisecond slice over and over and decides on the fly which actions to run. The exact actions processed within a slice will ultimately derive the 2 main racetrack segments (code execution and rendering) that constitute a ‘frame’. User actions and Invalidation actions fill up the code segment track, while Render actions fill up the render segment track. Its important to note that actions will only occur at times predetermined by the Marshal, so a if you have a short running User action, the Marshal will still wait a few milliseconds before moving on to the Invalidate and Render actions.
The best way to illustrate which actions are run and how the elastic racetrack is created, is to look at how those slices are processed on a swf running at 5 fps, 25, fps, and 50 fps.
As you can see, the elastic racetrack performs different actions per frame and requires a different visual illustration depending on the framerate that the player is trying to synthesize. So for a swf running at 5 fps, each frame processed 10 User actions, 1 Invalidation action, and 1 Render action. At 25 fps, each frame processed 2 User actions, 1 Invalidation action, and 1 Render action. At 50 fps, each frame processed 1 User action, 1 Invalidation action, and 1 Render action. Whats important to note in the above chart is that some events are only available in certain slices. For instance, the Event.ENTER_FRAME event will only ever be dispatched in a slice that occurs at the start of a frame.
So what does this all mean? Theres a couple quick ideas to take away from this.
- Long running code execution or render segments can extend a given slice beyond 20 milliseconds. Elasticity will be applied to that particular slice and the duration of the frame may or may not be extended as a result. The Marshal may drop the number of slices that constitute a frame in order to keep the active framerate close to the compiled framerate.
- A swfs real framerate won’t exceed the Marshals rate defined for the player instance. You can set your compiled framerate at 120fps, but Flash will still only process 50 slices max that generate 50 render calls (more or less depending on the system config).
- Code can be executed more often then the compiled framerate. A swf compiled at 1 fps can execute a Timer or Mouse event in every slice, even though it will only render in the last slice. Additionally, if you choose, you can render to the screen sooner then the compiled framerate by calling updateAfterEvent() , but only within a Mouse, Timer, or Keyboard event handler. In this instance though, the Marshal will consider that the end of the frame and start a new frame on the next slice. Lastly, Flash will force an automatic render when mousing over any Sprite that has had its visual properties (x,y,width,height,etc..) changed, naturally this still occurs at the end of the slice and any prerender logic will still run.
- Compiling a framerate that isn’t a multiple of the total number of slices per second for your platform will cause irregular rendering as it tries to divide up the slices. If you were to compile in a framerate of 20 on a platform executing 50 slices per second, then the player has to render 2 frames every 5 slices and would follow a 3-2-3-2-3-2 slice-to-render rate.
These 4 facts are moving targets though, since for this article we’re working on a 20 millisecond slice that’s processed 50 times per second. In reality you’ll see time slices as low as 5 milliseconds or as high at 100 milliseconds and some of the math will change as a result.
If you’d like to test this model for yourself, the easiest route is to create a swf running at 1 fps and another at 100 fps both with a Timer object set on a 1 millisecond interval. Inside the Timer event handler change the x property of a display object and hook a bunch of getTimer() traces up to different player events like Mouse, EnterFrame, and Render and watch the carnage unfold in your console. The rest of the information you can’t derive from the results comes from alot of context about the player I’ve learned over the past 2 years and so isn’t as easily visible. If anyone has any information to help add to or correct the above model, please submit it in the comments.
Thanks to several readers who have clarified some of the differences between Flex and Flash as well as how the Flash API is able to change the default behaviors described above.
How can you go from saying that one broswer/OS combo appears to be carving out 19-20 ms slices to saying “let’s assume 20 for simplicity” to “it’s a fact that compiling a framerate that isn’t a multiple of 50 will cause irregular rendering?”
You’re right, its a bit confusing but I didn’t want to annotate every reference to the length of a time slice or how that divides into 1000 milliseconds to remind readers it may change depending on their system, I think that would’ve been more confusing. The point I wanted to make in the last section was that like a dvd player trying to use 3:2 pulldown to display a 24 fps movie on a 60 fps tv, so will flash try use different amounts of time slices to create the visual framerate, and the amounts may be different per frame causing irregular(although probably not noticable) rendering
BTW, on PC, flash works off a minimum 17 millisecond Marshal, which equates to roughly 60 fps, or a maximum 100 millisecond Marshal which equates to 10 fps. While the Mac appears to use constant length time slices of around 19, the PC will stretch them out depending on the user requested framerate most likely as an optimization technique.
Austin:
You’re right, he should rephrase that as “if your framerate is not divisible by the slice length for a system it may cause irregular rendering”… but that’s saying nothing, ‘cos we already knew we couldn’t trust on the length of the frame being the same as the framerate indicates.
But otherwise it’s a really interesting read. For instance I read that the fps affects the responsiveness of the textfield inputs for flex apps, and I’m sure it has to do with this. In that case people recommended 60fps as the optimal solution, since it doesn’t consume more cpu and makes the interface more responsive.
I’ve changed some of the wording to clarify how the math may change for different platforms.
That’s cool. Btw, I apologize for not complimenting you on the discussion first. I appreciate your explanation of the system. I read to the end, and my first thought was, “Oh no. Everyone is going to start saying that you have to use multiples of 50 fps,” and I didn’t think that’s exactly what you were trying to say.
interesting
but did you try those same tests
by changing this flag (in mxmlc for ex)
-default-script-limits
basically, the elaticity of the FPS should be influenced by those parameters
ex: if you set 20FPS, and wan to execute a lot of code or gfx rendering per frame the FPS will slow down, but the script-limits will cap how much the FPS can slow down
I would love to see those slice of execution in the profiler in FB3
Great article! Though I believe your tests are all in context of a Flex project, which can skew the results because of how Flex sets up the environment. If not then something else is going on.
On Mac, in Firefox:
1. I’ve also found that long code execution or heavy rendering slows events, and that Flash still tries to keep the framerate on target as much as possible.
2. From my experience Flash will TRY to match the framerate as close as performance will allow.
stage.frameRate = 1000;
addEventListener(Event.ENTER_FRAME, enterFrameHandler);
startTime = getTimer();
private function enterFrameHandler(event:Event):void
{
trace(getTimer()-startTime);
}
//output
29
42
45
50
54
59
63
69..
3. The Timer will also fire events as quickly as performance will allow.
timer = new Timer(1); // add listener, start timer, the rest…
// output
22
29
33
34
35
39
42..
NOTE:
Event.RENDER only triggers when
a) stage.invalidate() is called and
b) just before the Flash Player renders the display list – the actual rendering of pixels to the screen, not to be confused with Flex’s redrawing and layout of components. Flash Player renders to screen on every frame (enterFrame) AND whenever updateAfterEvent() forces a refresh. Whatever you’re experiencing with MOUSE_DOWN is a result of a MouseEvent listener calling the updateAfterEvent() somewhere in your (or some library your using like Flex) code.
4. Because of the great difference in our results I don’t have advice to offer on this yet.
None of my data discounts a Marshall segmenting processes by time. In fact it makes sense that the event firing system would be organized such to sustain overall performance. But how are our results so different, even IF you were using Flex? I’d love to see a presentation on the Flash Player event and rendering model as detailed as Gary Grossmans introduction to the new Garbage Collection in FP9.
tyler,
you’re right, i was mistaken in thinking that manipulating display objects automatically triggered a stage invalidate.
i fired up flash cs3 and noticed the same thing you and zwetan pointed out, but mostly what I’m seeing is that the Marshal is carving out 5 millisecond slices in the standalone player, but using 10 millisecond slices in the browser. I think this is important and I’ll add it to the original article. I’m still seeing the same behavior though with regards to how a frame is assembled. When publishing at 20 fps, I get a frame that is made up of 10 slices in the standalone player, and 5 slices in the browser.
Although, just as I noticed in Flex, the Marshal still uses the framerate to a degree to determine the length of each slice it should generate. At 1 fps in Flash CS3, I still see 100 millisecond slices. There must be internal logic in the Flash player that says a frame should never consist of more then a certian number of slices or else it will be wasting cpu.
I’ve updated some of the wording to reflect zwetan and tylers comments. One last thing added though, it appears that no matter what, flash will try to render when mousing over a dirty Sprite. updateAfterEvent() is not needed for this scenario so it looks like the Flash player has some internal triggers for forcing a render.
Wow, this really cleared up some things. Thanks a lot, Sean and all the commenters!
Cheers, Thomas
Useful article, thanks! Do you know anything about how webcam video interacts with all this? I’m drawing a Video() w/ camera attached behind my other Sprites. Under some system load, the video framerate will be crippled (~4fps) while my ENTER_FRAME events continue firing merrily at ~35fps. It seems like flash is throttling its video feed, but I don’t really know. For my application, it’s fine if it uses lots of cpu, but I haven’t figured out how to tell it that. Any ideas?
Excellent article! Gives great insight on the internal processes of AVM2.
Sean, do you have a contact form somewhere? Is there any way to contact you? I’m conducting research at Georgia Tech with the college of computing and would really like to get in touch with you.
Hey Sean,
Nice work. This is really insightful.
Would you dare to draw any conclusions from this information? Specifically, do you think there could exist for AS3 the infamous “Magic Frame Rates”? Are there situations where Timers are more efficient than ENTER_FRAME and vice-versa? Any chance we’ll ever know exactly how things might perform on different systems?
Great post!
@Tyler One thing to keep in mind about the Timer class, is that it does not depend on time but rather FPS. If you set your SWF to 0 fps and do redraws “manually”, you’ll gain a lot performance-wise but no timers will run… I discovered this while making a drawing program in Flash.
J
Great article, thanks for your clear explanation about The Marshal.
Now, I am wondering why on Mac Safari Flash Fullscreen Mode, I do get a constant FPS (this only happens when in Fullscreen with Safari), which equals the one defined at compile time (side note: everything is smooth). As soon as I exit Fullscreen Mode, I do get different FPS given the complexity of is displayed (the elasticity principle happens here).
Cedric, I imagine it has to do with the Flash player ignoring any restrictions the browser places on it when entering fullscreen. One feature that I’ve noticed is that the flash player will respect any commands from the browser to throttle the marshal like when its notified that its in a different tab or below the fold on a scrolling page.
thanks for this great article!
there is also a strange behaviour of the flashplayer regarding cpu utilisation in idle state. if you have a high framerate and many displayobjects on stage cpu goes up even if you don´t do anything. if you set mouseEnable to false you can decrease cpus usage. the lower the framerate the lower the cpu usage. if you move the mouse over a text input field it goes down to 0%.
in the old avm1 (as2) the cpu usage at idle was nearly 0%.
do you have any idea why flashplayer behaves that way? cpu utilisation should be 0% if nothing is going on in your application even at higher framerates and independent of the number of displayobjects on stage. i guess it has to do ith the way the operation systems mouse events are treated in the flashplayer…
it seems that at every enter frame the mouse event is passed to every display object.
here is an excerpt of the bug entry:
Test results:
AS3 code (-> AVM2)
AS3: framerate 24:
mouse over the stage: 3%
mouse over the textField: 0%
mouse over the one sprite: 3%
mouse over the stack of 1000 sprites: 3%
AS3: framerate 120:
mouse over the stage: 9-14%
mouse over the textField: 0%
mouse over the one sprite: 10-13%
mouse over the stack of 1000 sprites: 14-18%
AS3: framerate 1000:
mouse over the stage: 36-40%
mouse over the textField: 0%
mouse over the one sprite: 22-40%
mouse over the stack of 1000 sprites: 17-33%
AS2 code (-> AVM1)
AS2: framerate 24:
mouse over the stage: 0%
mouse over the textField: 0%
mouse over the one sprite: 0%
mouse over the stack of 1000 sprites: 0%
AS2: framerate 120:
mouse over the stage: 6%
mouse over the textField: 0%
mouse over the one sprite: 6%
mouse over the stack of 1000 sprites: 6%
AS2: framerate 1000:
mouse over the stage: 8%
mouse over the textField: 0%
mouse over the one sprite: 8%
mouse over the stack of 1000 sprites: 8%
for more details see (and vote for) this bug entry at adobe:
http://bugs.adobe.com/jira/browse/FP-1149
and an additional bug entry added by grant skinner: http://bugs.adobe.com/jira/browse/FP-2009
This is a good article, but it’s quite misleading the way you depict the “remainder” of each time slice as “Render Action”. This implies that the player will somehow tailor the time spent rendering to the time available, which is not the case. What really happens is that the player performs rendering operations for however long that takes, and then simply waits until the next poll from the container. As such, it would be much more accurate if you depicted some kind of “idle” phase after the rendering phase, and if you made clear the fact that whenever processing or rendering are heavy, Flash will happily let those phases exceed the duration of a normal slice.
Great read. As a freelance flash developer, I see a lot of other people’s code. Whenever stuff needs to be updated (and drawn) regularly, 95% of the code listens to the Event.ENTER_FRAME, and some 5% to a TimerEvent.TIMER. I personally like the timer event for updating numbers, and then execute the drawing code in an enter frame listener.
Yet when I read your article I start wondering whether calling stage.invalidate() and then listening to the Event.RENDER would be a better solution, since the drawing code will then be much closer to the actual render.
Do you have an opinion on this? Cheers // EP.
Hi,
Very good post!
Can I ask you two questions?
1) What’s a frame? Is it essentially a slot of time or something else? I think a frame is a logic unic linked with ActionScript code, isn’t true?
2) You say that render events happen before the invalidate action’s section. Do you mean that invalidate methods, like invalidateproperties() or invalidateSize() [into the Flex Framework], have already executed? In this case flags have already set.
Thanks in advance,
Regards.
Note a bitmapData.draw doesn’t count as a Render Action: you can do some neat stuff if you don’t have your app (or just a part of it) on the stage, and manually draw() your content to a Bitmap. Of course this has some implications (no added-to-stage and such like) but if you run a custom animation system it offers great control if you hook it up to a Timer.
could you tell me about how this related to the flex components life cycle?