Gumbo Shader Based Bitmap Effects - Functional and Design Specification

Glossary

Bitmap: A snapshot or image of some state of the animation target. Bitmaps are used to represent the before and after state of the target, and the shader manipulates these bitmaps to transition from one to the other.

Shader: A Pixel Bender pixel-shader program that operates on two image inputs to transition over time from one to the other. These shaders are compiled for Flash using the Pixel Bender Toolkit. The bitmap transition effects bundled with the SDK (e.g., FxCrossFade, FxWipe) will use their own shaders internally, but developers can choose to use their own shaders instead, either to supply to the FxAnimateShaderTransition base class or to use in their own custom subclass effect.

Summary and Background

This feature is intended to create a framework for powerful transitions between states of an application, by using the new Shader capability in FlashPlayer 10.

Currently, all transitions in Flex happen by changing properties on the target objects, such as alpha, (x,y) or (width,height). Sometimes a more cinematic effect is desired, such as a dissolve, a Wipe, or even more complex visuals like page-curls.

In order to have this flexible capability, we need a couple of crucial elements: - infrastructure that makes it possible, even easy, to run an effect that operates not on the target objects themselves, but instead on the bitmap representations of those objects. That way, we can dynamically alter the appearance of the bitmap without regard to the typical constraints of the components. - infrastructure that makes it possible to run Pixel Bender shaders on the bitmaps. Canned shaders will be supplied with our built-in effects, but it also needs to be possible for developers to supply their own shaders for more custom effects, without requiring them to write custom subclasses or to do a lot of work outside of writing the actual shaders they want.

Also, we need to provide some canonical effects that we expect people to simply use out of the box. At a minimum, we should have a CrossFade effect and a Wipe effect (an Fx effect version of the four old Wipe* effects). We should also consider providing new versions of the other Mask effects, such as Dissolve and Iris.

Usage Scenarios

For the most part, developers would choose one of the canned effects, such as FxWipe. They would use these effects in a transition, typically, where they would be able to simply set up the effect in the transition and leave it to Flex to calculate the before/after bitmaps required.

It is also possible, although probably much less typical, to run these effects manually, supplying bitmaps where necessary.

Advanced developers, or those wanting more custom-looking effects, might develop their own shaders and possibly their own FxAnimateShaderTransition subclasses to get very custom transition effects.

Detailed Description

There are two distinct parts to this feature: - Shader: The shader developer writes a Pixel Bender shader that adheres to certain conventions (basically: an initial image4 input that goes unused (workaround for Flash player behavior), two image4 inputs called 'from' and 'to' and one float parameter called 'progress' that animates between the two images). - Effect: The base class, FxAnimateShaderTransition, takes two BitmapData properties, one Object property (for the shader bytecode, in either ByteArray or Class form), and an optional shaderProperties Object map. These properties can be populated by the application, but they will otherwise choose defaults that make sense. The bitmaps are determined by the before/after state of each target (which really only makes sense when running in a transition - manually-run effects should supply one or both bitmaps instead). The shader is assigned a default value by subclasses of FxAnimateShaderTransition; other subclasses or callers of FxAnimateShaderTransition directly should supply their own custom shader instead. The shaderProperties map is only necessary if there are other properties of the Shader that need to be supplied as each Shader instance is created. For example, a Shader that required a color parameter could have it set by this mechanism.

When the effect is started, a Shader is created for each effect instance around the supplied shader bytecode, the before/after bitmaps are created (if not already supplied), and the shaderProperties are used to populate the parameters of the Shader. Internally, a ShaderFilter object is created around this Shader. Then an animation is started which operates on the shader, varying the 'progress' parameter from 0 to 1 during the course of the animation. As the progress parameter is updated on the shader, the shader is set as the filters on the target object, which will result in Flash showing the resulting shader-produced bitmap in place of the actual target DisplayObject.

When the animation finishes, the ShaderFilter is removed as a filter on the target and the target is displayed normally.

The trick here that makes it all work, as described above, is that there is a hidden/unused input parameter that fakes-out Flash. Flash currently has a requirement that a ShaderFilter set on a DisplayObject will use that display object as the first input to the shader and as the output from the shader. In our case, we do not want to use the display object directly as an input, because it is in an indeterminant state during the transition, where some properties may be set to the pre-transition state and some to the post-transition state. For example, a button that changes its label in a state change will automatically set the post-change text before the transition starts. For this reason, we wish to use only the bitmaps we captured during the transition setup as inputs, so we need to supply the input that Flash expects (the display object) but use the bitmap inputs instead. The workaround is to have the shader require an initial input that is not actually used - Flash will assign that input to the display object, but pixels from that object will not be used, so they will not adversely affect the results.

An additional note about this workaround: it is not sufficient to supply simply an input parameter that is not used at all: Pixel Bender compilation is smart enough that it will optimize this parameter out and Flash will assign one of the other real inputs to the display object instead, which is not what we want. So in addition to simply declaring a first input, the shader writer needs to then use that parameter in some simple way to defeat the optimizing compiler.

We hope to have Flash change in future releases so that we can get rid of this hack.

For even more information, here is another workaround that we originally had, where we could use a more standard Shader without the extra input, but had to then use a second Shader under the hood to hook up the first one to the display object. This is not in the code any longer, but is included here only for informational purposes:
bq. there are two shaders involved in the animation. The first one simply animates between two BitmapData objects. The second applies the result of the first as a parameter to a ShaderFilter which simply displays that result. The reason for this two-tier approach is in the way that Flash ShaderFilter works. ShaderFilter takes at least one image4 input, which is always the target that the filter is applied to. In our case, we cannot use the target object as input to the first shader because that target object may not be in an appropriate state to use. For example, suppose we change the 'label' property in a state change and want to animate between the two states. the way that transitions in Flex work is that we actually switch to the second state before running the effect. And if there is no effect running which knows about that specific property change, then that property simply stays in the after state. So if we use the target object as input to the shader, then we'll actually get the 'after' state at the beginning, which is clearly not what we want. So the fix is to run the original shader on the vanilla BitmapData objects, and to then use that result as input to the internal shader that essentially just takes the bitmap as output, ignoring the DisplayObject input parameter.

API Description

/**
 * This effect animates a transition between two bitmaps,
 * one representing the start state (<code>bitmapFrom</code>), and
 * the other representing the end state (<code>bitmapTo</code>).
 * 
 * <p>The animation is performed by running a Pixel Bender shader,
 * supplied by the <code>shader</code> property,
 * using the two bitmaps as input. If either bitmap is
 * not supplied, that value will be determined dynamically from
 * either the appropriate state of the target in a transition or, 
 * if the effect is not running in a transition, from the 
 * target directly. If
 * the effect is run in a transition and the target object either
 * goes away or comes into existence during that state change,
 * then a fully-transparent bitmap will be used to represent
 * that object when it does not exist.</p>
 */
public class FxAnimateShaderTransition extends FxAnimate
{
    public function FxAnimateShaderTransition(target:Object=null);

    /**
     * The bitmap data representing the start state of this effect.
     * If this property is not set, it will be calculated automatically
     * when the effect is played by grabbing a snapshot of the target
     * object, or by using a transparent bitmap if the object does not
     * exist in the start state of a transition.
     */
    public var bitmapFrom:BitmapData;

    /**
     * The bitmap data representing the end state of this effect.
     * If this property is not set, it will be calculated automatically
     * when the effect is played by grabbing a snapshot of the target
     * object, or by using a transparent bitmap if the object does not
     * exist in the end state of a transition.
     */
    public var bitmapTo:BitmapData;

    /**
     * The bytecode that a <code>Shader</code> object will use
     * for running the transition between the two bitmaps. This
     * property can be represented as either a ByteArray or as a
     * Class representing a ByteArray (which is what results 
     * when you embed a resource).
     * 
     * <p>The shader can have arbitrary functionality and inputs, but 
     * must, at a minimum, have three <code>image4</code> inputs.
     * The first input, which can be named anything, should go
     * unused by your shader code - it exists only to satisfy the
     * Flash requirement of assigning a filtered object to the
     * first input. Note that inputs that go completely unused in a
     * shader kernel may be optimized out, so your kernel code should
     * at least reference this input to keep it around.</p>
     * 
     * <p>There must be at least two other inputs
     * named <code>from</code> and <code>to</code> and one 
     * <code>float</code> parameter named <code>progress</code>, where
     * <code>from/to</code> are the before/after bitmaps, respectively,
     * and <code>progress</code> is the elapsed fraction of
     * the effect.</p>
     * 
     * <p>Also, there are two optional parameters, <code>width</code>
     * and <code>height</code>, which if they exist in the shader
     * will be automatically set to the width and height for the
     * effect instance target.</p>
     * 
     * <p>See the Pixel Bender Toolkit documentation for more
     * information on writing shaders for Flash.</p>
     * 
     * @example To play an effect that uses a fictional Pixel Bender pbj 
     * file MyShader.pbj, which takes a single <code>direction</code>
     * parameter, the calling code could do this:
     * @example <listing version="3.0">
     *   <a href="Embed%28source%3D%26quot%3BMyShader.pbj%26quot%3B%2C%20mimeType%3D%26quot%3Bapplication/octet-stream%26quot%3B%29">Embed(source="MyShader.pbj", mimeType="application/octet-stream")</a>
     *   private var ShaderCodeClass:Class;
     *   var shaderEffect = new FxAnimateShaderTransition();
     *   shaderEffect.shaderCode = ShaderCodeClass;
     *   shaderEffect.shaderProperties = {direction : 1};</listing>
     * or in MXML code, this:<listing version="3.0">
     *   &lt;FxAnimateShaderTransition 
     *       shaderCode="&64;Embed(source='MyShader.pbj', mimeType='application/octet-stream')"
     *       shaderProperties="{{direction : 1}}}"/&gt;
     * </listing>
     * 
     * @see flash.display.Shader
     */
    public var shaderCode:Object;

    /**
     * A map of parameter name/value pairs that the shader
     * will set its data values to prior to playing. For example,
     * to set a parameter named <code>direction</code> in a
     * shader with a Pixel Bender pbj file in Wipe.pbj, the calling 
     * code could do this:
     * @example <listing version="3.0">
     *   <a href="Embed%28source%3D%26quot%3BWipe.pbj%26quot%3B%2C%20mimeType%3D%26quot%3Bapplication/octet-stream%26quot%3B%29">Embed(source="Wipe.pbj", mimeType="application/octet-stream")</a>
     *   private var WipeShader:Class;
     *   var shaderEffect = new FxAnimateShaderTransition();
     *   shaderEffect.shaderCode = new WipeShader();
     *   shaderEffect.shaderProperties = {direction : 1};</listing>
     */
    public var shaderProperties:Object;
}

public class FxAnimateShaderTransitionInstance extends FxAnimateInstance
{
    public function FxAnimateBitmapInstance(target:Object);
    public var bitmapFrom:BitmapData;
    public var bitmapTo:BitmapData;
    public var shaderCode:ByteArray;
    public var shaderProperties:Object;
    /**
     * The Shader that is created using the <code>shaderBytecode</code>
     * property as the underlying bytecode. Each instance needs its
     * own separate Shader, but can share the bytecode. On instance
     * creation, we create the Shader that the instance will use.
     */
    protected var shader:Shader;    
    /**
     * The filter wrapped around the instance's <code>shader</code>
     * property. This filter is assigned to the <code>filters</code>
     * property of the target object with every update during the animation,
     * so that animated updates to the underlying shader are reflected
     * in the filter applied to the display object that the user sees.
     */
    protected var shaderFilter:ShaderFilter;
}

/**
 * This class performs a bitmap transition effect by running a
 * directional 'wipe' between the first and second bitmaps.
 * This wipe exposes the second bitmap over the course of the 
 * animation in a direction indicated by the <code>direction</code>
 * property.
 * 
 * <p>The underlying bitmap effect is run by a Pixel Bender shader 
 * that is loaded by the effect. There is no need to supply a shader
 * to this effect since it uses its own by default. However, if
 * a different Wipe behavior is desired, a different shader may be
 * supplied, as long as it adheres to the following constraints: 
 * obey the constraints specified for the <code>shaderCode</code>
 * property of FxAnimateShaderTransition, and supply three additional
 * parameters. The extra parameters required by the FxWipe shader 
 * are an int <code>direction</code> parameter, 
 * whose values mean the same as the related String properties
 * in the FxWipe class, and floating point parameters
 * <code>imageWidth</code> and <code>imageHeight</code>. All of these
 * parameters will be set on the shader when the effect starts playing,
 * so the parameters need to exist and do something appropriate in
 * order for the effect to function correctly.</p>
 * 
 * @see mx.effects.FxAnimateShaderTransition#shaderCode
 */
public class FxWipe extends FxAnimateShaderTransition
{
    /**
     * The direction that the wipe will move during the animation, 
     * one of RIGHT, LEFT, UP, or DOWN. Other values will result in
     * undefined behavior. If no direction is supplied, a default
     * of RIGHT will be assumed;
     * @see WipeDirection#RIGHT
     * @see WipeDirection#UP
     * @see WipeDirection#LEFT
     * @see WipeDirection#DOWN
     */
    public var direction:String;

    public function FxWipe(target:Object=null)
}

public class FxWipeInstance extends FxAnimateBitmapInstance
{
    public var direction:String;
}

/**
 *  The WipeDirection class defines the values
 *  for the <code>direction</code> property of the FxWipe class.
 *
 *  @see mx.effects.FxWipe
 */
public class WipeDirection
{
    /**
     * Wipe direction that starts at the left and moves right
     */
    public static const RIGHT:String = "right";

    /**
     * Wipe direction that starts at the bottom and moves up
     */
    public static const UP:String = "up";

    /**
     * Wipe direction that starts at the right and moves left
     */
    public static const LEFT:String = "left";

    /**
     * Wipe direction that starts at the top and moves down
     */
    public static const DOWN:String = "down";
}

/**
 * This class performs a bitmap transition effect by running a
 * 'crossfade' between the first and second bitmaps.
 * The crossfade blends the two over the course of the 
 * animation such that, at any point in the animation, where the 
 * elapsed and eased fraction of the animation is f and the pixel
 * values in the first and second bitmaps are v1 and v2, the resulting
 * pixel value v for any pixel in the image will be
 * <code>v = v1 * (1 - f) + v2 * f</code>.
 * 
 * <p>The underlying bitmap effect is run by a Pixel Bender shader 
 * that is loaded by the effect. There is no need to supply a shader
 * to this effect since it uses its own by default. However, if
 * a different Crossfade behavior is desired, a different shader may be
 * supplied, as long as it adheres to the constraints specified 
 * for the <code>shaderCode</code> property of FxAnimateBitmap.</p>
 * 
 * @see mx.effects.FxAnimateShaderTransition#shaderCode
 */
public class FxCrossFade extends FxAnimateShaderTransition
{
    public function FxCrossFade(target:Object=null)
}

B Features

The main focus of this feature is the base FxAnimateShaderTransitionfunctionality of snapshotting before/after bitmaps of the target and running a Shader on the results, and the FxWipe/FxCrossFade subclasses to perform standard Wipe and Crossfade effects. We might also want to supply new Fx versions of the following old effects, once this base functionality is complete:

• Dissolve: Perform a color-dissolve on the target
• Iris: Perform an iris effect on the target

Examples and Usage

This is an example of using FxCrossFade and four versions of FxWipe on five different target objects:

    <transitions>
        <Transition>
            <Parallel duration="1000">
                <FxCrossFade targets="{<a href="button">button</a>}"/>
                <FxWipe targets="{<a href="buttonwr">buttonwr</a>}" direction="{WipeDirection.RIGHT}"/>
                <FxWipe targets="{<a href="buttonwu">buttonwu</a>}" direction="{WipeDirection.UP}"/>
                <FxWipe targets="{<a href="buttonwl">buttonwl</a>}" direction="{WipeDirection.LEFT}"/>
                <FxWipe targets="{<a href="buttonwd">buttonwd</a>}" direction="{WipeDirection.DOWN}"/>
            </Parallel>
        </Transition>
    </transitions>

Here is one of the target objects. The only change between states is the label:

    <Button id="button" x="200" y="100" width="100"
        label="crossfade" label.state2="CROSSFADE"
        click="currentState=(currentState=='state1')?'state2':'state1'"/>

When the state changes, the default transition runs and all buttons are animated according to the effect being run on them.

Here is an example of running a wipe-right effect manually, loading a local shader and setting the properties directly by using the FxAnimateShaderTransition superclass:

                <FxAnimateShaderTransition target="{buttonwd}" 
                    shaderCode="@Embed(source='shaders/FxWipe.pbj', mimeType='application/octet-stream')"
                    shaderProperties="{{width:buttonwd.width, height:buttonwd.height, direction:0}}"/>

Additional Implementation Details

There will be certain constraints with using this effect, due to either Flash player implementation details or the fact that we're using bitmap representations instead of live DisplayObjects: - snapshotting text is not the same as rendering it onscreen, at least not when cleartype is enabled. Therefore, some text objects may look different in transitions than they do live on the screen. - Targets that modify in size or orientation during a state change may have issues. Hopefully this will work with Parallel effects that Move/Resize/etc the targets, but there may be some problems with how PixelBender expects the from/to bitmaps to be of the same size. - Performance could be an issue for some effects. In some benchmark tests, we have seen performance degrade significantly as the size of the target object increases. For example, a crossfade shader operating on a 50×50 target object was gated only by the underlying performance of the flash Timer, and was able to get rates of over 80 fps. But as the size increased to 500×500, the performance dipped to about half of that rate, and at 900×900, the performance was down to about 12 fps.

Prototype Work

Most of the coding is done and the API above reflects changes that have occurred in the review and coding process.

Compiler Work

None

Web Tier Compiler Impact

None

Flex Feature Dependencies

None

Backwards Compatibility

Syntax changes

None

Behavior

None. Note that the new version of FxWipe is significantly different than the old Wipe effect. For example, there is no sizing behavior of these new bitmap effects. Also, there is just one FxWipe class that takes a direction property, as opposed to the previous four subclasses that each had a hard-coded direction.
Note, however, that none of these differences affect apps that use the old effects; it's just something that developers used to the old effects should know when they try to use the new ones.

Warnings/Deprecation

None

Accessibility

None

Performance

There are two performance concerns here (that I know of): - Complex shaders, or shaders that are operating on very large bitmaps, may take longer to run each frame and could thus result in animations of a much lower frame rate. As noted above, a simple shader performing a cross-fade was able to achieve rates of over 80 fps (limited only by the underlying Timer performance) when running on a 50×50 target object, but had rates of only 12 fps when running on a target object of size 900×900. - Flash appears to have an issue running multiple simultaneous shaders in the background. This is no longer an issue with our implementation, since we are now using ShaderFilter instead of ShaderJob as we were using in an earlier iteration. But it's a Flash bug that they should fix, and which may apply to us if we implement more complex shading approaches in the future that use asynchronous ShaderJobs.

Globalization

None

Localization

Compiler Features

None

Framework Features

None

Issues and Recommendations

As noted above, mainly the issue about Flash support for multiple simultaneous ShaderJobs.

Documentation

We might consider putting a sample shader in the docs since it won't be obvious how to do this from the ASDocs spec. We should also defer to the Pixel Bender documentation for shader development in general.