3D Effects Support - Functional and Design Specification

Glossary

• matrix: the player class for representing 2D geometric transformations (x, y, scaleX, scaleY, rotation).
• matrix3D: the player class for representing 3D geometric transformations (add z, scaleZ, and rotationX, rotationY). Added in FP10.
• projection matrix: the matrix used to take a object in 3D and flatten it down into a bitmap that can be rendered on screen. By default, defined on the root of the display list, but can also be defined on any aribtrary parent in the display hierarchy, effectively converting all of the 3D content of that parent into a 2D bitmap rendered as a child.
• convenience transform properties: the properties x, y, scaleX, scaleY, rotation (z, rotationX, rotationY, scaleZ). These values are combined in a standard equation to form the actual matrix transforms used for calculation.
• layout matrix: the (for now, 2D) transform taken used by flex during layout. A 100×20 rectangle with a layout matrix rotated 90 degrees will be treated by layout as though it were a 20×100 rectangle. the layout matrix can be assigned directly, or is built from the convenience transform properties defined on components.
• offset transform: the 3D transform properties concatenated on to the layout transform properties to determine the actual transform used by the component for rendering (by the flash player, typically). The offset transform is not a matrix, but a set of deltas to the common layout convenience properties.

Summary and Background

Flash Player 10 has added support for applying arbitrary 3D transforms to any element in the flash display list. True 3D rendering can be used to create attractive and engaging effects, transitions, and interfaces, and are something we wholeheartedly want to support in flex.

Rather than forcing developers to create explicit modal switches between the 3D world and the 2D world, the player has instead chosen to essentially turn the entire display list into a 3D environment. Ignoring backwards compatibility, performance, and implementation details, the display list in FP10 is essentially an entire stack of postcards sitting on the z=0 plane in 3D space. Any individual element in the display list can be 'popped' out of the z=0 plane by simply changing one of its new 3D transform properties.

Developers can create a modal switch from a 3D world back to a 2D world by attaching a projection matrix to any arbitrary node in the display list. All 3D content beneath the projection matrix uses it to render into an offscreen bitmap, which is then rendered as just another bitmap in the z=0 plane of the node's parent, effectively switching back into a 2D world.

This approach creates some interesting challenges for the flex layout system. Flex's layout system is inherently 2D, with individual layout managers sizing and positioning children on the z=0 plane based on the assumption that they lie completely flat against it.

This spec, then, outlines what changes to Flex's layout system is necessary to enable common uses of the new 3D transform, and the necessary resulting changes to the core component API.

Overview

The SDK should support 3D transforms in multiple ways:

true 3D layouts

Some of the sexier flash content out there uses programmatic, data driven 3D layout of inherently 2D content (images, text, etc). This must be something easy to do in Flex. A customer should be able to drop a layout onto a Group that can take ownership of the 3D transform of its children (just as a layout should be able to take ownership of the 2D transform of its children). Just as with 2D layouts, it is the parent's responsibility to appropriately assign the actual width and height of the children.

The main difficultly in doing this in Flex is the relationship between the layout transform of the child and the measured sizes reported to the parent during layout. A parent that takes control over the transform of its children must be able to determine and set the size of the component in its own untransformed coordinate system, and assign final values to its layout transform.

A simple example is the way we treat x/y properties today. Developers can set x/y and they will be respected by layouts if appropriate (Canvas). Other layouts (Vertical) ignore the user assigned values, and x/y are properties that a developer can set if they choose.

Status: see Proposal below

post layout transforms

Our layouts are inherently 2D, assuming that all of their children lie completely within the z=0 plane. When a child lies within the z=0 plane, the projection matrix becomes a no-op -- the bounds of the component in the parent coordinate space remain the same, regardless of where the component ends up on screen relative to the projection matrix.

This allows our layout system to automatically transform measured and actual sizes between the child and parent coordinate space, regardless of what the component's transform is, as long as it remains within the z=0 space.

putting a 3D transform on an item, however, pops the component out of the z=0 space, and disrupts our layout system. In these cases, Flex will continue to perform layout as though the component continued to lie in the z=0 plane, effectively ignoring the additional 3D transformations applied to the component. These transformations need to persist through additional layout passes, so semi-permanent effects can be applied even as other actions perturb the application (i.e., a vertical list with items that subtly twist in the breeze, or card-flip effects that flip a component over even as it resizes).

Status: Proposal Below

order-independent layering

Flash determines the layer of a display object based on child order in the display list: a DO layers above its parent and previous siblings, and below its subsequent siblings and parent siblings. This rule applied even to components with 3D transforms applied; it is up to the developer to manage child order to guarantee the right effect when using 3D effects.

This is problematic in flex, where child order dictates layout behavior as well as depth. i.e., in Flex 3, there is no way for a developer to make a child be the topmost element in a VBox (first child) and the topmost layered child (last child).

In Gumbo, we need to provide developers with a way to control the layering of children in a Group, independent of child order.

Status: Proposal Below

2D/3D joins

If we support 3D layouts, they will still in most cases need to live within a 2D based parent layout. A parent with a 3D based layout has a similar problem living in a 2D layout as a child with a 3D matrix -- how can the 2D preferred size of the content be measured when its 2D representation depends on the transform of all of its parents (and the projection matrix defined at the root).
The solution is to support a way to explicitly transform a 3D space into a 2D space for layout purposes. In the flash player, this is done by assigning a projection matrix to an arbitrary display object; a DO with a projection matrix renders all of its children into an off screen bitmap using the projection, effectively flattening itself back into a 2D object.

When a Group has an associated projection matrix, it should be able to effectively translate its measured and assign sizes back and forth from its 3D internal space to its parent's 2D coordiante space using (and potentially modifying) the projection.

Status: Not in Gumbo.

Usage Scenarios

3D transitions

Developer Tom is building a portal application. He'd like to spice it up by adding some nice transitions as the user navigates through the application. In particular, he likes the 'card flip' effect he's seen on various applications. When the user clicks the 'options' button on any of his individual portlets, he wants to have it flip over in place in 3D around the Y axis, without disturbing any of the surroundling portlets, to show an options panel on the back.

simple 3D animations

Developer Sue is creating a simple widget for navigating around her site. The goal is to make the various links in the widget appear like a stack of signs on an old signpost. The basic layout is a simple vertical stack, but she wants each element to subtly rotate back and forth around the vertical center as they 'blow in the wind.'

3D layouts

Developer Bob is creating a dynamic ecommerce site to sell products from around the globe. His application consists of a typical bit of e-store chrome -- navigation UI, shopping cart UI, etc. -- surrounding a 3D image of the earth, with a thumbnail of each product attached to its country of origin. As the globe spins, the products rotate in and out of view with their country.

Detailed Description

True 3D layout

Flex's layout system to date has had a mixed policy on what control a layout has over the transform of its children. Generally speaking, flex allows layouts to control the translation (x,y) of its children, but not the scale or rotation. Layouts typically deal with their children in a post-transform coordinate system, thinking of the child as a simple rectangle that can be positioned and sized, leaving the details of how that interacts with the actual component transform up to the framework.

True 3D layouts, on the other hand, require complete control over the transform of a component. Almost any 3D layout you can imagine needs to be able to control the rotation of the component, and likely the scale as well. This presents a challenge; some layouts want to be completely ignorant of how the child is being transformed, while some layouts need fine grained control.

To address both style of layouts, the API a component presents to the layout must support:

1. the ability to query preferred sizes in parent coordinate space.
2. the ability to set 'actual size' in parent coordinate space.
3. the ability to query preferred sizes in child coordinate space.
4. the ability to set actual size in child coordinate space.
5. the ability to set the 'actual' layout transform without triggering a revalidation.

These APIs are being added and addressed as part of the layout APIs spec. The only additional work is to make sure that the layout can legally provide a matrix3D as the actual layout transform.

post layout transforms

Gumbo will support post layout transform with the new offsets property.

Every component in Gumbo (UIComponents and GraphicElements) will support a single new property 'offsets.' of type 'TransformOffsets.' A developer can apply changes to the component's layout after all parent specified layout is done by modifying the offset values.

For example, to slide an item in a vertical list 10 pixels to the right of its siblings, scale it vertically in a way that doesn't affect its sibling layout, and move it up:

var item:UIComponent = group.items<a href="3">3</a>;
item.offsets = new TransformOffsets();
item.offsets.x = 10;
item.offsets.y = -10;
item.offsets.scaleY = 1.2;

Accessing the transform

A developer uses the following properties to access the transform of a component:

• x, y, z, scaleX, scaleY, scaleZ, rotationX, etc.: convenience properties used to read and write the layout matrix. Note that as with the flash player, if the developer sets the layout matrix directly, these values will be decomposed from the matrix as best as possible.
• layoutMatrix, layoutMatrix3D: new properties used to read / write the matrix used for layout calculations. If these properties are set directly, they will be decomposed into the associated transform properties. If the individual transform properties are modified, these properties will update to reflect the recomposed matrices.
• offsets.x, y, scaleX, etc: convenience properties used to read and write the offsets applied to the layout transform properties for rendering.

2D vs. 3D

The flash player uses different rendering paths for content with 2D and 3D transforms applied to them. Anything with a 3D transform applied to it is first rendered into an offscreen buffer, then transformed in 3D, then composited into the standard flash 2D pipeline.

This is fine for components that have explicit 3D transforms applied, but causes unacceptable rendering degradation for components that don't have any reason to be in 3D (i.e., components that lie flat in the z=0 plane).

We need to solve this problem in two ways:

1. keep 3D transforms pay-as-you-go, just as they are in the flash player; even when using offsets, a component should only use 3D once 3D properties have been modified.
2. drop back into 2D transforms whenever possible; once in 3D mode, periodically check the rotation and translation properties and drop back to 2D mode when possible.

building the matrices

The various matrices of a component in Gumbo are computed as follows:

component.layoutMatrix
1. translate(-transformX,-transformY);
2. scale(scaleX, scaleY);
3. rotate(0,0,rotationZ);
4. translate (transformX + x, transformY + y);

or The layout matrix explicitly assigned by the user.

component.layoutMatrix3D
1. translate(-transformX,-transformY,0);
2. scale(scaleX, scaleY, scaleZ);
3. rotate(rotationX, rotationY, rotationZ);
4. translate(transformX + x, transformY + y, transformZ + z);
The 3D layout matrix explicitly assigned by the user.

actual final component transform:

if offsets are unset, use the layoutMatrix or layoutMatrix3D as the final computed transform.

if the 3D convenience properties are no-ops or unset:
1. translate(-transformX,-transformY);
2. scale(scaleX * offsets.scaleX, scaleY * offsets.scaleY);
3. rotate(0,0, rotation + offsets.rotationZ)
4. translate (transformX + x + offsets.x, transformY + y + offsets.y);

if the 3D convenience properties are set:

1. translate(-transformX,-transformY,-transformZ);
2. scale(scaleX * offsets.scaleX, scaleY * offsets.scaleY, scaleZ * offsets.scaleZ);
3. rotate(rotationX + offsets.rotationX, rotationY + offsets.rotationY, rotation + offsets.rotationZ)
4. translate (transformX + x + offsets.x, transformY + y + offsets.y, transformZ + z + offsets.z);

order independent layering

Flash dictates the layering of components based on child order, and there is no getting around that. The only way for us separate layering from component ordering then is to separate component ordering from flash display object ordering.

Fortunately, the integration of graphics in Gumbo has provided a natural place for us to do this. For performance reasons, the GraphicElements added in Gumbo do not derive from DisplayObject -- instead, they draw into display objects, allowing us to aggregate multiple graphic elements into a single Shape.

This means that the items in a Gumbo Group are not necessarily DisplayObjects, and may not even correspond 1-1 to the display objects that get handed down to the underlying player implementation. The components and display object children are already decoupled.

order independent layering, then, can be simply described as sorting the component items based on a new 'layer' property before mapping them into underlying display objects.

performance

order-independent layering is a uncommon enough feature that it should be pay-as-you-go on a per component basis. Specifically:

• the layering code should assume that items will generally be rendered in item order. An item with a non-zero layer should be treated as an exception. Specifically, no additional work (esp. sorting of the items) should be done unless a non-zero layer is discovered on a child item.

API Description

// Note that all properties below are get/set functions, written as vars for brevity's sake.

/**
* An interface representing a slew of transform objects. defining an interface for these allows 
code that manipulates transforms (specifically, effects) to work with either a component's 
layout transform or offset transform agnostically.
*/
public interface ITransformable{

/** the translation component of the transformation */
public var x,y,z:Number;

/** the scale component of the transformation */
public var scaleX,scaleY,scaleZ:Number;


/** the rotation component of the transformation */
public var rotationX,rotationY,rotationZ:Number;

}

/**
* The common interface between UIComponent and GraphicElement
*/
public interface IVisualItem {

/** the matrix used for 2D layout of this component */
public var layoutMatrix:Matrix;

/** the matrix used for 3D layout of this component */
public var layoutMatrix3D:Matrix3D;

/** the center of transformation for both layout and offset transforms on
*   this component */
<a href="get/set">get/set</a> public var transformX,transformY,transformZ:Number;

}

/** a TransformOffset object represents a set of adjustments made
* to the transform of a component _after_ the parent has had a chance
* to adjust the size/transform of the component during a layout pass.
*/
public class TransformOffsets implements ITransformable
{
public var x,y,z:Number;
public var scaleX,scaleY,scaleZ:Number;
public var rotationX,rotationY,rotationZ:Number;
}

public class UIComponent implements ITransformable
{
// ... previous fields ...

/**
*  The convenience transform properties that are used to compose the
*  transform using when the parent performs layout on the element.  
*  If any of these properties are set directly, the layout matrix is 
*  rebuilt from the current values of these properties. if either 
*  the matrix or matrix3D properties are set directly, these values are 
*  populated from the decomposed matrix.
*/
<a href="get/set">get/set</a> public var x,y,z:Number;
<a href="get/set">get/set</a> public var scaleX,scaleY,scaleZ:Number;
<a href="get/set">get/set</a> public var rotationX,rotationY,rotationZ:Number;
<a href="get/set">get/set</a> public var transformX,transformY,transformZ:Number;

/**
*  The 2D matrix representing the layout matrix for this element.  Typically, this 
*  is built from the 2D convenience properties defined above.  If this property 
*  is assigned directly, the 2D convenience properties are decomposed out of 
*  this matrix. the 3D properties are left alone.
*/
<a href="get/set">get/set</a> public var layoutMatrix:Matrix;
/**
*  The 3D matrix representing the 3D layout matrix for this element.  Typically, 
*  this is built from the convenience properties defined above.  If this 
*  property is assigned directly, the convenience properties are decomposed 
*  out of this matrix.
*/
<a href="get/set">get/set</a> public var layoutMatrix3D:Matrix;

/**
* The offsets applied to this element's transform after layout occurs.  
* Defaults to null.
*/
<a href="get/set">get/set</a> public var offsets:TransformOffsets;

/**
* The depth at which this element will appear among its siblings.  Elements with 
* equal layerDepth appear in the order they appear in their parent's item list.  
* higher values appear above lower values.  Defaults to 0. Note that this property 
* only has affect when the element appears inside components with support for the 
* layerDepth property.  This is true for Groups, Skins, and new Gumbo skinned 
* components, but not true for classic Halo containers and random 
* UIComponents.
*/
<a href="get/set">get/set</a> public var layerDepth:Number;

/**
* a function used by a child component to inform its parent that its layerDepth has 
* changed, possibly requiring a change to the layering of the parent's items.  While 
* this API exists on UIComponent, it has no effect on the base class.  Derived  
* classes must override to add implemenation if they wish to support arbitrary 
* layering of their items.
*/
public function invalidateLayering();
}


public class GraphicElement implements ITransformable
{
// ... previous fields ...

/**
*  The convenience transform properties that are used to compose the transform 
*  using when the parent performs layout on the element.  If any of these 
*  properties are set directly, the layout matrix is rebuilt from the current 
*  values of these properties. if either the matrix or matrix3D properties are 
*  set directly, these values are populated from the decomposed matrix.
*/
<a href="get/set">get/set</a> public var x,y,z:Number;
<a href="get/set">get/set</a> public var scaleX,scaleY,scaleZ:Number;
<a href="get/set">get/set</a> public var rotationX,rotationY,rotationZ:Number;
<a href="get/set">get/set</a> public var transformX,transformY,transformZ:Number;

/**
*  The 2D matrix representing the layout matrix for this element.  
*  Typically, this is built from the 2D convenience properties 
*  defined above.  If this property is assigned directly, the 
*  2D convenience properties are decomposed out of this matrix. 
*  the 3D properties are left alone.
*/
<a href="get/set">get/set</a> public var layoutMatrix:Matrix;
/**
*  The 3D matrix representing the 3D layout matrix for this element.  
*  Typically, this is built from the convenience properties defined 
*  above.  If this property is assigned directly, the convenience 
*  properties are decomposed out of this matrix.
*/
<a href="get/set">get/set</a> public var layoutMatrix3D:Matrix;

/**
* The offsets applied to this element's transform after layout occurs.
* Defaults to null.
*/
<a href="get/set">get/set</a> public var offsets:TransformOffsets;

/**
* The depth at which this element will appear among its siblings.  
* Elements with equal layerDepth appear in the order they appear in 
* their parent's item list.  higher values appear above lower values.  
* Defaults to 0. Note that this property only has affect when the 
* element appears inside components with support for the layerDepth 
* property.  This is true for Groups, Skins, and new Gumbo skinned 
* components, but not true for classic Halo containers and 
* random UIComponents.
*/
<a href="get/set">get/set</a> public var layerDepth:Number;

}

B Features

• Support in new animations for manipulating offset transforms properties
• Support for pre-transform size offsets (i.e., offset.width, offset.height)
• support for layering outside of the the constraints of who your parent is.
• support for proper measurment / actual size calculation when a Group has a projection
  matrix defined on it.

Examples and Usage

TBD

Additional Implementation Details

performance

transform offsets are a useful feature, but one that should incur no more than a trivial cost for the majority of components that won't take avantage of it.

The offsets property of our components will default to null, indicating that the layout and rendering behavior of the component should be one in the same.

When the offset property is null, all of the standard transform properties on UIComponent route to the standard transform functionality as defined by the flash player. accessing the layout and render matrix properties both simply return the underlying transform from the flash player.

Assigning a TransformOffset object to the offsets property, the transform properties of the component kick into high gear:

• standard transform convenience properties (scale,translation,rotation) store their values in hidden fields in the TransformOffset object.
• during commitProperties, the standard properties are combined to calculate a layout matrix. the layout matrix is also cached in the TransformOffset object.
• before updateDisplayList is called, the cached standard properties are combined with the offset transform properties to build the render transform, which is also cached in the TransformOffset. The render transform is either applied to the displayObject, or used during the rendering.
• all of these updates will be appropriately gated by invalidation flags for the transform.

Prototype Work

Prototype work can be found here.

Compiler Work

None

Web Tier Compiler Impact

None

Flex Feature Dependencies

Heavy Dependence/Interaction with ongoing layout enhancements in Gumbo.

Backwards Compatibility

Flex 3 has limited support for developers setting the transform object of a component directly. Some developers might be directly manipulating the transform, although it has the potential to break the layout system in fundamental ways.

As long as a component is not using any transform offsets, setting the transform will continue to work as before. But once a component is assigned an offset, the underlying player transform no longer maps directly to the layout convenience properties seen and used by the flex layout API.

Otherwise, this feature presents no aditional challenges to backwards compatibility that are not already presented by the existing layout changes.

Syntax changes

None.

Behavior

None.

Warnings/Deprecation

None.

Accessibility

QUESTION: Do screen readers walk the display list directly, or does it rely on intervention from the developer to indicate next/previous child? If so, changing the layer of a component might affect the order in which a screen reader sees the components. Is that a problem? Is there any way around it?

Performance

Impact on overall performance should be trivial. See implementation section for details.

Globalization

No impact.

Localization

None.

Compiler Features

None.

Framework Features

None.

Issues and Recommendations

• there's really two matrices people might care about…the matrix used for layout, and the matrix used for rendering, which is the composite of the layout and offset properties. Do we need to provide access to the final actual matrix?

Documentation

Describe any documentation issues or any tips for the doc team.

QA

Important tests: - test layout/transform performance of components with no offsets applied, make sure the impact is trivial. - test layout/transform performance of components with offsets applied, measure and track the impact. - test layout behavior of component with no offsets applied, vs. with no-op offsets applied. behavior should be the same. - test effect of setting layout properties directly vs. setting matrix. make sure that both behave as expected, and that mixing back and forth between matrix and convenience properties behaves as expected. - test rendering of components when not using offsets. Should be unaffected. - test rendering of components when only using 2D offsets. Should be unaffected. - test rendering of components when 3D transform properties have been assigned, and then removed (i.e., 3D transform properties are no-ops). Rendering shoudl be identical to having never set them. Same for 3D offset properties, and matrix properties.