Menu ▾ ▴

GL Shapes

Peter Maersk-Moller

GL Shapes (Snowmix-0.5.x)

This page has been obsoleted by the Snowmix GL Shape Guide on the new Snowmix website.

Introduction

GL Shapes or glshapes enables use of OpenGL in Snowmix for video mixing. With OpenGL Snowmix can create hardware accelerated 2D and 3D effects. Hardware acceleration require the hardware to support OpenGL on the system Snowmix is running. If hardware acceleration is not supported, Snowmix will fall back to Off Screen CPU based rendering for OpenGL support. GL shapes will become available in Snowmix-0.5.0 to be released during 2015H1.

GL Context

The GL Context is an OpenGL context within Snowmix enabling OpenGL graphics. Snowmix supports the Off Screen Mesa (OSMesa) and the GLX/X-Windows (GLX) context. The OSMesa context is CPU based. The GLX context is GPU based and requires a local graphic adapter supporting OpenGL. Further more the local machine has to support and run X-Windows as well as allowing Snowmix to open a window within its X-Window session. Using a GPU can offer great performance, not only because a GPU can have a very high performance executing otherwise complicated heavy load CPU tasks, but also because it can do so in parallel with the CPU perhaps doing other tasks. However it comes at a cost of not only complexicity and the cost of learning a new skill being OpenGL programming, but it also usually require transferring large amount of data from CPU memory to GPU and back again. While the transfer to the GPU is DMA and queue based, transferring from GPU to CPU memory is currently CPU intensive.

GL Shapes

Snowmix allows defintion of glshapes being a list of primarily OpenGL graphic primitives/commands similar to creating list of Cairo Graphics primitives/commands for regular shapes. A GL shape is defined by its unique glshape Id. Each glshape needs a glshape place holder defined with the command glshape add. This is shown in the example below:

    glshape add 1 My first glshape

Here the glshape place holder 1 is defined and given the name 'My first glshape'. After the glshape has been created, it can now be added commands to its list. The commands are primarily OpenGL primtives with only a few exceptions. The commands all have the form:

    glshape COMMAND_NAME <shape id> ......

Such commands will add the command to end of the glshape command list identified by the shape id. It is important to understand that commands added to a glshape are parsed and added to a glshape command list, but they are not executed at that moment. For the commands in a glshapes command list to be executed, a glshape has to be placed and overlayed. Below is an example of creating a glshape command list with commands:

    glshape add 1 My first glshape
    glshape loadidentity 1
    glshape matrixmode 1 projection
    glshape enable 1 blend

The glshape command list can be inspected using the command glshape list <glshape id> as shown below:

    gl list 1
    STAT:  shape 1 ops 3 name My first glshape
    STAT:  -  1 loadidentity
    STAT:  -  2 matrixmode projection 
    STAT:  -  3 enable mode blend  
    STAT:

The command glshape entry can be used to move commands within a glshape command list as well as used for deleting a specific command in a glshape command list.

The command glshape modify can be used to change numbers and setting of some commands in a glshape command list.

If the glshape 1 is later executed, that is if the glshape is placed and overlayed, the following OpenGL C/C++ commands are excuted in whatever context is active at that moment:

A glshape definition can later be used as the basis for a placed glshape like a shape is used for placed shape in Snowmix.

GL Textures

Textures in Snowmix are images the shader can use as a surface on 2D and 3D objects. Snowmix supports 2D and Cube textures. The source of textures in Snowmix are images, video feed frames and the main mixer frame. Textures are used to move image data from CPU memory into the graphical context or GPU.

Defining a texture To use a texture in Snowmix, a place holder within Snowmix has to be created. This is done using the command glshape texture add. Each texture place holder is identified by its unique texture Id, which is an integer number starting from 0. In the following example, 3 textures are created. First the place holders for the textures are created:

    glshape texture add 1 Texture for Mixer Frame
    glshape texture add 2 Texture for Video Feed #1
    glshape texture add 3 Texture for Loaded Image #5

The syntax for the command is as follows:

    glshape texture add [<texture id> [<texture name>]]

The command texture add can not be added to a glshape command list although technically it is possible to workaround this limitation.

A texture place holder is deleted using the command glshape texture add with only the texture id argument.

Setting the source for a texture Before using a texture, the source of the texture must be defined. This is done using the command glshape texture source. In the example below, the source is set for the 3 texture place holder we just defined above:

    glshape texture source 1 frame 0
    glshape texture source 2 feed 1
    glshape texture source 3 image 5

In this example we defined texture 1 to be sourced by the mixer frame 0 (only one exist). We also set texture 2 and 3 to be sourced by feed 1 and image 5 respectively. Feed 1 and image 5 must exist at this point. The source of a texture can be removed setting the source to none as shown in this example:

    glshape texture source 2 none

The syntax for the *texture source command is as follows:

    glshape texture source [<texture id> (feed <feed id>|frame <frame id>|image <image id>)|none [<id>]]

There is no OpenGL equivalent to the commands. The command texture source can not be added to a glshape command list although it is possible to workaround this limitation.

Binding a texture The actual moving of image data from CPU to the context or GPU is not taking place until a texture is bound. This is done using the command glshape texture bind. The command needs to part of a glshape command list and that glshape has to be placed and overlayed. Below is shown the full syntax for the glshape texture bind command:

glshape texture bind <glshape id> <texture id> [<min> <mag>] [2d | cube]

When a glshape is executed and it reaches the texture bind command, it will allocate a an internal texture in the context or GPU. Such a ressource is limited by the amount of RAM available for the context/GPU. After an internal texture has been generated, the image data is loaded to the texture in the context or GPU. The texture bind allocation is equivalent to the the following OpenGl commands:

In the following example glshape 1 is set to create a texture, source it, bind it and use it on a four sided polygon, a quad. This glshape can then later be placed and overlayed. When a placed glshape is overlayed, its command list is executed line by line.

    1  glshape add 1 This is a list of shape commands
    2  glshape enable 1 texture_2d           # Enable 2D textures.
    3  glshape texture bind 1 3 linear nearest  # Bind texture 3 as current texture
    4  glshape begin 1 quads                 # Begin a 4 point polygon
    5    #glshape normal 1 0 0 1               # Setting the normal for the quad
    6    glshape texcoord 1 0 0              # Using current texture lower left
    7    glshape vertex 1 -0.5 -0.5 0        # Setting Lower left corner
    8    glshape texcoord 1 1 0              # Using current texture lower right
    9    glshape vertex 1 0.5 -0.5 0         # Setting Lower right corner
    10   glshape texcoord 1 1 1              # Using current texture upper right
    11   glshape vertex 1 0.5 0.5 0          # Setting Upper right corner
    12   glshape texcoord 1 0 1              # Using current texture upper left
    13   glshape vertex 1 -0.5 0.5 0         # Setting Upper left corner
    14 glshape end 1                         # Ends the definition of the quad polygon
    15 glshape disable 1 texture_2d          # Disabling the 2D Texture feature

The first line defines a glshape place holder with the Id 1 and the name 'This is a list of shape commands'.

The second line enables the OpenGL feature TEXTURE_2D using glEnable(GL_TEXTURE_2D);.

The third line defines texture 3 to be bound as the current texture. When the shape list is executed, this is the line that will initiate image data to e send from the CPU memory to the context or GPU. See above for the relation between the texture bind command and OpenGL commands.

In the 4th line a definition of a quad polygons is defined. In this case we only define one, but it could contain definition of more than one. A valid begin definition must later end with an end definition. The command is equivalent to the OpenGL command glBegin(GL_QUADS).

The 5th line is defining the normal of the quad defined. The normal is a vector usually perpendicular to plane of the quad. In this case, the normal definition is commented out. The command is equivalent to the OpenGL command glNormal(x,y,z).

The next four vertex commands defines the four 3D (x,y,z) coordinates of the quad. Note that the points are defined counter clockwise. The commands are equivalent to glVertex(x,y,..) (www.opengl.org).

Before each vertex command is a texcoord command defining how the texture is intended to be merged onto the quad. Each texcoord commands takes a set of 2d coordinates s and r, where s=0 means the left side of the image and r=0 means the bottom of the image. These values are fraction of the image and must be between 0 and 1. The OpenGL equivalent is glTexCoord(s,r,..) (www.opengl.org).

In the 14th line we have the corresponding end command to the previous begin command. The OpenGL equivalent is glEnd().

Finally the command list ends by disabling the feature TEXTURE_2D.

The above defined shape can be listed using the command gl list 1:

    gl list 1
    STAT:  shape 1 ops 13 name This is a list of shape commands
    STAT:  -  1 enable mode texture_2d
    STAT:  -  2 texture bind id 3 min linear mag nearest type 2d : Texture 3 Texture for Loaded Image #5, image 5
    STAT:  -  3 begin quads
    STAT:  -  4  texcoord2 0.0000,0.0000
    STAT:  -  5  vertex3 -0.5000,-0.5000,0.0000
    STAT:  -  6  texcoord2 1.0000,0.0000
    STAT:  -  7  vertex3 0.5000,-0.5000,0.0000
    STAT:  -  8  texcoord2 1.0000,1.0000
    STAT:  -  9  vertex3 0.5000,0.5000,0.0000
    STAT:  - 10  texcoord2 0.0000,1.0000
    STAT:  - 11  vertex3 -0.5000,0.5000,0.0000
    STAT:  - 12 end
    STAT:  - 13 disable mode texture_2d
    STAT:

The above example can be rewritten into two shapes adding flexibility:

    1  glshape add This a Quad shape with texcoords
    2  glshape begin 1 quads                 # Begin a 4 point polygon
    3    #glshape normal 1 0 0 1             # Setting the normal for the quad
    4    glshape texcoord 1 0 0              # Using current texture lower left
    5    glshape vertex 1 -0.5 -0.5 0        # Setting Lower left corner
    6    glshape texcoord 1 1 0              # Using current texture lower right
    7    glshape vertex 1 0.5 -0.5 0         # Setting Lower right corner
    8    glshape texcoord 1 1 1              # Using current texture upper right
    9    glshape vertex 1 0.5 0.5 0          # Setting Upper right corner
    10   glshape texcoord 1 0 1              # Using current texture upper left
    11   glshape vertex 1 -0.5 0.5 0         # Setting Upper left corner
    12 glshape end 1                         # Ends the definition of the quad polygon

    13 glshape add 2 This is a glshape including another shape
    14 glshape enable 2 texture_2d           # Enable 2D textures.
    15 glshape texture bind 2 3 linear nearest  # Bind texture 3 as current texture
    16 glshape inshape 2 1                   # include glshape 1
    17 glshape disable 2 texture_2d          # Disabling the 2D Texture feature

The benefit is that it is quite easy to change the glshape included in line 16 to another glshape with another outline or otherwise other settings.

GL Vectors

In Snowmix Vector constants can be created (and changed) using the commands glshape vector add and glshape vector value. The evctors can have 2, 3 and 4 dimensions. Internally the vectors are saved as either integers, floats or doubles depending on the settings for compiling Snowmix. The vectors are needed when defining light and materials. The syntax for the two vector commands are:

    glshape vector add [<vector id> [<vector name>]]
    glshape vector value [<vector id> <x> <y> [<z> <w>]]]

Both the add and the value commands when used without argument, will listed created vectors name and values respectively. If the add command is used with the vector id argument, but no name is given, the vector with that id will be deleted. The maximum number of vectors Snowmix can have, is currently fixed and set in the file src/opengl_video.h.

The GL vectors have no direct equivalent in OpenGL, but several OpenGL commands are taking an array of values as arguments, which is what we are doing in Snowmix too.

GL Lights and materials

Defining light constants

Light commands

Vertex Buffer Objects

The OpenGL commands glBegin and glEnd are deprecated in OpenGL 3.0 though still supported. In OpenGL 3.1 the commands has been removed. For now the commands will stay in Snowmix. Instead of glBegin/glEnd a Vertex Buffer Objects or VBO for short must be created and used. The benefit of VBO over the older glBegin/glEnd method is that the data points for Vertex only has to be sent to the GPU once and whenever they change if ever. An additional benefit of using a VBO over the glBegin/glEnd commands is that in a VBO the data points can be reused thus lowering the number of data points needed to be send from CPU to GPU. Reusing datapoints is possible when a VBO defines more than one polygon and at least two polygons share one or more datapoints.

Previous a Quad Vertex with texture was defined using the glBegin/glEnd equivalent. Here the same quad is being defined as a VBO:

    glshape vbo add 5 Quad with texture
    glshape vbo config 5 static quads t2 v3
    glshape vbo data 5 0.0 0.0 -0.5 -0.5 0.0  1.0 0.0 0.5 -0.5 0.0  1.0 1.0 0.5 0.5 0.0  0.0 1.0  -0.5 0.5 0.0
    glshape vbo indices 5 0 1 2 3

The first line adds/creates a VBO place holder with the vbo_id 5 and the name "Quad with texture".

The second line sets up the VBO 5 as a static VBO containing quad polygon or polygns with the elements t2 and v3 meaning a 2-dimensional texture ( glTexCoord) on a 3-dimensional vertex (glVertex3). This means that each dataset in the VBO consist of 5 data points. The first two datapoints are the texture datapoints and the next 3 are the vertex datapoints.

The third line defines for VBO 5 four datapoint sets.

The fifth line defines defines how the VBO should use the datapoint sets by defining an indexing. What is says is that when drawing/executing the VBO, the GPU should use the datasets in the order 0, 1, 2, 3. In this case, there is no benefit using indexing as there is only defined one quad. However if we add a second plane into the z-plane that share the right side of the first quad, we can add extra index points:

glshape vbo data 5 0.0 0.0 0.5 -0.5 1.0  0.0 1.0 0.5 0.5 1.0
glshape vbo indices 5 1 4 5 2

Now when the VBO is used, two four-sided polygons sharing two points are drawn. First the datasets 0,1,2,3 are used for the first quad and then the datasets 1,4,5,2 are used for the second polygon. In this case we saved 2 data sets at the expense of extra index information. The benefit will increase with the number of adjacent polygons sharing datapoints.

In a glshape a defined VBO can be referenced as shown in the following reusing the example in GL texture section (see the GL Texture section):

    13 glshape add 2 This is a glshape including another shape
    14 glshape enable 2 texture_2d           # Enable 2D textures.
    15 glshape texture bind 2 3 linear nearest  # Bind texture 3 as current texture
    16 glshape vbo 2 5                       # include VBO 5
    17 glshape disable 2 texture_2d          # Disabling the 2D Texture feature

The first time glshape 2 is overlayed, it will reference VBO 5. If this is the first time the VBO is referenced, the VBO's datapoints will be sent to the GPU. In future references when overlaying a glshape containing VBO 5, fewer OpenGL calls are needed and less data has to be sent from CPU to GPU.

Vertex Array Objects

VBOs in Snowmix are automatically wrapped in a Vertex Array Object or VAO so when overlaying a glshape Snowmix containing a VBO, all Snowmix has to do is binding the VAO containing the VBO. However the first time the VBO is referenced in a placed glshape, the VBO is first defined for the VAO sending the VBO data is sent to the GPU. It all happens automatically and is done with only a couple of OpenGL calls minimizing CPU load significantly.

Adding a functionality to combine several VBOs in one VAO is under consideration.

Placed GL Shapes

Copying back from Graphic Adapter.

OpenGL Ressources

MongoDB Logo MongoDB