can't get bones/skinning to work

[Kenny Riddile]

I'm trying to use assimp to convert collada models to a custom format and I'm having trouble getting the skeleton data out in the form I want.  The custom format is loaded into a hierarchical skeleton structure where each node has a bone matrix and a list of children.  This structure is then traversed to create an array of bone matrices that are uploaded to a simple skinning shader.  I am confident that the custom skeletal structure and skinning shader are correct.  I can't seem to figure out how to get the skeletal data out of assimp in the form that I want, because the model ends up looking like junk.  I would really appreciate a small code example for what I'm trying to do cause this is really annoying me :)

[Anonymous]

I have the exact same problem.

I managed to successfully load a textured model and created my own custom format by the use of Assimp. This was no issue at all because there's good example code available from the Assimp site.

But when trying to implement bones with skinning, I just see junk similar to kfriddile.

Would it be possible to offer a simple example? Maybe directly in the Assimp documentation or as a seperate example project like the SimpleOpenGL example?

I'd really appreciate that.

Thanks,
Thomas

[Kim Kulling]

Hi,
hm, hopefully we will find some time for that task. Or has anyone else a small example for us? We would appreciate this very much as well :-). The main problem building such an example is the time but we will do our very best.

Kimmi

[Icebone1000]

Hi, Id like to know if this is an issue, a "lack of handling bones" on the sdk, or just a difficult task..
I have been on hell, I feel like cuting my own balls with my teeths!  Such a common task but still you cant find any thing concrete about it on internet!!

I have just asked few weeks(last months?) what archives are good to work with bonned animation(skeletal/skinned, whatever), two names commed out: FBX and COLLADA…I went to FBX cause I just hate xml(who dont?)…I have been crying since that day, anyone with psycopathics apptitudes should avoid that sdk, or for sure their pets will suffer..After 4 weeks of pure pain, feeling like you are the only moron in the world who is using the damn FBX sdk, I gived up! I jusp putted 4 weeks of my live away!! GODDAMMIT!!!
Now Im asking for you..Its possible to import mesh(vertex, normals, uv, textures) and animation data( bones hierarchy, vertices IDs, vertices weights, tranformations matrix(per keyframe), keyframe timing info)  using assimp?
Cause since it looks you could be able to, my skinned mesh exported from maya2010 as dae(COLLADA) doesnt animate on the assimp viewer, witch is alredy a VERY warning thing in my opinion..
YES IM DESPERATE, AND YES, MY ENGLISH IS TERRIBLE

Thanks for your grasp, if any.

[Thomas Ziegenhagen]

Well… bone animations are no trivial topic! I'll try my best to explain the basics here, but I already did several times here and at other places and I therefore know that the devil hides in the details. But I CAN'T help you with the details! Because too much of it all depends on YOUR coordinate systems, YOUR matrix classes, YOUR quaternion setup. That's why the following instructions are vague by design. Please try to figure out your math background upfront or the following description won't help you.

What you need to know:
- Your coordinate system: lefthanded or righthanded, which roughly translated to DirectX or OpenGL
- In which order of sequence you need to multiply matrices.
- Have working vector, matrix and quaternion classes at hand - if not, use Assimp's classes.

What AssimpView does - and I suggest you do the same:

Node animations consist of two independent steps. First is the change of node transformations over time. The second step is the deformation of a skinned mesh to adhere to the current skeleton pose.

You see that these two steps are independent? The only connection is that a skeleton of a skinned mesh is a hierarchy of nodes. Therefore a node animation *can be* a bone animation, but it doesn't have to. It is only a bone animation if the nodes in question are part of a skeleton node hierarchy. Because cameras or lights are also attached to nodes, you could also script a camera movement by a node animation. But for simplicity's sake I'll stick to skeleton animations here.

So do the first step first. An assimp scene can contain zero to many animations, each of them describing a certain movement of the scene's nodes over time. Select a single animation! Mixing multiple animations can also be done, but we first try to get the basics working!

A single animation contains of one to many animation channels. Each channel desribes the movement of a single node over time. So the first thing you do when starting an animation is to look up which node is affected by the animation channel. But how do you learn which node the channel wants to move? You search through the scene node hierarchy by name. If you find a node whose name is the very same as the channel's node name, you have found the target node for that channel.

You can now resolve the channel for a certain point in time - which means to find the corresponding values the channel specifies for a given point in time. Each channel contains three time lines: one for the node's position, one for the node's rotation, and one for its scaling. The rotation is the most important part, then comes position, scaling is rarely ever used. You go through each of these three sequences, search the key-value-pair for the current point in time, and construct a new transformation matrix from those three. This new transformation matrix *replaces* the node's local transformation matrix.

Per order of sequence:
- calculate the current point in time - it has to be a value between zero and aiAnimation::mDuration
- for each animation channel do:
-- find the node which the channel affects, by searching recursivly through all scene nodes and comparing node names to aiNodeAnim::mNodeName
-- for each of the three rotation, position and scaling, find the key which is closest in time to the current time point
-- use the values from the three "current" keys to construct a transformation matrix
-- copy this transformation matrix over to the node.

or in pseudo code:

const aiAnimation* anim = theAnimYouWantToUse;
double currentTime = fmod( GetTimeInSeconds() * timeScale, anim->mDuration);
for( size_t a = 0; a < anim->mNumChannels; ++a)
{
  const aiNodeAnim* channel = anim->mChannels[a];
  aiVector3D curPosition;
  aiQuaternion curRotation;
  // scaling purposefully left out 
  // find the node which the channel affects
  aiNode* targetNode = FindNodeRecursivelyByName( scene->mRootNode, channel->mNodeName);
  // find current position
  size_t posIndex = 0;
  while( 1 )
  {
    // break if this is the last key - there are no more keys after this one, we need to use it
    if( posIndex +1 >= channel->mNumPositionKeys )
      break;
    // break if the next key lies in the future - the current one is the correct one then
    if( channel->mPositionKeys[posIndex + 1].mTime > currentTime )
      break;
  } 
  // maybe add a check here if the anim has any position keys at all
  curPosition = channel->mPositionKeys[posIndex].mValue;
  // same goes for rotation, but I shorten it now
  size_t rotIndex = 0;
  while( 1 )
  {
    if( rotIndex +1 >= channel->mNumRotationKeys )
      break;
    if( channel->mRotationKeys[rotIndex + 1].mTime > currentTime )
      break;
  } 
  curRotation = channel->mRotationKeys[posIndex].mValue;
  // now build a transformation matrix from it. First rotation, thenn push position in it as well. 
  aiMatrix4x4 trafo = curRotation.GetMatrix();
  trafo.a4 = curPosition.x; trafo.b4 = curPosition.y; trafo.c4 = curPosition.z;
  // assign this transformation to the node
  targetNode->mTransformation = trafo;
}

If you have this step working, you should see your nodes move. I suggest adding a debug renderer which draws the nodes using some coloured lines or similar. DO NOT CONTINUE UNTIL THIS STEP IS WORKING.

And of course there's a lot of room for optimisations. For example, you don't have to search for the affected node for each channel in each frame! Do it once and store the pointer somewhere. You also might want to add interpolation between current and next key - anims look much smoother with interpolations, and quaternions are especially well suited for interpolation. I left out all of these to keep the example short.

[Thomas Ziegenhagen]

I had some copy&paste errors, but you can figure out that one for yourself :-) And sourceforge doesn't allow editing own posts.

[Thomas Ziegenhagen]

Second step: deform a skinned mesh so that it conforms to the current pose of a skeleton. This has to be done for each mesh you want to see animated, but that's most probably just one. And you have the choice to do it on a CPU or a GPU.

If you draw a skinned mesh without any deformations, just by rendering all the vertices, you see the mesh in so-called bind pose. This is the pose in which the artist modelled the mesh and in which he connected all the vertices to one or more bones. Do this "no deformations" rendering just once to check if your rendering works and if the imported mesh looks correctly.

To do the actual skinning you need to calculate a local transformation matrix for each bone, the so-called bone matrix. The bone matrix describes how a vertex moves/changes if the bone is in the current position instead of the bind position. So in bind pose all bone matrices are identity! It's only when you rotate or move a bone away from its bind position, its corresponding bone matrix becomes non-identity. And if you transform the vertices which are affected by this bone using its bone matrix, you see that the vertices move away from their bind positions. The mesh deforms in conformance to the skeleton. And that's how vertex skinning works.

So at each frame you need to calculate the bone matrix for each bone. You do this by using the bone's so-called offset matrix or inverse bind matrix. The offset matrix transforms from mesh coordinates to the bone's local coordinate system, it's a mesh-to-bone matrix. You then concatenate the node's transformation matrix to it, and that of its parent, and so on until you're back at the mesh coordinate system. The node transformation chain from node to parent to parent… describes the bone-to-mesh matrix - that's the exact counterpart to the offset matrix. So if your skeleton nodes are in bind pose, both matrices cancel out and you get an identity matrix for the bone matrix. Please note: that's exactly what we described above! But as soon as the node or its parents move away from the bind position, the bone matrix contains the "difference" between the bind pose and the current skeleton pose. And that's what we need to calculate our vertices.

In order of sequence:
- for each bone
-- find the corresponding node by name
-- calculate the node's transformation from local node to mesh coordinates
-- multiply bone's offset matrix with this node-to-mesh matrix to get the bone matrix

or in code

const aiMesh* mesh = theMeshYouWantToRender();
// calculate bone matrices
std::vector<aiMatrix4x4> boneMatrices( mesh->mNumBones);
for( size_t a = 0; a < mesh->mNumBones; ++a)
{
  const aiBone* bone = mesh->mBones[a];
  
  // find the corresponding node by again looking recursively through the node hierarchy for the same name
  aiNode* node = FindNodeRecursivelyByName( scene->mRootNode, bone->mName);
  
  // start with the mesh-to-bone matrix 
  boneMatrices[a] = bone->mOffsetMatrix;
  // and now append all node transformations down the parent chain until we're back at mesh coordinates again
  const aiNode* tempNode = node;
  while( tempNode)
  {
    boneMatrices[a] *= tempNode->mTransformation;   // check your matrix multiplication order here!!!
    tempNode = tempNode->mParent;
  }
}

You now have an array of bone matrices. The following steps differ between hardware and software skinning, whether you're doing it on CPU or GPU. I suggest using the GPU for this whenever you can, but GPU skinning has limits on the maximum number of bones in a mesh, or on the number of bones affecting any single vertex, so you'll probably have to implement software skinning at some time anyways. To keep this posts managable I'll restrict myself to software skinning.

What we need to do now is to iterate over all vertex influences in all bones and accumulate the vertex' deformed data. A vertex can be affected by more than a single bone - you simply sum up all transformed vertices weighted by the weight of the bone vertex weight. That looks like:

// all using the results from the previous code snippet
std::vector<aiVector3D> resultPos( mesh->mNumVertices); 
std::vector<aiVector3D> resultNorm( mesh->mNumVertices);
// loop through all vertex weights of all bones
for( size_t a = 0; a < mesh->mNumBones; ++a)
{
  const aiBone* bone = mesh->mBones[a];
  const aiMatrix4x4& posTrafo = boneMatrices[a];
  aiMatrix3x3 normTrafo = aiMatrix3x3( posTrafo); // 3x3 matrix, contains the bone matrix without the translation, only with rotation and possibly scaling
  for( size_t b = 0; b < bone->mNumWeights; ++b)
  {
    const aiVertexWeight& weight = bone->mWeights[b];
    size_t vertexId = weight.mVertexId; 
    const aiVector3D& srcPos = mesh->mVertices[vertexId];
    const aiVector3D& srcNorm = mesh->mNormals[vertexId];
    resultPos[vertexId] += weight.mWeight * (posTrafo * srcPos);
    resultNorm[vertexId] += weight.mWeight * (normTrafo * srcNorm);
  }
}
// now upload the result position and normal along with the other vertex attributes into a dynamic vertex buffer, VBO or whatever

I hope this makes things clearer now. I know that this is not the exact solution. The "normal" variant of the bone matrix is usually calculated by a more complicated method. And you usually do GPU skinning anyways, using a different approach. But well… this should be enough to get you started, and I certainly hope it's a good base to sort out all the matrix and coordinate system hassles you'll undoubtly encounter on your way. Just to repeat this important message: THIS IS NO CODE TO COPY AND WORK OUT OF THE BOX! You'll have to read the text, read the code, adapt it to your coordinate systems and math classes, and then it might work!

good luck.

[Kenny Riddile]

I did finally get it working as you can see :)

http://www.youtube.com/watch?v=-mB2HF_bCOg

[Anonymous]

ulfjorensen: Thanks very very much for this great description! That's exactly what I was looking for!! I'll definitly try it out as soon as I have time.

kfriddile: Looking really great! Where did you get this model and animation? Or did you do it by yourself?

[Kenny Riddile]

kfriddile: Looking really great!

Thanks!

Where did you get this model and animation? Or did you do it by yourself?

That's the hell knight model from Doom3.  All of the game's models and animations are in md5 format, so I just wrote a converter that uses assimp to convert them to my own proprietary format.  It converts collada data just as well, but those are the only two formats I've tried so far.

[Kenny Riddile]

Heh…quote fail :)

[Alexander Gessler]

I'm glad to see the md5 importer  actually used ;-)

[Icebone1000]

Very impressive, Im using a cilinder with 3 bones on my app to start lol
..I hope I will be the next with a working animation.

[Icebone1000]

Can someone help me with one thing…
I had understand that an aiAnimation hold an array of aiNodeAnims, each aiNodeAnim being responsable for one bone node..
But exporting a collada file from maya2010, I get an aiAnimation for each keyframe…each with only one aiNodeAnim..is that some export setting that is screwing my animation?

[Thomas Ziegenhagen]

This issue should already be fixed. Get the latest version from the SVN repository and try again. If that still happens with the most recent version, please send the Collada file in question to me (thomas (at) dreamworlds (dot) de) so I can fix it.

[Thomas Ziegenhagen]

I hate the fact that I can't edit my posts. I'd like to add one thing: this is definitely a trait of the exporter plugin you use. If you find some export options, try them. It might solve the problem. But I already encountered several files of the type you described, so I also added a workaround to the Collada loader which handles these corner cases. This workaround had been added after the last binary release, therefore you need to fetch the latest version from the Assimp SVN repository to get it.

[Icebone1000]

The fbx sdk is pursuing me(the default maya collada exporter)..
I downloaded the openCollada plugin( from opencollada.org) and it works just fine(at least until what I have donne it seems ok).
I dont know what/where exactly I have to download at the SVN repository, to me is just a lot of crazy stuff..But I will keep the release version until get stucked..The version I have is the 1.1.
Im sending  the file to you anyway..in case you curious. Thanks for the information.

[Icebone1000]

-fake edit-
Id like to add that  none of the options I tried on the fbx plugin changed the issue, the best I did with it was keep the number of keyframes(by default it was adding keyframes per frame..) and then I downloaded the openCollada one..

[Icebone1000]

Hi again.
I got into the part where I have to consider what happens if the current instant in time falls after the last bone`s keyframe time, but inside annimation total duration.
I did a test collada file to test it on maya, I concluded the aiAnimBehaviour should be the constant one(get the near one, in that case would mean: stay with the last). But I get the default  aiAnimBehaviour, is that just my misinterpretation ?
Looking on maya, after the last keyframe of the bone, the last transformation is keeped, not reseted( witch is what I understand by default behaviour : reset to local transformation ).
I just need a clarification.

[Thomas Ziegenhagen]

To my knowledge all animations have a duration which is equal the time of the last anim key. At least the Collada anim loader is coded that way… I'd really worry if you find Collada animations which have a larger duration than the last key time :-)

In case you really encounter such an animation, you have the choice to either keep the last frame or to wrap around and interpolate between the last frame and the first frame of the animation. Which of the two cases you want depends on the context of the animation - it's an application decision, not an asset definition. The aiAnimBehaviour can be safely ignored in my opinion, and I don't know if any loader actually writes it.

[Icebone1000]

To my knowledge all animations have a duration which is equal the time of the last anim key

The last key for sure, but bones dont have the same number of keyframes, for example: you can have a shoulder joint and a elbow and hand as hierarchy, you can have an animation where only the shoulder get some rotation keyframes, but elbow and hand got none or some few(less)..(at least, if you do that way on maya, and export it to collada, it will keep your original keyframes)
For now I will just keep the last(or save a collada with equal number of keys per bone) ;D

[Icebone1000]

I think Im close!! The sequence is very close, but is still messy, but Im alredy  happy..lol
I`ll describe my recursive animate method here, so if you please, can see any mistake Im doing, I feel is something little(I hope):
-I get the frame duration and pass to my anim function
-my skeletal structure(that holds the hierarchy and its keyframes, along with all transformations) accumulates those seconds, and "modulates" it to the total anim duration(to get current time on the timeline, in a cyclical way).
-so it enter recursion, passing as param the root bone, an parent absolute transformation( in the first call, it is a identity matrix), and the current time.

So in each recursion, the current bone absolute matrix is passed as param as the new parent absolute matrix..
heres how I calculate the absolute matrix:

//interpolate between current and previous keyframe:--

//generate 't' value:(current time - prev time)/keyframe duration
FLOAT t = ((FLOAT)secAccPerTake_p
- pCurrentBone_p->pTransformation.fABSTime_sec )
/ pCurrentBone_p->pTransformation.fDuration_sec;

//slerp quaternions:
XMVECTOR qSlerpedQuaternion = XMQuaternionSlerp(
pCurrentBone_p->pTransformation.qQuatLocalTransformation,
pCurrentBone_p->pTransformation.qQuatLocalTransformation,
t );
//-----------------------------------
//create abs transformation:
//concatenate offset * quaternion * parent abs matrix:
mParentNowABS_p =
pCurrentBone_p->mOffsetTransformation
* ( XMMatrixRotationQuaternion( qSlerpedQuaternion )
* mParentNowABS_p );
//update bone final matrix for skinning:
pCurrentBone_p->mFinal = mParentNowABS_p;

for( UINT i = 0; i < pCurrentBone_p->iNChilds; i++ ){
AnimateBones_nodedown( secAccPerTake_p, pCurrentBone_p->ppChilds_, mParentNowABS_p );
}
_

Any logical or math mistake?

[Thomas Ziegenhagen]

This looks ok, as far as I can tell. Check the values of t with a debugger. Also check the order of sequence of your matrix multiplications. Apart from that I think it looks ok.

[Icebone1000]

wooooooooooooooooooooooooooooooot /o/
wooooooooooooooooooooooooooooooot \o\
I did it!!!!! yuhuuuuuuuuuuuu!!!!
xDDDDD
Some notes:
-The offset matrix just account to the final per bone matrix, it is NOT to be concatenate with the parent absolute passed to its children!
-Using JUST rotations doesnt work, when u use just rotations from the quaternions you lose the position information, so your rotation will be relative just to the offset matrix..this is not right,  the offset matrix main job is make the bind position not influence vertices..Your bind position DO ALWAYS have a translation part, so just remember to take the translation part of the local matrix(the default, in the case u using just rotations) and concatenate it with the quaternion:

mParentNowABS_p = 
                    (XMMatrixRotationQuaternion( qSlerpedQuaternion )
                    * XMMatrixTranslationFromVector( pCurrentBone_p->mLocalTransformation.r[3] ))
                    * mParentNowABS_p;
pCurrentBone_p->mFinal = pCurrentBone_p->mOffsetTransformation * mParentNowABS_p;

=D!!
I hope it will work with other files too xD, since im just testing with the one I did since the begining.

[Icebone1000]

Thanks everyone for the great help!!