Unreal Engine C++: Skeletal Mesh doc sheet
Skeletal Mesh editor official Doc
Disclaimer
The following may not be up to date, for now it should roughly apply to UE > 5.1.
This document could contain mistakes, always test out things in small sample projects first.
This sheet mainly caters towards programmers, it's essentially a shared memo, that I wrote from an Engineer's viewpoint on the topic.
Intro
There are mainly 2 options to represent a mesh asset in Unreal static mesh and skeletal mesh. While only static mesh (as of today) can use Nanite, skeletal meshes are the only way to represent skeletal deformation and to smoothly deform a triangular mesh. Note: you can associate static meshes to joints as a trick to use Nanite, this would only work for robot like rig where each part moves rigidly without deformation (this is exactly what Epic did in the first sample of UE 5.0 with the "ancient robot").
Before reading this sheet, I recommend to:
- Get familiar with skeletal meshes in Unreal's UI first.
(For instance importing some FBX file containing joint animations into Unreal's editor, opening Unreal's asset into the skeletal editor etc.) - Create a C++ project with an AActor equiped with a USkeletalMesh component to test out things with draw debug helper, ex:
Code snippet to debug skeletal mesh transforms
(if "component", "AActor" is gibberish to you; you should get familiar with those concepts first elsewhere)
Skeletal Mesh overview
Roughly speaking we have:
SkeletalMesh
// RefToLocal = WorldJointMat . WorldBindPoseMat^-1 = SkinningMatToApplyToVertices
ReferenceToLocalMatrices[nb_joints_for_the_entire_skeleton]
VertexBuffer
LOD[0]
LOD[1]
LOD[2]
MeshChunk[0] // MeshSection 0
ChunkMatrices[nb_joints_influencing_the_section]
MeshChunk[1] // MeshSection 1
ChunkMatrices[nb_joints_influencing_the_section]
In other words, a mesh has several LOD levels (LOD0, LOD1, LOD2 etc.). Each LOD is composed of “sections” (also called “chunks”), which represents a sub-part of the mesh assigned to a specific material. In the skeletal editor you can find the LOD’s sections and choose to highlight or only display a specific section:
Each mesh section/chunk is rendered by a vertex factory that stores the chunkMatrices, the offset in the vertex buffer where the chunk is located and so on.
Although there is only one skeleton associated to a single array of skinning matrices (i.e. ReferenceToLocalMatrices[]),
Each mesh section is rendered in a separate draw call, where only the sub-sets of bones deforming this particular mesh section is considered and uploaded.
Which means each section stores a separate buffer of chunkMatrices[] built from the main ReferenceToLocalMatrices[] buffer.
Typically each section/chunk setup is done by its corresponding "vertex factory".
Data Structure
Note 1: It is important to keep in mind Unreal runs multiple threads, 2 of which are the "GameThread" and the "RenderThread". To avoid race condition data used in the game thread is duplicated at each frame. So we have 2 data structure that keep in synch together.
Note 2: some of the data or methods are only available when running in the editor, the code wrapped into WITH_EDITOR or WITH_EDITORONLY_DATA macros is not available in the game runtime (after packaging/exporting the game).
USkeletalMesh
FSkeletalMeshModel // Imported data (EDITOR ONLY)
FSkeletalMeshRenderData // data prepared for rendering
Here is the more detailed view, both structure roughly follow the pattern ModelPointer->LODs[LODIndex].Sections[i]:
// UE 4.26 – 5.1
USkeletalMesh // (SkeletalMesh.h)
// ------------------
// GetImportedModel(): // #if WITH_EDITORONLY_DATA
// ------------------
FSkeletalMeshModel* ImportedModel; // (Rendering/SkeletalMeshModel.h) #if WITH_EDITORONLY_DATA
FSkeletalMeshLODModel LODModels[] // (Rendering/SkeletalMeshLODModel.h) #if WITH_EDITOR
FImportedSkinWeightProfileData SkinWeightProfiles[]; // #if WITH_EDITORONLY_DATA
// (Rendering/SkeletalMeshLODModel.h) #if WITH_EDITOR
FSkelMeshSourceSectionUserData UserSectionsData[]
stuff
...
// (Rendering/SkeletalMeshLODModel.h) #if WITH_EDITOR
FSkelMeshSection Sections[]
//Index in the TMap: FSkeletalMeshLODModel::UserSectionsData
int32 OriginalDataSectionIndex
// Skin weights, vertex position etc.
// #if WITH_EDITOR
FSoftSkinVertex SoftVertices[];
// ------------------------
//GetResourceForRendering():
// ------------------------
FskeletalMeshRenderData* SkeletalMeshRenderData; // (SkeletalMeshRenderData.h)
FSkeletalMeshLODRenderData LODRenderData[]; // (SkeletalMeshLODRenderData.h)
FSkinWeightVertexBuffer SkinWeightVertexBuffer;
// Skin weight profile data structures,
// can contain multiple profiles and their runtime FSkinWeightVertexBuffer
FSkinWeightProfilesData SkinWeightProfilesData;
FSkelMeshRenderSection RenderSections[]; // (SkeletalMeshLODRenderData.h)
Components
USkeletalMesh is also available via various components:
USkinnedMeshComponent : public UMeshComponent
USkinnedAsset* GetSkinnedAsset() const;
FSkeletalMeshObject* MeshObject; // #include "SkeletalRenderPublic.h"
USkeletalMeshComponent : public USkinnedMeshComponent
USkeletalMesh* SkeletalMeshAsset;
USkeletalMesh* GetSkeletalMeshAsset() const;
Look up vertices on CPU (FskeletalMeshRenderData)
The code below shows how to extract vertex position for each mesh section.
This can be executed on the game thread. The main ways to access FskeletalMeshRenderData are:
1) USkeletalMesh->GetResourceForRendering(); 2) USkeletalMeshComponent->MeshObject; 3) USkinnedMeshComponent->MeshObject;
// 1) From skinned mesh TObjectPtr<USkinnedMeshComponent> skinnedMesh = nullptr; const FSkeletalMeshObject* skelMeshObject = skinnedMesh->MeshObject; const FSkeletalMeshRenderData& renderData = skelMeshObject->GetSkeletalMeshRenderData(); // 2) From a skeletal mesh TObjectPtr<USkeletalMeshComponent> skeletalMesh = nullptr; const FSkeletalMeshObject* skelMeshObject = skeletalMesh->MeshObject; const FSkeletalMeshRenderData& renderData = skelMeshObject->GetSkeletalMeshRenderData(); // 3) From USkeletalMesh USkeletalMesh skeletalMesh = nullptr; const FSkeletalMeshRenderData& renderData = skeletalMesh->GetResourceForRendering();
Which allows you to look up vertices:
const& FSkeletalMeshRenderData renderData = ...;
// Get Active LOD
const int32 lodIndex = skelMeshObject->GetLOD();
// Or look up every LODs:
for (int32 lodIndex = 0; lodIndex < renderData.LODRenderData.Num(); ++lodIndex)
{
const FSkeletalMeshLODRenderData* lodRenderData = &renderData.LODRenderData[lodIndex];
uint32 numSections = lodRenderData->RenderSections.Num();
for (uint32 sectionsIndex = 0; sectionsIndex < numSections; ++sectionsIndex)
{
const FSkelMeshRenderSection& section = lodRenderData->RenderSections[sectionsIndex];
const FPositionVertexBuffer& vertexBuffer =
lodRenderData->StaticVertexBuffers.PositionVertexBuffer;
TArray<FVector> positions;
positions.SetNum(section.NumVertices);
for (uint32 vix = 0; vix < section.NumVertices; ++vix)
{
const int32 vertexBufferIndex = section.GetVertexBufferIndex() + vix;
const FVector& vertexPosition =
(FVector)vertexBuffer.VertexPosition(vertexBufferIndex);
// local position / object coordinates
positions[vix] = vertexPosition;
// World position:
FVector worldPosition =
skeletalMesh->GetActorTransform().TransformPosition(vertexPositionLocal);
}
}
}
This would be the un-animated position (a.k.a position in rest-pose / T-pose / reference pose) as skinning animation is usually computed on the GPU
// Fill out your copyright notice in the Description page of Project Settings.
#pragma once
#include "CoreMinimal.h"
#include "GameFramework/Actor.h"
#include "MyActor.generated.h"
UCLASS()
class BONEMATRICESTEST_API AMyActor : public AActor
{
GENERATED_BODY()
public:
// Sets default values for this actor's properties
AMyActor();
UPROPERTY(EditAnywhere);
USkeletalMeshComponent* MySkel;
protected:
// Called when the game starts or when spawned
virtual void BeginPlay() override;
public:
// Called every frame
virtual void Tick(float DeltaTime) override;
};
// Fill out your copyright notice in the Description page of Project Settings.
#include "MyActor.h"
#include "DrawDebugHelpers.h"
#include "SkeletalRenderPublic.h"
// Sets default values
AMyActor::AMyActor()
{
// Set this actor to call Tick() every frame. You can turn this off to improve performance if you don't need it.
PrimaryActorTick.bCanEverTick = true;
}
// Called when the game starts or when spawned
void AMyActor::BeginPlay()
{
Super::BeginPlay();
if (MySkel == nullptr)
return;
FTransform actorWorldTransform = this->GetActorTransform();
const FSkeletalMeshObject* skelMeshObject = MySkel->MeshObject;
const FSkeletalMeshRenderData& renderData = skelMeshObject->GetSkeletalMeshRenderData();
// Get Active LOD
const int32 lodIndex = skelMeshObject->GetLOD();
// Or look up every LODs:
//for (int32 lodIndex = 0; lodIndex < renderData.LODRenderData.Num(); ++lodIndex)
{
const FSkeletalMeshLODRenderData* lodRenderData = &renderData.LODRenderData[lodIndex];
uint32 numSections = lodRenderData->RenderSections.Num();
for (uint32 sectionsIndex = 0; sectionsIndex < numSections; ++sectionsIndex)
{
const FSkelMeshRenderSection& section = lodRenderData->RenderSections[sectionsIndex];
const FPositionVertexBuffer& vertexBuffer =
lodRenderData->StaticVertexBuffers.PositionVertexBuffer;
//TArray<FVector> positions;
//positions.SetNum(section.NumVertices);
for (uint32 vix = 0; vix < section.NumVertices; ++vix)
{
const int32 vertexBufferIndex = section.GetVertexBufferIndex() + vix;
const FVector& vertexPositionLocal = (FVector)vertexBuffer.VertexPosition(vertexBufferIndex);
const FVector& worldPosition = actorWorldTransform.TransformPosition(vertexPositionLocal);
//positions[vix] = vertexPosition;
DrawDebugPoint(GetWorld(), worldPosition, 30, FColor(52, 220, 239), true);
}
}
}
}
// Called every frame
void AMyActor::Tick(float DeltaTime)
{
Super::Tick(DeltaTime);
}
Look up LOD & Sections (FSkeletalMeshModel)
This would be only available in the Editor and not the game runtime after packaging:
#include "Engine/SkeletalMesh.h"
#include "Rendering/SkeletalMeshModel.h"
TObjectPtr<USkeletalMeshComponent> SkeletalMesh;
USkeletalMesh* SkeletalMeshPtr = SkeletalMesh->GetSkeletalMeshAsset();
FSkeletalMeshModel* SourceModel = SkeletalMeshPtr->GetImportedModel();
for (int32 LODIndex = 0; LODIndex < SourceModel->LODModels.Num(); ++LODIndex) {
for (int32 i = 0; i < SourceModel->LODModels[LODIndex].Sections.Num(); ++i) {
SourceModel->LODModels[LODIndex].Sections[i].MaterialIndex = NewIndex;
++NewIndex;
}
}
FBX to UE data mapping
Consider a FBX file generated from Maya with several mesh objects, each attached to a skincluster:
FBX file
MeshObject[0]
...
MeshObject[n]
Skincluster
Each object and associated skin cluster will be translated in UE as a mesh section. As mentioned earlier each mesh chunk/section will be processed and skinned separately in a dedicated "vertex factory".
Keywords to look up to find things related to skin cluster import:
FbxCluster* Cluster = Skin->GetCluster(ClusterIndex); lClusterCount =( (FbxSkin *)FbxMesh->GetDeformer(i, FbxDeformer::eSkin))->GetClusterCount();
Duplicated vertices
When importing a mesh, for instance via .fbx, vertices may get duplicated (a.k.a split).
For instance, to allow the definition of several normals at the same vertex, a typical way to light sharp edges. It can also occur when we need to associate multiple uv coordinates for the same vertex,
this happens whenever a vertex lies on a texture seam.
In other words, for each mesh object (a.k.a mesh section in UE), the vertex count displayed in Maya (before exporting) is likely to be different to the number of vertices of each mesh section you will find in UE's USkeletalMesh.
This means at least two things:
- Importing vertex data likely requires you to mirror how UE duplicates vertex data.
- In vertex-based simulation it's likely you need to only treat original vertices
-
If you import new vertex data into UE, say exporting a vertex weight map (e.g. a float that represent the mass of each vertex) from a DCC tool like Maya and importing it into UE,
your custom C++ UE importer will likely need to duplicate vertex data exactly like UE does in his (ex FBX) import code.
-
For instance if you implement a custom cloth simulation, you would want to base the simulation on the original topology
to avoid breaking the simulation and optimize computation. Only after doing the accurate and heavy computation would you copy
the final vertex position to the duplicated vertices for rendering.
Fortunatly, UE keeps the mapping between imported vertices and render vertices (i.e. duplicated vertices).
Terminology
It's quite a confusing topic, so we will define our own terminology (different from official UE notations). Let's consider 2 vertex buffers, we name the associated vertex index as follows:
TArray<FVector> VertexListWhenImported <- [importVertIdx]; // Same as DCC tool, maya etc. TArray<FVector> VertexListAfterDuplication <- [renderVertIdx]; // Duplicated and ready for rendering.
In addition, we can convert both index type to either:
"Global index" <-> vertex index inside the LOD buffer of concatenated mesh sections "Local index" <-> vertex index inside a specific mesh section
OverlappingVertices & DuplicatedVerticesBuffer buffers
Warning: In this paragraph everything is expressed in local renderVertIdx (as opposed to importedVertIdx).
TMap<int32, TArray<int32>> FSkelMeshSection::OverlappingVertices; // Same as above but with a "flat" memory layout FDuplicatedVerticesBuffer FSkelMeshRenderSection::DuplicatedVerticesBuffer
Here is where both buffers are accessible relative to the skeletal mesh:
// UE 4.26 – 5.1
USkeletalMesh
// ------------------
// GetImportedModel()
// (Editor only)
FSkeletalMeshModel* ImportedModel;
FSkeletalMeshLODModel LODModels[]
FSkelMeshSection Sections[]
TMap<int32, TArray<int32>> OverlappingVertices;
// -------------------------
// GetResourceForRendering()
// Both editor & game runtime
FskeletalMeshRenderData* SkeletalMeshRenderData;
FSkeletalMeshLODRenderData LODRenderData[];
FSkelMeshRenderSection RenderSections[];
FDuplicatedVerticesBuffer DuplicatedVerticesBuffer; // GPU friendly memory layout
They are both the same with different memory layout. The map OverlappingVertices associates a vertex index (in local renderVertIdx)
to a sub-list of vertex indices (in local renderVertIdx) with same coordinates.
OverlappingVertices
In other words:
TMap<int32, TArray<int32>> FSkelMeshSection::OverlappingVertices;
Maps any vertex index to all vertices that share the same position (all indices are local renderVertIdx).
OverlappingVertices[localRenderVertIdx_i]
=
sub list of localRenderVertIdx_j that share the same coordinates as localRenderVertIdx_i;
FDuplicatedVerticesBuffer
The memory flat version of OverlappingVertices is FDuplicatedVerticesBuffer FSkelMeshRenderSection::DuplicatedVerticesBuffer;
FIndexLengthPair
int Index;
int Length;
FDuplicatedVerticesBuffer
// Look up table:
// Maps each vertex to a sub array stored in ‘FDuplicatedVerticesBuffer::DupVertData[]’
// DupVertIndexData[localRenderVert] =
// {.Length == number of duplicated vertices; .Index == start index in 'DupVertData[]'}
TSkeletalMeshVertexData<FIndexLengthPair> DupVertIndexData;
// flat data:
// Concatanation of all 'TArray<int32>' from 'TMap<int32, TArray<int32>> OverlappingVertices'
// Example:
// let's rename: OverlappingVertices := Ov;
// DupVertData[] =
// {Ov[ 0 ][0], Ov[ 0 ][1], Ov[ 0 ][2];
// Ov[ 1 ][0], Ov[ 1 ][1], Ov[ 1 ][2], Ov[1][3], Ov[1][4];
// ...
// Ov[vix][0], Ov[vix][1], Ov[vix][2];
// ...
// }
TSkeletalMeshVertexData<uint32> DupVertData;
Sample code to look up FDuplicatedVerticesBuffer:
// All vertex indices here refer to render vertices
// (as opposed to imported vertices)
bool notDup = false;
// Look up every vertices of the current mesh section:
for (uint32 vix = 0; vix < RenderSection.NumVertices; ++vix)
{
const int32 globalVertIdx = RenderSection.GetVertexBufferIndex() + vix;
const auto& listDescriptor = RenderSection.DuplicatedVerticesBuffer.DupVertIndexData[vix];
const FPositionVertexBuffer& vertexBuff = LodRenderData->StaticVertexBuffers.PositionVertexBuffer
const FVector& VertexPosition = (FVector)vertexBuff.VertexPosition(globalVertIdx);
// look up sub list of duplicated vertices:
for (uint32 l = listDescriptor.Index; l < listDescriptor.Length; ++l)
{
const int32 localDupIdx = RenderSection.DuplicatedVerticesBuffer.DupVertData[l];
int32 globalRenderVertIdx = RenderSection.GetVertexBufferIndex() + localDupIdx;
const FVector& DupPosition = (FVector)vertexBuff.VertexPosition(globalRenderVertIdx);
FVector res = DupPosition - VertexPosition;
if (res.Length() > 0.000001f)
notDup = true;
}
}
// should turn out true if the mesh has duplicated vertices:
notDup;
Bonus: FDuplicatedVerticesBuffer is computed with:
void FDuplicatedVerticesBuffer::Init(
const int32 NumVertices,
const TMap<int32,
TArray<int32>>& OverlappingVertices);
NewRenderSection.DuplicatedVerticesBuffer.Init(
ModelSection.NumVertices,
ModelSection.OverlappingVertices);
MeshToImportVertexMap
Map between importVertIdx and renderVertIdx is found in:
// Warning: global index. // (as opposed to local indices relative to a mesh section) FSkeletalMeshLODModel::MeshToImportVertexMap[renderVertIdx] = importVertIdx
Here is where both buffers are accessible relative to the skeletal mesh:
// UE 4.26 – 5.1
USkeletalMesh
// ------------------
// GetImportedModel()
// (Editor only)
FSkeletalMeshModel* ImportedModel;
FSkeletalMeshLODModel LODModels[]
MeshToImportVertexMap[renderVertIdx]
FSkelMeshSection Sections[]
...
// -------------------------
// GetResourceForRendering()
// Both editor & game runtime
FskeletalMeshRenderData* SkeletalMeshRenderData;
...
That you typically access as follows:
TObjectPtr<USkeletalMeshComponent> SkeletalMesh const USkeletalMesh* Mesh = SkeletalMesh->GetSkeletalMeshAsset(); const FSkeletalMeshModel* SkeletalMeshModel = Mesh->GetImportedModel(); int32 importVertIdx = SkeletalMeshModel->LODModels[LODIndex].MeshToImportVertexMap[renderVertIdx];
Code sample:
const FSkeletalMeshModel* SkeletalMeshModel;
// Look up LODs:
for (int32 LodIndex = 0; LodIndex < LODSkinWeights.Num(); ++LodIndex)
{
const FSkeletalMeshLODModel& LODModel = SkeletalMeshModel->LODModels[LodIndex];
//Note:
uint32 NumFbxVerts = LODModel.MaxImportVertex;
// Lookup global renderVertIdx
for (uint32 renderVertIdx = 0; renderVertIdx < LODModel.NumVertices; ++renderVertIdx)
{
// Warning: both importVertIdx and renderVertIdx are global here:
const int32 importVertIdx = LODModel.MeshToImportVertexMap[renderVertIdx];
...
}
}
Bonus:
When importing a FBX file:
FMeshUtilities::BuildSkeletalModelFromChunks(){
LODModel.MeshToImportVertexMap.Add(RawVertIndex);
}
// Which is based on the data LODPointToRawMap / PointToRawMap :
TArray<int32> LODPointToRawMap;
SkeletalMeshImportData.CopyLODImportData(
LODPoints,
LODWedges,
LODFaces,
LODInfluences,
LODPointToRawMap)
{
LODPointToRawMap = PointToRawMap;
}
// PointToRawMap is filled when duplicating vertices of a smoothing group:
void FSkeletalMeshImportData::SplitVerticesBySmoothingGroups(){
PointToRawMap[NewPointIndex] = p;
}
int32 UnFbx::FFbxImporter::DoUnSmoothVerts(
FSkeletalMeshImportData &ImportData,
bool bDuplicateUnSmoothWedges)
{
ImportData.PointToRawMap[NewPointIndex] = p;
}
This may happen in other places (My intuition tells me it may happen for UV seams) Note that as a vertex gets split/duplicated other data such as skin weights are also duplicated.
Convert "renderVertIdx" from "global" to "local"
To convert a renderVertIdx from global to a local index
relative to a mesh section use:
void FSkeletalMeshLODModel::GetSectionFromVertexIndex(
int32 InVertIndex, // global renderVertIdx
int32& OutSectionIndex, // Section's index
int32& OutVertIndex) // *local* renderVertIdx (relative to OutSectionIndex)
Code sample to Look up LODs and global renderVertIdx inside to then convert to local renderVertIdx:
const FSkeletalMeshModel* SkeletalMeshModel;
// Look up LODs:
for (int32 LodIndex = 0; LodIndex < LODSkinWeights.Num(); ++LodIndex) {
const FSkeletalMeshLODModel& LODModel = SkeletalMeshModel->LODModels[LodIndex];
// Lookup global renderVertIdx
for (uint32 renderVertIdx = 0; renderVertIdx < LODModel.NumVertices; ++renderVertIdx) {
// Convert to local indices relative to the mesh section/chunk:
int32 sectionIdx;
int32 localrenderVertIdx;
LODModel.GetSectionFromVertexIndex(
/*in: */ renderVertIdx,
/*out: */ sectionIdx,
/*out: */ localrenderVertIdx);
...
}
}
Convert "importVertIdx" from "global" to "local"
Function to operate the conversion global importVertIx to
local importVertIx:
TTuple<int32/*out SectionIndex*/, int32/*out local ImportVertIdx*/>
ToLocalFBXIndex(
int32 globalImportVertIdx,
const FSkeletalMeshLODModel& LODModel)
{
int32 outSectionIndex = -1;
int32 outLocalImportVertIndex = -1;
check( LODModel.ImportedMeshInfos.Num() == LODModel.Sections.Num() );
int32 numChunks = LODModel.ImportedMeshInfos.Num();
int32 vertCount = 0;
for (int32 sectionCount = 0; sectionCount < numChunks; sectionCount++)
{
const FSkelMeshImportedMeshInfo& section = LODModel.ImportedMeshInfos[sectionCount];
outSectionIndex = sectionCount;
check(section.StartImportedVertex == vertCount);
// Is it in section's range?
if (globalImportVertIdx < vertCount + section.NumVertices)
{
outLocalImportVertIndex = globalImportVertIdx - vertCount;
return MakeTuple(outSectionIndex, outLocalImportVertIndex);
}
vertCount += section.NumVertices;
}
return MakeTuple(outSectionIndex, outLocalImportVertIndex);
}
{
int32 localFBXVertIdx;
int32 sectionFBXIdx;
Tie(sectionFBXIdx, localFBXVertIdx) = ToLocalFBXIndex(globalFBXIdx, LODModel);
}
Mesh sections by name
You can find the names of the sub-meshes (a.k.a section or chunk) that compose a USkeletalMesh. For instance,
when exporting several objects from Maya into a .FBX file each object's name is associated to
a UE mesh section as follows:
const FSkeletalMeshLODModel& LODModel
int32 NumChunks = LODModel.ImportedMeshInfos.Num();
for (int32 SectionCount = 0; SectionCount < NumChunks; SectionCount++)
{
const FSkelMeshImportedMeshInfo& section = LODModel.ImportedMeshInfos[SectionCount];
FName = section.Name;
}
Joint data (Matrices, transformations etc.)
Remember that FMatrix multiplication is applied from left to right in Unreal's cpu code:
vec x M1 x M2 x ... where M1 is applied first then M2 etc.
Get bone index from name in UE:
// GetBoneIndex(FName boneName) int boneIndex = skeletalMesh->GetBoneIndex(FName boneName);
Get bone local transformation (according to its parent bone):
// GetBoneSpaceTransforms() TArray<FTransform> localBoneTransforms = skeletalMesh->GetBoneSpaceTransforms(); FTransform localTransform = localBoneTransforms[boneIndex]; FMatrix44f localMatrix = FMatrix44f( localTransform.ToMatrixWithScale() );
Get global transformations of the bone (i.e. world coordinates of the scene)
// GetBoneMatrix(int idx) // GetBoneTransform(int idx) // Both lines are equivalent: FMatrix44f mat = FMatrix44f( skeletalMesh->GetBoneMatrix(boneIndex) ); FMatrix44f mat = FMatrix44f( skeletalMesh->GetBoneTransform(boneIndex).ToMatrixWithScale() );
Get transformation of a bone in component space (object coordinates)
// GetBoneTransform(int idx, FTransform::Identity) // Transformation of the joint expressed relative to the skeletal mesh object coordinates, // as opposed to the scene (world) coordinates FTransform tr = skeletalMesh->GetBoneTransform(boneIndex, FTransform::Identity); FMatrix44f mat = FMatrix44f( tr.ToMatrixWithScale() );
Joint sockets
The joints of a skeletal mesh can be complemented by "sockets". A socket is a special type of joint that you insert in the skeletal hierarchy and is parented to some original joint of your imported model. This is useful to bind props for instance. See official doc about the UI to use sockets.
You can access the transformation value of a socket:
// TODO: describe more of the API TArray<USkeletalMeshSocket*> GetActiveSocketList() const; RootComponent->GetSocketLocation()
Code snippet to debug skeletal mesh transforms
UCLASS()
class BONEMATRICESTEST_API AMyActor : public AActor
{
GENERATED_BODY()
public:
// Sets default values for this actor's properties
AMyActor();
UPROPERTY(EditAnywhere);
USkeletalMeshComponent* MySkel;
UPROPERTY(EditAnywhere, Category = "Locations")
FVector LocationOne;
protected:
// Called when the game starts or when spawned
virtual void BeginPlay() override;
public:
// Called every frame
virtual void Tick(float DeltaTime) override;
};
#include "MyActor.h"
#include "DrawDebugHelpers.h"
// Sets default values
AMyActor::AMyActor()
{
PrimaryActorTick.bCanEverTick = true;
LocationOne = FVector(0, 0, 0);
}
// Called when the game starts or when spawned
void AMyActor::BeginPlay()
{
Super::BeginPlay();
//MySkel->GetBoneMatrix();
int32 idx = MySkel->GetBoneIndex("hand_l");
if (idx > -1)
{
FTransform identity = FTransform::Identity;
FTransform tr = MySkel->GetBoneTransform(idx, FTransform::Identity);
FMatrix m;
FVector Loc = tr.GetLocation();
FQuat Rot = tr.GetRotation();
FRotator EulerRot = Rot.Rotator();
FVector Scale = tr.GetScale3D();
LocationOne = Loc;
DrawDebugPoint(GetWorld(), LocationOne, 30, FColor(52, 220, 239), true);
DrawDebugCoordinateSystem(GetWorld(), Loc, EulerRot, 30.f, true);
}
}
// Called every frame
void AMyActor::Tick(float DeltaTime)
{
Super::Tick(DeltaTime);
}
Skinning
See skinning cheat sheet for general math.
Bind pose / T-Pose
In Unreal Engine the reference pose of a skeletal mesh designates:
- "bind pose"
- "rest pose"
- "T-Pose"
Sometimes abreviated as ref you may see names like refskeleton, refmatrix or ReferenceToLocal as a way to signal it makes use of the reference pose.
C++ to find the reference pose (a.k.a bind pose)
#include "ReferenceSkeleton.h" FReferenceSkeleton refSkeleton = USkeletalMesh->RefSkeleton;
FReferenceSkeleton stores the T-pose as a list of transforms in local coordinates (relative to parent joint):
#include "ReferenceSkeleton.h"
FTransform getRefPoseBoneTransforms(USkeletalMeshComponent* skelMesh, FName boneName)
{
FTransform boneTransform;
FReferenceSkeleton refSkel;
refSkel = skelMesh->SkeletalMesh->RefSkeleton;
boneTransform = refSkel.GetRefBonePose()[refSkel.FindBoneIndex(boneName)];
return boneTransform;
}
Now you can compute from this the bind-pose in component space (object coordinates). Which means expressing each joint's transform relative to the component / object coordinates:
/// @return The bone transform for bone 'boneIdx' at rest-pose in component space
FTransform get_ref_pose_single_bone_comp_space(const FReferenceSkeleton& inSkel, int32 boneIdx)
{
// Local transform (relative to parent joint)
FTransform resultBoneTransform = inSkel.GetRefBonePose()[boneIdx];
auto refBoneInfo = inSkel.GetRefBoneInfo();
while (boneIdx)
{
resultBoneTransform *= inSkel.GetRefBonePose()[refBoneInfo[boneIdx].ParentIndex];
boneIdx = refBoneInfo[boneIdx].ParentIndex;
}
return resultBoneTransform;
}
/// @return the list of bone transforms at rest-pose in component space
TArray<FTransform> get_ref_pose_bone_comp_space(const USkeletalMeshComponent* inSkelComp)
{
const FReferenceSkeleton& refSkeleton = inSkelComp->SkeletalMesh->RefSkeleton;
const int32 poseNum = refSkeleton.GetRefBonePose().Num();
TArray<FTransform> outResult;
outResult.Reset();
outResult.AddUninitialized(poseNum);
// Compute global transform for each joint?
for (int32 i = 0; i < poseNum; i++) {
outResult[i] = get_ref_pose_single_bone_comp_space(refSkeleton, i);
}
return outResult;
}
To get the world coordinates of the bones, for instance in an AActor, you would need to multiply by the transform expressing the world coordinates of that actor:
const USkeletalMeshComponent* inSkelComp = ...;
const FReferenceSkeleton& refSkel = inSkelComp->SkeletalMesh->RefSkeleton;
FTransform actorWorldTransform = this->GetActorTransform();
FTransform tr = get_ref_pose_single_bone_comp_space(
refSkel,
refSkel.FindBoneIndex(name));
// World coordinates:
tr = tr * actorWorldTransform;
original article.
// Fill out your copyright notice in the Description page of Project Settings.
#pragma once
#include "CoreMinimal.h"
#include "GameFramework/Actor.h"
#include "MyActor.generated.h"
UCLASS()
class BONEMATRICESTEST_API AMyActor : public AActor
{
GENERATED_BODY()
public:
// Sets default values for this actor's properties
AMyActor();
UPROPERTY(EditAnywhere);
USkeletalMeshComponent* MySkel;
protected:
// Called when the game starts or when spawned
virtual void BeginPlay() override;
public:
// Called every frame
virtual void Tick(float DeltaTime) override;
};
// Fill out your copyright notice in the Description page of Project Settings.
#include "MyActor.h"
#include "DrawDebugHelpers.h"
#include "SkeletalRenderPublic.h"
#include "ReferenceSkeleton.h"
// Sets default values
AMyActor::AMyActor()
{
// Set this actor to call Tick() every frame. You can turn this off to improve performance if you don't need it.
PrimaryActorTick.bCanEverTick = true;
}
FTransform GetRefPoseBoneTransforms(USkeletalMeshComponent* SkelMesh, FName BoneName) {
FTransform BoneTransform;
FReferenceSkeleton RefSkel;
RefSkel = SkelMesh->SkeletalMesh->RefSkeleton;
BoneTransform = RefSkel.GetRefBonePose()[RefSkel.FindBoneIndex(BoneName)];
return BoneTransform;
}
/// @return The bone transform for bone 'boneIdx' at rest-pose in component space
FTransform get_ref_pose_single_bone_comp_space_transform(const FReferenceSkeleton& inSkel, int32 boneIdx)
{
// Local transform (relative to parent joint)
FTransform resultBoneTransform = inSkel.GetRefBonePose()[boneIdx];
auto refBoneInfo = inSkel.GetRefBoneInfo();
while (boneIdx)
{
resultBoneTransform *= inSkel.GetRefBonePose()[refBoneInfo[boneIdx].ParentIndex];
boneIdx = refBoneInfo[boneIdx].ParentIndex;
}
return resultBoneTransform;
}
/// @return the list of bone transforms at rest-pose in component space
TArray<FTransform> get_ref_pose_bone_comp_space_transform(const USkeletalMeshComponent* inSkelComp)
{
const FReferenceSkeleton& refSkeleton = inSkelComp->SkeletalMesh->RefSkeleton;
const int32 poseNum = refSkeleton.GetRefBonePose().Num();
TArray<FTransform> outResult;
outResult.Reset();
outResult.AddUninitialized(poseNum);
// Compute global transform for each joint?
for (int32 i = 0; i < poseNum; i++) {
outResult[i] = get_ref_pose_single_bone_comp_space_transform(refSkeleton, i);
}
return outResult;
}
// Called when the game starts or when spawned
void AMyActor::BeginPlay()
{
Super::BeginPlay();
if (MySkel == nullptr)
return;
FTransform actorWorldTransform = this->GetActorTransform();
// UE puppet's bone names:
TArray<FName> BoneNames = {
"pelvis",
"spine_01",
"spine_02",
"spine_03",
"spine_04",
"spine_05",
"clavicle_l",
"upperarm_l",
"lowerarm_l",
"hand_l",
"index_01_l",
"index_02_l",
"index_03_l"
};
const FReferenceSkeleton& RefSkel = MySkel->SkeletalMesh->RefSkeleton;
for (FName name : BoneNames) {
FTransform tr = GetRefPoseBoneTransforms(MySkel, name);
FRotator EulerRot = tr.GetRotation().Rotator();
FVector Loc = tr.GetLocation();
DrawDebugCoordinateSystem(GetWorld(), Loc, EulerRot, 40.f, true);
DrawDebugPoint(GetWorld(), Loc, 25, FColor(52, 220, 239), true);
tr = get_ref_pose_single_bone_comp_space_transform(RefSkel, RefSkel.FindBoneIndex(name)) * actorWorldTransform;
EulerRot = tr.GetRotation().Rotator();
Loc = tr.GetLocation();
DrawDebugCoordinateSystem(GetWorld(), Loc, EulerRot, 40.f, true);
DrawDebugPoint(GetWorld(), Loc, 25, FColor(220, 52, 239), true);
}
}
// Called every frame
void AMyActor::Tick(float DeltaTime)
{
Super::Tick(DeltaTime);
}
Skin matrices (lbsReferenceToLocal)
A "skin matrix" is a matrix that takes a vertex in global position and transforms it to its animated/deformed position. In unreal engine skin matrices are often designated under the name ReferenceToLocal. You can get skin matrices as follows in c++:
TArray<FMatrix44f> lbsReferenceToLocal;
int32 lodIndex;
SkeletalMeshComponent::GetCurrentRefToLocalMatrices(
lbsReferenceToLocal,
lodIndex
);
Each element of lbsReferenceToLocal[] contains:
\[ W . B^{-1} \]
where:
- \( W \) is the joint's position (world space a.k.a joint's global transform)
- \( B \) is the joint's bind matrix (reference matrix): world space transformation when bound to the T-pose.
Execution paths / implementation
Skinning is implemented 3 ways on:
- CPU: designated as CPUSkin in UE
- vertex shader (GPU): designated as Skin vertex factory in UE
- compute shader (GPU): designated as skin cache in UE
Mainly located in:
- CPUSkin: SkeletalRenderCPUSkin.cpp
- Skin vertex factory: GpuSkinVertexFactory.ush GPUSkinVertexFactory.cpp
- skin cache: GPUSkinCache.cpp
CPU skinning
As far as I know cpu skinning is only used in specific scenarios in the editor and not the game runtime. It mainly gets enabled when displaying the skeletal mesh normals or tangent vectors or skinning weights. For instance you can force it through the menus:
more at "force cpu skinning.docx"
Note: on the CPU side skinning is also computed in many other places:/ A non exhaustive list of duplicated CPU skinning:
USkinnedMeshComponent::ComputeSkinnedPositions() USkinnedMeshComponent::ComputeSkinnedTangentBasis() /** Simple, CPU evaluation of a vertex's skinned position helper function */ template <bool bCachedMatrices> FVector GetTypedSkinnedVertexPosition() /** Simple, CPU evaluation of a vertex's skinned position helper function */ void GetTypedSkinnedTangentBasis() ClothingMeshUtils::SkinPhysicsMesh(…)
GPU skinning
Upload of skinning matrices to GPU
Skinning matrices are uploaded for Compute Shader (ue skin cache) and Vertex Shader (ue vertex factory)
via the same procedure: UpdateBoneData() see below for a small overview:
// Engine\Source\Runtime\Engine\Private\GPUSkinVertexFactory.cpp
bool FGPUBaseSkinVertexFactory::FShaderDataType::UpdateBoneData() {
// takes CPU arrays RefTolocal and upload to GPU
// to Bufffer<float> BoneMatrix in the shader
// equivalent of glMap() (i.e. lockBuffer) is used for the upload.
// GPU buffer are in a pool
// see GetBoneBufferForReading() GetBoneBufferForWriting() (see below)
// Don't forget that we only upload matrices related to a mesh section.
// and not every matrices of the USKeletalMesh.
}
class FGPUBaseSkinVertexFactory : public FvertexFactory {
struct FShaderDataType {
const FVertexBufferAndSRV& GetBoneBufferForReading(bool bPrevious, uint32 FrameNumber) const
FVertexBufferAndSRV& GetBoneBufferForWriting(uint32 FrameNumber) {
return BoneBuffer[alternate(FrameNumber)]
}
private:
// double buffered bone positions+orientations
// to support normal rendering and velocity (new-old position) rendering
FVertexBufferAndSRV BoneBuffer[2];
The “Skin Cache” a.k.a Compute shader
In the compute shader (GpuSkinCacheComputeShader.usf) we can access the global transformation to be blended with skin weights and applied to the vertices:
Buffer<float4> BoneMatrices;
It is structured as follows:
// BoneMatrices[boneIndex * 3 + 0] == first row // BoneMatrices[boneIndex * 3 + 1] == second row // BoneMatrices[boneIndex * 3 + 2] == third row // No fourth row since in affine transformation it's alway (0 0 0 1) Adding define to the compute shader (GPUSkinCache.cpp) static void TGPUSkinCacheCS::ModifyCompilationEnvironment() Compute shader dispatch:(GPUSkinCache) void FGPUSkinCache::DispatchUpdateSkinning()
Enable compute shader over vertex shader in the Editor's console:
> r.SkinCache.CompileShaders 1
The “Vertex factory” a.k.a Vertex shader
(GpuSkinVertexFactory.ush) because of motion blur we need to compute skinning for current and previous frame in the vertex shader, so we find the bone matrices as follows (same structure as CS version):
// The buffer of bone matrices (4x3) is stored as 3 ‘float4’ behind each other,
// all chunks of a skeletal mesh in one, each skeletal mesh has it's own buffer
Buffer<float4> BoneMatrices;
// The previous bone matrix buffer (same format as above)
Buffer<float4> PreviousBoneMatrices;
// Adding defines to the vertex shader (skin vertex factory) GPUSkinVertexFactory.cpp:
void TGPUSkinVertexFactory<…>::ModifyCompilationEnvironment(…, FShaderCompilerEnvironment& OutEnvironment )
{
FVertexFactory::ModifyCompilationEnvironment(Platform, Material, OutEnvironment);
OutEnvironment.SetDefine(TEXT("USE_DQS"),1.0);
}
Editor, UI, Slates
One can add a combo-box selector to an existing Skeletal Mesh asset as follows:
/* If your class/Actor/UObject etc. is already part of a UI system, and associated for instance to a "detail" pannel, then thanks to introspection magic you will only need to add the UPROPERTY which will automatically add the corresponding combo box element in the your existing pannel/menu (creating the menu itself is much more involved I think) */ UPROPERTY(EditAnywhere, Category = Mesh) TObjectPtr<USkinnedAsset> skinnedAsset; UPROPERTY(EditAnywhere, Category = Mesh) TObjectPtr<USkeletalMesh> skeletalMesh; // Similarly: UPROPERTY(EditAnywhere, Category = Mesh) TObjectPtr<USkeletalMeshComponent> skeletalMeshComponent;
Todo
There is no guarantee I'll ever complete it, but I still need to document how skin weight data is organized:
- Explain skin weights data in mesh sections
- convertion of skin weights joint indices from local (mesh section) to global (LOD level) array of joints?
- How multiple skin weight profiles can be imported, and the data layout
- Sample code to a simple LBS example that makes use of skin weights and display mesh with draw helper debug points.
No comments