How to access the SkinnedMeshRenderer behind ObiSoftbodySkinner?

[kodra]

As far as I can tell, once a ObiSoftbodySkinner is bound to a SkinnedMeshRenderer, the mesh is still rendered even I disable the SkinnedMeshRenderer at runtime. So I assume the ObiSoftbodySkinner "takes it over" at runtime and the SkinnedMeshRenderer is no longer in use. (plz correct me if it's not how it works)

However, there are some operations that we usually can only access through the SkinnedMeshRenderer. For example, changing the blendshape weights, switching the material, etc.

How do we do these in the case where ObiSoftbodySkinner presents?

[josemendez]

As far as I can tell, once a ObiSoftbodySkinner is bound to a SkinnedMeshRenderer, the mesh is still rendered even I disable the SkinnedMeshRenderer at runtime. So I assume the ObiSoftbodySkinner "takes it over" at runtime and the SkinnedMeshRenderer is no longer in use. (plz correct me if it's not how it works)

In few words, yes that’s how it works, except for one important difference: Obi will batch together multiple softbody skinned meshes by combining them, in order to be able to deform a lot of geometry efficiently in parallel. Deforming each individual mesh in the main thread is just not an option, specially not when performing simulation in the GPU.

So there’s not a single “internal SkinnedMeshRenderer” per ObiSoftbodySkinner, but just a few of them for all ObiSoftbodySkinners.

However, there are some operations that we usually can only access through the SkinnedMeshRenderer. For example, changing the blendshape weights, switching the material, etc.

Blendshapes are not supported. Never have been and probably never will due to how softbody deformation clashes with pose-driven deformation: blendshapes change the rest shape of the base mesh which would require re-generating the blueprint every time a blendshape weight changes, something that cannot be done in realtime.

How do we do these in the case where ObiSoftbodySkinner presents?

Modifying the SkinnedMeshRenderer parameters (material, shadow casting options, etc) and then marking softbody rendering as dirty in the ObiSolver component. This will trigger a re-batching of softbody mesh data (merging multiiple meshes together, see above) which is mildy expensive, so it’s not recommended to do this every frame.

I’ll whip up an example of this for you on Monday.

Cheers!

[kodra]

In few words, yes that’s how it works, except for one important difference: Obi will batch together multiple softbody skinned meshes by combining them, in order to be able to deform a lot of geometry efficiently in parallel. Deforming each individual mesh in the main thread is just not an option, specially not when performing simulation in the GPU.

So there’s not a single “internal SkinnedMeshRenderer” per ObiSoftbodySkinner, but just a few of them for all ObiSoftbodySkinners.

Understood!

Blendshapes are not supported. Never have been and probably never will due to how softbody deformation clashes with pose-driven deformation: blendshapes change the rest shape of the base mesh which would require re-generating the blueprint every time a blendshape weight changes, something that cannot be done in realtime.

I'm a little confused about this. I thought the Softbody, when used with SoftbodySkinner, acts very similarly to skeleton animation: each particle is just like a "bone" in skeleton, and when a particle moves, the vertices with non-zero weights binding to that particle moves along. Just like how skeleton animation works, right?

Then why can't it support blendshapes? I understand blendshapes work on the rest shape of the mesh. But they don't require the whole skeleton to be regenerated, in the case of skeleton animation.

My use case is that the mesh is only partially driven by softbody, similar to "ElasticCharacter" sample: the arms are driven by the softbody, but the legs are driven by skeleton animation. Let's say I have a blendshape that changes the shape of legs; is it possible to make them work together?

[josemendez]

I'm a little confused about this. I thought the Softbody, when used with SoftbodySkinner, acts very similarly to skeleton animation: each particle is just like a "bone" in skeleton, and when a particle moves, the vertices with non-zero weights binding to that particle moves along. Just like how skeleton animation works, right?

Not really. What you describe is only the very last step, and is applied after blendshapes would. Softbody simulation works using shape-matching, which tries to match the current shape of the softbody to the original/rest shape of the mesh. This involves preprocessing a lot of data about the original shape, which is stored in the blueprint.

Blendshapes modify the original shape of the mesh, prior to skeletal deformation. The reason is that you want to be able to alter the shape of the mesh and *then* have it animate, otherwise the mesh would blend towards the blendshape at rest.

So the desired order would be:

Rest mesh -> blend shapes/recalculate rest mesh -> lerp(softbody simulation,skeletal animation, influence)

Note that skeletal animation does not happen either before or after softbody simulation, but at the same time. Then, the painted per-vertex influence is used to choose between both.

[kodra]

Not really. What you describe is only the very last step. Softbody simulation works using shape-matching, which tries to match the current shape of the softbody to the original shape of the mesh. This involves preprocessing a lot of data about the original shape, which is stored in the blueprint.

Blendshapes modify the original shape of the mesh, prior to skeletal deformation. The reason is that you want to be able to alter the shape of the mesh and *then* have it animate, otherwise the mesh would blend towards the blendshape at rest.

So the desired order would be:

Original mesh -> blend shapes -> lerp(softbody simulation,skeletal animation, influence)

Note that skeletal animation does not happen either before or after softbody simulation, but at the same time. Then, the painted per-vertex influence is used to choose between both.

Now I'm even more confused.

> Blendshapes modify the original shape of the mesh, prior to skeletal deformation. The reason is that you want to be able to alter the shape of the mesh and *then* have it animate, otherwise the mesh would blend towards the blendshape at rest.

Yes, I understand this.

> Softbody simulation works using shape-matching, which tries to match the current shape of the softbody to the original shape of the mesh.

But... Shape-matching constraint should match the current shape of the softbody to the original shape of the softbody mesh, which is generated at the moment the user presses "Generate" in Blueprint editor, shouldn't it? It shouldn't take the skinned mesh, which is used for rendering and can be much much more higher-poly than the softbody mesh, into consideration.

But the blendshape works on skinned mesh. So why does changing blendshape have to affect the softbody, or more broadly speaking, affect the simulation in any way?

[josemendez]

But the blendshape works on skinned mesh.

Nope, that's not how blend shapes work at all. They work on the base mesh, prior to any skinning or skeletal deformation.

Imagine you make a blend shape for a facial expression. If you were to apply blend shapes on the skinned mesh (after skeletal animation), rotating the neck to the left then applying the blend shape would make the face morph into the original position facial expression, looking forward: it would "override" animation.

Same happens with softbody deformation. If you were to apply blend shapes after softbody simulation, the mesh would completely ignore the deformation due to the softbody and morph into the original, rigid blend shape. At this point you’d have blendshapes, but no softbody simulation.

So blend shapes must be applied *before* both skeletal animation and softbody simulation, which in the case of a softbody means recalculating the rest shape as modified by the blend shape.

[kodra]

Nope, that's not how blend shapes work at all. They work on the base mesh, prior to any skinning or skeletal deformation.

Imagine you make a blend shape for a facial expression. If you were to apply blend shapes on the skinned mesh (after skeletal animation), rotating the neck to the left then applying the blend shape would make the face morph into the original position facial expression, looking forward: it would "override" animation.

Same happens with softbody deformation. If you were to apply blend shapes after softbody simulation, the mesh would completely ignore the deformation due to the softbody and morph into the original, rigid blend shape.

So blend shapes must be applied *before* both skeletal animation and softbody simulation, which in the case of a softbody means recalculating the rest shape as modified by the blend shape.

You misunderstood what I meant. I'm 99.9% confident that I understand how blendshapes work, as someone who has implemented this feature from scratch in a custom engine. Unless Unity's blendshapes are very abnormal, but I don't believe it's the case.

When I said blendshapes work on skinned mesh, I were not trying to say them work on mesh after deformation. I know blendshapes work on the rest shape. I meant that blendshape work on the mesh that is skinned (has vertices weights painted), not the softbody mesh. By "softbody mesh" I mean the thing is simulated, which is just a bunch of particles with constraints between them.

Shape-matching constraints work on the softbody mesh, so it should not be affected by blendshapes.

[josemendez]

I know blendshapes work on the rest shape. I meant that blendshape work on the mesh that is skinned (has vertices weights painted), not the softbody mesh. By "softbody mesh" I mean the thing is simulated, which is just a bunch of particles with constraints between them.

Precisely: the “softbody mesh” is generated *from* the “skinned mesh” (ie, the high resolution, base mesh, which is the one affected by blendshapes). If you change the base mesh, it stands to reason the softbody mesh must be updated/re-generated.

Shape-matching constraints work on the softbody mesh, so it should not be affected by blendshapes.

In that case you’d have a mesh with blendshapes and no visible softbody deformation: a regular SkinnedMeshRenderer.

I believe what you want is not softbody simulation on blendshapes, but softbody simulation *or* blendshapes. This is easily accomplished by separating the face/blendshape affected areas into their own SkinnedMeshRenderer.

It would be possible to have blendshapes work in areas of a softbody where the painted softbody influence is zero, but that can harly be considered “blendshape support for softbodies” imho.

Let me know whether this is your use case, I can try and hack <blendshapes for zero softbody influence areas> in for the next beta of v7. Something like:

lerp(softbody simulation,rest mesh -> blendshapes -> skeletal animation, influence)

[kodra]

In that case you’d have a mesh with blendshapes and no visible softbody deformation: a regular SkinnedMeshRenderer.

I believe what you want is not softbody simulation on blendshapes, but softbody simulation *or* blendshapes. This is easily accomplished by separating the face/blendshape affected areas into their own SkinnedMeshRenderer.

It would be possible to have blendshapes work in areas of a softbody where the painted softbody influence is zero, but that can harly be considered “blendshape support for softbodies” imho.

"It would be possible to have blendshapes work in areas of a softbody where the painted softbody influence is zero, but that can harly be considered “blendshape support for softbodies” imho."

 Uh, yes, that's what I want, and I thought I've made it clear when I stated several posts ago:

My use case is that the mesh is only partially driven by softbody, similar to "ElasticCharacter" sample: the arms are driven by the softbody, but the legs are driven by skeleton animation. Let's say I have a blendshape that changes the shape of legs; is it possible to make them work together?

[josemendez]

My use case is that the mesh is only partially driven by softbody, similar to "ElasticCharacter" sample: the arms are driven by the softbody, but the legs are driven by skeleton animation. Let's say I have a blendshape that changes the shape of legs; is it possible to make them work together?

My apologies! That paragraph went completely unnoticed.

Currently the only way to do this is to separate the blendshape-affected areas into their own mesh, unaffected by softbody simulation.