WaitForAllTasks randomly uses too much CPU, and performance optimization suggestions

WaitForAllTasks randomly uses too much CPU, and performance optimization suggestions - Printable Version

+- Obi Official Forum (https://obi.virtualmethodstudio.com/forum)
+-- Forum: Obi Users Category (https://obi.virtualmethodstudio.com/forum/forum-1.html)
+--- Forum: Obi Rope (https://obi.virtualmethodstudio.com/forum/forum-4.html)
+--- Thread: WaitForAllTasks randomly uses too much CPU, and performance optimization suggestions (/thread-569.html)

WaitForAllTasks randomly uses too much CPU, and performance optimization suggestions - Devver - 30-04-2018

Hello, I'm developing a VR project using the awesome ObiRope and would like to ask some questions and share a few optimizations suggestions that helped lowering CPU usage on my project. Sonrisa

The main issue: Oni.WaitForAllTasks() usually uses around 0.6ms of CPU time, but at unpredictable intervals (a few or many seconds) it takes an extra long time for a single frame: sometimes 5ms, sometimes 15ms, or even 20ms.

Here are some images from the profiler, using the bundled sample scene "FreightLift", unchanged. The higher peak is 20ms, the smaller ones are around 10ms. (This wasn't in VR)

[Image: p9x6zji.png]

I used the Profiler.BeginSample function around Oni.WaitForAllTasks to show it easier on the profiler, like this in the ObiArbiter's WaitForAllSolvers method, at line 41:

Code:
UnityEngine.Profiling.Profiler.BeginSample("WaitForAllTasks");

Oni.WaitForAllTasks();

UnityEngine.Profiling.Profiler.EndSample();

Otherwise you can just look at the ObiSolver.FixedUpdate() times in the profiler.

This would be (maybe) acceptable for a desktop application, but in VR tracking is totally lost for some frames each time, at random intervals, making it unbearable.

Even if this couldn't be prevented somehow, it would be great if could at least be run asynchronously or split into more frames when it's taking too long. Even if it could delay rope calculations for a few frames when that happens, it would be way better than entirely freezing the game in VR.

I'm using ObiRope 3.4 with Unity 5.6.3f1 on Windows 10.
This happens using any of the 3 simulation order options on Obi Solver.
The tests above were made using a clean project with just ObiRope imported.

Also, for this project I'm currently using kinematicForParticles as true for everything, making the ropes work just as visuals and they don't physically affect anything for now (no pin constraints, just handles). I'm also using a fixed timestep value that matches the drawing update (both at 90fps), so they're synched, and like that it seems that changing the simulation order doesn't change performance. (PS: None of those changes were made to the clean project above for testing)
Do you have any suggestions to increase performance (any settings or maybe disabling anything unnecessary) for this use case?

Thank you in advance!

Performance optimization suggestions

CPU performance is very important for this VR project, so anything I could optimize helps. Here are a few changes that got me a few milliseconds back. I hope this can help someone. (The numbers below are examples taken from my machine)

1. Anisotropy doesn't seem to be needed for only ropes, but this call below still uses 0.9ms each frame, so I commented it out. This breaks the debug particle render shader, so I'll reenable this if I need to do some debugging, but not on the final build.

Location: ObiSolver.cs, line 557.

Code:
// Oni.GetParticleAnisotropies(oniSolver,anisotropies,anisotropies.Length,0);

2. ObiRopeThickRenderMode's Update will keep recreating the rope's mesh every frame even if it's sleeping, not moving. That's around 1.2ms for 6 short ropes in the "RopeShowcase" scene, and will increase with length, resolution, smoothing and more ropes. I check if the rope has actually moved before allowing it to recalculate the mesh, by changing this at line 32 on the ObiRopeThickRenderMode.cs file, just above "rope.SmoothCurvesFromParticles();":

Code:
private Vector3[] previousPositions = new Vector3[0];

public override void Update(Camera camera){

    if (rope.section == null || rope.ropeMesh == null)

        return;

    // Skip mesh recalculation if rope has not changed.

    bool updateMesh = false;

    if(previousPositions.Length != rope.positions.Length)

    {

        updateMesh = true;

        previousPositions = new Vector3[rope.positions.Length];

    }

    else

    {

        for(int i = 0; i < rope.positions.Length; i++)

        {

            Vector3 newPosition = rope.GetParticlePosition(i);

            if(previousPositions[i] != newPosition)

            {

                previousPositions[i] = newPosition;

                updateMesh = true;

            }

        }

    }

    if(!updateMesh)

        return;

    rope.SmoothCurvesFromParticles();

...

It would be easier to just check if the rope is sleeping, but I couldn't find any exposed method to detect this. Also, this may need some higher damping and sleep threshold values on the Obi Solver object to be effective, or the ropes may take too long to sleep.

3. When using ObiRigidbory and pin constraints, the ropes never let Unity rigidbody objects to get to sleep (they keep slightly moving the objects forever). I don't know if this is a bug, but it will force the engine to keep calculating physics for objects that aren't moving anymore, usually set to sleep until touched again, removing them from unnecessary physical calculations.
Replacing the ObiRigidbody's UpdateVelocities() method with this code can fix that. You may need to tweak those thresholds.

Code:
public override void UpdateVelocities(object sender, EventArgs e){

    // kinematic rigidbodies are passed to Obi with zero velocity, so we must ignore the new velocities calculated by the solver:

    if (Application.isPlaying && (unityRigidbody.isKinematic || !kinematicForParticles)){

        Oni.GetRigidbodyVelocity(oniRigidbody,ref oniVelocities);

        Vector3 newVelocity = oniVelocities.linearVelocity - velocity;

        Vector3 newAngularVelocity = oniVelocities.angularVelocity - angularVelocity;

        bool updateVelocity = false;

        bool updateAngularVelocity = false;

        for(int i = 0; i < 3; i++){

            if(Mathf.Abs(newVelocity[i]) > 0.13f)

                updateVelocity = true;

            if(Mathf.Abs(newAngularVelocity[i]) > 2f)

                updateAngularVelocity = true;

        }

        if(updateVelocity)

            unityRigidbody.velocity += oniVelocities.linearVelocity - velocity;

        if(updateAngularVelocity)

            unityRigidbody.angularVelocity += oniVelocities.angularVelocity - angularVelocity;

    }

}

4. Like some people here, I needed to use ObiColliders with objects with multiple colliders, but that wouldn't work anymore. So I made this ObiCompoundCollider class. You can add it to objects in the editor as usual, or programmatically using the static AddCollider method. You don't need to add it specifically to objects with rigidbodies only; just add it to any gameobject, and it will search for children with colliders. Triggers colliders will be ignored.

Code:
using UnityEngine;

using Obi;

public class ObiCompoundCollider : MonoBehaviour

{

    public ObiCollisionMaterial material;

    public int phase = 0;

    public float thickness = 0;

    public bool useDistanceFields = false;

    void Awake()

    {

        foreach(Collider collider in GetComponentsInChildren<Collider>())

            AddCollider(collider, this);

        Destroy(this);

    }

    public static void AddCollider(Collider collider, ObiCompoundCollider obiCompoundCollider)

    {

        AddCollider(collider,

            obiCompoundCollider.material,

            obiCompoundCollider.phase,

            obiCompoundCollider.thickness,

            obiCompoundCollider.useDistanceFields);

    }

    public static void AddCollider(Collider collider, ObiCollisionMaterial material = null, int phase = 0, float thickness = 0, bool useDistanceFields = false)

    {

        if(collider.isTrigger)

            return;

        ObiCollider obiCollider = collider.GetComponent<ObiCollider>();

        if(obiCollider == null || obiCollider.GetCollider() != collider)

        {

            collider.gameObject.SetActive(false);

            obiCollider = collider.gameObject.AddComponent<ObiCollider>();

            obiCollider.SetCollider(collider);

            obiCollider.CollisionMaterial = material;

            obiCollider.phase = phase;

            obiCollider.thickness = thickness;

            obiCollider.UseDistanceFields = useDistanceFields;

            collider.gameObject.SetActive(true);

        }

    }

}

You need to also edit these files below to make it work.

In ObiCollider.cs:

Change the IsUnityColliderEnabled() method to this:

Code:
protected override bool IsUnityColliderEnabled(){

    Collider collider = ((Collider)unityCollider);

    return collider.enabled && !collider.isTrigger;

}

Change the Awake() method to this:

Code:
protected override void Awake(){

    if (unityCollider == null)

        unityCollider = GetComponent<Collider>();

    if (unityCollider == null)

        return;

    base.Awake();

}

Add these new methods:

Code:
public void SetCollider(Collider collider){

    unityCollider = collider;

}

public Collider GetCollider(){

    return (Collider)unityCollider;

}

In ObiColliderBase.cs, add these variables:

Code:
protected static Transform previousTransform = null;

protected static bool previousHasChanged = false;

And change the UpdateIfNeeded() method like this:

Code:
private void UpdateIfNeeded(object sender, EventArgs e){

    if (unityCollider != null){

        // update the collider:

        bool unityColliderEnabled = IsUnityColliderEnabled();

        if(previousTransform == unityCollider.transform)

            unityCollider.transform.hasChanged = previousHasChanged;

        previousTransform = unityCollider.transform;

        previousHasChanged = unityCollider.transform.hasChanged;

        if (unityCollider.transform.hasChanged || 

            phase != oldPhase ||

            thickness != oldThickness || 

            unityColliderEnabled != wasUnityColliderEnabled){

...

With that, if you want you can also do this once at runtime to add Obi colliders to every static collider in the scene.

Code:
foreach(Collider collider in FindObjectsOfType<Collider>())

{

    if(collider.attachedRigidbody == null)

        ObiCompoundCollider.AddCollider(collider);

}

I hope this helps, and thanks again!

RE: WaitForAllTasks randomly uses too much CPU, and performance optimization suggestions - josemendez - 02-05-2018

Hi there!

Sorry for taking some time to reply to this, but that was a long post to read! Guiño

Regarding the WaitForAllTasks spikes: this method is where the physics computation itself takes place. It runs by default in FixedUpdate, which is usually called only once per frame but can be called by Unity as many times as necessary to get the physics update to be in sync with the game world's time. If for some reason a given frame takes longer to render, the next frame will call FixedUpdate multiple times.

This behavior is completely normal in all fixed-timestep physics engines (and should be expected). The maximum amount of time that is spent in physics in any given frame can be capped in Unity: ProjectSettings->Time->max fixed timestep. Note that in some less powerful devices, you can enter a situation known as the "death spiral", see the following video for my pitiful attempt of an explanation:

Performance tips:
- Use "line" rendering mode with the line material provided for much cheaper rope rendering (camera oriented triangle strips instead of full-3D extruded geometry).
- Use low rope resolution values (less particles will be generated) and crank up rope smoothing to compensate.
- Use less distance constraint iterations, and use a couple substeps instead.

We are currently polishing Obi's collider system, so expect compound colliders to be back in the next version.

RE: WaitForAllTasks randomly uses too much CPU, and performance optimization suggestions - Devver - 02-05-2018

Hi, josemendez, thanks for your answer. Sonrisa

However, please notice in the screenshot, that that's not related to the fixed timestep and multiple calls in the same frame, as that spike is for just 1 call in that frame. We're using the same timing for update and fixed update calls, as you can see below. Also, here's one more screenshot showing a spike using these timings (for 60fps) on the clean project:

[Image: LZ4gXKI.png]

As you can see, it's narrowed down to a single call specifically to Oni.WaitForAllTasks(), and doesn't seem to be affected by Unity's own physics calculations.
There are no such spikes whatsoever when temporarily disabling the ObiSolver component, and we aren't affecting any rigidbodies with ropes in this project, they're purely visual, and all colliders are set as kinematicForParticles.

Changing the "Maximum Allowed Timestep" doesn't change this behavior in any way, as it's already just 1 call in those frames. There's a better official documentation on that value in this post: https://forum.unity.com/threads/what-is-maximum-allowed-timestep.330116/#post-2139556
I've also tried setting it to the minimum and higher values, just to be sure, to no avail.

I can send you the project so you can take a closer look if you can. (It's really just a clean new project though).

These high spikes are rendering the application unusable in VR, and I really need your help with that.

Thanks again.

RE: WaitForAllTasks randomly uses too much CPU, and performance optimization suggestions - josemendez - 03-05-2018

(02-05-2018, 05:55 PM)Devver Wrote: Hi, josemendez, thanks for your answer.

However, please notice in the screenshot, that that's not related to the fixed timestep and multiple calls in the same frame, as that spike is for just 1 call in that frame. We're using the same timing for update and fixed update calls, as you can see below. Also, here's one more screenshot showing a spike using these timings (for 60fps) on the clean project:

As you can see, it's narrowed down to a single call specifically to Oni.WaitForAllTasks(), and doesn't seem to be affected by Unity's own physics calculations.
There are no such spikes whatsoever when temporarily disabling the ObiSolver component, and we aren't affecting any rigidbodies with ropes in this project, they're purely visual, and all colliders are set as kinematicForParticles.

Changing the "Maximum Allowed Timestep" doesn't change this behavior in any way, as it's already just 1 call in those frames. There's a better official documentation on that value in this post: https://forum.unity.com/threads/what-is-maximum-allowed-timestep.330116/#post-2139556
I've also tried setting it to the minimum and higher values, just to be sure, to no avail.

I can send you the project so you can take a closer look if you can. (It's really just a clean new project though).

These high spikes are rendering the application unusable in VR, and I really need your help with that.

Thanks again.

Hi,

Sure, you can send the project to support (at) virtualmethodstudio.com for me to take a look at it. We haven't seen this behavior before, frame times stay quite stable in all platforms we've tried (including Oculus and Vive).

RE: WaitForAllTasks randomly uses too much CPU, and performance optimization suggestions - Devver - 03-05-2018

(03-05-2018, 08:34 AM)josemendez Wrote: Hi,

Sure, you can send the project to support (at) virtualmethodstudio.com for me to take a look at it. We haven't seen this behavior before, frame times stay quite stable in all platforms we've tried (including Oculus and Vive).

Thanks, happy to hear that's not supposed to happen! Will send it asap!

EDIT: Sent!

RE: WaitForAllTasks randomly uses too much CPU, and performance optimization suggestions - Devver - 05-05-2018

Hi josemendez, did you receive my email?

RE: WaitForAllTasks randomly uses too much CPU, and performance optimization suggestions - StevenHolwerda - 01-10-2019

Has anyone figured this out? I get the same spike over a single call, killing my VR performance.