#178 Reinit/reposition with speed > 0 may cause NaN issues and crash (YASIM propeller planes)

[Anonymous]

Originally created by: grg.gdsr (code.google.com)

I was flying a b1900d on auto pilot from KSFO south west over the pacific. All of a sudden the plane seemed to bank steeply and stop. FlightGear became unresponsive. I was flying at 6x speed rate. I was able to kill the main fgfs window with gnome system monitor but it took a while because my whole system was also slowed. I had started FlightGear with the command line, not with fgrun. Also had started terrasync from the command line as well.
What steps will reproduce the problem?
1. I cannot determine at the moment what steps make it happen, apart from what I wrote above. Seems to be random.

What is the expected output?
The plane to continue flying without sudden banking or becoming stuck in one position.
What do you see instead?
Plane banks and gets stuck in one position. Doesn't seem to move.

Any output in the console (black window)? Yes:
Could not find at least one of the following objects for animation:

terminal_2.

Could not find at least one of the following objects for animation:

terminal_2

AL Error (fx): Invalid Value at pitch and gain
AL Error (fx): Invalid Value at pitch and gain
AL Error (fx): Invalid Value at pitch and gain
Warning: TangentSpaceGenerator: unknown primitive mode 9
Warning: TangentSpaceGenerator: unknown primitive mode 9
Warning: TangentSpaceGenerator: unknown primitive mode 9
Warning: TangentSpaceGenerator: unknown primitive mode 9
Warning: TangentSpaceGenerator: unknown primitive mode 9
Warning: TangentSpaceGenerator: unknown primitive mode 9
Warning: TangentSpaceGenerator: unknown primitive mode 9
Warning: TangentSpaceGenerator: unknown primitive mode 9
Warning: TangentSpaceGenerator: unknown primitive mode 9
Warning: TangentSpaceGenerator: unknown primitive mode 9
Warning: TangentSpaceGenerator: unknown primitive mode 9
Warning: TangentSpaceGenerator: unknown primitive mode 9
Warning: TangentSpaceGenerator: unknown primitive mode 9
Warning: TangentSpaceGenerator: unknown primitive mode 9
Warning: TangentSpaceGenerator: unknown primitive mode 9
CullVisitor::apply(Geode&) detected NaN,
    depth=nan, center=(0 0 0),
    matrix={
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
}

CullVisitor::apply(Geode&) detected NaN,
    depth=nan, center=(0 0 0),
    matrix={
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
}

CullVisitor::apply(Geode&) detected NaN,
    depth=nan, center=(6623.53 -5187.35 7.26094),
    matrix={
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
}

CullVisitor::apply(Geode&) detected NaN,
    depth=nan, center=(0.835048 0.203138 1.3498),
    matrix={
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
}

CullVisitor::apply(Geode&) detected NaN,
    depth=nan, center=(6640.9 -5207.29 6.66881),
    matrix={
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
}

CullVisitor::apply(Geode&) detected NaN,
    depth=nan, center=(0 0 35040),
    matrix={
    -nan -nan -nan -nan
    nan nan nan nan
    nan nan nan nan
    nan nan nan nan
}

CullVisitor::apply(Geode&) detected NaN,
    depth=nan, center=(1146.61 -37656.6 -18632.6),
    matrix={
    -nan -nan -nan -nan
    nan nan nan nan
    nan nan nan nan
    nan nan nan nan
}

CullVisitor::apply(Geode&) detected NaN,
    depth=nan, center=(7.07422 9.16406 3.33594),
    matrix={
    -nan -nan -nan -nan
    nan nan nan nan
    nan nan nan nan
    nan nan nan nan
}

CullVisitor::apply(Geode&) detected NaN,
    depth=nan, center=(3.8147e-06 0.821896 357.847),
    matrix={
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
}

CullVisitor::apply(Geode&) detected NaN,
    depth=nan, center=(0 1.60786 342.207),
    matrix={
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
}

CullVisitor::apply(Geode&) detected NaN,
    depth=nan, center=(0 2.32357 311.612),
    matrix={
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
}

CullVisitor::apply(Geode&) detected NaN,
    depth=nan, center=(0 2.93773 267.397),
    matrix={
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
}

CullVisitor::apply(Geode&) detected NaN,
    depth=nan, center=(1.52588e-05 3.42349 211.496),
    matrix={
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
}

CullVisitor::apply(Geode&) detected NaN,
    depth=nan, center=(0 3.75961 146.351),
    matrix={
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
}

CullVisitor::apply(Geode&) detected NaN,
    depth=nan, center=(0 3.93144 74.8104),
    matrix={
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
}

CullVisitor::apply(Geode&) detected NaN,
    depth=nan, center=(0 3.93144 -1.71661e-05),
    matrix={
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
}

CullVisitor::apply(Geode&) detected NaN,
    depth=nan, center=(0 3.93144 -74.8104),
    matrix={
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
}

CullVisitor::apply(Geode&) detected NaN,
    depth=nan, center=(0 3.75963 -146.351),
    matrix={
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
}

CullVisitor::apply(Geode&) detected NaN,
    depth=nan, center=(0 3.42348 -211.496),
    matrix={
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
}

CullVisitor::apply(Geode&) detected NaN,
    depth=nan, center=(0 2.93773 -267.397),
    matrix={
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
}

CullVisitor::apply(Geode&) detected NaN,
    depth=nan, center=(0 2.32358 -311.612),
    matrix={
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
}

CullVisitor::apply(Geode&) detected NaN,
    depth=nan, center=(0 1.60786 -342.207),
    matrix={
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
    -nan -nan -nan -nan
}

What version of the product are you using (when using GIT version, please
mention date or Hudson build number)? Version 2.2.0 14th November.

What operating system and graphics card? Ununtu 10.04 64bit. Nvidia Geforce 7950GT.

Please provide any additional information below.

[Anonymous]

Originally posted by: grg.gdsr (code.google.com)

I was able to reproduce the bug:
1. Take of from KSFO with auto pilot set for Christmas Island, alt at 3000ft, ias at 200kts.
2. Once aircraft has autopilot turned on and altitude levelled and speed attained and heading stabilised. Open Location, Position Aircraft in Air.
3. Select radio button longitude. Type in longitude 127, latitude 35.
4. Type in altitude 3000, airspeed 200 and  leave the heading as it is already at.  
5. Click OK.
FlightGear immediately crashes with CullVisitor::apply(Geode&) detected NaN.

[Anonymous]

Originally posted by: bre... (code.google.com)@gmail.com

Even easier to reproduce: KSFO startup, speed rate 6x, reposition in air@127/35, 3000ft/200ias causes the sim to run amok. Looks like heavy memory corruption with all kinds of resulting issues all over the sim. Root cause not so easy to track...

Labels: type.crash
Cc: zakalawe
Status: Accepted

[Anonymous]

Originally posted by: bre... (code.google.com)@gmail.com

(No comment was entered for this change.)

Summary: Reinit/reposition may cause NaN issues and crash

[Anonymous]

Originally posted by: bre... (code.google.com)@gmail.com

I was looked into this. Problem is YASIM (it's YASIM, not JSBSim for this plane!). All it takes is an initial non-zero speed (tested with speed 200). Reset/reposition is not required: starting fgfs with an initial velocity ( --vc=200 ) triggers the same/similar issues.

It doesn't show with other FDMs, so it seems to be a genuine YASIM problem. When the FDM is created with an initial velocity, it starts calculating an insane flight path (warp-driven launch to outer space...). Eventually this results in NaNs. Since this can also be triggered on intial start-up, it's probably not caused by memory corruption or some properties not being reset.
=> YASIM must be fixed to work with initial velocities...

Summary: Reinit/reposition with speed > 0 may cause NaN issues and crash (YASIM planes)
Labels: YASIM FDM

[Anonymous]

Originally posted by: bre... (code.google.com)@gmail.com

(No comment was entered for this change.)

Labels: Priority-High

[Anonymous]

Originally posted by: gijsrooy

Seems to be fixed in current Git. If anyone can confirm we can close this issue.

Status: Testing

[Anonymous]

Originally posted by: bcoco... (code.google.com)@gmail.com

Launching FG with the following command makes it unresponsive. No NaNs are issued however. FG just freezes.

fgfs --aircraft=b1900d --altitude=10000 --vc=200

Status: Accepted

[Anonymous]

Originally posted by: bre... (code.google.com)@gmail.com

The bit about freezing is resolved. Now we're getting NaNs... :-)
http://www.gitorious.org/fg/flightgear/commit/68dec9af2dae7a46d2ad6a6be18150035ef1299c

[Anonymous]

Originally posted by: bre... (code.google.com)@gmail.com

I had another look into this earlier. Adding to comment #4 I can now limit this issue to YASim propeller aircraft. YASim jets are not affected. The issue is caused by YASim's "TurbineEngine.cpp" and "Propeller.cpp". These produce huge force and torque values when an initial speed > 0 is set. YASim stabilizes when the initial speed isn't too large (a few knots), but it the forces/torque are too huge when the initial speed is in the range of >100kts - and result in the aircraft spinning wildly and being propelled into the infinity of space... A few comments in the code also suggest it was never intended to be started with an initial speed != 0.
Unfortunately I can't see any obvious fix here. It's not as easy as initializing the initial speed somewhere.
I need to leave that for someone who knows more about the FDM.

Summary: Reinit/reposition with speed > 0 may cause NaN issues and crash (YASIM propeller planes)

[Anonymous]

Originally posted by: bre... (code.google.com)@gmail.com

Issue 466 has been merged into this issue.

[Anonymous]

Originally posted by: i... (code.google.com)@beware.dropbear.id.au

Try --model-hz=240 or more. The default is 120. I just spent quite a bit of time tracing execution for a model which would go unstable at full throttle (showing very similar symptoms). An amusing detail is that on the last iteration before actual floating overflows occurred, the plane had a prop rotating at over 500000rpm a flying at a speed of 2 1/4 light years per second!

Anyway, the different derivatives in the Runge and Kutta algorithm were wildly different, which meant that dt was too large. Doubling model-hz halves dt and did the trick for me. Even if this still doesn't fix the case of an actual discontinuity in state, it might work for the original problem: "all of a sudden the plane seemed to bank steeply and stop."

[Anonymous]

Originally posted by: gijsrooy

Still there with current Git. Setting --model-hz=240; the model indeed loads...

Labels: -type.crash -FDM

[Anonymous]

Originally posted by: i... (code.google.com)@beware.dropbear.id.au

Ah, comment 13 is a little ambiguous.

[Anonymous]

Originally posted by: gijsrooy

The actual issue is still there with current Git. So positioning a YASim aircraft with speed > 0 crashes FlightGear. If I set --model-hz=240, the issue is gone, like you mentioned.

But that is of course not a real fix...

[Anonymous]

Originally posted by: bre... (code.google.com)@gmail.com

Issue 869 has been merged into this issue.

[Anonymous]

Originally posted by: i... (code.google.com)@beware.dropbear.id.au

Re: comments 11-15, setting --model-hz=240 may not be ideal, but it is a work around and a pointer to what may be a better solution. It seems to me that the iterations step is probably dependent on the model. A highly sensitive model likely needs a smaller iteration step size, so it would make sense for the default iteration step to be a per model parameter.

An better, but more difficult solution would be to make the step size dynamically adjust. There is an established family of Runge and Kutta algorithms with adaptive step size.

[Anonymous]

Originally posted by: clrc... (code.google.com)@gmail.com

Even better solution that works without needing a reworking of YASIM.

Consider leaving the IAS entry blank.

I have had great success with this tactic restarting any of the turbo props with an in-air start. I only enter the LAT, LON, and Heading in the case of turbo props and I only include IAS on jets.

-Ray