#2681 AI wake turbolence seems to invalidate the FDM (segfault/abort)

[Benedikt Hallinger]

Hi there,
when flying with AI and wake turbolence generation turned on (5 miles radius), my flightgear crashes after a while. More specific it seems to crash, when I enter regions with turbolence.
This happens with the C172, but also with an experimental branch of the C182s (where I just added the JSBSim external forces definitions!), so I highly suspect this is a core bug, and not aircraft sided.

To reproduce:
- Start at keflavik and enable AI and wake turbolence generation.
- Let the AI start up
- Fly the C172 behind a jet

Now I get, seemingly random distributed, two cases (but one of them triggers reliably); either i get the Error "FDM has become invalid", or a plain segfault.

I recompiled next a few days ago.

[James Turner]

We see some of these crashes on Sentry, sometimes the matrix used to compute the turbulence fails to solve.

[Wayne Bragg]

It, seems to be failing more consistently as far as I can tell.
Has anyone noticed the effect actually working when in the wake?

[Wayne Bragg]

I have verified that in the case of the c172p. it crashes the FDM the moment the aircraft is exposed to the wake of the AI aircraft. Something has failed completely. A fairly easy test is to use KLAX R 24L and wait to the AI craft to taxi onto the runway and as soon as it passes and I think gets the onGround() flag cleared it crashed.

[Wayne Bragg]

After a little more testing, at least with the c172 and c182 the AI wake is broken, it crashes the FDM as soon as the conditions are met that it is active.

Does anyone know which aircraft this was originally written for that used to work so I can test it as well?

[Wayne Bragg]

I tried a totally unrelated aircraft, Saab 37, it crashed as well.

[peridotite]

Going back into older LTS territory (As far back as 2020.3.5 so far) I can get this to crash fairly predictably:

• Startup at KSFO in the C172 with morning setting, AI aircraft & wake turned on
• Takeoff & fly, an airliner will spawn & make its way to the end of the runway (JetBlue from memory)
• Line up behind it while it takes off
• FGFS will crash (Usually FDM invalid on the C172) as soon as it leaves the ground
• lots of NaNs in the logs

I have a copy of Bertrands hacking/yasim-wake-turb branch which was used to add yasim support in the hackathon from over a year ago - This doesn't crash with either the C172 or any Yasim aircraft (Katana tested). I have a vague memory of Bertrand fixing stuff on the fly

I'm not sure this branch will merge cleanly anymore due to some changes done in AIBase to fix vs (a0cb70688b84e1426bc8732f3add1ae44835c835 I think). In any case it also needs some additional changes to the yasim tests which were disabled due to not compiling at the time

[Anonymous]

2020.3.13.

1. Set AI wake radius to a huge value
2. Try to catch up with some AI craft

SEGFAULT:

#0  0x00005555561c0b0f in nr_gaussj ()
#1  0x00005555561bdfe4 in WakeMesh::WakeMesh(double, double) ()
#2  0x00005555561bfcf0 in AIWakeGroup::AddAI(FGAIAircraft*) ()
#3  0x000055555618fa56 in FDMShell::update(double) ()
#4  0x00005555565ef9fb in SGSubsystemGroup::Member::update (this=0x5555578d71a0, delta_time_sec=<optimized out>) at /home/mike/src/git/flightgear/simgear/simgear/structure/subsystem_mgr.cxx:838
#5  0x00005555565f559f in SGSubsystemGroup::update (this=0x5555570513d0, delta_time_sec=0.0083333333333333332) at /home/mike/src/git/flightgear/simgear/simgear/structure/subsystem_mgr.cxx:422
#6  0x00005555565eb62a in SGSubsystemMgr::update (this=0x555557051010, delta_time_sec=0.36666666666666664) at /home/mike/src/git/flightgear/simgear/simgear/structure/subsystem_mgr.cxx:990
#7  0x0000555556368df9 in fgMainLoop() ()
#8  0x00005555562c2599 in fgOSMainLoop() ()
#9  0x000055555636cea7 in fgMainInit(int, char**) ()
#10 0x0000555555b8b229 in main ()
#0  0x00005555561c0b0f in nr_gaussj ()
#1  0x00005555561bdfe4 in WakeMesh::WakeMesh(double, double) ()
#2  0x00005555561bfcf0 in AIWakeGroup::AddAI(FGAIAircraft*) ()
#3  0x000055555618fa56 in FDMShell::update(double) ()
#4  0x00005555565ef9fb in SGSubsystemGroup::Member::update (this=0x5555578d71a0, delta_time_sec=<optimized out>) at /home/mike/src/git/flightgear/simgear/simgear/structure/subsystem_mgr.cxx:838
#5  0x00005555565f559f in SGSubsystemGroup::update (this=0x5555570513d0, delta_time_sec=0.0083333333333333332) at /home/mike/src/git/flightgear/simgear/simgear/structure/subsystem_mgr.cxx:422
#6  0x00005555565eb62a in SGSubsystemMgr::update (this=0x555557051010, delta_time_sec=0.36666666666666664) at /home/mike/src/git/flightgear/simgear/simgear/structure/subsystem_mgr.cxx:990
#7  0x0000555556368df9 in fgMainLoop() ()
#8  0x00005555562c2599 in fgOSMainLoop() ()
#9  0x000055555636cea7 in fgMainInit(int, char**) ()
#10 0x0000555555b8b229 in main ()

[James Turner]

Pinging BConni becuase this is his code I think. I also have many crashes on Sentry, it's always in nr_gaussj with an invalid matrix. But as with this one, I can't see /why/ the matrix is invalid, so far.

[Bertrand Coconnier]

Most likely the aircraft intercepts wake vortex lines.
Any chance to get the line at which the code fails (requires to compile with debug infos) ?

[James Turner]

There's a bunch of these on the crask-reporting site but due to some other issues, hard for me to find them right now. I'll keep trying but I think it's due to a mis-sized maxtrix in gr_gaussj.

[Bertrand Coconnier]

I have just pushed a pair of commits to the next branch. Hopefully, this should avoid the crashes and report faulty aircraft to the log.
Let me know if you find something in the log about disabled wake.

[Richard Harrison]

@bcoconni

Applying to 2020.3.12 I get the following errors: I haven't tried this on next yet.

.\fgfs.log: 107.82 [WARN]:flight src\FDM\AIWake\WakeMesh.cxx:75: Failed to build wake mesh. AI: ( span:2.29179e-312, chord:3.2526e-86) wake will be ignored.
.\fgfs.log: 107.82 [ALRT]:flight src\FDM\AIWake\AIWakeGroup.cxx:54: Created mesh for ID: #9
.\fgfs.log: 132.70 [WARN]:flight src\FDM\AIWake\WakeMesh.cxx:75: Failed to build wake mesh. AI: ( span:2.29179e-312, chord:3.2526e-86) wake will be ignored.
.\fgfs.log: 132.70 [ALRT]:flight src\FDM\AIWake\AIWakeGroup.cxx:54: Created mesh for ID: #16
.\fgfs.log: 140.71 [WARN]:flight src\FDM\AIWake\WakeMesh.cxx:75: Failed to build wake mesh. AI: ( span:0, chord:0) wake will be ignored.
.\fgfs.log: 140.71 [ALRT]:flight src\FDM\AIWake\AIWakeGroup.cxx:54: Created mesh for ID: #30
.\fgfs.log: 141.75 [WARN]:flight src\FDM\AIWake\WakeMesh.cxx:75: Failed to build wake mesh. AI: ( span:0, chord:0) wake will be ignored.
.\fgfs.log: 141.75 [ALRT]:flight src\FDM\AIWake\AIWakeGroup.cxx:54: Created mesh for ID: #40
.\fgfs.log: 142.30 [WARN]:flight src\FDM\AIWake\WakeMesh.cxx:75: Failed to build wake mesh. AI: ( span:5.33591e-322, chord:0) wake will be ignored.
.\fgfs.log: 142.30 [ALRT]:flight src\FDM\AIWake\AIWakeGroup.cxx:54: Created mesh for ID: #22
.\fgfs.log: 143.34 [WARN]:flight src\FDM\AIWake\WakeMesh.cxx:75: Failed to build wake mesh. AI: ( span:2.29179e-312, chord:3.2526e-86) wake will be ignored.
.\fgfs.log: 143.34 [ALRT]:flight src\FDM\AIWake\AIWakeGroup.cxx:54: Created mesh for ID: #42
.\fgfs.log: 191.66 [WARN]:flight src\FDM\AIWake\WakeMesh.cxx:75: Failed to build wake mesh. AI: ( span:0, chord:0) wake will be ignored.
.\fgfs.log: 191.66 [ALRT]:flight src\FDM\AIWake\AIWakeGroup.cxx:54: Created mesh for ID: #73
.\fgfs.log: 191.80 [WARN]:flight src\FDM\AIWake\WakeMesh.cxx:75: Failed to build wake mesh. AI: ( span:0, chord:0) wake will be ignored.
.\fgfs.log: 191.80 [ALRT]:flight src\FDM\AIWake\AIWakeGroup.cxx:54: Created mesh for ID: #74
.\fgfs.log: 193.09 [WARN]:flight src\FDM\AIWake\WakeMesh.cxx:75: Failed to build wake mesh. AI: ( span:2.29179e-312, chord:3.2526e-86) wake will be ignored.
.\fgfs.log: 193.09 [ALRT]:flight src\FDM\AIWake\AIWakeGroup.cxx:54: Created mesh for ID: #90
.\fgfs.log: 193.23 [WARN]:flight src\FDM\AIWake\WakeMesh.cxx:75: Failed to build wake mesh. AI: ( span:2.29179e-312, chord:3.2526e-86) wake will be ignored.
.\fgfs.log: 193.23 [ALRT]:flight src\FDM\AIWake\AIWakeGroup.cxx:54: Created mesh for ID: #92
.\fgfs.log: 193.42 [WARN]:flight src\FDM\AIWake\WakeMesh.cxx:75: Failed to build wake mesh. AI: ( span:2.29179e-312, chord:3.2526e-86) wake will be ignored.
.\fgfs.log: 193.42 [ALRT]:flight src\FDM\AIWake\AIWakeGroup.cxx:54: Created mesh for ID: #94
.\fgfs.log: 193.44 [WARN]:flight src\FDM\AIWake\WakeMesh.cxx:75: Failed to build wake mesh. AI: ( span:2.29179e-312, chord:3.2526e-86) wake will be ignored.
.\fgfs.log: 193.44 [ALRT]:flight src\FDM\AIWake\AIWakeGroup.cxx:54: Created mesh for ID: #93
.\fgfs.log: 193.47 [ALRT]:flight src\FDM\AIWake\AIWakeGroup.cxx:94: Deleted mesh for aircraft #94
.\fgfs.log: 196.22 [WARN]:flight src\FDM\AIWake\WakeMesh.cxx:75: Failed to build wake mesh. AI: ( span:0, chord:0) wake will be ignored.
.\fgfs.log: 196.22 [ALRT]:flight src\FDM\AIWake\AIWakeGroup.cxx:54: Created mesh for ID: #112
.\fgfs.log: 200.37 [WARN]:flight src\FDM\AIWake\WakeMesh.cxx:75: Failed to build wake mesh. AI: ( span:0, chord:0) wake will be ignored.
.\fgfs.log: 200.37 [ALRT]:flight src\FDM\AIWake\AIWakeGroup.cxx:54: Created mesh for ID: #59
.\fgfs.log: 201.11 [WARN]:flight src\FDM\AIWake\WakeMesh.cxx:75: Failed to build wake mesh. AI: ( span:5.33591e-322, chord:0) wake will be ignored.
.\fgfs.log: 201.11 [ALRT]:flight src\FDM\AIWake\AIWakeGroup.cxx:54: Created mesh for ID: #127
.\fgfs.log: 201.23 [WARN]:flight src\FDM\AIWake\WakeMesh.cxx:75: Failed to build wake mesh. AI: ( span:5.33591e-322, chord:0) wake will be ignored.
.\fgfs.log: 201.23 [ALRT]:flight src\FDM\AIWake\AIWakeGroup.cxx:54: Created mesh for ID: #128
.\fgfs.log: 210.17 [WARN]:flight src\FDM\AIWake\WakeMesh.cxx:75: Failed to build wake mesh. AI: ( span:0, chord:0) wake will be ignored.
.\fgfs.log: 210.17 [ALRT]:flight src\FDM\AIWake\AIWakeGroup.cxx:54: Created mesh for ID: #19
.\fgfs.log: 219.84 [ALRT]:flight src\FDM\AIWake\AIWakeGroup.cxx:94: Deleted mesh for aircraft #40
.\fgfs.log: 240.70 [ALRT]:flight src\FDM\AIWake\AIWakeGroup.cxx:94: Deleted mesh for aircraft #30
.\fgfs.log: 257.73 [WARN]:flight src\FDM\AIWake\WakeMesh.cxx:75: Failed to build wake mesh. AI: ( span:2.29179e-312, chord:3.2526e-86) wake will be ignored.
.\fgfs.log: 257.73 [ALRT]:flight src\FDM\AIWake\AIWakeGroup.cxx:54: Created mesh for ID: #94
.\fgfs.log: 260.50 [WARN]:flight src\FDM\AIWake\WakeMesh.cxx:75: Failed to build wake mesh. AI: ( span:0, chord:0) wake will be ignored.
.\fgfs.log: 260.50 [ALRT]:flight src\FDM\AIWake\AIWakeGroup.cxx:54: Created mesh for ID: #146
.\fgfs.log: 260.92 [WARN]:flight src\FDM\AIWake\WakeMesh.cxx:75: Failed to build wake mesh. AI: ( span:5.33591e-322, chord:0) wake will be ignored.
.\fgfs.log: 260.92 [ALRT]:flight src\FDM\AIWake\AIWakeGroup.cxx:54: Created mesh for ID: #149
.\fgfs.log: 266.54 [WARN]:flight src\FDM\AIWake\WakeMesh.cxx:75: Failed to build wake mesh. AI: ( span:5.33591e-322, chord:0) wake will be ignored.
.\fgfs.log: 266.54 [ALRT]:flight src\FDM\AIWake\AIWakeGroup.cxx:54: Created mesh for ID: #152

this then results in all of the forces and moments that are used in JSBSim.cxx being NaN; so I changed the code to guard against this.

      SGVec3d f = mesh->GetForce(get_wake_group(),
                                 SGVec3d(uvw(1), uvw(2), uvw(3)),
                                 Atmosphere->GetDensity());
      const SGVec3d& m = mesh->GetMoment();
      if (isnan(f[0]) || isnan(f[1]) || isnan(f[2]) ||
          isnan(m[0]) || isnan(m[1]) || isnan(m[2])) {
          printf("AWI: **NAN** F:(%7.4f,%7.4f,%7.4f) M:(%7.4f,%7.4f,%7.4f)\n", f[0], f[1], f[2], m[0], m[1], m[2]);
          if (_fmag) _fmag->setDoubleValue(0.0);
          if (_mmag) _mmag->setDoubleValue(0.0);
      } else {
          if (_fmag) _fmag->setDoubleValue(1.0);
          if (_fbx) _fbx->setDoubleValue(f[0]);
          if (_fby) _fby->setDoubleValue(f[1]);
          if (_fbz) _fbz->setDoubleValue(f[2]);
          ...

[James Turner]

Thanks Bertrand, I'll take a look and let you know what I see.

[Benedikt Hallinger]

Which commits are those specifically?
Just tried with todays next and still got an invalid FDM at kevlavik

[James Turner]

@benih I might have missed something, invalid FDM is not the same as the wake turbulence crash, at least based on the reports I saw.

[Wayne Bragg]

Wow, I hadn't tried this since this ticket was created. As soon as I entered a wake the c172p stopped dead in its tracks, in the air, behind the AI. Nothing crashed, the sim continued to animate the scene including the AI aircraft but the c172p just went dead, no sound no animation just sitting there in the air not moving. Best I can tell the log at that point looked like this.

373.43 [DBUG]:sound Sound volume too large for 'engine': 1.34694 -> clipping to 1.0
373.47 [WARN]:flight Failed to build wake mesh. AI: ( span:7.566e-307, chord:9.34602e-307) wake will be ignored.
373.47 [ALRT]:flight Created mesh for ID: #67
373.47 [ALRT]:autopilot AP input: read NaN from:/instrumentation[0]/turn-indicator[0]/indicated-turn-rate[0]
373.47 [ALRT]:autopilot AP input: read NaN from:/autopilot[0]/internal[0]/predicted-turn-rate[0]
373.47 [ALRT]:autopilot AP input: read NaN from:/autopilot[0]/internal[0]/predicted-turn-rate[0]
373.47 [ALRT]:autopilot AP input: read NaN from:/systems[0]/static[0]/pressure-inhg[0]
373.47 [ALRT]:autopilot AP input: read NaN from:/systems[0]/static[0]/pressure-inhg[0]
373.47 [ALRT]:autopilot AP input: read NaN from:/autopilot[0]/internal[0]/pressure-rate-raw[0]
373.47 [ALRT]:autopilot AP input: read NaN from:/autopilot[0]/internal[0]/pressure-rate-raw[0]
373.47 [ALRT]:autopilot AP input: read NaN from expression
373.47 [ALRT]:autopilot AP input: read NaN from:/systems[0]/static[0]/pressure-inhg[0]
373.47 [ALRT]:autopilot AP input: read NaN from:/systems[0]/static[0]/pressure-inhg[0]
373.47 [ALRT]:autopilot AP input: read NaN from expression
373.47 [ALRT]:autopilot AP input: read NaN from expression
373.47 [ALRT]:autopilot AP input: read NaN from expression
373.47 [ALRT]:autopilot AP input: read NaN from expression
373.47 [ALRT]:autopilot AP input: read NaN from:/autopilot[0]/internal[0]/g-damped[0]
373.47 [ALRT]:autopilot AP input: read NaN from:/autopilot[0]/internal[0]/g-damped[0]
373.47 [ALRT]:autopilot AP input: read NaN from:/autopilot[0]/internal[0]/pressure-rate[0]
373.47 [ALRT]:autopilot AP input: read NaN from:/autopilot[0]/internal[0]/pressure-rate[0]
373.47 [ALRT]:general caught exception processing subsystem:flight
message:FGGroundCache::get_agl: NaN position input
373.47 [ALRT]:autopilot AP input: read NaN from:/instrumentation[0]/turn-indicator[0]/indicated-turn-rate[0]
373.47 [ALRT]:autopilot AP input: read NaN from:/autopilot[0]/internal[0]/predicted-turn-rate[0]
373.47 [ALRT]:autopilot AP input: read NaN from:/autopilot[0]/internal[0]/predicted-turn-rate[0]
373.47 [ALRT]:autopilot AP input: read NaN from:/systems[0]/static[0]/pressure-inhg[0]
373.47 [ALRT]:autopilot AP input: read NaN from:/systems[0]/static[0]/pressure-inhg[0]
373.47 [ALRT]:autopilot AP input: read NaN from:/autopilot[0]/internal[0]/pressure-rate-raw[0]
373.47 [ALRT]:autopilot AP input: read NaN from:/autopilot[0]/internal[0]/pressure-rate-raw[0]
373.47 [ALRT]:autopilot AP input: read NaN from expression

[Wayne Bragg]

This is not a crash, I was in the debugger when this happened and it is not causing any exceptions. The c172p FDM just stops!

[Wayne Bragg]

One more. I added AI wake to the J3Cub to see if it did anything different and it produced the same phenomena.

134.91 [WARN]:flight Failed to build wake mesh. AI: ( span:0, chord:0) wake will be ignored.
134.91 [ALRT]:flight Created mesh for ID: #76
134.93 [ALRT]:autopilot AP input: read NaN from:/instrumentation[0]/turn-indicator[0]/indicated-turn-rate[0]
134.93 [ALRT]:autopilot AP input: read NaN from:/it-stec55x[0]/internal[0]/predicted-turn-rate[0]
134.93 [ALRT]:autopilot AP input: read NaN from:/it-stec55x[0]/internal[0]/predicted-turn-rate[0]
134.93 [ALRT]:autopilot AP input: read NaN from:/systems[0]/static[0]/pressure-inhg[0]
134.93 [ALRT]:autopilot AP input: read NaN from:/systems[0]/static[0]/pressure-inhg[0]
134.93 [ALRT]:autopilot AP input: read NaN from:/it-stec55x[0]/internal[0]/pressure-rate-raw[0]
134.93 [ALRT]:autopilot AP input: read NaN from:/it-stec55x[0]/internal[0]/pressure-rate-raw[0]
134.93 [ALRT]:autopilot AP input: read NaN from expression
134.93 [ALRT]:autopilot AP input: read NaN from expression
134.93 [ALRT]:autopilot AP input: read NaN from expression
134.93 [ALRT]:autopilot AP input: read NaN from expression
134.93 [ALRT]:autopilot AP input: read NaN from expression
134.93 [ALRT]:autopilot AP input: read NaN from:/systems[0]/static[0]/pressure-inhg[0]
134.93 [ALRT]:autopilot AP input: read NaN from:/systems[0]/static[0]/pressure-inhg[0]
134.93 [ALRT]:autopilot AP input: read NaN from expression
134.93 [ALRT]:autopilot AP input: read NaN from expression
134.93 [ALRT]:autopilot AP input: read NaN from:/it-stec55x[0]/input[0]/vs[0]
134.93 [ALRT]:autopilot AP input: read NaN from:/it-stec55x[0]/input[0]/vs[0]
134.93 [ALRT]:autopilot AP input: read NaN from:/it-stec55x[0]/internal[0]/vs[0]
134.93 [ALRT]:autopilot AP input: read NaN from:/it-stec55x[0]/internal[0]/vs[0]
134.93 [ALRT]:autopilot AP input: read NaN from expression
134.93 [ALRT]:autopilot AP input: read NaN from expression
134.93 [ALRT]:autopilot AP input: read NaN from:/it-stec55x[0]/internal[0]/g-damped[0]
134.93 [ALRT]:autopilot AP input: read NaN from:/it-stec55x[0]/internal[0]/g-damped[0]
134.93 [ALRT]:autopilot AP input: read NaN from:/orientation[0]/r-body[0]
134.93 [ALRT]:autopilot AP input: read NaN from:/orientation[0]/r-body[0]
134.93 [ALRT]:autopilot AP input: read NaN from:/it-stec55x[0]/internal[0]/yaw-rate-low[0]
134.93 [ALRT]:autopilot AP input: read NaN from:/it-stec55x[0]/internal[0]/yaw-rate-low[0]
134.93 [ALRT]:autopilot High pass filter output is NaN.
134.93 [ALRT]:autopilot High pass filter output is NaN.
134.93 [ALRT]:general caught exception processing subsystem:flight
message:FGGroundCache::get_agl: NaN position input
134.93 [ALRT]:autopilot AP input: read NaN from:/instrumentation[0]/turn-indicator[0]/indicated-turn-rate[0]
134.93 [ALRT]:autopilot AP input: read NaN from:/it-stec55x[0]/internal[0]/predicted-turn-rate[0]
134.93 [ALRT]:autopilot AP input: read NaN from:/it-stec55x[0]/internal[0]/predicted-turn-rate[0]
134.93 [ALRT]:autopilot AP input: read NaN from:/systems[0]/static[0]/pressure-inhg[0]
134.93 [ALRT]:autopilot AP input: read NaN from:/systems[0]/static[0]/pressure-inhg[0]

[Benedikt Hallinger]

...Is it intentional this ticket is marked as "fixed" when the last comment still shows errors...?

[Wayne Bragg]

I would like the opportunity to test any fixes to the code before marking this as solved. When was code changed or fixed? I may need to pull the appropriate code base and rebuild to test.

[Benedikt Hallinger]

When was code changed or fixed

no idea if any

[Benedikt Hallinger]

Tried to reproduce. I got no crash, but also no turbulence, despite the box beeing ticket.