#21 opennopd will start consuming 100% of CPU core after issuing show sessions

[Levente Peres]

Description:

When I open the CLI and type show sessions, the table will stop drawing after about 2-3 rows and you get no prompt, only CTRLC will exit the CLI. After that, you can see in 'top' that some of the CPU cores will be 100% utilized continously by opennopd.

How to reproduce:

Not sure. Build a network with NAT-ed, Masqueraded connections going through NOPs, and have about 10 pepole connect... that's how I got it.

Debug info:

None I could collect.

[Justin Yaple]

Does this just randomly happen?

[Levente Peres]

Yeah looks like it... OR it COULD happen when there're lots of masqueraded connections going on.

Anyway just now I did a test with only single client doing anything sensible and it didn't happen

tcpbp-ac:~ # opennop
opennopd# show sessions
--------------------------------------------------------------------------------------
|  Index  |   Client IP    | Client Port |    Server IP   | Server Port | Optimizing |
--------------------------------------------------------------------------------------
|  23437  | 10.20.5.2      |   139       | 10.30.1.10     |   36048     |     Yes    |
|  53451  | 10.20.5.2      |   139       | 10.30.1.10     |   51803     |     Yes    |

Last time this happened there were - don't remember exactly but - at least 10 NAT-ed, masqueraded client machines going on about their business,

EDIT : actually that's weird... where's port 445?
EDIT2: ah well maybe clients don't always use it... I'm forwarding a whole array of legacy Samba ports...

[Levente Peres]

Ooops got a bad one...

It even died.

opennopd# show sessions
--------------------------------------------------------------------------------------
|  Index  |   Client IP    | Client Port |    Server IP   | Server Port | Optimizing |
--------------------------------------------------------------------------------------

/var/log/messages says:

Feb 28 19:17:33 tcpbp-ac opennopd: CLI: Current command [sessions].
Feb 28 19:17:33 tcpbp-ac opennopd: CLI: Command has no parameters.
Feb 28 19:17:33 tcpbp-ac opennopd: sessionmanager.c Larger ID:
Feb 28 19:17:33 tcpbp-ac opennopd: Binary Dump:
D4000D18 | 56CA5E2C                             | V.^,
Feb 28 19:17:33 tcpbp-ac opennopd: sessionmanager.c Smaller ID:
Feb 28 19:17:33 tcpbp-ac opennopd: Binary Dump:
D4000D2C | 87624101                             | .bA.
Feb 28 19:17:33 tcpbp-ac opennopd: sessionmanager.c Larger ID:
Feb 28 19:17:33 tcpbp-ac opennopd: Binary Dump:
D4000DF8 | 56CA5E2C                             | V.^,
Feb 28 19:17:33 tcpbp-ac opennopd: sessionmanager.c Smaller ID:
Feb 28 19:17:33 tcpbp-ac opennopd: Binary Dump:
D4000E0C | 87624101                             | .bA.
Feb 28 19:17:45 tcpbp-ac opennopd: CLI: Begin processing a command.
Feb 28 19:17:45 tcpbp-ac opennopd: CLI: Locating [exit] command.
Feb 28 19:17:45 tcpbp-ac opennopd: CLI: Current command [exit].
Feb 28 19:17:45 tcpbp-ac opennopd: CLI: Command has no parameters.
Feb 28 19:17:45 tcpbp-ac opennopd: Closing cli connection.

[Justin Yaple]

Yah. I think that the "show sessions" function is going to need a total rewrite. It has way to many caveats.

It only shows sessions if it knows what one is the client & server. There could be other sessions being tracked and they just won't appear.

I'm not tracking usage of the session records so while one process is reading the session another could be closing it. This might be why the "show sessions" command sometimes locks, stalls, crashes, freezes or just doesn't work correctly.

Its just some quick speculation right now. There does seem to be some problem with that particular function though.

[Levente Peres]

Ok. I'll just avoid that particular command for the time being then... no big deal right now...

[Justin Yaple]

I just fixed some bugs in the session manager that could have been causing this. This is only fixed in my feature branch right now.

[dec372]