#1553 Possible memory leak tightvnc server Win10 Enterprise 2016

[BH001]

Hello,

We are noticing excessive memory consumption with long term client connections to tvncserver.exe. We observe ever increasing memory consumption at ~2GB after 2 days with ~2-3 persistent client connections.

This is on a machine running Windows 10 Enterprise LTSB 2016 Build 1607 displaying a frequently updated/changing foreground application. Please see attached screenshot.

Thank you

[Anton]

Hi!
What was the server version?

[BH001]

Hello,

My apologies. The server version was 2.8.59. On the client/viewer side, at least one connection was using 2.8.27.

[Damien Cymbal]

I believe I am possibly hitting the same issue, but this is on Windows 10 Home Edition, so I do not feel it is specific to the Windows release. What I notice appears to be a runaway thread problem which leads to the memory exhaustion.

TightVNC Server For Windows Version 2.8.59 (built Dec 17, 2020 at 12:34:01)
Windows 10 Home Version 2004 OS Build 19041.1165
Experience Windows Feature Experience Pack 120.2212.3530.0

When the server starts, it typically is running with 24-25 threads. What I notice is that st some point(s) threads seem to start being allocated wildly and not released. As an example, at this point I have 33k+ of threads allocated to tvnserver.exe. The majority of these all show the same stack:

ntoskrnl.exe!KeSynchronizeExecution+0x5b66
ntoskrnl.exe!KeWaitForSingleObject+0x1460
ntoskrnl.exe!KeWaitForSingleObject+0x98f
ntoskrnl.exe!KeWaitForSingleObject+0x233
ntoskrnl.exe!ExWaitForRundownProtectionRelease+0x7dd
ntoskrnl.exe!KeWaitForSingleObject+0x3a29
ntoskrnl.exe!KeSynchronizeExecution+0x3140
ntoskrnl.exe!LpcRequestPort+0x3a48
ntoskrnl.exe!KeSynchronizeExecution+0x67a8
ntoskrnl.exe!KeSynchronizeExecution+0x6710
ntdll.dll!RtlUserThreadStart

All these threads report Wait:Suspended

I have seen this continue to allocated threads into even higher amounts until it effectively deadlocks the Windows system with swapping until I basically need to just power it off due to response times being so slow.

My suspicion is that these bursts of allocations seem to happen when the VNC session is basically idle (and possibly when the host screen has gone to idle background screen) but I have not definitively locked down if this is the exact pattern.

I increased logging to 9, but I did not see anything obvious or tell-tale from my general skimming of the log file. The only "thread" occurrence I see if a repeated

[ 3788/ 3568] 2021-09-05 13:28:40:806   Update sender thread of client #0 is awake

I am connected from a Remmina 1.4.2 client (Ubuntu Ubuntu 20.04.3 LTS). Interestingly I use this same client to connect to a the same version tvnserver.exe running on a Windows 10 Enterprise server (Version 20H2, OS build 19042.1165, Feature Experience Pack 120.2212.3530.0) and I do not observe this behavior. I have compared the 2 server's configurations' via the UI and I do not see any differences. If there is something I can do to provide more information (debug build, additional logging, etc.) let me know and I will try to provide.

NOTE: disconnected the client from my 33k thread tvnserver.exe and it immediately dropped down to 14 threads.

[Anton]

Hi,
could you look for crash.dmp file in
C:\Windows\System32\config\systemprofile\AppData\Roaming\TightVNC\ ?
and test the version https://www.tightvnc.com/download/2.8.63/tightvnc-2.8.63-gpl-setup-64bit.msi

[Damien Cymbal]

So far my testing with the 2.8.63 release has shown good results. I have not observed the runaway thread / OOM issues with this and they were fairly easy to reproduce with the previous version.

[Damien Cymbal]

I let it run overnight and the system did bluescreen. This time I did not get a dump in TightVNC ( have seen them at times generated, and I typically clear them out because they are so large). System did generate a MEMORY.DMP, it looks like one of the DX drivers bombed based on a procexp64 query (running to monitor thread/mem of tvnserver.exe). Possibly due to resource constraints?

I will try the version listed above and see if I can repro and will keep an eye out for a TVNC specific dump file.

P.S. I also updated my Windows system to the latest build and applied all updates prior to this run: Windows 10 Home Version 21H1 OS build 19043.1202
Experience Windows Feature Experience Pack 120.2212.3530.0

||0:0: kd> !analyze -v
************************************************************

*                        Bugcheck Analysis                    ************************************************************

SYSTEM_SERVICE_EXCEPTION (3b)
An exception happened while executing a system service routine.
Arguments:
Arg1: 00000000c0000005, Exception code that caused the bugcheck
Arg2: fffff8072eca2b70, Address of the instruction which caused the bugcheck
Arg3: ffffef8db48cead0, Address of the context record for the exception that caused the bugcheck
Arg4: 0000000000000000, zero.

Debugging Details:
------------------
KEY_VALUES_STRING: 1
    Key  : Analysis.CPU.Sec
    Value: 3
    Key  : Analysis.DebugAnalysisProvider.CPP
    Value: Create: 8007007e on HP-ALL-IN-ONE
    Key  : Analysis.DebugData
    Value: CreateObject
    Key  : Analysis.DebugModel
    Value: CreateObject
    Key  : Analysis.Elapsed.Sec
    Value: 3
    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 74
    Key  : Analysis.System
    Value: CreateObject

BUGCHECK_CODE:  3b
BUGCHECK_P1: c0000005
BUGCHECK_P2: fffff8072eca2b70
BUGCHECK_P3: ffffef8db48cead0
BUGCHECK_P4: 0

CONTEXT:  ffffef8db48cead0 -- (.cxr 0xffffef8db48cead0)
rax=ffff968e22ed91a0 rbx=ffff968e232c5050 rcx=0000000000000000
rdx=0000000000000000 rsi=0000000000000007 rdi=ffff968e1839af30
rip=fffff8072eca2b70 rsp=ffffef8db48cf4d0 rbp=ffff968e1f10e000
 r8=0000000000000000  r9=ffffef8db48cf638 r10=0000000000000007
r11=ffffef8db48cf560 r12=ffff968e1f10c000 r13=0000000000000006
r14=ffffc20d4e9edde0 r15=0000000000000080
iopl=0         nv up ei pl nz na pe nc
cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00050202
dxgmms2!VidSchQueryProcessNodeStatistics+0x30:
fffff807`2eca2b70 48833a00        cmp     qword ptr [rdx],0 ds:002b:00000000`00000000=????????????????
Resetting default scope

BLACKBOXBSD: 1 (!blackboxbsd)
BLACKBOXNTFS: 1 (!blackboxntfs)
BLACKBOXPNP: 1 (!blackboxpnp)
BLACKBOXWINLOGON: 1

PROCESS_NAME:  procexp64.exe

STACK_TEXT:  
ffffef8d`b48cf4d0 fffff807`2dafe59d : ffff968e`00000000 ffff968e`23290118 ffff968e`23290001 ffffef8d`b48cf970 : dxgmms2!VidSchQueryProcessNodeStatistics+0x30
ffffef8d`b48cf500 fffff807`2dce26a5 : 00000000`00000000 ffff968e`1f10c000 ffffc20d`4e9edde0 ffffc20d`00000001 : dxgkrnl!VIDSCH_EXPORT::VidSchQueryProcessNodeStatistics+0x75
ffffef8d`b48cf540 fffff807`2dce5ee6 : ffff968e`23290080 ffffef8d`b48cfa80 ffff968e`1f10c000 00000000`00000000 : dxgkrnl!QueryProcessStatistics+0x1f5
ffffef8d`b48cf580 fffff807`2dce5a6b : 000001a3`400aba80 00000000`00000080 00000000`00000020 00000000`00000000 : dxgkrnl!DxgkQueryStatisticsInternal+0x46e
ffffef8d`b48cf9d0 fffff807`12a08bb5 : 00000000`00000000 00000000`00000001 00000000`00000640 ffffef8d`b48cfa80 : dxgkrnl!DxgkQueryStatistics+0xb
ffffef8d`b48cfa00 00007ffd`b81e5604 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x25
0000007b`d71fab68 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ffd`b81e5604

SYMBOL_NAME:  dxgmms2!VidSchQueryProcessNodeStatistics+30
MODULE_NAME: dxgmms2
IMAGE_NAME:  dxgmms2.sys
IMAGE_VERSION:  10.0.19041.1202
STACK_COMMAND:  .cxr 0xffffef8db48cead0 ; kb
BUCKET_ID_FUNC_OFFSET:  30
FAILURE_BUCKET_ID:  0x3B_c0000005_dxgmms2!VidSchQueryProcessNodeStatistics
OS_VERSION:  10.0.19041.1
BUILDLAB_STR:  vb_release
OSPLATFORM_TYPE:  x64
OSNAME:  Windows 10
FAILURE_ID_HASH:  {05101c9a-cae4-5c97-3c67-545ae2196081}
Followup:     MachineOwner
---------