Re: [ipmiutil-developers] Win64: heap leak crashing ipmiutil 3.13

[Rob S. <rob...@nu...>]

Using “ipmiutil fru -x” I get a 100% reproduction rate. The debug log of one attempt is attached. All five attempts I did are the same: they stop at:

...
get_sysinfo(1,2) j=2 len=15
ipmi_cmdraw_ms(cmd=59,netfn=6,lun=0,sa=20,sdata=4) RequestResponse ret=0
ipmi_cmdraw_ms: req data(4): 00 01 03 00
ipmi_cmdraw_ms: CompletionCode 80 returned
ipmi_cmdraw_ms: resp data(2): 80 d0
ccode 80: Invalid Session Handle or Empty Buffer

\Rob

From: Andy Cress <and...@gm...>
Date: Wednesday, 10 July 2019 at 03:53
To: Rob Scheepens <rob...@nu...>
Cc: "ipm...@li..." <ipm...@li...>, Abhijit Sunil Betigeri <abh...@nu...>, Anupam Chakraborty <anu...@nu...>, Naga Chandana <nag...@nu...>
Subject: Re: [ipmiutil-developers] Win64: heap leak crashing ipmiutil 3.13


Sure the source is available.  Here is the link for that version.
  http://sourceforge.net/projects/ipmiutil/files/ipmiutil-3.1.3.tar.gz [sourceforge.net]<https://urldefense.proofpoint.com/v2/url?u=http-3A__sourceforge.net_projects_ipmiutil_files_ipmiutil-2D3.1.3.tar.gz&d=DwMFaQ&c=s883GpUCOChKOHiocYtGcg&r=OMged-t_5I_fmfpUaT3vaA06lgLL_alYnDQJxHmXz64&m=DIsjsjlFIOJRbicayrSFqCCQVzTlqRjaQ04bd8WjicM&s=46h52Eb8jp-lu4UCCD6qH7HnvJnVOWctoQ20aQcztPQ&e=>

One other clue would be to run this command on the system where the dumps occur:
   ipmiutil fru -x
and send me the (debug) output.
The output would show a good bit of where it fails, if it fails frequently.

Andy


On Mon, Jul 8, 2019 at 7:04 AM Rob Scheepens <rob...@nu...<mailto:rob...@nu...>> wrote:
Hi Andy,

The commandline is “'"C:\Program Files\sourceforge\ipmiutil\ipmiutil.exe" fru' “. Reproduction is fairly reliable, see timestamps of the user dumps:

07/05/2019  12:38 AM        24,044,684 ipmiutil.exe-dumps.zip
07/08/2019  01:55 AM        65,382,554 ipmiutil.exe.10568.dmp
07/07/2019  11:55 PM        65,374,160 ipmiutil.exe.11612.dmp
07/07/2019  10:56 PM        65,370,628 ipmiutil.exe.13348.dmp
07/08/2019  02:56 AM        65,386,532 ipmiutil.exe.14328.dmp
07/08/2019  12:55 AM        65,378,874 ipmiutil.exe.15160.dmp
07/08/2019  01:57 AM        65,408,636 ipmiutil.exe.9272.dmp

Instead of PDBs, can I get the source code somewhere and line it up in WinDbg? I’ve uploaded two dumps to https://we.tl/t-FGq9I8scLA [we.tl]<https://urldefense.proofpoint.com/v2/url?u=https-3A__we.tl_t-2DFGq9I8scLA&d=DwMFaQ&c=s883GpUCOChKOHiocYtGcg&r=OMged-t_5I_fmfpUaT3vaA06lgLL_alYnDQJxHmXz64&m=DIsjsjlFIOJRbicayrSFqCCQVzTlqRjaQ04bd8WjicM&s=n89zN5bGfGYqXi2WHJ2X2TXmMQ2QiJashFzYCJx8lSs&e=>.

Iirc we recently switched from imbdrv to ipmidrv because of an issue. @Abhijit: can you (dis)confirm?

\Rob

From: Andy Cress <and...@gm...<mailto:and...@gm...>>
Date: Saturday, 6 July 2019 at 14:27
To: Rob Scheepens <rob...@nu...<mailto:rob...@nu...>>
Cc: "ipm...@li...<mailto:ipm...@li...>" <ipm...@li...<mailto:ipm...@li...>>, Abhijit Sunil Betigeri <abh...@nu...<mailto:abh...@nu...>>, Anupam Chakraborty <anu...@nu...<mailto:anu...@nu...>>, Naga Chandana <nag...@nu...<mailto:nag...@nu...>>
Subject: Re: [ipmiutil-developers] Win64: heap leak crashing ipmiutil 3.13

Rob,

Unfortunately, I haven't been saving the pdb files for each version.
My first guess as to the cause would be the MS Wbem layer with IPMIDRV.SYS.  It wouldn't be the first bug in that combination.
It probably would not happen with the IMBDRV.SYS instead.

However, we need to debug this further.
First, can you tell me which ipmiutil function was being used when this occurred?
Is it random, or is it reproducable?

Andy


On Fri, Jul 5, 2019 at 5:04 AM Rob Scheepens <rob...@nu...<mailto:rob...@nu...>> wrote:
Hello All,

Recently I encountered ipmiutil winx64 crashes on Windows Server 2019. I am yet to figure out what the trigger is, but after enabling application verifier I got the following stack:

0:000> !heap -p -a 0xd02fec0
    address 000000000d02fec0 found in
    _DPH_HEAP_ROOT @ 1c01000
    in busy allocation (  DPH_HEAP_BLOCK:         UserAddr         UserSize -         VirtAddr         VirtSize)
                                 bf67d68:          d02fec0              140 -          d02f000             2000
          fastprox!CEnumProxyBuffer::`vftable'
    00007ffc37f56cf7 ntdll!RtlDebugAllocateHeap+0x000000000000003f
    00007ffc37efca9e ntdll!RtlpAllocateHeap+0x000000000009d23e
    00007ffc37e5da21 ntdll!RtlpAllocateHeapInternal+0x0000000000000991
    00007ffc1d98be42 vrfcore!VfCoreRtlAllocateHeap+0x0000000000000022
    00007ffc189587c0 vfbasics!AVrfpRtlAllocateHeap+0x0000000000000130
    00007ffc2787d3aa fastprox!CEnumFactoryBuffer::XEnumFactory::CreateProxy+0x000000000000007a
    00007ffc36d9114f combase!CStdMarshal::CreateProxy+0x000000000000019f [onecore\com\combase\dcomrem\marshal.cxx @ 6551]
    00007ffc36d93b69 combase!CStdMarshal::MakeCliIPIDEntry+0x0000000000000069 [onecore\com\combase\dcomrem\marshal.cxx @ 2840]
    00007ffc36d944ea combase!CStdMarshal::UnmarshalIPID+0x000000000000007a [onecore\com\combase\dcomrem\marshal.cxx @ 2404]
    00007ffc36d97500 combase!CStdMarshal::UnmarshalObjRef+0x0000000000000170 [onecore\com\combase\dcomrem\marshal.cxx @ 2272]
    00007ffc36d8a7e3 combase!CoUnmarshalInterface+0x0000000000000483 [onecore\com\combase\dcomrem\coapi.cxx @ 1931]
    00007ffc36deb563 combase!NdrExtInterfacePointerUnmarshall+0x00000000000001b3 [onecore\com\combase\ndr\ndrole\oleaux.cxx @ 1244]
    00007ffc37553c54 rpcrt4!NdrPointerUnmarshall+0x0000000000000284
    00007ffc37553cc6 rpcrt4!NdrPointerUnmarshall+0x00000000000002f6
    00007ffc37558cb3 rpcrt4!NdrpClientUnMarshal+0x0000000000000433
    00007ffc37511ea5 rpcrt4!NdrpClientCall2+0x0000000000000475
    00007ffc36deaa87 combase!ObjectStublessClient+0x00000000000001d7 [onecore\com\combase\ndr\ndrole\amd64\stblsclt.cxx @ 368]
    00007ffc36e5c7b2 combase!ObjectStubless+0x0000000000000042 [onecore\com\combase\ndr\ndrole\amd64\stubless.asm @ 176]
    00007ffc27876171 fastprox!CWbemSvcWrapper::XWbemServices::CreateInstanceEnum+0x0000000000000091
    0000000140061ae1 ipmiutil+0x0000000000061ae1
    00000001400621b1 ipmiutil+0x00000000000621b1
    000000014000b9ee ipmiutil+0x000000000000b9ee
    0000000140001144 ipmiutil+0x0000000000001144
    000000014006f40b ipmiutil+0x000000000006f40b
    00007ffc35217974 kernel32!BaseThreadInitThunk+0x0000000000000014
    00007ffc37eba271 ntdll!RtlUserThreadStart+0x0000000000000021

Image info:

    Loaded symbol image file: ipmiutil.exe
    Image path: C:\Program Files\sourceforge\ipmiutil\ipmiutil.exe
    Image name: ipmiutil.exe
    Browse all global symbols  functions  data
    Timestamp:        Thu Sep 13 09:43:24 2018 (5B9A93AC)
    CheckSum:         00000000
    ImageSize:        00156000

Are there (private) PDB files available for this version of ipmiutil?

\Rob

_______________________________________________
ipmiutil-developers mailing list
ipm...@li...<mailto:ipm...@li...>
https://lists.sourceforge.net/lists/listinfo/ipmiutil-developers [lists.sourceforge.net]<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_ipmiutil-2Ddevelopers&d=DwMFaQ&c=s883GpUCOChKOHiocYtGcg&r=OMged-t_5I_fmfpUaT3vaA06lgLL_alYnDQJxHmXz64&m=0uA8_RO-wJLFP7eq8M8A0oBuHRUvGSaTJhQ21-gXp8o&s=25IBWbV6u_-JYOLfV01hQa-o1e_XdBnKNKqRW-_PNoM&e=>