Menu
▾
▴
Re: [Capstone-users] Memory Leak?
|
From: Jurriaan B. <jur...@gm...> - 2014-12-04 13:04:16
|
I was unaware of this approach - nice! +1 for using this as a patch ;) Jurriaan On 12/04/2014 02:01 PM, Jan Newger wrote: > I think the easiest fix would be to free the memory from a finally > clause, as described here: > > https://docs.python.org/2/reference/simple_stmts.html#the-yield-statement > > Like this (untested): > > def disasm(self, code, offset, count=0): > all_insn = ctypes.POINTER(_cs_insn)() > '''if not _python2: > print(code) > code = code.encode() > print(code)''' > res = _cs.cs_disasm(self.csh, code, len(code), offset, count, > ctypes.byref(all_insn)) > if res > 0: > try: > for i in range(res): > yield CsInsn(self, all_insn[i]) > finally: > _cs.cs_free(all_insn, res) > else: > status = _cs.cs_errno(self.csh) > if status != CS_ERR_OK: > raise CsError(status) > return > yield > > On 12/04/2014 01:07 AM, Jasiel Spelman wrote: >> Nguyen, >> >> Here is example code that hopefully better illustrates Juriaan's >> point/what Jan is seeing: >> >> def gen_example(): >> print '[gen_example] START' >> for i in xrange(2): >> print '[gen_example] before yield: ', i >> yield i >> print '[gen_example] after yield: ', i >> print '[gen_example] END' >> >> print "Typical case" >> for i in gen_example(): >> print >> >> print "Jan's case" >> for i in gen_example(): >> if i == 1: >> break >> print >> >> >> Typical case >> [gen_example] START >> [gen_example] before yield: 0 >> >> [gen_example] after yield: 0 >> [gen_example] before yield: 1 >> >> [gen_example] after yield: 1 >> [gen_example] END >> >> >> Jan's case >> [gen_example] START >> [gen_example] before yield: 0 >> >> [gen_example] after yield: 0 >> [gen_example] before yield: 1 >> >> >> On Wed, Dec 3, 2014 at 5:49 PM, Nguyen Anh Quynh <aq...@gm... >> <mailto:aq...@gm...>> wrote: >> >> >> >> On Thu, Dec 4, 2014 at 6:32 AM, Jurriaan Bremer >> <jur...@gm... <mailto:jur...@gm...>> wrote: >> >> The 'yield' keyword allows one to fetch new records/rows (in >> this case >> disassembled instructions) on-demand. Doing an early exit (e.g., >> quitting from a for-loop iterating over a function that yield's >> after >> only, say, 2 out of 10 items) will prematurely exit the function as >> well. Because, of course, why would Python calculate the latter 8 >> results when they're not used in the first place? This is also >> useful >> for never-ending functions - I suppose calculating digits of Pi >> would be >> a 'good' example. >> >> Anyway, so your cs_free() call is never reached in this case - you >> should switch to cs_free()'ing every row after each iteration >> through >> the for loop. >> >> >> but the loop is called upon the number of successfully disassembled >> instructions, >> so there is no where in the code that might possibly quit the loop >> prematurely. >> >> thanks, >> >> Q >> >> On 12/04/2014 12:27 AM, Nguyen Anh Quynh wrote: >> > >> > >> > On Thu, Dec 4, 2014 at 4:42 AM, Jan Newger <jan...@ne... <mailto:jan...@ne...> >> > <mailto:jan...@ne... <mailto:jan...@ne...>>> wrote: >> > >> > This is the python implementation of the disasm function >> (starting at >> > line 791): >> > >> > def disasm(self, code, offset, count=0): >> > all_insn = ctypes.POINTER(_cs_insn)() >> > '''if not _python2: >> > print(code) >> > code = code.encode() >> > print(code)''' >> > res = _cs.cs_disasm(self.csh, code, len(code), >> offset, count, >> > ctypes.byref(all_insn)) >> > if res > 0: >> > for i in range(res): >> > yield CsInsn(self, all_insn[i]) >> > _cs.cs_free(all_insn, res) >> > else: >> > status = _cs.cs_errno(self.csh) >> > if status != CS_ERR_OK: >> > raise CsError(status) >> > return >> > yield >> > >> > I'm really no python expert, but from what I see you >> apparently need to >> > free the instruction instances manually. However, if client >> code stops >> > enumeration over the instructions prematurely, then >> _cs.cs_free() is >> > never invoked, and thus memory is leaked, right? >> > >> > >> > yes the problem must be with Python binding but not the core. >> > however, in the above code, cs_free() is called after the "for" loop, >> > so i dont see how memleak can happen "prematurely". >> > >> > >> > thanks. >> > >> > >> > >> > >> > >> > On 03.12.2014 22:25, Jan Newger wrote: >> > > It seems the equivalent C implementation is not affected by >> the mem >> > > leak, which is to be expected, since the memory is >> explicitly freed >> > > anyways, and the group checking boils down to comparing an >> integer >> > value. >> > > >> > > If I had to guess, I'd suspect that in the python case the >> group >> > > checking code introduces a spurious reference to the >> instruction >> > > instance(?) which cannot be claimed by the GC. >> > > >> > > On 03.12.2014 16:57, Jan Newger wrote: >> > >> No I haven't tried to reproduce the mem leak with C. >> > >> It already took me a considerable amount of time to come >> up with this >> > >> minimal example. >> > >> >> > >> On 12/03/2014 04:53 PM, Capstone Engine wrote: >> > >>> >> > >>> >> > >>> On Wed, Dec 3, 2014 at 10:32 PM, Jan Newger >> > <jan...@ne... <mailto:jan...@ne...> >> <mailto:jan...@ne... <mailto:jan...@ne...>> >> > >>> <mailto:jan...@ne... >> <mailto:jan...@ne...> <mailto:jan...@ne... >> <mailto:jan...@ne...>>>> >> > wrote: >> > >>> >> > >>> >> > >>> Yes, it's using the latest version. >> > >>> OS was win7 x64 running python 2.7 with 32bit libraries. >> > >>> >> > >>> >> > >>> this is interesting. have you tried to code the same >> program in >> > C to see >> > >>> if the mem leak issue still happens? >> > >>> >> > >>> >> > >>> thanks. >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> On 12/03/2014 04:31 PM, Nguyen Anh Quynh wrote: >> > >>> > >> > >>> > >> > >>> > On Wed, Dec 3, 2014 at 4:57 PM, Jan Newger >> > <jan...@ne... <mailto:jan...@ne...> >> <mailto:jan...@ne... <mailto:jan...@ne...>> >> > <mailto:jan...@ne... <mailto:jan...@ne...> >> <mailto:jan...@ne... <mailto:jan...@ne...>>> >> > >>> > <mailto:jan...@ne... >> <mailto:jan...@ne...> >> > <mailto:jan...@ne... <mailto:jan...@ne...>> >> <mailto:jan...@ne... <mailto:jan...@ne...> >> > <mailto:jan...@ne... >> <mailto:jan...@ne...>>>>> wrote: >> > >>> > >> > >>> > Hey, >> > >>> > >> > >>> > I was playing around with a few python scripts >> (using >> > >>> capstone among >> > >>> > other things) and always ran out of memory - and I >> > have no >> > >>> freaking idea >> > >>> > why. >> > >>> > >> > >>> > >> > >>> > is this with the latest 3.0 version? >> > >>> > >> > >>> > thanks, >> > >>> > Q >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> > The code is really short: >> > >>> > >> > >>> > >> > >>> > from capstone import Cs >> > >>> > from capstone import CS_ARCH_X86 >> > >>> > from capstone import CS_MODE_32 >> > >>> > from capstone import CS_GRP_JUMP >> > >>> > from capstone import CS_GRP_CALL >> > >>> > from capstone import CS_GRP_RET >> > >>> > from capstone.x86_const import X86_INS_JNE, >> X86_INS_JMP >> > >>> > >> > >>> > """ >> > >>> > 0x401000: push ecx >> > >>> > 0x401001: pop ecx >> > >>> > 0x401002: mov eax, dword ptr [esp + >> 0x18] >> > >>> > 0x401006: mov eax, dword ptr [eax] >> > >>> > 0x401008: sar eax, 0 >> > >>> > 0x40100b: xor edi, eax >> > >>> > 0x40100d: nop >> > >>> > 0x40100e: add dword ptr [esp + 0x18], 4 >> > >>> > 0x401013: nop >> > >>> > 0x401014: dec word ptr [esp + 0x14] >> > >>> > 0x401019: shld edi, ecx, 0 >> > >>> > 0x40101d: jne 0x401000 >> > >>> > """ >> > >>> > def get_code(): >> > >>> > CODE = >> > >>> > >> > >>> >> > >> "\x51\x59\x8B\x44\x24\x18\x8B\x00\xC1\xF8\x00\x33\xF8\x90\x83\x44\x24\x18\x04\x90\x66\xFF\x4C\x24\x14\x0F\xA4\xCF\x00\x75\xE1" >> > >>> > >> > >>> > return CODE >> > >>> > >> > >>> > def is_branch(instr): >> > >>> > for group in branch_groups: >> > >>> > if group in instr.groups: >> > >>> > return True >> > >>> > return False >> > >>> > #return False >> > >>> > >> > >>> > # Disassemble until we hit basic block end. >> > >>> > def disasm(code): >> > >>> > disasm = Cs(CS_ARCH_X86, CS_MODE_32) >> > >>> > disasm.detail = True >> > >>> > address = 0x401000 >> > >>> > for instr in disasm.disasm(code, address): >> > >>> > print "0x%x:\t%s\t%s" % (instr.address, >> > instr.mnemonic, >> > >>> > instr.op_str) >> > >>> > if is_branch(instr): >> > >>> > break >> > >>> > >> > >>> > branch_groups = [CS_GRP_JUMP, CS_GRP_CALL, >> CS_GRP_RET] >> > >>> > code = get_code() >> > >>> > while True: >> > >>> > disasm(code) >> > >>> > >> > >>> > >> > >>> > That code goes out of memory after a few >> seconds. The >> > super >> > >>> weird thing >> > >>> > is, that if I change the implementation of >> > "is_branch(instr)" >> > >>> to simply >> > >>> > return False all the time, then the program >> does not >> > go out >> > >>> of memory! >> > >>> > Does anyone have an idea what's going on? >> > >>> > >> > >>> > Best >> > >>> > Jan >> > >>> > >> > >>> > >> > >>> >> > >> ------------------------------------------------------------------------------ >> > >>> > Download BIRT iHub F-Type - The Free >> Enterprise-Grade >> > BIRT Server >> > >>> > from Actuate! Instantly Supercharge Your Business >> > Reports and >> > >>> Dashboards >> > >>> > with Interactivity, Sharing, Native Excel >> Exports, App >> > >>> Integration & >> > >>> > more >> > >>> > Get technology previously reserved for >> billion-dollar >> > >>> corporations, FREE >> > >>> > >> > >>> >> > >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> > >>> > _______________________________________________ >> > >>> > Capstone-users mailing list >> > >>> > Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>> >> > >>> <mailto:Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>>> >> > >>> > <mailto:Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>> >> > >>> <mailto:Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>>>> >> > >>> > >> https://lists.sourceforge.net/lists/listinfo/capstone-users >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> >> > >> ------------------------------------------------------------------------------ >> > >>> > Download BIRT iHub F-Type - The Free Enterprise-Grade >> > BIRT Server >> > >>> > from Actuate! Instantly Supercharge Your Business >> Reports and >> > >>> Dashboards >> > >>> > with Interactivity, Sharing, Native Excel Exports, App >> > >>> Integration & more >> > >>> > Get technology previously reserved for billion-dollar >> > >>> corporations, FREE >> > >>> > >> > >>> >> > >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> > >>> > >> > >>> > >> > >>> > >> > >>> > _______________________________________________ >> > >>> > Capstone-users mailing list >> > >>> > Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>> >> > >>> <mailto:Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>>> >> > >>> > >> https://lists.sourceforge.net/lists/listinfo/capstone-users >> > >>> > >> > >>> >> > >>> >> > >>> >> > >> ------------------------------------------------------------------------------ >> > >>> Download BIRT iHub F-Type - The Free Enterprise-Grade >> BIRT >> > Server >> > >>> from Actuate! Instantly Supercharge Your Business Reports >> > and Dashboards >> > >>> with Interactivity, Sharing, Native Excel Exports, App >> > Integration & >> > >>> more >> > >>> Get technology previously reserved for billion-dollar >> > corporations, FREE >> > >>> >> > >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> > >>> _______________________________________________ >> > >>> Capstone-users mailing list >> > >>> Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>> >> > >>> <mailto:Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>>> >> > >>> https://lists.sourceforge.net/lists/listinfo/capstone-users >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >> ------------------------------------------------------------------------------ >> > >>> Download BIRT iHub F-Type - The Free Enterprise-Grade >> BIRT Server >> > >>> from Actuate! Instantly Supercharge Your Business Reports and >> > Dashboards >> > >>> with Interactivity, Sharing, Native Excel Exports, App >> > Integration & more >> > >>> Get technology previously reserved for billion-dollar >> > corporations, FREE >> > >>> >> > >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> > >>> >> > >>> >> > >>> >> > >>> _______________________________________________ >> > >>> Capstone-users mailing list >> > >>> Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>> >> > >>> https://lists.sourceforge.net/lists/listinfo/capstone-users >> > >>> >> > >> >> > >> >> > >> >> > >> ------------------------------------------------------------------------------ >> > >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT >> Server >> > >> from Actuate! Instantly Supercharge Your Business Reports and >> > Dashboards >> > >> with Interactivity, Sharing, Native Excel Exports, App >> > Integration & more >> > >> Get technology previously reserved for billion-dollar >> > corporations, FREE >> > >> >> > >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> > >> _______________________________________________ >> > >> Capstone-users mailing list >> > >> Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>> >> > >> https://lists.sourceforge.net/lists/listinfo/capstone-users >> > >> >> > > >> > > >> > > >> > >> ------------------------------------------------------------------------------ >> > > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT >> Server >> > > from Actuate! Instantly Supercharge Your Business Reports and >> > Dashboards >> > > with Interactivity, Sharing, Native Excel Exports, App >> Integration >> > & more >> > > Get technology previously reserved for billion-dollar >> > corporations, FREE >> > > >> > >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> > > _______________________________________________ >> > > Capstone-users mailing list >> > > Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>> >> > > https://lists.sourceforge.net/lists/listinfo/capstone-users >> > > >> > >> > >> > >> ------------------------------------------------------------------------------ >> > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server >> > from Actuate! Instantly Supercharge Your Business Reports and >> Dashboards >> > with Interactivity, Sharing, Native Excel Exports, App >> Integration & >> > more >> > Get technology previously reserved for billion-dollar >> corporations, FREE >> > >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> > _______________________________________________ >> > Capstone-users mailing list >> > Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>> >> > https://lists.sourceforge.net/lists/listinfo/capstone-users >> > >> > >> > >> > >> > >> ------------------------------------------------------------------------------ >> > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server >> > from Actuate! Instantly Supercharge Your Business Reports and >> Dashboards >> > with Interactivity, Sharing, Native Excel Exports, App >> Integration & more >> > Get technology previously reserved for billion-dollar >> corporations, FREE >> > >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> > >> > >> > >> > _______________________________________________ >> > Capstone-users mailing list >> > Cap...@li... >> <mailto:Cap...@li...> >> > https://lists.sourceforge.net/lists/listinfo/capstone-users >> > >> >> ------------------------------------------------------------------------------ >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server >> from Actuate! Instantly Supercharge Your Business Reports and Dashboards >> with Interactivity, Sharing, Native Excel Exports, App Integration & >> more >> Get technology previously reserved for billion-dollar corporations, FREE >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> _______________________________________________ >> Capstone-users mailing list >> Cap...@li... >> <mailto:Cap...@li...> >> https://lists.sourceforge.net/lists/listinfo/capstone-users >> >> >> ------------------------------------------------------------------------------ >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server >> from Actuate! Instantly Supercharge Your Business Reports and Dashboards >> with Interactivity, Sharing, Native Excel Exports, App Integration & >> more >> Get technology previously reserved for billion-dollar corporations, FREE >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> _______________________________________________ >> Capstone-users mailing list >> Cap...@li... >> <mailto:Cap...@li...> >> https://lists.sourceforge.net/lists/listinfo/capstone-users >> >> >> >> >> ------------------------------------------------------------------------------ >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server >> from Actuate! Instantly Supercharge Your Business Reports and Dashboards >> with Interactivity, Sharing, Native Excel Exports, App Integration & more >> Get technology previously reserved for billion-dollar corporations, FREE >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> >> >> >> _______________________________________________ >> Capstone-users mailing list >> Cap...@li... >> https://lists.sourceforge.net/lists/listinfo/capstone-users >> > > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & more > Get technology previously reserved for billion-dollar corporations, FREE > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > _______________________________________________ > Capstone-users mailing list > Cap...@li... > https://lists.sourceforge.net/lists/listinfo/capstone-users > |
