Menu
▾
▴
Re: [Capstone-users] Memory Leak?
|
From: Jan N. <jan...@ne...> - 2014-12-04 13:01:53
|
I think the easiest fix would be to free the memory from a finally clause, as described here: https://docs.python.org/2/reference/simple_stmts.html#the-yield-statement Like this (untested): def disasm(self, code, offset, count=0): all_insn = ctypes.POINTER(_cs_insn)() '''if not _python2: print(code) code = code.encode() print(code)''' res = _cs.cs_disasm(self.csh, code, len(code), offset, count, ctypes.byref(all_insn)) if res > 0: try: for i in range(res): yield CsInsn(self, all_insn[i]) finally: _cs.cs_free(all_insn, res) else: status = _cs.cs_errno(self.csh) if status != CS_ERR_OK: raise CsError(status) return yield On 12/04/2014 01:07 AM, Jasiel Spelman wrote: > Nguyen, > > Here is example code that hopefully better illustrates Juriaan's > point/what Jan is seeing: > > def gen_example(): > print '[gen_example] START' > for i in xrange(2): > print '[gen_example] before yield: ', i > yield i > print '[gen_example] after yield: ', i > print '[gen_example] END' > > print "Typical case" > for i in gen_example(): > print > > print "Jan's case" > for i in gen_example(): > if i == 1: > break > print > > > Typical case > [gen_example] START > [gen_example] before yield: 0 > > [gen_example] after yield: 0 > [gen_example] before yield: 1 > > [gen_example] after yield: 1 > [gen_example] END > > > Jan's case > [gen_example] START > [gen_example] before yield: 0 > > [gen_example] after yield: 0 > [gen_example] before yield: 1 > > > On Wed, Dec 3, 2014 at 5:49 PM, Nguyen Anh Quynh <aq...@gm... > <mailto:aq...@gm...>> wrote: > > > > On Thu, Dec 4, 2014 at 6:32 AM, Jurriaan Bremer > <jur...@gm... <mailto:jur...@gm...>> wrote: > > The 'yield' keyword allows one to fetch new records/rows (in > this case > disassembled instructions) on-demand. Doing an early exit (e.g., > quitting from a for-loop iterating over a function that yield's > after > only, say, 2 out of 10 items) will prematurely exit the function as > well. Because, of course, why would Python calculate the latter 8 > results when they're not used in the first place? This is also > useful > for never-ending functions - I suppose calculating digits of Pi > would be > a 'good' example. > > Anyway, so your cs_free() call is never reached in this case - you > should switch to cs_free()'ing every row after each iteration > through > the for loop. > > > but the loop is called upon the number of successfully disassembled > instructions, > so there is no where in the code that might possibly quit the loop > prematurely. > > thanks, > > Q > > On 12/04/2014 12:27 AM, Nguyen Anh Quynh wrote: > > > > > > On Thu, Dec 4, 2014 at 4:42 AM, Jan Newger <jan...@ne... <mailto:jan...@ne...> > > <mailto:jan...@ne... <mailto:jan...@ne...>>> wrote: > > > > This is the python implementation of the disasm function > (starting at > > line 791): > > > > def disasm(self, code, offset, count=0): > > all_insn = ctypes.POINTER(_cs_insn)() > > '''if not _python2: > > print(code) > > code = code.encode() > > print(code)''' > > res = _cs.cs_disasm(self.csh, code, len(code), > offset, count, > > ctypes.byref(all_insn)) > > if res > 0: > > for i in range(res): > > yield CsInsn(self, all_insn[i]) > > _cs.cs_free(all_insn, res) > > else: > > status = _cs.cs_errno(self.csh) > > if status != CS_ERR_OK: > > raise CsError(status) > > return > > yield > > > > I'm really no python expert, but from what I see you > apparently need to > > free the instruction instances manually. However, if client > code stops > > enumeration over the instructions prematurely, then > _cs.cs_free() is > > never invoked, and thus memory is leaked, right? > > > > > > yes the problem must be with Python binding but not the core. > > however, in the above code, cs_free() is called after the "for" loop, > > so i dont see how memleak can happen "prematurely". > > > > > > thanks. > > > > > > > > > > > > On 03.12.2014 22:25, Jan Newger wrote: > > > It seems the equivalent C implementation is not affected by > the mem > > > leak, which is to be expected, since the memory is > explicitly freed > > > anyways, and the group checking boils down to comparing an > integer > > value. > > > > > > If I had to guess, I'd suspect that in the python case the > group > > > checking code introduces a spurious reference to the > instruction > > > instance(?) which cannot be claimed by the GC. > > > > > > On 03.12.2014 16:57, Jan Newger wrote: > > >> No I haven't tried to reproduce the mem leak with C. > > >> It already took me a considerable amount of time to come > up with this > > >> minimal example. > > >> > > >> On 12/03/2014 04:53 PM, Capstone Engine wrote: > > >>> > > >>> > > >>> On Wed, Dec 3, 2014 at 10:32 PM, Jan Newger > > <jan...@ne... <mailto:jan...@ne...> > <mailto:jan...@ne... <mailto:jan...@ne...>> > > >>> <mailto:jan...@ne... > <mailto:jan...@ne...> <mailto:jan...@ne... > <mailto:jan...@ne...>>>> > > wrote: > > >>> > > >>> > > >>> Yes, it's using the latest version. > > >>> OS was win7 x64 running python 2.7 with 32bit libraries. > > >>> > > >>> > > >>> this is interesting. have you tried to code the same > program in > > C to see > > >>> if the mem leak issue still happens? > > >>> > > >>> > > >>> thanks. > > >>> > > >>> > > >>> > > >>> > > >>> On 12/03/2014 04:31 PM, Nguyen Anh Quynh wrote: > > >>> > > > >>> > > > >>> > On Wed, Dec 3, 2014 at 4:57 PM, Jan Newger > > <jan...@ne... <mailto:jan...@ne...> > <mailto:jan...@ne... <mailto:jan...@ne...>> > > <mailto:jan...@ne... <mailto:jan...@ne...> > <mailto:jan...@ne... <mailto:jan...@ne...>>> > > >>> > <mailto:jan...@ne... > <mailto:jan...@ne...> > > <mailto:jan...@ne... <mailto:jan...@ne...>> > <mailto:jan...@ne... <mailto:jan...@ne...> > > <mailto:jan...@ne... > <mailto:jan...@ne...>>>>> wrote: > > >>> > > > >>> > Hey, > > >>> > > > >>> > I was playing around with a few python scripts > (using > > >>> capstone among > > >>> > other things) and always ran out of memory - and I > > have no > > >>> freaking idea > > >>> > why. > > >>> > > > >>> > > > >>> > is this with the latest 3.0 version? > > >>> > > > >>> > thanks, > > >>> > Q > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > The code is really short: > > >>> > > > >>> > > > >>> > from capstone import Cs > > >>> > from capstone import CS_ARCH_X86 > > >>> > from capstone import CS_MODE_32 > > >>> > from capstone import CS_GRP_JUMP > > >>> > from capstone import CS_GRP_CALL > > >>> > from capstone import CS_GRP_RET > > >>> > from capstone.x86_const import X86_INS_JNE, > X86_INS_JMP > > >>> > > > >>> > """ > > >>> > 0x401000: push ecx > > >>> > 0x401001: pop ecx > > >>> > 0x401002: mov eax, dword ptr [esp + > 0x18] > > >>> > 0x401006: mov eax, dword ptr [eax] > > >>> > 0x401008: sar eax, 0 > > >>> > 0x40100b: xor edi, eax > > >>> > 0x40100d: nop > > >>> > 0x40100e: add dword ptr [esp + 0x18], 4 > > >>> > 0x401013: nop > > >>> > 0x401014: dec word ptr [esp + 0x14] > > >>> > 0x401019: shld edi, ecx, 0 > > >>> > 0x40101d: jne 0x401000 > > >>> > """ > > >>> > def get_code(): > > >>> > CODE = > > >>> > > > >>> > > > "\x51\x59\x8B\x44\x24\x18\x8B\x00\xC1\xF8\x00\x33\xF8\x90\x83\x44\x24\x18\x04\x90\x66\xFF\x4C\x24\x14\x0F\xA4\xCF\x00\x75\xE1" > > >>> > > > >>> > return CODE > > >>> > > > >>> > def is_branch(instr): > > >>> > for group in branch_groups: > > >>> > if group in instr.groups: > > >>> > return True > > >>> > return False > > >>> > #return False > > >>> > > > >>> > # Disassemble until we hit basic block end. > > >>> > def disasm(code): > > >>> > disasm = Cs(CS_ARCH_X86, CS_MODE_32) > > >>> > disasm.detail = True > > >>> > address = 0x401000 > > >>> > for instr in disasm.disasm(code, address): > > >>> > print "0x%x:\t%s\t%s" % (instr.address, > > instr.mnemonic, > > >>> > instr.op_str) > > >>> > if is_branch(instr): > > >>> > break > > >>> > > > >>> > branch_groups = [CS_GRP_JUMP, CS_GRP_CALL, > CS_GRP_RET] > > >>> > code = get_code() > > >>> > while True: > > >>> > disasm(code) > > >>> > > > >>> > > > >>> > That code goes out of memory after a few > seconds. The > > super > > >>> weird thing > > >>> > is, that if I change the implementation of > > "is_branch(instr)" > > >>> to simply > > >>> > return False all the time, then the program > does not > > go out > > >>> of memory! > > >>> > Does anyone have an idea what's going on? > > >>> > > > >>> > Best > > >>> > Jan > > >>> > > > >>> > > > >>> > > > ------------------------------------------------------------------------------ > > >>> > Download BIRT iHub F-Type - The Free > Enterprise-Grade > > BIRT Server > > >>> > from Actuate! Instantly Supercharge Your Business > > Reports and > > >>> Dashboards > > >>> > with Interactivity, Sharing, Native Excel > Exports, App > > >>> Integration & > > >>> > more > > >>> > Get technology previously reserved for > billion-dollar > > >>> corporations, FREE > > >>> > > > >>> > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > >>> > _______________________________________________ > > >>> > Capstone-users mailing list > > >>> > Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > >>> <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>>> > > >>> > <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > >>> <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>>>> > > >>> > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > ------------------------------------------------------------------------------ > > >>> > Download BIRT iHub F-Type - The Free Enterprise-Grade > > BIRT Server > > >>> > from Actuate! Instantly Supercharge Your Business > Reports and > > >>> Dashboards > > >>> > with Interactivity, Sharing, Native Excel Exports, App > > >>> Integration & more > > >>> > Get technology previously reserved for billion-dollar > > >>> corporations, FREE > > >>> > > > >>> > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > >>> > > > >>> > > > >>> > > > >>> > _______________________________________________ > > >>> > Capstone-users mailing list > > >>> > Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > >>> <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>>> > > >>> > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > >>> > > > >>> > > >>> > > >>> > > > ------------------------------------------------------------------------------ > > >>> Download BIRT iHub F-Type - The Free Enterprise-Grade > BIRT > > Server > > >>> from Actuate! Instantly Supercharge Your Business Reports > > and Dashboards > > >>> with Interactivity, Sharing, Native Excel Exports, App > > Integration & > > >>> more > > >>> Get technology previously reserved for billion-dollar > > corporations, FREE > > >>> > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > >>> _______________________________________________ > > >>> Capstone-users mailing list > > >>> Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > >>> <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>>> > > >>> https://lists.sourceforge.net/lists/listinfo/capstone-users > > >>> > > >>> > > >>> > > >>> > > >>> > > > ------------------------------------------------------------------------------ > > >>> Download BIRT iHub F-Type - The Free Enterprise-Grade > BIRT Server > > >>> from Actuate! Instantly Supercharge Your Business Reports and > > Dashboards > > >>> with Interactivity, Sharing, Native Excel Exports, App > > Integration & more > > >>> Get technology previously reserved for billion-dollar > > corporations, FREE > > >>> > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > >>> > > >>> > > >>> > > >>> _______________________________________________ > > >>> Capstone-users mailing list > > >>> Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > >>> https://lists.sourceforge.net/lists/listinfo/capstone-users > > >>> > > >> > > >> > > >> > > > ------------------------------------------------------------------------------ > > >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT > Server > > >> from Actuate! Instantly Supercharge Your Business Reports and > > Dashboards > > >> with Interactivity, Sharing, Native Excel Exports, App > > Integration & more > > >> Get technology previously reserved for billion-dollar > > corporations, FREE > > >> > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > >> _______________________________________________ > > >> Capstone-users mailing list > > >> Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > >> https://lists.sourceforge.net/lists/listinfo/capstone-users > > >> > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT > Server > > > from Actuate! Instantly Supercharge Your Business Reports and > > Dashboards > > > with Interactivity, Sharing, Native Excel Exports, App > Integration > > & more > > > Get technology previously reserved for billion-dollar > > corporations, FREE > > > > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > > _______________________________________________ > > > Capstone-users mailing list > > > Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > > > > > > ------------------------------------------------------------------------------ > > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > > from Actuate! Instantly Supercharge Your Business Reports and > Dashboards > > with Interactivity, Sharing, Native Excel Exports, App > Integration & > > more > > Get technology previously reserved for billion-dollar > corporations, FREE > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > _______________________________________________ > > Capstone-users mailing list > > Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > > > > > > > ------------------------------------------------------------------------------ > > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > > from Actuate! Instantly Supercharge Your Business Reports and > Dashboards > > with Interactivity, Sharing, Native Excel Exports, App > Integration & more > > Get technology previously reserved for billion-dollar > corporations, FREE > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > > > > > > > _______________________________________________ > > Capstone-users mailing list > > Cap...@li... > <mailto:Cap...@li...> > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & > more > Get technology previously reserved for billion-dollar corporations, FREE > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > _______________________________________________ > Capstone-users mailing list > Cap...@li... > <mailto:Cap...@li...> > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & > more > Get technology previously reserved for billion-dollar corporations, FREE > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > _______________________________________________ > Capstone-users mailing list > Cap...@li... > <mailto:Cap...@li...> > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & more > Get technology previously reserved for billion-dollar corporations, FREE > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > > > _______________________________________________ > Capstone-users mailing list > Cap...@li... > https://lists.sourceforge.net/lists/listinfo/capstone-users > |
