Menu
▾
▴
Re: [Capstone-users] Memory Leak?
|
From: Jan N. <jan...@ne...> - 2014-12-04 14:09:16
|
I don't have the dev environment for this in place, it would mean additional time for me to set this up. On 12/04/2014 02:54 PM, Nguyen Anh Quynh wrote: > > > On Thu, Dec 4, 2014 at 8:01 PM, Jan Newger <jan...@ne... > <mailto:jan...@ne...>> wrote: > > I think the easiest fix would be to free the memory from a finally > clause, as described here: > > https://docs.python.org/2/reference/simple_stmts.html#the-yield-statement > > Like this (untested): > > def disasm(self, code, offset, count=0): > all_insn = ctypes.POINTER(_cs_insn)() > '''if not _python2: > print(code) > code = code.encode() > print(code)''' > res = _cs.cs_disasm(self.csh, code, len(code), offset, count, > ctypes.byref(all_insn)) > if res > 0: > try: > for i in range(res): > yield CsInsn(self, all_insn[i]) > finally: > _cs.cs_free(all_insn, res) > else: > status = _cs.cs_errno(self.csh) > if status != CS_ERR_OK: > raise CsError(status) > return > yield > > > can you submit a pull request on Github? > > thanks, > Q > > > > > > > On 12/04/2014 01:07 AM, Jasiel Spelman wrote: > > Nguyen, > > > > Here is example code that hopefully better illustrates Juriaan's > > point/what Jan is seeing: > > > > def gen_example(): > > print '[gen_example] START' > > for i in xrange(2): > > print '[gen_example] before yield: ', i > > yield i > > print '[gen_example] after yield: ', i > > print '[gen_example] END' > > > > print "Typical case" > > for i in gen_example(): > > print > > > > print "Jan's case" > > for i in gen_example(): > > if i == 1: > > break > > print > > > > > > Typical case > > [gen_example] START > > [gen_example] before yield: 0 > > > > [gen_example] after yield: 0 > > [gen_example] before yield: 1 > > > > [gen_example] after yield: 1 > > [gen_example] END > > > > > > Jan's case > > [gen_example] START > > [gen_example] before yield: 0 > > > > [gen_example] after yield: 0 > > [gen_example] before yield: 1 > > > > > > On Wed, Dec 3, 2014 at 5:49 PM, Nguyen Anh Quynh > <aq...@gm... <mailto:aq...@gm...> > > <mailto:aq...@gm... <mailto:aq...@gm...>>> wrote: > > > > > > > > On Thu, Dec 4, 2014 at 6:32 AM, Jurriaan Bremer > > <jur...@gm... <mailto:jur...@gm...> > <mailto:jur...@gm... <mailto:jur...@gm...>>> > wrote: > > > > The 'yield' keyword allows one to fetch new records/rows (in > > this case > > disassembled instructions) on-demand. Doing an early exit > (e.g., > > quitting from a for-loop iterating over a function that > yield's > > after > > only, say, 2 out of 10 items) will prematurely exit the > function as > > well. Because, of course, why would Python calculate the > latter 8 > > results when they're not used in the first place? This is > also > > useful > > for never-ending functions - I suppose calculating digits > of Pi > > would be > > a 'good' example. > > > > Anyway, so your cs_free() call is never reached in this > case - you > > should switch to cs_free()'ing every row after each iteration > > through > > the for loop. > > > > > > but the loop is called upon the number of successfully > disassembled > > instructions, > > so there is no where in the code that might possibly quit the > loop > > prematurely. > > > > thanks, > > > > Q > > > > On 12/04/2014 12:27 AM, Nguyen Anh Quynh wrote: > > > > > > > > > On Thu, Dec 4, 2014 at 4:42 AM, Jan Newger > <jan...@ne... <mailto:jan...@ne...> > <mailto:jan...@ne... <mailto:jan...@ne...>> > > > <mailto:jan...@ne... > <mailto:jan...@ne...> <mailto:jan...@ne... > <mailto:jan...@ne...>>>> wrote: > > > > > > This is the python implementation of the disasm function > > (starting at > > > line 791): > > > > > > def disasm(self, code, offset, count=0): > > > all_insn = ctypes.POINTER(_cs_insn)() > > > '''if not _python2: > > > print(code) > > > code = code.encode() > > > print(code)''' > > > res = _cs.cs_disasm(self.csh, code, len(code), > > offset, count, > > > ctypes.byref(all_insn)) > > > if res > 0: > > > for i in range(res): > > > yield CsInsn(self, all_insn[i]) > > > _cs.cs_free(all_insn, res) > > > else: > > > status = _cs.cs_errno(self.csh) > > > if status != CS_ERR_OK: > > > raise CsError(status) > > > return > > > yield > > > > > > I'm really no python expert, but from what I see you > > apparently need to > > > free the instruction instances manually. However, if > client > > code stops > > > enumeration over the instructions prematurely, then > > _cs.cs_free() is > > > never invoked, and thus memory is leaked, right? > > > > > > > > > yes the problem must be with Python binding but not the core. > > > however, in the above code, cs_free() is called after the > "for" loop, > > > so i dont see how memleak can happen "prematurely". > > > > > > > > > thanks. > > > > > > > > > > > > > > > > > > On 03.12.2014 22:25, Jan Newger wrote: > > > > It seems the equivalent C implementation is not > affected by > > the mem > > > > leak, which is to be expected, since the memory is > > explicitly freed > > > > anyways, and the group checking boils down to > comparing an > > integer > > > value. > > > > > > > > If I had to guess, I'd suspect that in the python > case the > > group > > > > checking code introduces a spurious reference to the > > instruction > > > > instance(?) which cannot be claimed by the GC. > > > > > > > > On 03.12.2014 16:57, Jan Newger wrote: > > > >> No I haven't tried to reproduce the mem leak with C. > > > >> It already took me a considerable amount of time to > come > > up with this > > > >> minimal example. > > > >> > > > >> On 12/03/2014 04:53 PM, Capstone Engine wrote: > > > >>> > > > >>> > > > >>> On Wed, Dec 3, 2014 at 10:32 PM, Jan Newger > > > <jan...@ne... <mailto:jan...@ne...> > <mailto:jan...@ne... <mailto:jan...@ne...>> > > <mailto:jan...@ne... <mailto:jan...@ne...> > <mailto:jan...@ne... <mailto:jan...@ne...>>> > > > >>> <mailto:jan...@ne... > <mailto:jan...@ne...> > > <mailto:jan...@ne... <mailto:jan...@ne...>> > <mailto:jan...@ne... <mailto:jan...@ne...> > > <mailto:jan...@ne... <mailto:jan...@ne...>>>>> > > > wrote: > > > >>> > > > >>> > > > >>> Yes, it's using the latest version. > > > >>> OS was win7 x64 running python 2.7 with 32bit > libraries. > > > >>> > > > >>> > > > >>> this is interesting. have you tried to code the same > > program in > > > C to see > > > >>> if the mem leak issue still happens? > > > >>> > > > >>> > > > >>> thanks. > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> On 12/03/2014 04:31 PM, Nguyen Anh Quynh wrote: > > > >>> > > > > >>> > > > > >>> > On Wed, Dec 3, 2014 at 4:57 PM, Jan Newger > > > <jan...@ne... <mailto:jan...@ne...> > <mailto:jan...@ne... <mailto:jan...@ne...>> > > <mailto:jan...@ne... <mailto:jan...@ne...> > <mailto:jan...@ne... <mailto:jan...@ne...>>> > > > <mailto:jan...@ne... > <mailto:jan...@ne...> <mailto:jan...@ne... > <mailto:jan...@ne...>> > > <mailto:jan...@ne... <mailto:jan...@ne...> > <mailto:jan...@ne... <mailto:jan...@ne...>>>> > > > >>> > <mailto:jan...@ne... > <mailto:jan...@ne...> > > <mailto:jan...@ne... <mailto:jan...@ne...>> > > > <mailto:jan...@ne... > <mailto:jan...@ne...> <mailto:jan...@ne... > <mailto:jan...@ne...>>> > > <mailto:jan...@ne... <mailto:jan...@ne...> > <mailto:jan...@ne... <mailto:jan...@ne...>> > > > <mailto:jan...@ne... > <mailto:jan...@ne...> > > <mailto:jan...@ne... > <mailto:jan...@ne...>>>>>> wrote: > > > >>> > > > > >>> > Hey, > > > >>> > > > > >>> > I was playing around with a few python > scripts > > (using > > > >>> capstone among > > > >>> > other things) and always ran out of > memory - and I > > > have no > > > >>> freaking idea > > > >>> > why. > > > >>> > > > > >>> > > > > >>> > is this with the latest 3.0 version? > > > >>> > > > > >>> > thanks, > > > >>> > Q > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > The code is really short: > > > >>> > > > > >>> > > > > >>> > from capstone import Cs > > > >>> > from capstone import CS_ARCH_X86 > > > >>> > from capstone import CS_MODE_32 > > > >>> > from capstone import CS_GRP_JUMP > > > >>> > from capstone import CS_GRP_CALL > > > >>> > from capstone import CS_GRP_RET > > > >>> > from capstone.x86_const import X86_INS_JNE, > > X86_INS_JMP > > > >>> > > > > >>> > """ > > > >>> > 0x401000: push ecx > > > >>> > 0x401001: pop ecx > > > >>> > 0x401002: mov eax, dword ptr > [esp + > > 0x18] > > > >>> > 0x401006: mov eax, dword ptr > [eax] > > > >>> > 0x401008: sar eax, 0 > > > >>> > 0x40100b: xor edi, eax > > > >>> > 0x40100d: nop > > > >>> > 0x40100e: add dword ptr [esp > + 0x18], 4 > > > >>> > 0x401013: nop > > > >>> > 0x401014: dec word ptr [esp + > 0x14] > > > >>> > 0x401019: shld edi, ecx, 0 > > > >>> > 0x40101d: jne 0x401000 > > > >>> > """ > > > >>> > def get_code(): > > > >>> > CODE = > > > >>> > > > > >>> > > > > > > "\x51\x59\x8B\x44\x24\x18\x8B\x00\xC1\xF8\x00\x33\xF8\x90\x83\x44\x24\x18\x04\x90\x66\xFF\x4C\x24\x14\x0F\xA4\xCF\x00\x75\xE1" > > > >>> > > > > >>> > return CODE > > > >>> > > > > >>> > def is_branch(instr): > > > >>> > for group in branch_groups: > > > >>> > if group in instr.groups: > > > >>> > return True > > > >>> > return False > > > >>> > #return False > > > >>> > > > > >>> > # Disassemble until we hit basic block end. > > > >>> > def disasm(code): > > > >>> > disasm = Cs(CS_ARCH_X86, CS_MODE_32) > > > >>> > disasm.detail = True > > > >>> > address = 0x401000 > > > >>> > for instr in disasm.disasm(code, > address): > > > >>> > print "0x%x:\t%s\t%s" % > (instr.address, > > > instr.mnemonic, > > > >>> > instr.op_str) > > > >>> > if is_branch(instr): > > > >>> > break > > > >>> > > > > >>> > branch_groups = [CS_GRP_JUMP, CS_GRP_CALL, > > CS_GRP_RET] > > > >>> > code = get_code() > > > >>> > while True: > > > >>> > disasm(code) > > > >>> > > > > >>> > > > > >>> > That code goes out of memory after a few > > seconds. The > > > super > > > >>> weird thing > > > >>> > is, that if I change the implementation of > > > "is_branch(instr)" > > > >>> to simply > > > >>> > return False all the time, then the program > > does not > > > go out > > > >>> of memory! > > > >>> > Does anyone have an idea what's going on? > > > >>> > > > > >>> > Best > > > >>> > Jan > > > >>> > > > > >>> > > > > >>> > > > > > > ------------------------------------------------------------------------------ > > > >>> > Download BIRT iHub F-Type - The Free > > Enterprise-Grade > > > BIRT Server > > > >>> > from Actuate! Instantly Supercharge > Your Business > > > Reports and > > > >>> Dashboards > > > >>> > with Interactivity, Sharing, Native Excel > > Exports, App > > > >>> Integration & > > > >>> > more > > > >>> > Get technology previously reserved for > > billion-dollar > > > >>> corporations, FREE > > > >>> > > > > >>> > > > > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > > >>> > > _______________________________________________ > > > >>> > Capstone-users mailing list > > > >>> > Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > > <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>>> > > > >>> <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > > <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>>>> > > > >>> > > <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > > <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>>> > > > >>> <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > > <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>>>>> > > > >>> > > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > > > ------------------------------------------------------------------------------ > > > >>> > Download BIRT iHub F-Type - The Free > Enterprise-Grade > > > BIRT Server > > > >>> > from Actuate! Instantly Supercharge Your > Business > > Reports and > > > >>> Dashboards > > > >>> > with Interactivity, Sharing, Native Excel > Exports, App > > > >>> Integration & more > > > >>> > Get technology previously reserved for > billion-dollar > > > >>> corporations, FREE > > > >>> > > > > >>> > > > > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > > >>> > > > > >>> > > > > >>> > > > > >>> > _______________________________________________ > > > >>> > Capstone-users mailing list > > > >>> > Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > > <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>>> > > > >>> <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > > <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>>>> > > > >>> > > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > >>> > > > > >>> > > > >>> > > > >>> > > > > > > ------------------------------------------------------------------------------ > > > >>> Download BIRT iHub F-Type - The Free > Enterprise-Grade > > BIRT > > > Server > > > >>> from Actuate! Instantly Supercharge Your > Business Reports > > > and Dashboards > > > >>> with Interactivity, Sharing, Native Excel > Exports, App > > > Integration & > > > >>> more > > > >>> Get technology previously reserved for > billion-dollar > > > corporations, FREE > > > >>> > > > > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > > >>> _______________________________________________ > > > >>> Capstone-users mailing list > > > >>> Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > > <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>>> > > > >>> <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > > <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>>>> > > > >>> > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > > > > ------------------------------------------------------------------------------ > > > >>> Download BIRT iHub F-Type - The Free Enterprise-Grade > > BIRT Server > > > >>> from Actuate! Instantly Supercharge Your Business > Reports and > > > Dashboards > > > >>> with Interactivity, Sharing, Native Excel Exports, App > > > Integration & more > > > >>> Get technology previously reserved for billion-dollar > > > corporations, FREE > > > >>> > > > > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > > >>> > > > >>> > > > >>> > > > >>> _______________________________________________ > > > >>> Capstone-users mailing list > > > >>> Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > > <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>>> > > > >>> > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > >>> > > > >> > > > >> > > > >> > > > > > > ------------------------------------------------------------------------------ > > > >> Download BIRT iHub F-Type - The Free > Enterprise-Grade BIRT > > Server > > > >> from Actuate! Instantly Supercharge Your Business > Reports and > > > Dashboards > > > >> with Interactivity, Sharing, Native Excel Exports, App > > > Integration & more > > > >> Get technology previously reserved for billion-dollar > > > corporations, FREE > > > >> > > > > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > > >> _______________________________________________ > > > >> Capstone-users mailing list > > > >> Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > > <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>>> > > > >> > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > >> > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > > Download BIRT iHub F-Type - The Free > Enterprise-Grade BIRT > > Server > > > > from Actuate! Instantly Supercharge Your Business > Reports and > > > Dashboards > > > > with Interactivity, Sharing, Native Excel Exports, App > > Integration > > > & more > > > > Get technology previously reserved for billion-dollar > > > corporations, FREE > > > > > > > > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > > > _______________________________________________ > > > > Capstone-users mailing list > > > > Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > > <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>>> > > > > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > Download BIRT iHub F-Type - The Free Enterprise-Grade > BIRT Server > > > from Actuate! Instantly Supercharge Your Business > Reports and > > Dashboards > > > with Interactivity, Sharing, Native Excel Exports, App > > Integration & > > > more > > > Get technology previously reserved for billion-dollar > > corporations, FREE > > > > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > > _______________________________________________ > > > Capstone-users mailing list > > > Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > > <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>>> > > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT > Server > > > from Actuate! Instantly Supercharge Your Business Reports and > > Dashboards > > > with Interactivity, Sharing, Native Excel Exports, App > > Integration & more > > > Get technology previously reserved for billion-dollar > > corporations, FREE > > > > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > > > > > > > > > > > _______________________________________________ > > > Capstone-users mailing list > > > Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > > > > ------------------------------------------------------------------------------ > > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > > from Actuate! Instantly Supercharge Your Business Reports and > Dashboards > > with Interactivity, Sharing, Native Excel Exports, App > Integration & > > more > > Get technology previously reserved for billion-dollar > corporations, FREE > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > _______________________________________________ > > Capstone-users mailing list > > Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > > > ------------------------------------------------------------------------------ > > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > > from Actuate! Instantly Supercharge Your Business Reports and > Dashboards > > with Interactivity, Sharing, Native Excel Exports, App > Integration & > > more > > Get technology previously reserved for billion-dollar > corporations, FREE > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > _______________________________________________ > > Capstone-users mailing list > > Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > > > > > > > ------------------------------------------------------------------------------ > > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > > from Actuate! Instantly Supercharge Your Business Reports and > Dashboards > > with Interactivity, Sharing, Native Excel Exports, App > Integration & more > > Get technology previously reserved for billion-dollar > corporations, FREE > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > > > > > > > _______________________________________________ > > Capstone-users mailing list > > Cap...@li... > <mailto:Cap...@li...> > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & > more > Get technology previously reserved for billion-dollar corporations, FREE > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > _______________________________________________ > Capstone-users mailing list > Cap...@li... > <mailto:Cap...@li...> > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & more > Get technology previously reserved for billion-dollar corporations, FREE > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > > > _______________________________________________ > Capstone-users mailing list > Cap...@li... > https://lists.sourceforge.net/lists/listinfo/capstone-users > |
