Menu
▾
▴
capstone-users — Capstone mailing list for users
You can subscribe to this list here.
| 2013 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
(11) |
Dec
(11) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2014 |
Jan
(12) |
Feb
(3) |
Mar
(7) |
Apr
(4) |
May
(31) |
Jun
(2) |
Jul
(4) |
Aug
(2) |
Sep
(16) |
Oct
(13) |
Nov
(2) |
Dec
(25) |
| 2015 |
Jan
(28) |
Feb
(9) |
Mar
(7) |
Apr
(1) |
May
(3) |
Jun
(1) |
Jul
(3) |
Aug
(12) |
Sep
|
Oct
(11) |
Nov
(4) |
Dec
|
| 2016 |
Jan
(4) |
Feb
|
Mar
(8) |
Apr
|
May
(2) |
Jun
(2) |
Jul
(1) |
Aug
|
Sep
|
Oct
|
Nov
(1) |
Dec
|
| 2017 |
Jan
|
Feb
(1) |
Mar
(3) |
Apr
(2) |
May
(4) |
Jun
(6) |
Jul
(9) |
Aug
(2) |
Sep
(1) |
Oct
|
Nov
|
Dec
|
| 2018 |
Jan
|
Feb
|
Mar
(7) |
Apr
|
May
|
Jun
|
Jul
(2) |
Aug
(3) |
Sep
|
Oct
(2) |
Nov
(7) |
Dec
(2) |
| 2019 |
Jan
(1) |
Feb
(1) |
Mar
(5) |
Apr
(1) |
May
(1) |
Jun
(4) |
Jul
(6) |
Aug
(2) |
Sep
|
Oct
|
Nov
|
Dec
(4) |
| 2020 |
Jan
|
Feb
|
Mar
|
Apr
|
May
(1) |
Jun
(2) |
Jul
|
Aug
|
Sep
|
Oct
|
Nov
(1) |
Dec
|
| 2022 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
(2) |
Jul
(1) |
Aug
(3) |
Sep
|
Oct
|
Nov
|
Dec
|
| 2023 |
Jan
(1) |
Feb
|
Mar
(1) |
Apr
|
May
(1) |
Jun
|
Jul
(1) |
Aug
|
Sep
|
Oct
|
Nov
(1) |
Dec
|
| 2024 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(1) |
Nov
|
Dec
|
|
From: Jurriaan B. <jur...@gm...> - 2014-12-04 13:04:16
|
I was unaware of this approach - nice! +1 for using this as a patch ;) Jurriaan On 12/04/2014 02:01 PM, Jan Newger wrote: > I think the easiest fix would be to free the memory from a finally > clause, as described here: > > https://docs.python.org/2/reference/simple_stmts.html#the-yield-statement > > Like this (untested): > > def disasm(self, code, offset, count=0): > all_insn = ctypes.POINTER(_cs_insn)() > '''if not _python2: > print(code) > code = code.encode() > print(code)''' > res = _cs.cs_disasm(self.csh, code, len(code), offset, count, > ctypes.byref(all_insn)) > if res > 0: > try: > for i in range(res): > yield CsInsn(self, all_insn[i]) > finally: > _cs.cs_free(all_insn, res) > else: > status = _cs.cs_errno(self.csh) > if status != CS_ERR_OK: > raise CsError(status) > return > yield > > On 12/04/2014 01:07 AM, Jasiel Spelman wrote: >> Nguyen, >> >> Here is example code that hopefully better illustrates Juriaan's >> point/what Jan is seeing: >> >> def gen_example(): >> print '[gen_example] START' >> for i in xrange(2): >> print '[gen_example] before yield: ', i >> yield i >> print '[gen_example] after yield: ', i >> print '[gen_example] END' >> >> print "Typical case" >> for i in gen_example(): >> print >> >> print "Jan's case" >> for i in gen_example(): >> if i == 1: >> break >> print >> >> >> Typical case >> [gen_example] START >> [gen_example] before yield: 0 >> >> [gen_example] after yield: 0 >> [gen_example] before yield: 1 >> >> [gen_example] after yield: 1 >> [gen_example] END >> >> >> Jan's case >> [gen_example] START >> [gen_example] before yield: 0 >> >> [gen_example] after yield: 0 >> [gen_example] before yield: 1 >> >> >> On Wed, Dec 3, 2014 at 5:49 PM, Nguyen Anh Quynh <aq...@gm... >> <mailto:aq...@gm...>> wrote: >> >> >> >> On Thu, Dec 4, 2014 at 6:32 AM, Jurriaan Bremer >> <jur...@gm... <mailto:jur...@gm...>> wrote: >> >> The 'yield' keyword allows one to fetch new records/rows (in >> this case >> disassembled instructions) on-demand. Doing an early exit (e.g., >> quitting from a for-loop iterating over a function that yield's >> after >> only, say, 2 out of 10 items) will prematurely exit the function as >> well. Because, of course, why would Python calculate the latter 8 >> results when they're not used in the first place? This is also >> useful >> for never-ending functions - I suppose calculating digits of Pi >> would be >> a 'good' example. >> >> Anyway, so your cs_free() call is never reached in this case - you >> should switch to cs_free()'ing every row after each iteration >> through >> the for loop. >> >> >> but the loop is called upon the number of successfully disassembled >> instructions, >> so there is no where in the code that might possibly quit the loop >> prematurely. >> >> thanks, >> >> Q >> >> On 12/04/2014 12:27 AM, Nguyen Anh Quynh wrote: >> > >> > >> > On Thu, Dec 4, 2014 at 4:42 AM, Jan Newger <jan...@ne... <mailto:jan...@ne...> >> > <mailto:jan...@ne... <mailto:jan...@ne...>>> wrote: >> > >> > This is the python implementation of the disasm function >> (starting at >> > line 791): >> > >> > def disasm(self, code, offset, count=0): >> > all_insn = ctypes.POINTER(_cs_insn)() >> > '''if not _python2: >> > print(code) >> > code = code.encode() >> > print(code)''' >> > res = _cs.cs_disasm(self.csh, code, len(code), >> offset, count, >> > ctypes.byref(all_insn)) >> > if res > 0: >> > for i in range(res): >> > yield CsInsn(self, all_insn[i]) >> > _cs.cs_free(all_insn, res) >> > else: >> > status = _cs.cs_errno(self.csh) >> > if status != CS_ERR_OK: >> > raise CsError(status) >> > return >> > yield >> > >> > I'm really no python expert, but from what I see you >> apparently need to >> > free the instruction instances manually. However, if client >> code stops >> > enumeration over the instructions prematurely, then >> _cs.cs_free() is >> > never invoked, and thus memory is leaked, right? >> > >> > >> > yes the problem must be with Python binding but not the core. >> > however, in the above code, cs_free() is called after the "for" loop, >> > so i dont see how memleak can happen "prematurely". >> > >> > >> > thanks. >> > >> > >> > >> > >> > >> > On 03.12.2014 22:25, Jan Newger wrote: >> > > It seems the equivalent C implementation is not affected by >> the mem >> > > leak, which is to be expected, since the memory is >> explicitly freed >> > > anyways, and the group checking boils down to comparing an >> integer >> > value. >> > > >> > > If I had to guess, I'd suspect that in the python case the >> group >> > > checking code introduces a spurious reference to the >> instruction >> > > instance(?) which cannot be claimed by the GC. >> > > >> > > On 03.12.2014 16:57, Jan Newger wrote: >> > >> No I haven't tried to reproduce the mem leak with C. >> > >> It already took me a considerable amount of time to come >> up with this >> > >> minimal example. >> > >> >> > >> On 12/03/2014 04:53 PM, Capstone Engine wrote: >> > >>> >> > >>> >> > >>> On Wed, Dec 3, 2014 at 10:32 PM, Jan Newger >> > <jan...@ne... <mailto:jan...@ne...> >> <mailto:jan...@ne... <mailto:jan...@ne...>> >> > >>> <mailto:jan...@ne... >> <mailto:jan...@ne...> <mailto:jan...@ne... >> <mailto:jan...@ne...>>>> >> > wrote: >> > >>> >> > >>> >> > >>> Yes, it's using the latest version. >> > >>> OS was win7 x64 running python 2.7 with 32bit libraries. >> > >>> >> > >>> >> > >>> this is interesting. have you tried to code the same >> program in >> > C to see >> > >>> if the mem leak issue still happens? >> > >>> >> > >>> >> > >>> thanks. >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> On 12/03/2014 04:31 PM, Nguyen Anh Quynh wrote: >> > >>> > >> > >>> > >> > >>> > On Wed, Dec 3, 2014 at 4:57 PM, Jan Newger >> > <jan...@ne... <mailto:jan...@ne...> >> <mailto:jan...@ne... <mailto:jan...@ne...>> >> > <mailto:jan...@ne... <mailto:jan...@ne...> >> <mailto:jan...@ne... <mailto:jan...@ne...>>> >> > >>> > <mailto:jan...@ne... >> <mailto:jan...@ne...> >> > <mailto:jan...@ne... <mailto:jan...@ne...>> >> <mailto:jan...@ne... <mailto:jan...@ne...> >> > <mailto:jan...@ne... >> <mailto:jan...@ne...>>>>> wrote: >> > >>> > >> > >>> > Hey, >> > >>> > >> > >>> > I was playing around with a few python scripts >> (using >> > >>> capstone among >> > >>> > other things) and always ran out of memory - and I >> > have no >> > >>> freaking idea >> > >>> > why. >> > >>> > >> > >>> > >> > >>> > is this with the latest 3.0 version? >> > >>> > >> > >>> > thanks, >> > >>> > Q >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> > The code is really short: >> > >>> > >> > >>> > >> > >>> > from capstone import Cs >> > >>> > from capstone import CS_ARCH_X86 >> > >>> > from capstone import CS_MODE_32 >> > >>> > from capstone import CS_GRP_JUMP >> > >>> > from capstone import CS_GRP_CALL >> > >>> > from capstone import CS_GRP_RET >> > >>> > from capstone.x86_const import X86_INS_JNE, >> X86_INS_JMP >> > >>> > >> > >>> > """ >> > >>> > 0x401000: push ecx >> > >>> > 0x401001: pop ecx >> > >>> > 0x401002: mov eax, dword ptr [esp + >> 0x18] >> > >>> > 0x401006: mov eax, dword ptr [eax] >> > >>> > 0x401008: sar eax, 0 >> > >>> > 0x40100b: xor edi, eax >> > >>> > 0x40100d: nop >> > >>> > 0x40100e: add dword ptr [esp + 0x18], 4 >> > >>> > 0x401013: nop >> > >>> > 0x401014: dec word ptr [esp + 0x14] >> > >>> > 0x401019: shld edi, ecx, 0 >> > >>> > 0x40101d: jne 0x401000 >> > >>> > """ >> > >>> > def get_code(): >> > >>> > CODE = >> > >>> > >> > >>> >> > >> "\x51\x59\x8B\x44\x24\x18\x8B\x00\xC1\xF8\x00\x33\xF8\x90\x83\x44\x24\x18\x04\x90\x66\xFF\x4C\x24\x14\x0F\xA4\xCF\x00\x75\xE1" >> > >>> > >> > >>> > return CODE >> > >>> > >> > >>> > def is_branch(instr): >> > >>> > for group in branch_groups: >> > >>> > if group in instr.groups: >> > >>> > return True >> > >>> > return False >> > >>> > #return False >> > >>> > >> > >>> > # Disassemble until we hit basic block end. >> > >>> > def disasm(code): >> > >>> > disasm = Cs(CS_ARCH_X86, CS_MODE_32) >> > >>> > disasm.detail = True >> > >>> > address = 0x401000 >> > >>> > for instr in disasm.disasm(code, address): >> > >>> > print "0x%x:\t%s\t%s" % (instr.address, >> > instr.mnemonic, >> > >>> > instr.op_str) >> > >>> > if is_branch(instr): >> > >>> > break >> > >>> > >> > >>> > branch_groups = [CS_GRP_JUMP, CS_GRP_CALL, >> CS_GRP_RET] >> > >>> > code = get_code() >> > >>> > while True: >> > >>> > disasm(code) >> > >>> > >> > >>> > >> > >>> > That code goes out of memory after a few >> seconds. The >> > super >> > >>> weird thing >> > >>> > is, that if I change the implementation of >> > "is_branch(instr)" >> > >>> to simply >> > >>> > return False all the time, then the program >> does not >> > go out >> > >>> of memory! >> > >>> > Does anyone have an idea what's going on? >> > >>> > >> > >>> > Best >> > >>> > Jan >> > >>> > >> > >>> > >> > >>> >> > >> ------------------------------------------------------------------------------ >> > >>> > Download BIRT iHub F-Type - The Free >> Enterprise-Grade >> > BIRT Server >> > >>> > from Actuate! Instantly Supercharge Your Business >> > Reports and >> > >>> Dashboards >> > >>> > with Interactivity, Sharing, Native Excel >> Exports, App >> > >>> Integration & >> > >>> > more >> > >>> > Get technology previously reserved for >> billion-dollar >> > >>> corporations, FREE >> > >>> > >> > >>> >> > >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> > >>> > _______________________________________________ >> > >>> > Capstone-users mailing list >> > >>> > Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>> >> > >>> <mailto:Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>>> >> > >>> > <mailto:Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>> >> > >>> <mailto:Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>>>> >> > >>> > >> https://lists.sourceforge.net/lists/listinfo/capstone-users >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> > >> > >>> >> > >> ------------------------------------------------------------------------------ >> > >>> > Download BIRT iHub F-Type - The Free Enterprise-Grade >> > BIRT Server >> > >>> > from Actuate! Instantly Supercharge Your Business >> Reports and >> > >>> Dashboards >> > >>> > with Interactivity, Sharing, Native Excel Exports, App >> > >>> Integration & more >> > >>> > Get technology previously reserved for billion-dollar >> > >>> corporations, FREE >> > >>> > >> > >>> >> > >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> > >>> > >> > >>> > >> > >>> > >> > >>> > _______________________________________________ >> > >>> > Capstone-users mailing list >> > >>> > Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>> >> > >>> <mailto:Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>>> >> > >>> > >> https://lists.sourceforge.net/lists/listinfo/capstone-users >> > >>> > >> > >>> >> > >>> >> > >>> >> > >> ------------------------------------------------------------------------------ >> > >>> Download BIRT iHub F-Type - The Free Enterprise-Grade >> BIRT >> > Server >> > >>> from Actuate! Instantly Supercharge Your Business Reports >> > and Dashboards >> > >>> with Interactivity, Sharing, Native Excel Exports, App >> > Integration & >> > >>> more >> > >>> Get technology previously reserved for billion-dollar >> > corporations, FREE >> > >>> >> > >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> > >>> _______________________________________________ >> > >>> Capstone-users mailing list >> > >>> Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>> >> > >>> <mailto:Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>>> >> > >>> https://lists.sourceforge.net/lists/listinfo/capstone-users >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >> ------------------------------------------------------------------------------ >> > >>> Download BIRT iHub F-Type - The Free Enterprise-Grade >> BIRT Server >> > >>> from Actuate! Instantly Supercharge Your Business Reports and >> > Dashboards >> > >>> with Interactivity, Sharing, Native Excel Exports, App >> > Integration & more >> > >>> Get technology previously reserved for billion-dollar >> > corporations, FREE >> > >>> >> > >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> > >>> >> > >>> >> > >>> >> > >>> _______________________________________________ >> > >>> Capstone-users mailing list >> > >>> Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>> >> > >>> https://lists.sourceforge.net/lists/listinfo/capstone-users >> > >>> >> > >> >> > >> >> > >> >> > >> ------------------------------------------------------------------------------ >> > >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT >> Server >> > >> from Actuate! Instantly Supercharge Your Business Reports and >> > Dashboards >> > >> with Interactivity, Sharing, Native Excel Exports, App >> > Integration & more >> > >> Get technology previously reserved for billion-dollar >> > corporations, FREE >> > >> >> > >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> > >> _______________________________________________ >> > >> Capstone-users mailing list >> > >> Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>> >> > >> https://lists.sourceforge.net/lists/listinfo/capstone-users >> > >> >> > > >> > > >> > > >> > >> ------------------------------------------------------------------------------ >> > > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT >> Server >> > > from Actuate! Instantly Supercharge Your Business Reports and >> > Dashboards >> > > with Interactivity, Sharing, Native Excel Exports, App >> Integration >> > & more >> > > Get technology previously reserved for billion-dollar >> > corporations, FREE >> > > >> > >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> > > _______________________________________________ >> > > Capstone-users mailing list >> > > Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>> >> > > https://lists.sourceforge.net/lists/listinfo/capstone-users >> > > >> > >> > >> > >> ------------------------------------------------------------------------------ >> > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server >> > from Actuate! Instantly Supercharge Your Business Reports and >> Dashboards >> > with Interactivity, Sharing, Native Excel Exports, App >> Integration & >> > more >> > Get technology previously reserved for billion-dollar >> corporations, FREE >> > >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> > _______________________________________________ >> > Capstone-users mailing list >> > Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>> >> > https://lists.sourceforge.net/lists/listinfo/capstone-users >> > >> > >> > >> > >> > >> ------------------------------------------------------------------------------ >> > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server >> > from Actuate! Instantly Supercharge Your Business Reports and >> Dashboards >> > with Interactivity, Sharing, Native Excel Exports, App >> Integration & more >> > Get technology previously reserved for billion-dollar >> corporations, FREE >> > >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> > >> > >> > >> > _______________________________________________ >> > Capstone-users mailing list >> > Cap...@li... >> <mailto:Cap...@li...> >> > https://lists.sourceforge.net/lists/listinfo/capstone-users >> > >> >> ------------------------------------------------------------------------------ >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server >> from Actuate! Instantly Supercharge Your Business Reports and Dashboards >> with Interactivity, Sharing, Native Excel Exports, App Integration & >> more >> Get technology previously reserved for billion-dollar corporations, FREE >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> _______________________________________________ >> Capstone-users mailing list >> Cap...@li... >> <mailto:Cap...@li...> >> https://lists.sourceforge.net/lists/listinfo/capstone-users >> >> >> ------------------------------------------------------------------------------ >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server >> from Actuate! Instantly Supercharge Your Business Reports and Dashboards >> with Interactivity, Sharing, Native Excel Exports, App Integration & >> more >> Get technology previously reserved for billion-dollar corporations, FREE >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> _______________________________________________ >> Capstone-users mailing list >> Cap...@li... >> <mailto:Cap...@li...> >> https://lists.sourceforge.net/lists/listinfo/capstone-users >> >> >> >> >> ------------------------------------------------------------------------------ >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server >> from Actuate! Instantly Supercharge Your Business Reports and Dashboards >> with Interactivity, Sharing, Native Excel Exports, App Integration & more >> Get technology previously reserved for billion-dollar corporations, FREE >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> >> >> >> _______________________________________________ >> Capstone-users mailing list >> Cap...@li... >> https://lists.sourceforge.net/lists/listinfo/capstone-users >> > > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & more > Get technology previously reserved for billion-dollar corporations, FREE > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > _______________________________________________ > Capstone-users mailing list > Cap...@li... > https://lists.sourceforge.net/lists/listinfo/capstone-users > |
|
From: Jan N. <jan...@ne...> - 2014-12-04 13:01:53
|
I think the easiest fix would be to free the memory from a finally clause, as described here: https://docs.python.org/2/reference/simple_stmts.html#the-yield-statement Like this (untested): def disasm(self, code, offset, count=0): all_insn = ctypes.POINTER(_cs_insn)() '''if not _python2: print(code) code = code.encode() print(code)''' res = _cs.cs_disasm(self.csh, code, len(code), offset, count, ctypes.byref(all_insn)) if res > 0: try: for i in range(res): yield CsInsn(self, all_insn[i]) finally: _cs.cs_free(all_insn, res) else: status = _cs.cs_errno(self.csh) if status != CS_ERR_OK: raise CsError(status) return yield On 12/04/2014 01:07 AM, Jasiel Spelman wrote: > Nguyen, > > Here is example code that hopefully better illustrates Juriaan's > point/what Jan is seeing: > > def gen_example(): > print '[gen_example] START' > for i in xrange(2): > print '[gen_example] before yield: ', i > yield i > print '[gen_example] after yield: ', i > print '[gen_example] END' > > print "Typical case" > for i in gen_example(): > print > > print "Jan's case" > for i in gen_example(): > if i == 1: > break > print > > > Typical case > [gen_example] START > [gen_example] before yield: 0 > > [gen_example] after yield: 0 > [gen_example] before yield: 1 > > [gen_example] after yield: 1 > [gen_example] END > > > Jan's case > [gen_example] START > [gen_example] before yield: 0 > > [gen_example] after yield: 0 > [gen_example] before yield: 1 > > > On Wed, Dec 3, 2014 at 5:49 PM, Nguyen Anh Quynh <aq...@gm... > <mailto:aq...@gm...>> wrote: > > > > On Thu, Dec 4, 2014 at 6:32 AM, Jurriaan Bremer > <jur...@gm... <mailto:jur...@gm...>> wrote: > > The 'yield' keyword allows one to fetch new records/rows (in > this case > disassembled instructions) on-demand. Doing an early exit (e.g., > quitting from a for-loop iterating over a function that yield's > after > only, say, 2 out of 10 items) will prematurely exit the function as > well. Because, of course, why would Python calculate the latter 8 > results when they're not used in the first place? This is also > useful > for never-ending functions - I suppose calculating digits of Pi > would be > a 'good' example. > > Anyway, so your cs_free() call is never reached in this case - you > should switch to cs_free()'ing every row after each iteration > through > the for loop. > > > but the loop is called upon the number of successfully disassembled > instructions, > so there is no where in the code that might possibly quit the loop > prematurely. > > thanks, > > Q > > On 12/04/2014 12:27 AM, Nguyen Anh Quynh wrote: > > > > > > On Thu, Dec 4, 2014 at 4:42 AM, Jan Newger <jan...@ne... <mailto:jan...@ne...> > > <mailto:jan...@ne... <mailto:jan...@ne...>>> wrote: > > > > This is the python implementation of the disasm function > (starting at > > line 791): > > > > def disasm(self, code, offset, count=0): > > all_insn = ctypes.POINTER(_cs_insn)() > > '''if not _python2: > > print(code) > > code = code.encode() > > print(code)''' > > res = _cs.cs_disasm(self.csh, code, len(code), > offset, count, > > ctypes.byref(all_insn)) > > if res > 0: > > for i in range(res): > > yield CsInsn(self, all_insn[i]) > > _cs.cs_free(all_insn, res) > > else: > > status = _cs.cs_errno(self.csh) > > if status != CS_ERR_OK: > > raise CsError(status) > > return > > yield > > > > I'm really no python expert, but from what I see you > apparently need to > > free the instruction instances manually. However, if client > code stops > > enumeration over the instructions prematurely, then > _cs.cs_free() is > > never invoked, and thus memory is leaked, right? > > > > > > yes the problem must be with Python binding but not the core. > > however, in the above code, cs_free() is called after the "for" loop, > > so i dont see how memleak can happen "prematurely". > > > > > > thanks. > > > > > > > > > > > > On 03.12.2014 22:25, Jan Newger wrote: > > > It seems the equivalent C implementation is not affected by > the mem > > > leak, which is to be expected, since the memory is > explicitly freed > > > anyways, and the group checking boils down to comparing an > integer > > value. > > > > > > If I had to guess, I'd suspect that in the python case the > group > > > checking code introduces a spurious reference to the > instruction > > > instance(?) which cannot be claimed by the GC. > > > > > > On 03.12.2014 16:57, Jan Newger wrote: > > >> No I haven't tried to reproduce the mem leak with C. > > >> It already took me a considerable amount of time to come > up with this > > >> minimal example. > > >> > > >> On 12/03/2014 04:53 PM, Capstone Engine wrote: > > >>> > > >>> > > >>> On Wed, Dec 3, 2014 at 10:32 PM, Jan Newger > > <jan...@ne... <mailto:jan...@ne...> > <mailto:jan...@ne... <mailto:jan...@ne...>> > > >>> <mailto:jan...@ne... > <mailto:jan...@ne...> <mailto:jan...@ne... > <mailto:jan...@ne...>>>> > > wrote: > > >>> > > >>> > > >>> Yes, it's using the latest version. > > >>> OS was win7 x64 running python 2.7 with 32bit libraries. > > >>> > > >>> > > >>> this is interesting. have you tried to code the same > program in > > C to see > > >>> if the mem leak issue still happens? > > >>> > > >>> > > >>> thanks. > > >>> > > >>> > > >>> > > >>> > > >>> On 12/03/2014 04:31 PM, Nguyen Anh Quynh wrote: > > >>> > > > >>> > > > >>> > On Wed, Dec 3, 2014 at 4:57 PM, Jan Newger > > <jan...@ne... <mailto:jan...@ne...> > <mailto:jan...@ne... <mailto:jan...@ne...>> > > <mailto:jan...@ne... <mailto:jan...@ne...> > <mailto:jan...@ne... <mailto:jan...@ne...>>> > > >>> > <mailto:jan...@ne... > <mailto:jan...@ne...> > > <mailto:jan...@ne... <mailto:jan...@ne...>> > <mailto:jan...@ne... <mailto:jan...@ne...> > > <mailto:jan...@ne... > <mailto:jan...@ne...>>>>> wrote: > > >>> > > > >>> > Hey, > > >>> > > > >>> > I was playing around with a few python scripts > (using > > >>> capstone among > > >>> > other things) and always ran out of memory - and I > > have no > > >>> freaking idea > > >>> > why. > > >>> > > > >>> > > > >>> > is this with the latest 3.0 version? > > >>> > > > >>> > thanks, > > >>> > Q > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > The code is really short: > > >>> > > > >>> > > > >>> > from capstone import Cs > > >>> > from capstone import CS_ARCH_X86 > > >>> > from capstone import CS_MODE_32 > > >>> > from capstone import CS_GRP_JUMP > > >>> > from capstone import CS_GRP_CALL > > >>> > from capstone import CS_GRP_RET > > >>> > from capstone.x86_const import X86_INS_JNE, > X86_INS_JMP > > >>> > > > >>> > """ > > >>> > 0x401000: push ecx > > >>> > 0x401001: pop ecx > > >>> > 0x401002: mov eax, dword ptr [esp + > 0x18] > > >>> > 0x401006: mov eax, dword ptr [eax] > > >>> > 0x401008: sar eax, 0 > > >>> > 0x40100b: xor edi, eax > > >>> > 0x40100d: nop > > >>> > 0x40100e: add dword ptr [esp + 0x18], 4 > > >>> > 0x401013: nop > > >>> > 0x401014: dec word ptr [esp + 0x14] > > >>> > 0x401019: shld edi, ecx, 0 > > >>> > 0x40101d: jne 0x401000 > > >>> > """ > > >>> > def get_code(): > > >>> > CODE = > > >>> > > > >>> > > > "\x51\x59\x8B\x44\x24\x18\x8B\x00\xC1\xF8\x00\x33\xF8\x90\x83\x44\x24\x18\x04\x90\x66\xFF\x4C\x24\x14\x0F\xA4\xCF\x00\x75\xE1" > > >>> > > > >>> > return CODE > > >>> > > > >>> > def is_branch(instr): > > >>> > for group in branch_groups: > > >>> > if group in instr.groups: > > >>> > return True > > >>> > return False > > >>> > #return False > > >>> > > > >>> > # Disassemble until we hit basic block end. > > >>> > def disasm(code): > > >>> > disasm = Cs(CS_ARCH_X86, CS_MODE_32) > > >>> > disasm.detail = True > > >>> > address = 0x401000 > > >>> > for instr in disasm.disasm(code, address): > > >>> > print "0x%x:\t%s\t%s" % (instr.address, > > instr.mnemonic, > > >>> > instr.op_str) > > >>> > if is_branch(instr): > > >>> > break > > >>> > > > >>> > branch_groups = [CS_GRP_JUMP, CS_GRP_CALL, > CS_GRP_RET] > > >>> > code = get_code() > > >>> > while True: > > >>> > disasm(code) > > >>> > > > >>> > > > >>> > That code goes out of memory after a few > seconds. The > > super > > >>> weird thing > > >>> > is, that if I change the implementation of > > "is_branch(instr)" > > >>> to simply > > >>> > return False all the time, then the program > does not > > go out > > >>> of memory! > > >>> > Does anyone have an idea what's going on? > > >>> > > > >>> > Best > > >>> > Jan > > >>> > > > >>> > > > >>> > > > ------------------------------------------------------------------------------ > > >>> > Download BIRT iHub F-Type - The Free > Enterprise-Grade > > BIRT Server > > >>> > from Actuate! Instantly Supercharge Your Business > > Reports and > > >>> Dashboards > > >>> > with Interactivity, Sharing, Native Excel > Exports, App > > >>> Integration & > > >>> > more > > >>> > Get technology previously reserved for > billion-dollar > > >>> corporations, FREE > > >>> > > > >>> > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > >>> > _______________________________________________ > > >>> > Capstone-users mailing list > > >>> > Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > >>> <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>>> > > >>> > <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > >>> <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>>>> > > >>> > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > ------------------------------------------------------------------------------ > > >>> > Download BIRT iHub F-Type - The Free Enterprise-Grade > > BIRT Server > > >>> > from Actuate! Instantly Supercharge Your Business > Reports and > > >>> Dashboards > > >>> > with Interactivity, Sharing, Native Excel Exports, App > > >>> Integration & more > > >>> > Get technology previously reserved for billion-dollar > > >>> corporations, FREE > > >>> > > > >>> > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > >>> > > > >>> > > > >>> > > > >>> > _______________________________________________ > > >>> > Capstone-users mailing list > > >>> > Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > >>> <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>>> > > >>> > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > >>> > > > >>> > > >>> > > >>> > > > ------------------------------------------------------------------------------ > > >>> Download BIRT iHub F-Type - The Free Enterprise-Grade > BIRT > > Server > > >>> from Actuate! Instantly Supercharge Your Business Reports > > and Dashboards > > >>> with Interactivity, Sharing, Native Excel Exports, App > > Integration & > > >>> more > > >>> Get technology previously reserved for billion-dollar > > corporations, FREE > > >>> > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > >>> _______________________________________________ > > >>> Capstone-users mailing list > > >>> Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > >>> <mailto:Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>>> > > >>> https://lists.sourceforge.net/lists/listinfo/capstone-users > > >>> > > >>> > > >>> > > >>> > > >>> > > > ------------------------------------------------------------------------------ > > >>> Download BIRT iHub F-Type - The Free Enterprise-Grade > BIRT Server > > >>> from Actuate! Instantly Supercharge Your Business Reports and > > Dashboards > > >>> with Interactivity, Sharing, Native Excel Exports, App > > Integration & more > > >>> Get technology previously reserved for billion-dollar > > corporations, FREE > > >>> > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > >>> > > >>> > > >>> > > >>> _______________________________________________ > > >>> Capstone-users mailing list > > >>> Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > >>> https://lists.sourceforge.net/lists/listinfo/capstone-users > > >>> > > >> > > >> > > >> > > > ------------------------------------------------------------------------------ > > >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT > Server > > >> from Actuate! Instantly Supercharge Your Business Reports and > > Dashboards > > >> with Interactivity, Sharing, Native Excel Exports, App > > Integration & more > > >> Get technology previously reserved for billion-dollar > > corporations, FREE > > >> > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > >> _______________________________________________ > > >> Capstone-users mailing list > > >> Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > >> https://lists.sourceforge.net/lists/listinfo/capstone-users > > >> > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT > Server > > > from Actuate! Instantly Supercharge Your Business Reports and > > Dashboards > > > with Interactivity, Sharing, Native Excel Exports, App > Integration > > & more > > > Get technology previously reserved for billion-dollar > > corporations, FREE > > > > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > > _______________________________________________ > > > Capstone-users mailing list > > > Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > > > > > > ------------------------------------------------------------------------------ > > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > > from Actuate! Instantly Supercharge Your Business Reports and > Dashboards > > with Interactivity, Sharing, Native Excel Exports, App > Integration & > > more > > Get technology previously reserved for billion-dollar > corporations, FREE > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > _______________________________________________ > > Capstone-users mailing list > > Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > > > > > > > ------------------------------------------------------------------------------ > > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > > from Actuate! Instantly Supercharge Your Business Reports and > Dashboards > > with Interactivity, Sharing, Native Excel Exports, App > Integration & more > > Get technology previously reserved for billion-dollar > corporations, FREE > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > > > > > > > _______________________________________________ > > Capstone-users mailing list > > Cap...@li... > <mailto:Cap...@li...> > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & > more > Get technology previously reserved for billion-dollar corporations, FREE > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > _______________________________________________ > Capstone-users mailing list > Cap...@li... > <mailto:Cap...@li...> > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & > more > Get technology previously reserved for billion-dollar corporations, FREE > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > _______________________________________________ > Capstone-users mailing list > Cap...@li... > <mailto:Cap...@li...> > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & more > Get technology previously reserved for billion-dollar corporations, FREE > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > > > _______________________________________________ > Capstone-users mailing list > Cap...@li... > https://lists.sourceforge.net/lists/listinfo/capstone-users > |
|
From: Jasiel S. <jas...@gm...> - 2014-12-04 00:08:04
|
Nguyen,
Here is example code that hopefully better illustrates Juriaan's point/what
Jan is seeing:
def gen_example():
print '[gen_example] START'
for i in xrange(2):
print '[gen_example] before yield: ', i
yield i
print '[gen_example] after yield: ', i
print '[gen_example] END'
print "Typical case"
for i in gen_example():
print
print "Jan's case"
for i in gen_example():
if i == 1:
break
print
Typical case
[gen_example] START
[gen_example] before yield: 0
[gen_example] after yield: 0
[gen_example] before yield: 1
[gen_example] after yield: 1
[gen_example] END
Jan's case
[gen_example] START
[gen_example] before yield: 0
[gen_example] after yield: 0
[gen_example] before yield: 1
On Wed, Dec 3, 2014 at 5:49 PM, Nguyen Anh Quynh <aq...@gm...> wrote:
>
>
> On Thu, Dec 4, 2014 at 6:32 AM, Jurriaan Bremer <jur...@gm...>
> wrote:
>
>> The 'yield' keyword allows one to fetch new records/rows (in this case
>> disassembled instructions) on-demand. Doing an early exit (e.g.,
>> quitting from a for-loop iterating over a function that yield's after
>> only, say, 2 out of 10 items) will prematurely exit the function as
>> well. Because, of course, why would Python calculate the latter 8
>> results when they're not used in the first place? This is also useful
>> for never-ending functions - I suppose calculating digits of Pi would be
>> a 'good' example.
>>
>> Anyway, so your cs_free() call is never reached in this case - you
>> should switch to cs_free()'ing every row after each iteration through
>> the for loop.
>>
>>
> but the loop is called upon the number of successfully disassembled
> instructions,
> so there is no where in the code that might possibly quit the loop
> prematurely.
>
> thanks,
>
> Q
>
> On 12/04/2014 12:27 AM, Nguyen Anh Quynh wrote:
> >
> >
> > On Thu, Dec 4, 2014 at 4:42 AM, Jan Newger <jan...@ne...
> > <mailto:jan...@ne...>> wrote:
> >
> > This is the python implementation of the disasm function (starting at
> > line 791):
> >
> > def disasm(self, code, offset, count=0):
> > all_insn = ctypes.POINTER(_cs_insn)()
> > '''if not _python2:
> > print(code)
> > code = code.encode()
> > print(code)'''
> > res = _cs.cs_disasm(self.csh, code, len(code), offset, count,
> > ctypes.byref(all_insn))
> > if res > 0:
> > for i in range(res):
> > yield CsInsn(self, all_insn[i])
> > _cs.cs_free(all_insn, res)
> > else:
> > status = _cs.cs_errno(self.csh)
> > if status != CS_ERR_OK:
> > raise CsError(status)
> > return
> > yield
> >
> > I'm really no python expert, but from what I see you apparently need
> to
> > free the instruction instances manually. However, if client code
> stops
> > enumeration over the instructions prematurely, then _cs.cs_free() is
> > never invoked, and thus memory is leaked, right?
> >
> >
> > yes the problem must be with Python binding but not the core.
> > however, in the above code, cs_free() is called after the "for" loop,
> > so i dont see how memleak can happen "prematurely".
> >
> >
> > thanks.
> >
> >
> >
> >
> >
> > On 03.12.2014 22:25, Jan Newger wrote:
> > > It seems the equivalent C implementation is not affected by the mem
> > > leak, which is to be expected, since the memory is explicitly freed
> > > anyways, and the group checking boils down to comparing an integer
> > value.
> > >
> > > If I had to guess, I'd suspect that in the python case the group
> > > checking code introduces a spurious reference to the instruction
> > > instance(?) which cannot be claimed by the GC.
> > >
> > > On 03.12.2014 16:57, Jan Newger wrote:
> > >> No I haven't tried to reproduce the mem leak with C.
> > >> It already took me a considerable amount of time to come up with
> this
> > >> minimal example.
> > >>
> > >> On 12/03/2014 04:53 PM, Capstone Engine wrote:
> > >>>
> > >>>
> > >>> On Wed, Dec 3, 2014 at 10:32 PM, Jan Newger
> > <jan...@ne... <mailto:jan...@ne...>
> > >>> <mailto:jan...@ne... <mailto:jan...@ne...>>>
> > wrote:
> > >>>
> > >>>
> > >>> Yes, it's using the latest version.
> > >>> OS was win7 x64 running python 2.7 with 32bit libraries.
> > >>>
> > >>>
> > >>> this is interesting. have you tried to code the same program in
> > C to see
> > >>> if the mem leak issue still happens?
> > >>>
> > >>>
> > >>> thanks.
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On 12/03/2014 04:31 PM, Nguyen Anh Quynh wrote:
> > >>> >
> > >>> >
> > >>> > On Wed, Dec 3, 2014 at 4:57 PM, Jan Newger
> > <jan...@ne... <mailto:jan...@ne...>
> > <mailto:jan...@ne... <mailto:jan...@ne...>>
> > >>> > <mailto:jan...@ne...
> > <mailto:jan...@ne...> <mailto:jan...@ne...
> > <mailto:jan...@ne...>>>> wrote:
> > >>> >
> > >>> > Hey,
> > >>> >
> > >>> > I was playing around with a few python scripts (using
> > >>> capstone among
> > >>> > other things) and always ran out of memory - and I
> > have no
> > >>> freaking idea
> > >>> > why.
> > >>> >
> > >>> >
> > >>> > is this with the latest 3.0 version?
> > >>> >
> > >>> > thanks,
> > >>> > Q
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> > The code is really short:
> > >>> >
> > >>> >
> > >>> > from capstone import Cs
> > >>> > from capstone import CS_ARCH_X86
> > >>> > from capstone import CS_MODE_32
> > >>> > from capstone import CS_GRP_JUMP
> > >>> > from capstone import CS_GRP_CALL
> > >>> > from capstone import CS_GRP_RET
> > >>> > from capstone.x86_const import X86_INS_JNE,
> X86_INS_JMP
> > >>> >
> > >>> > """
> > >>> > 0x401000: push ecx
> > >>> > 0x401001: pop ecx
> > >>> > 0x401002: mov eax, dword ptr [esp + 0x18]
> > >>> > 0x401006: mov eax, dword ptr [eax]
> > >>> > 0x401008: sar eax, 0
> > >>> > 0x40100b: xor edi, eax
> > >>> > 0x40100d: nop
> > >>> > 0x40100e: add dword ptr [esp + 0x18], 4
> > >>> > 0x401013: nop
> > >>> > 0x401014: dec word ptr [esp + 0x14]
> > >>> > 0x401019: shld edi, ecx, 0
> > >>> > 0x40101d: jne 0x401000
> > >>> > """
> > >>> > def get_code():
> > >>> > CODE =
> > >>> >
> > >>>
> >
> "\x51\x59\x8B\x44\x24\x18\x8B\x00\xC1\xF8\x00\x33\xF8\x90\x83\x44\x24\x18\x04\x90\x66\xFF\x4C\x24\x14\x0F\xA4\xCF\x00\x75\xE1"
> > >>> >
> > >>> > return CODE
> > >>> >
> > >>> > def is_branch(instr):
> > >>> > for group in branch_groups:
> > >>> > if group in instr.groups:
> > >>> > return True
> > >>> > return False
> > >>> > #return False
> > >>> >
> > >>> > # Disassemble until we hit basic block end.
> > >>> > def disasm(code):
> > >>> > disasm = Cs(CS_ARCH_X86, CS_MODE_32)
> > >>> > disasm.detail = True
> > >>> > address = 0x401000
> > >>> > for instr in disasm.disasm(code, address):
> > >>> > print "0x%x:\t%s\t%s" % (instr.address,
> > instr.mnemonic,
> > >>> > instr.op_str)
> > >>> > if is_branch(instr):
> > >>> > break
> > >>> >
> > >>> > branch_groups = [CS_GRP_JUMP, CS_GRP_CALL, CS_GRP_RET]
> > >>> > code = get_code()
> > >>> > while True:
> > >>> > disasm(code)
> > >>> >
> > >>> >
> > >>> > That code goes out of memory after a few seconds. The
> > super
> > >>> weird thing
> > >>> > is, that if I change the implementation of
> > "is_branch(instr)"
> > >>> to simply
> > >>> > return False all the time, then the program does not
> > go out
> > >>> of memory!
> > >>> > Does anyone have an idea what's going on?
> > >>> >
> > >>> > Best
> > >>> > Jan
> > >>> >
> > >>> >
> > >>>
> >
> ------------------------------------------------------------------------------
> > >>> > Download BIRT iHub F-Type - The Free Enterprise-Grade
> > BIRT Server
> > >>> > from Actuate! Instantly Supercharge Your Business
> > Reports and
> > >>> Dashboards
> > >>> > with Interactivity, Sharing, Native Excel Exports, App
> > >>> Integration &
> > >>> > more
> > >>> > Get technology previously reserved for billion-dollar
> > >>> corporations, FREE
> > >>> >
> > >>>
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> > >>> > _______________________________________________
> > >>> > Capstone-users mailing list
> > >>> > Cap...@li...
> > <mailto:Cap...@li...>
> > >>> <mailto:Cap...@li...
> > <mailto:Cap...@li...>>
> > >>> > <mailto:Cap...@li...
> > <mailto:Cap...@li...>
> > >>> <mailto:Cap...@li...
> > <mailto:Cap...@li...>>>
> > >>> >
> https://lists.sourceforge.net/lists/listinfo/capstone-users
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>>
> >
> ------------------------------------------------------------------------------
> > >>> > Download BIRT iHub F-Type - The Free Enterprise-Grade
> > BIRT Server
> > >>> > from Actuate! Instantly Supercharge Your Business Reports
> and
> > >>> Dashboards
> > >>> > with Interactivity, Sharing, Native Excel Exports, App
> > >>> Integration & more
> > >>> > Get technology previously reserved for billion-dollar
> > >>> corporations, FREE
> > >>> >
> > >>>
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> > >>> >
> > >>> >
> > >>> >
> > >>> > _______________________________________________
> > >>> > Capstone-users mailing list
> > >>> > Cap...@li...
> > <mailto:Cap...@li...>
> > >>> <mailto:Cap...@li...
> > <mailto:Cap...@li...>>
> > >>> >
> https://lists.sourceforge.net/lists/listinfo/capstone-users
> > >>> >
> > >>>
> > >>>
> > >>>
> >
> ------------------------------------------------------------------------------
> > >>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT
> > Server
> > >>> from Actuate! Instantly Supercharge Your Business Reports
> > and Dashboards
> > >>> with Interactivity, Sharing, Native Excel Exports, App
> > Integration &
> > >>> more
> > >>> Get technology previously reserved for billion-dollar
> > corporations, FREE
> > >>>
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> > >>> _______________________________________________
> > >>> Capstone-users mailing list
> > >>> Cap...@li...
> > <mailto:Cap...@li...>
> > >>> <mailto:Cap...@li...
> > <mailto:Cap...@li...>>
> > >>> https://lists.sourceforge.net/lists/listinfo/capstone-users
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> >
> ------------------------------------------------------------------------------
> > >>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> > >>> from Actuate! Instantly Supercharge Your Business Reports and
> > Dashboards
> > >>> with Interactivity, Sharing, Native Excel Exports, App
> > Integration & more
> > >>> Get technology previously reserved for billion-dollar
> > corporations, FREE
> > >>>
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> > >>>
> > >>>
> > >>>
> > >>> _______________________________________________
> > >>> Capstone-users mailing list
> > >>> Cap...@li...
> > <mailto:Cap...@li...>
> > >>> https://lists.sourceforge.net/lists/listinfo/capstone-users
> > >>>
> > >>
> > >>
> > >>
> >
> ------------------------------------------------------------------------------
> > >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> > >> from Actuate! Instantly Supercharge Your Business Reports and
> > Dashboards
> > >> with Interactivity, Sharing, Native Excel Exports, App
> > Integration & more
> > >> Get technology previously reserved for billion-dollar
> > corporations, FREE
> > >>
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> > >> _______________________________________________
> > >> Capstone-users mailing list
> > >> Cap...@li...
> > <mailto:Cap...@li...>
> > >> https://lists.sourceforge.net/lists/listinfo/capstone-users
> > >>
> > >
> > >
> > >
> >
> ------------------------------------------------------------------------------
> > > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> > > from Actuate! Instantly Supercharge Your Business Reports and
> > Dashboards
> > > with Interactivity, Sharing, Native Excel Exports, App Integration
> > & more
> > > Get technology previously reserved for billion-dollar
> > corporations, FREE
> > >
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> > > _______________________________________________
> > > Capstone-users mailing list
> > > Cap...@li...
> > <mailto:Cap...@li...>
> > > https://lists.sourceforge.net/lists/listinfo/capstone-users
> > >
> >
> >
> >
> ------------------------------------------------------------------------------
> > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> > from Actuate! Instantly Supercharge Your Business Reports and
> Dashboards
> > with Interactivity, Sharing, Native Excel Exports, App Integration &
> > more
> > Get technology previously reserved for billion-dollar corporations,
> FREE
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> > _______________________________________________
> > Capstone-users mailing list
> > Cap...@li...
> > <mailto:Cap...@li...>
> > https://lists.sourceforge.net/lists/listinfo/capstone-users
> >
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> > from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> > with Interactivity, Sharing, Native Excel Exports, App Integration & more
> > Get technology previously reserved for billion-dollar corporations, FREE
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> >
> >
> >
> > _______________________________________________
> > Capstone-users mailing list
> > Cap...@li...
> > https://lists.sourceforge.net/lists/listinfo/capstone-users
> >
>
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
>
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> _______________________________________________
> Capstone-users mailing list
> Cap...@li...
> https://lists.sourceforge.net/lists/listinfo/capstone-users
>
>
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
>
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> _______________________________________________
> Capstone-users mailing list
> Cap...@li...
> https://lists.sourceforge.net/lists/listinfo/capstone-users
>
>
|
|
From: Nguyen A. Q. <aq...@gm...> - 2014-12-03 23:50:22
|
On Thu, Dec 4, 2014 at 6:32 AM, Jurriaan Bremer <jur...@gm...> wrote: > The 'yield' keyword allows one to fetch new records/rows (in this case > disassembled instructions) on-demand. Doing an early exit (e.g., > quitting from a for-loop iterating over a function that yield's after > only, say, 2 out of 10 items) will prematurely exit the function as > well. Because, of course, why would Python calculate the latter 8 > results when they're not used in the first place? This is also useful > for never-ending functions - I suppose calculating digits of Pi would be > a 'good' example. > > Anyway, so your cs_free() call is never reached in this case - you > should switch to cs_free()'ing every row after each iteration through > the for loop. > > but the loop is called upon the number of successfully disassembled instructions, so there is no where in the code that might possibly quit the loop prematurely. thanks, Q On 12/04/2014 12:27 AM, Nguyen Anh Quynh wrote: > > > On Thu, Dec 4, 2014 at 4:42 AM, Jan Newger <jan...@ne... > <mailto:jan...@ne...>> wrote: > > This is the python implementation of the disasm function (starting at > line 791): > > def disasm(self, code, offset, count=0): > all_insn = ctypes.POINTER(_cs_insn)() > '''if not _python2: > print(code) > code = code.encode() > print(code)''' > res = _cs.cs_disasm(self.csh, code, len(code), offset, count, > ctypes.byref(all_insn)) > if res > 0: > for i in range(res): > yield CsInsn(self, all_insn[i]) > _cs.cs_free(all_insn, res) > else: > status = _cs.cs_errno(self.csh) > if status != CS_ERR_OK: > raise CsError(status) > return > yield > > I'm really no python expert, but from what I see you apparently need to > free the instruction instances manually. However, if client code stops > enumeration over the instructions prematurely, then _cs.cs_free() is > never invoked, and thus memory is leaked, right? > > > yes the problem must be with Python binding but not the core. > however, in the above code, cs_free() is called after the "for" loop, > so i dont see how memleak can happen "prematurely". > > > thanks. > > > > > > On 03.12.2014 22:25, Jan Newger wrote: > > It seems the equivalent C implementation is not affected by the mem > > leak, which is to be expected, since the memory is explicitly freed > > anyways, and the group checking boils down to comparing an integer > value. > > > > If I had to guess, I'd suspect that in the python case the group > > checking code introduces a spurious reference to the instruction > > instance(?) which cannot be claimed by the GC. > > > > On 03.12.2014 16:57, Jan Newger wrote: > >> No I haven't tried to reproduce the mem leak with C. > >> It already took me a considerable amount of time to come up with this > >> minimal example. > >> > >> On 12/03/2014 04:53 PM, Capstone Engine wrote: > >>> > >>> > >>> On Wed, Dec 3, 2014 at 10:32 PM, Jan Newger > <jan...@ne... <mailto:jan...@ne...> > >>> <mailto:jan...@ne... <mailto:jan...@ne...>>> > wrote: > >>> > >>> > >>> Yes, it's using the latest version. > >>> OS was win7 x64 running python 2.7 with 32bit libraries. > >>> > >>> > >>> this is interesting. have you tried to code the same program in > C to see > >>> if the mem leak issue still happens? > >>> > >>> > >>> thanks. > >>> > >>> > >>> > >>> > >>> On 12/03/2014 04:31 PM, Nguyen Anh Quynh wrote: > >>> > > >>> > > >>> > On Wed, Dec 3, 2014 at 4:57 PM, Jan Newger > <jan...@ne... <mailto:jan...@ne...> > <mailto:jan...@ne... <mailto:jan...@ne...>> > >>> > <mailto:jan...@ne... > <mailto:jan...@ne...> <mailto:jan...@ne... > <mailto:jan...@ne...>>>> wrote: > >>> > > >>> > Hey, > >>> > > >>> > I was playing around with a few python scripts (using > >>> capstone among > >>> > other things) and always ran out of memory - and I > have no > >>> freaking idea > >>> > why. > >>> > > >>> > > >>> > is this with the latest 3.0 version? > >>> > > >>> > thanks, > >>> > Q > >>> > > >>> > > >>> > > >>> > > >>> > The code is really short: > >>> > > >>> > > >>> > from capstone import Cs > >>> > from capstone import CS_ARCH_X86 > >>> > from capstone import CS_MODE_32 > >>> > from capstone import CS_GRP_JUMP > >>> > from capstone import CS_GRP_CALL > >>> > from capstone import CS_GRP_RET > >>> > from capstone.x86_const import X86_INS_JNE, X86_INS_JMP > >>> > > >>> > """ > >>> > 0x401000: push ecx > >>> > 0x401001: pop ecx > >>> > 0x401002: mov eax, dword ptr [esp + 0x18] > >>> > 0x401006: mov eax, dword ptr [eax] > >>> > 0x401008: sar eax, 0 > >>> > 0x40100b: xor edi, eax > >>> > 0x40100d: nop > >>> > 0x40100e: add dword ptr [esp + 0x18], 4 > >>> > 0x401013: nop > >>> > 0x401014: dec word ptr [esp + 0x14] > >>> > 0x401019: shld edi, ecx, 0 > >>> > 0x40101d: jne 0x401000 > >>> > """ > >>> > def get_code(): > >>> > CODE = > >>> > > >>> > "\x51\x59\x8B\x44\x24\x18\x8B\x00\xC1\xF8\x00\x33\xF8\x90\x83\x44\x24\x18\x04\x90\x66\xFF\x4C\x24\x14\x0F\xA4\xCF\x00\x75\xE1" > >>> > > >>> > return CODE > >>> > > >>> > def is_branch(instr): > >>> > for group in branch_groups: > >>> > if group in instr.groups: > >>> > return True > >>> > return False > >>> > #return False > >>> > > >>> > # Disassemble until we hit basic block end. > >>> > def disasm(code): > >>> > disasm = Cs(CS_ARCH_X86, CS_MODE_32) > >>> > disasm.detail = True > >>> > address = 0x401000 > >>> > for instr in disasm.disasm(code, address): > >>> > print "0x%x:\t%s\t%s" % (instr.address, > instr.mnemonic, > >>> > instr.op_str) > >>> > if is_branch(instr): > >>> > break > >>> > > >>> > branch_groups = [CS_GRP_JUMP, CS_GRP_CALL, CS_GRP_RET] > >>> > code = get_code() > >>> > while True: > >>> > disasm(code) > >>> > > >>> > > >>> > That code goes out of memory after a few seconds. The > super > >>> weird thing > >>> > is, that if I change the implementation of > "is_branch(instr)" > >>> to simply > >>> > return False all the time, then the program does not > go out > >>> of memory! > >>> > Does anyone have an idea what's going on? > >>> > > >>> > Best > >>> > Jan > >>> > > >>> > > >>> > ------------------------------------------------------------------------------ > >>> > Download BIRT iHub F-Type - The Free Enterprise-Grade > BIRT Server > >>> > from Actuate! Instantly Supercharge Your Business > Reports and > >>> Dashboards > >>> > with Interactivity, Sharing, Native Excel Exports, App > >>> Integration & > >>> > more > >>> > Get technology previously reserved for billion-dollar > >>> corporations, FREE > >>> > > >>> > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > >>> > _______________________________________________ > >>> > Capstone-users mailing list > >>> > Cap...@li... > <mailto:Cap...@li...> > >>> <mailto:Cap...@li... > <mailto:Cap...@li...>> > >>> > <mailto:Cap...@li... > <mailto:Cap...@li...> > >>> <mailto:Cap...@li... > <mailto:Cap...@li...>>> > >>> > https://lists.sourceforge.net/lists/listinfo/capstone-users > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > ------------------------------------------------------------------------------ > >>> > Download BIRT iHub F-Type - The Free Enterprise-Grade > BIRT Server > >>> > from Actuate! Instantly Supercharge Your Business Reports and > >>> Dashboards > >>> > with Interactivity, Sharing, Native Excel Exports, App > >>> Integration & more > >>> > Get technology previously reserved for billion-dollar > >>> corporations, FREE > >>> > > >>> > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > >>> > > >>> > > >>> > > >>> > _______________________________________________ > >>> > Capstone-users mailing list > >>> > Cap...@li... > <mailto:Cap...@li...> > >>> <mailto:Cap...@li... > <mailto:Cap...@li...>> > >>> > https://lists.sourceforge.net/lists/listinfo/capstone-users > >>> > > >>> > >>> > >>> > ------------------------------------------------------------------------------ > >>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT > Server > >>> from Actuate! Instantly Supercharge Your Business Reports > and Dashboards > >>> with Interactivity, Sharing, Native Excel Exports, App > Integration & > >>> more > >>> Get technology previously reserved for billion-dollar > corporations, FREE > >>> > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > >>> _______________________________________________ > >>> Capstone-users mailing list > >>> Cap...@li... > <mailto:Cap...@li...> > >>> <mailto:Cap...@li... > <mailto:Cap...@li...>> > >>> https://lists.sourceforge.net/lists/listinfo/capstone-users > >>> > >>> > >>> > >>> > >>> > ------------------------------------------------------------------------------ > >>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > >>> from Actuate! Instantly Supercharge Your Business Reports and > Dashboards > >>> with Interactivity, Sharing, Native Excel Exports, App > Integration & more > >>> Get technology previously reserved for billion-dollar > corporations, FREE > >>> > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > >>> > >>> > >>> > >>> _______________________________________________ > >>> Capstone-users mailing list > >>> Cap...@li... > <mailto:Cap...@li...> > >>> https://lists.sourceforge.net/lists/listinfo/capstone-users > >>> > >> > >> > >> > ------------------------------------------------------------------------------ > >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > >> from Actuate! Instantly Supercharge Your Business Reports and > Dashboards > >> with Interactivity, Sharing, Native Excel Exports, App > Integration & more > >> Get technology previously reserved for billion-dollar > corporations, FREE > >> > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > >> _______________________________________________ > >> Capstone-users mailing list > >> Cap...@li... > <mailto:Cap...@li...> > >> https://lists.sourceforge.net/lists/listinfo/capstone-users > >> > > > > > > > ------------------------------------------------------------------------------ > > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > > from Actuate! Instantly Supercharge Your Business Reports and > Dashboards > > with Interactivity, Sharing, Native Excel Exports, App Integration > & more > > Get technology previously reserved for billion-dollar > corporations, FREE > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > _______________________________________________ > > Capstone-users mailing list > > Cap...@li... > <mailto:Cap...@li...> > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & > more > Get technology previously reserved for billion-dollar corporations, FREE > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > _______________________________________________ > Capstone-users mailing list > Cap...@li... > <mailto:Cap...@li...> > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & more > Get technology previously reserved for billion-dollar corporations, FREE > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > > > _______________________________________________ > Capstone-users mailing list > Cap...@li... > https://lists.sourceforge.net/lists/listinfo/capstone-users > ------------------------------------------------------------------------------ Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server from Actuate! Instantly Supercharge Your Business Reports and Dashboards with Interactivity, Sharing, Native Excel Exports, App Integration & more Get technology previously reserved for billion-dollar corporations, FREE http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk _______________________________________________ Capstone-users mailing list Cap...@li... https://lists.sourceforge.net/lists/listinfo/capstone-users |
|
From: Jurriaan B. <jur...@gm...> - 2014-12-03 23:33:00
|
The 'yield' keyword allows one to fetch new records/rows (in this case disassembled instructions) on-demand. Doing an early exit (e.g., quitting from a for-loop iterating over a function that yield's after only, say, 2 out of 10 items) will prematurely exit the function as well. Because, of course, why would Python calculate the latter 8 results when they're not used in the first place? This is also useful for never-ending functions - I suppose calculating digits of Pi would be a 'good' example. Anyway, so your cs_free() call is never reached in this case - you should switch to cs_free()'ing every row after each iteration through the for loop. Regards, Jurriaan On 12/04/2014 12:27 AM, Nguyen Anh Quynh wrote: > > > On Thu, Dec 4, 2014 at 4:42 AM, Jan Newger <jan...@ne... > <mailto:jan...@ne...>> wrote: > > This is the python implementation of the disasm function (starting at > line 791): > > def disasm(self, code, offset, count=0): > all_insn = ctypes.POINTER(_cs_insn)() > '''if not _python2: > print(code) > code = code.encode() > print(code)''' > res = _cs.cs_disasm(self.csh, code, len(code), offset, count, > ctypes.byref(all_insn)) > if res > 0: > for i in range(res): > yield CsInsn(self, all_insn[i]) > _cs.cs_free(all_insn, res) > else: > status = _cs.cs_errno(self.csh) > if status != CS_ERR_OK: > raise CsError(status) > return > yield > > I'm really no python expert, but from what I see you apparently need to > free the instruction instances manually. However, if client code stops > enumeration over the instructions prematurely, then _cs.cs_free() is > never invoked, and thus memory is leaked, right? > > > yes the problem must be with Python binding but not the core. > however, in the above code, cs_free() is called after the "for" loop, > so i dont see how memleak can happen "prematurely". > > > thanks. > > > > > > On 03.12.2014 22:25, Jan Newger wrote: > > It seems the equivalent C implementation is not affected by the mem > > leak, which is to be expected, since the memory is explicitly freed > > anyways, and the group checking boils down to comparing an integer > value. > > > > If I had to guess, I'd suspect that in the python case the group > > checking code introduces a spurious reference to the instruction > > instance(?) which cannot be claimed by the GC. > > > > On 03.12.2014 16:57, Jan Newger wrote: > >> No I haven't tried to reproduce the mem leak with C. > >> It already took me a considerable amount of time to come up with this > >> minimal example. > >> > >> On 12/03/2014 04:53 PM, Capstone Engine wrote: > >>> > >>> > >>> On Wed, Dec 3, 2014 at 10:32 PM, Jan Newger > <jan...@ne... <mailto:jan...@ne...> > >>> <mailto:jan...@ne... <mailto:jan...@ne...>>> > wrote: > >>> > >>> > >>> Yes, it's using the latest version. > >>> OS was win7 x64 running python 2.7 with 32bit libraries. > >>> > >>> > >>> this is interesting. have you tried to code the same program in > C to see > >>> if the mem leak issue still happens? > >>> > >>> > >>> thanks. > >>> > >>> > >>> > >>> > >>> On 12/03/2014 04:31 PM, Nguyen Anh Quynh wrote: > >>> > > >>> > > >>> > On Wed, Dec 3, 2014 at 4:57 PM, Jan Newger > <jan...@ne... <mailto:jan...@ne...> > <mailto:jan...@ne... <mailto:jan...@ne...>> > >>> > <mailto:jan...@ne... > <mailto:jan...@ne...> <mailto:jan...@ne... > <mailto:jan...@ne...>>>> wrote: > >>> > > >>> > Hey, > >>> > > >>> > I was playing around with a few python scripts (using > >>> capstone among > >>> > other things) and always ran out of memory - and I > have no > >>> freaking idea > >>> > why. > >>> > > >>> > > >>> > is this with the latest 3.0 version? > >>> > > >>> > thanks, > >>> > Q > >>> > > >>> > > >>> > > >>> > > >>> > The code is really short: > >>> > > >>> > > >>> > from capstone import Cs > >>> > from capstone import CS_ARCH_X86 > >>> > from capstone import CS_MODE_32 > >>> > from capstone import CS_GRP_JUMP > >>> > from capstone import CS_GRP_CALL > >>> > from capstone import CS_GRP_RET > >>> > from capstone.x86_const import X86_INS_JNE, X86_INS_JMP > >>> > > >>> > """ > >>> > 0x401000: push ecx > >>> > 0x401001: pop ecx > >>> > 0x401002: mov eax, dword ptr [esp + 0x18] > >>> > 0x401006: mov eax, dword ptr [eax] > >>> > 0x401008: sar eax, 0 > >>> > 0x40100b: xor edi, eax > >>> > 0x40100d: nop > >>> > 0x40100e: add dword ptr [esp + 0x18], 4 > >>> > 0x401013: nop > >>> > 0x401014: dec word ptr [esp + 0x14] > >>> > 0x401019: shld edi, ecx, 0 > >>> > 0x40101d: jne 0x401000 > >>> > """ > >>> > def get_code(): > >>> > CODE = > >>> > > >>> > "\x51\x59\x8B\x44\x24\x18\x8B\x00\xC1\xF8\x00\x33\xF8\x90\x83\x44\x24\x18\x04\x90\x66\xFF\x4C\x24\x14\x0F\xA4\xCF\x00\x75\xE1" > >>> > > >>> > return CODE > >>> > > >>> > def is_branch(instr): > >>> > for group in branch_groups: > >>> > if group in instr.groups: > >>> > return True > >>> > return False > >>> > #return False > >>> > > >>> > # Disassemble until we hit basic block end. > >>> > def disasm(code): > >>> > disasm = Cs(CS_ARCH_X86, CS_MODE_32) > >>> > disasm.detail = True > >>> > address = 0x401000 > >>> > for instr in disasm.disasm(code, address): > >>> > print "0x%x:\t%s\t%s" % (instr.address, > instr.mnemonic, > >>> > instr.op_str) > >>> > if is_branch(instr): > >>> > break > >>> > > >>> > branch_groups = [CS_GRP_JUMP, CS_GRP_CALL, CS_GRP_RET] > >>> > code = get_code() > >>> > while True: > >>> > disasm(code) > >>> > > >>> > > >>> > That code goes out of memory after a few seconds. The > super > >>> weird thing > >>> > is, that if I change the implementation of > "is_branch(instr)" > >>> to simply > >>> > return False all the time, then the program does not > go out > >>> of memory! > >>> > Does anyone have an idea what's going on? > >>> > > >>> > Best > >>> > Jan > >>> > > >>> > > >>> > ------------------------------------------------------------------------------ > >>> > Download BIRT iHub F-Type - The Free Enterprise-Grade > BIRT Server > >>> > from Actuate! Instantly Supercharge Your Business > Reports and > >>> Dashboards > >>> > with Interactivity, Sharing, Native Excel Exports, App > >>> Integration & > >>> > more > >>> > Get technology previously reserved for billion-dollar > >>> corporations, FREE > >>> > > >>> > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > >>> > _______________________________________________ > >>> > Capstone-users mailing list > >>> > Cap...@li... > <mailto:Cap...@li...> > >>> <mailto:Cap...@li... > <mailto:Cap...@li...>> > >>> > <mailto:Cap...@li... > <mailto:Cap...@li...> > >>> <mailto:Cap...@li... > <mailto:Cap...@li...>>> > >>> > https://lists.sourceforge.net/lists/listinfo/capstone-users > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > ------------------------------------------------------------------------------ > >>> > Download BIRT iHub F-Type - The Free Enterprise-Grade > BIRT Server > >>> > from Actuate! Instantly Supercharge Your Business Reports and > >>> Dashboards > >>> > with Interactivity, Sharing, Native Excel Exports, App > >>> Integration & more > >>> > Get technology previously reserved for billion-dollar > >>> corporations, FREE > >>> > > >>> > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > >>> > > >>> > > >>> > > >>> > _______________________________________________ > >>> > Capstone-users mailing list > >>> > Cap...@li... > <mailto:Cap...@li...> > >>> <mailto:Cap...@li... > <mailto:Cap...@li...>> > >>> > https://lists.sourceforge.net/lists/listinfo/capstone-users > >>> > > >>> > >>> > >>> > ------------------------------------------------------------------------------ > >>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT > Server > >>> from Actuate! Instantly Supercharge Your Business Reports > and Dashboards > >>> with Interactivity, Sharing, Native Excel Exports, App > Integration & > >>> more > >>> Get technology previously reserved for billion-dollar > corporations, FREE > >>> > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > >>> _______________________________________________ > >>> Capstone-users mailing list > >>> Cap...@li... > <mailto:Cap...@li...> > >>> <mailto:Cap...@li... > <mailto:Cap...@li...>> > >>> https://lists.sourceforge.net/lists/listinfo/capstone-users > >>> > >>> > >>> > >>> > >>> > ------------------------------------------------------------------------------ > >>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > >>> from Actuate! Instantly Supercharge Your Business Reports and > Dashboards > >>> with Interactivity, Sharing, Native Excel Exports, App > Integration & more > >>> Get technology previously reserved for billion-dollar > corporations, FREE > >>> > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > >>> > >>> > >>> > >>> _______________________________________________ > >>> Capstone-users mailing list > >>> Cap...@li... > <mailto:Cap...@li...> > >>> https://lists.sourceforge.net/lists/listinfo/capstone-users > >>> > >> > >> > >> > ------------------------------------------------------------------------------ > >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > >> from Actuate! Instantly Supercharge Your Business Reports and > Dashboards > >> with Interactivity, Sharing, Native Excel Exports, App > Integration & more > >> Get technology previously reserved for billion-dollar > corporations, FREE > >> > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > >> _______________________________________________ > >> Capstone-users mailing list > >> Cap...@li... > <mailto:Cap...@li...> > >> https://lists.sourceforge.net/lists/listinfo/capstone-users > >> > > > > > > > ------------------------------------------------------------------------------ > > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > > from Actuate! Instantly Supercharge Your Business Reports and > Dashboards > > with Interactivity, Sharing, Native Excel Exports, App Integration > & more > > Get technology previously reserved for billion-dollar > corporations, FREE > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > _______________________________________________ > > Capstone-users mailing list > > Cap...@li... > <mailto:Cap...@li...> > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & > more > Get technology previously reserved for billion-dollar corporations, FREE > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > _______________________________________________ > Capstone-users mailing list > Cap...@li... > <mailto:Cap...@li...> > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & more > Get technology previously reserved for billion-dollar corporations, FREE > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > > > _______________________________________________ > Capstone-users mailing list > Cap...@li... > https://lists.sourceforge.net/lists/listinfo/capstone-users > |
|
From: Nguyen A. Q. <aq...@gm...> - 2014-12-03 23:28:16
|
On Thu, Dec 4, 2014 at 4:42 AM, Jan Newger <jan...@ne...> wrote: > This is the python implementation of the disasm function (starting at > line 791): > > def disasm(self, code, offset, count=0): > all_insn = ctypes.POINTER(_cs_insn)() > '''if not _python2: > print(code) > code = code.encode() > print(code)''' > res = _cs.cs_disasm(self.csh, code, len(code), offset, count, > ctypes.byref(all_insn)) > if res > 0: > for i in range(res): > yield CsInsn(self, all_insn[i]) > _cs.cs_free(all_insn, res) > else: > status = _cs.cs_errno(self.csh) > if status != CS_ERR_OK: > raise CsError(status) > return > yield > > I'm really no python expert, but from what I see you apparently need to > free the instruction instances manually. However, if client code stops > enumeration over the instructions prematurely, then _cs.cs_free() is > never invoked, and thus memory is leaked, right? > yes the problem must be with Python binding but not the core. however, in the above code, cs_free() is called after the "for" loop, so i dont see how memleak can happen "prematurely". thanks. > On 03.12.2014 22:25, Jan Newger wrote: > > It seems the equivalent C implementation is not affected by the mem > > leak, which is to be expected, since the memory is explicitly freed > > anyways, and the group checking boils down to comparing an integer value. > > > > If I had to guess, I'd suspect that in the python case the group > > checking code introduces a spurious reference to the instruction > > instance(?) which cannot be claimed by the GC. > > > > On 03.12.2014 16:57, Jan Newger wrote: > >> No I haven't tried to reproduce the mem leak with C. > >> It already took me a considerable amount of time to come up with this > >> minimal example. > >> > >> On 12/03/2014 04:53 PM, Capstone Engine wrote: > >>> > >>> > >>> On Wed, Dec 3, 2014 at 10:32 PM, Jan Newger <jan...@ne... > >>> <mailto:jan...@ne...>> wrote: > >>> > >>> > >>> Yes, it's using the latest version. > >>> OS was win7 x64 running python 2.7 with 32bit libraries. > >>> > >>> > >>> this is interesting. have you tried to code the same program in C to > see > >>> if the mem leak issue still happens? > >>> > >>> > >>> thanks. > >>> > >>> > >>> > >>> > >>> On 12/03/2014 04:31 PM, Nguyen Anh Quynh wrote: > >>> > > >>> > > >>> > On Wed, Dec 3, 2014 at 4:57 PM, Jan Newger < > jan...@ne... <mailto:jan...@ne...> > >>> > <mailto:jan...@ne... <mailto:jan...@ne...>>> > wrote: > >>> > > >>> > Hey, > >>> > > >>> > I was playing around with a few python scripts (using > >>> capstone among > >>> > other things) and always ran out of memory - and I have no > >>> freaking idea > >>> > why. > >>> > > >>> > > >>> > is this with the latest 3.0 version? > >>> > > >>> > thanks, > >>> > Q > >>> > > >>> > > >>> > > >>> > > >>> > The code is really short: > >>> > > >>> > > >>> > from capstone import Cs > >>> > from capstone import CS_ARCH_X86 > >>> > from capstone import CS_MODE_32 > >>> > from capstone import CS_GRP_JUMP > >>> > from capstone import CS_GRP_CALL > >>> > from capstone import CS_GRP_RET > >>> > from capstone.x86_const import X86_INS_JNE, X86_INS_JMP > >>> > > >>> > """ > >>> > 0x401000: push ecx > >>> > 0x401001: pop ecx > >>> > 0x401002: mov eax, dword ptr [esp + 0x18] > >>> > 0x401006: mov eax, dword ptr [eax] > >>> > 0x401008: sar eax, 0 > >>> > 0x40100b: xor edi, eax > >>> > 0x40100d: nop > >>> > 0x40100e: add dword ptr [esp + 0x18], 4 > >>> > 0x401013: nop > >>> > 0x401014: dec word ptr [esp + 0x14] > >>> > 0x401019: shld edi, ecx, 0 > >>> > 0x40101d: jne 0x401000 > >>> > """ > >>> > def get_code(): > >>> > CODE = > >>> > > >>> > "\x51\x59\x8B\x44\x24\x18\x8B\x00\xC1\xF8\x00\x33\xF8\x90\x83\x44\x24\x18\x04\x90\x66\xFF\x4C\x24\x14\x0F\xA4\xCF\x00\x75\xE1" > >>> > > >>> > return CODE > >>> > > >>> > def is_branch(instr): > >>> > for group in branch_groups: > >>> > if group in instr.groups: > >>> > return True > >>> > return False > >>> > #return False > >>> > > >>> > # Disassemble until we hit basic block end. > >>> > def disasm(code): > >>> > disasm = Cs(CS_ARCH_X86, CS_MODE_32) > >>> > disasm.detail = True > >>> > address = 0x401000 > >>> > for instr in disasm.disasm(code, address): > >>> > print "0x%x:\t%s\t%s" % (instr.address, > instr.mnemonic, > >>> > instr.op_str) > >>> > if is_branch(instr): > >>> > break > >>> > > >>> > branch_groups = [CS_GRP_JUMP, CS_GRP_CALL, CS_GRP_RET] > >>> > code = get_code() > >>> > while True: > >>> > disasm(code) > >>> > > >>> > > >>> > That code goes out of memory after a few seconds. The super > >>> weird thing > >>> > is, that if I change the implementation of > "is_branch(instr)" > >>> to simply > >>> > return False all the time, then the program does not go out > >>> of memory! > >>> > Does anyone have an idea what's going on? > >>> > > >>> > Best > >>> > Jan > >>> > > >>> > > >>> > ------------------------------------------------------------------------------ > >>> > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT > Server > >>> > from Actuate! Instantly Supercharge Your Business Reports > and > >>> Dashboards > >>> > with Interactivity, Sharing, Native Excel Exports, App > >>> Integration & > >>> > more > >>> > Get technology previously reserved for billion-dollar > >>> corporations, FREE > >>> > > >>> > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > >>> > _______________________________________________ > >>> > Capstone-users mailing list > >>> > Cap...@li... > >>> <mailto:Cap...@li...> > >>> > <mailto:Cap...@li... > >>> <mailto:Cap...@li...>> > >>> > https://lists.sourceforge.net/lists/listinfo/capstone-users > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > ------------------------------------------------------------------------------ > >>> > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT > Server > >>> > from Actuate! Instantly Supercharge Your Business Reports and > >>> Dashboards > >>> > with Interactivity, Sharing, Native Excel Exports, App > >>> Integration & more > >>> > Get technology previously reserved for billion-dollar > >>> corporations, FREE > >>> > > >>> > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > >>> > > >>> > > >>> > > >>> > _______________________________________________ > >>> > Capstone-users mailing list > >>> > Cap...@li... > >>> <mailto:Cap...@li...> > >>> > https://lists.sourceforge.net/lists/listinfo/capstone-users > >>> > > >>> > >>> > >>> > ------------------------------------------------------------------------------ > >>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > >>> from Actuate! Instantly Supercharge Your Business Reports and > Dashboards > >>> with Interactivity, Sharing, Native Excel Exports, App Integration > & > >>> more > >>> Get technology previously reserved for billion-dollar > corporations, FREE > >>> > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > >>> _______________________________________________ > >>> Capstone-users mailing list > >>> Cap...@li... > >>> <mailto:Cap...@li...> > >>> https://lists.sourceforge.net/lists/listinfo/capstone-users > >>> > >>> > >>> > >>> > >>> > ------------------------------------------------------------------------------ > >>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > >>> from Actuate! Instantly Supercharge Your Business Reports and > Dashboards > >>> with Interactivity, Sharing, Native Excel Exports, App Integration & > more > >>> Get technology previously reserved for billion-dollar corporations, > FREE > >>> > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > >>> > >>> > >>> > >>> _______________________________________________ > >>> Capstone-users mailing list > >>> Cap...@li... > >>> https://lists.sourceforge.net/lists/listinfo/capstone-users > >>> > >> > >> > >> > ------------------------------------------------------------------------------ > >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > >> from Actuate! Instantly Supercharge Your Business Reports and Dashboards > >> with Interactivity, Sharing, Native Excel Exports, App Integration & > more > >> Get technology previously reserved for billion-dollar corporations, FREE > >> > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > >> _______________________________________________ > >> Capstone-users mailing list > >> Cap...@li... > >> https://lists.sourceforge.net/lists/listinfo/capstone-users > >> > > > > > > > ------------------------------------------------------------------------------ > > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > > with Interactivity, Sharing, Native Excel Exports, App Integration & more > > Get technology previously reserved for billion-dollar corporations, FREE > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > _______________________________________________ > > Capstone-users mailing list > > Cap...@li... > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & more > Get technology previously reserved for billion-dollar corporations, FREE > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > _______________________________________________ > Capstone-users mailing list > Cap...@li... > https://lists.sourceforge.net/lists/listinfo/capstone-users > |
|
From: Jay O. <ja...@ko...> - 2014-12-03 22:41:25
|
The _cs.cs_free call can be made within the special __del__ method (a destructor), but this is only safe in a reference-counting garbage collector (like Python) when there are no circular references. See documentation for more info: https://docs.python.org/2/reference/datamodel.html#object.__del__ This will fix the leak by freeing memory when the generator goes out of scope. Which, in your particular case, is an acceptable solution. I would recommend the GC not be relied upon for freeing memory, but use it instead as a safety net. On Wed, Dec 3, 2014 at 1:42 PM, Jan Newger <jan...@ne...> wrote: > This is the python implementation of the disasm function (starting at > line 791): > > def disasm(self, code, offset, count=0): > all_insn = ctypes.POINTER(_cs_insn)() > '''if not _python2: > print(code) > code = code.encode() > print(code)''' > res = _cs.cs_disasm(self.csh, code, len(code), offset, count, > ctypes.byref(all_insn)) > if res > 0: > for i in range(res): > yield CsInsn(self, all_insn[i]) > _cs.cs_free(all_insn, res) > else: > status = _cs.cs_errno(self.csh) > if status != CS_ERR_OK: > raise CsError(status) > return > yield > > I'm really no python expert, but from what I see you apparently need to > free the instruction instances manually. However, if client code stops > enumeration over the instructions prematurely, then _cs.cs_free() is > never invoked, and thus memory is leaked, right? > > On 03.12.2014 22:25, Jan Newger wrote: > > It seems the equivalent C implementation is not affected by the mem > > leak, which is to be expected, since the memory is explicitly freed > > anyways, and the group checking boils down to comparing an integer value. > > > > If I had to guess, I'd suspect that in the python case the group > > checking code introduces a spurious reference to the instruction > > instance(?) which cannot be claimed by the GC. > > > > On 03.12.2014 16:57, Jan Newger wrote: > >> No I haven't tried to reproduce the mem leak with C. > >> It already took me a considerable amount of time to come up with this > >> minimal example. > >> > >> On 12/03/2014 04:53 PM, Capstone Engine wrote: > >>> > >>> > >>> On Wed, Dec 3, 2014 at 10:32 PM, Jan Newger <jan...@ne... > >>> <mailto:jan...@ne...>> wrote: > >>> > >>> > >>> Yes, it's using the latest version. > >>> OS was win7 x64 running python 2.7 with 32bit libraries. > >>> > >>> > >>> this is interesting. have you tried to code the same program in C to > see > >>> if the mem leak issue still happens? > >>> > >>> > >>> thanks. > >>> > >>> > >>> > >>> > >>> On 12/03/2014 04:31 PM, Nguyen Anh Quynh wrote: > >>> > > >>> > > >>> > On Wed, Dec 3, 2014 at 4:57 PM, Jan Newger < > jan...@ne... <mailto:jan...@ne...> > >>> > <mailto:jan...@ne... <mailto:jan...@ne...>>> > wrote: > >>> > > >>> > Hey, > >>> > > >>> > I was playing around with a few python scripts (using > >>> capstone among > >>> > other things) and always ran out of memory - and I have no > >>> freaking idea > >>> > why. > >>> > > >>> > > >>> > is this with the latest 3.0 version? > >>> > > >>> > thanks, > >>> > Q > >>> > > >>> > > >>> > > >>> > > >>> > The code is really short: > >>> > > >>> > > >>> > from capstone import Cs > >>> > from capstone import CS_ARCH_X86 > >>> > from capstone import CS_MODE_32 > >>> > from capstone import CS_GRP_JUMP > >>> > from capstone import CS_GRP_CALL > >>> > from capstone import CS_GRP_RET > >>> > from capstone.x86_const import X86_INS_JNE, X86_INS_JMP > >>> > > >>> > """ > >>> > 0x401000: push ecx > >>> > 0x401001: pop ecx > >>> > 0x401002: mov eax, dword ptr [esp + 0x18] > >>> > 0x401006: mov eax, dword ptr [eax] > >>> > 0x401008: sar eax, 0 > >>> > 0x40100b: xor edi, eax > >>> > 0x40100d: nop > >>> > 0x40100e: add dword ptr [esp + 0x18], 4 > >>> > 0x401013: nop > >>> > 0x401014: dec word ptr [esp + 0x14] > >>> > 0x401019: shld edi, ecx, 0 > >>> > 0x40101d: jne 0x401000 > >>> > """ > >>> > def get_code(): > >>> > CODE = > >>> > > >>> > "\x51\x59\x8B\x44\x24\x18\x8B\x00\xC1\xF8\x00\x33\xF8\x90\x83\x44\x24\x18\x04\x90\x66\xFF\x4C\x24\x14\x0F\xA4\xCF\x00\x75\xE1" > >>> > > >>> > return CODE > >>> > > >>> > def is_branch(instr): > >>> > for group in branch_groups: > >>> > if group in instr.groups: > >>> > return True > >>> > return False > >>> > #return False > >>> > > >>> > # Disassemble until we hit basic block end. > >>> > def disasm(code): > >>> > disasm = Cs(CS_ARCH_X86, CS_MODE_32) > >>> > disasm.detail = True > >>> > address = 0x401000 > >>> > for instr in disasm.disasm(code, address): > >>> > print "0x%x:\t%s\t%s" % (instr.address, > instr.mnemonic, > >>> > instr.op_str) > >>> > if is_branch(instr): > >>> > break > >>> > > >>> > branch_groups = [CS_GRP_JUMP, CS_GRP_CALL, CS_GRP_RET] > >>> > code = get_code() > >>> > while True: > >>> > disasm(code) > >>> > > >>> > > >>> > That code goes out of memory after a few seconds. The super > >>> weird thing > >>> > is, that if I change the implementation of > "is_branch(instr)" > >>> to simply > >>> > return False all the time, then the program does not go out > >>> of memory! > >>> > Does anyone have an idea what's going on? > >>> > > >>> > Best > >>> > Jan > >>> > > >>> > > >>> > ------------------------------------------------------------------------------ > >>> > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT > Server > >>> > from Actuate! Instantly Supercharge Your Business Reports > and > >>> Dashboards > >>> > with Interactivity, Sharing, Native Excel Exports, App > >>> Integration & > >>> > more > >>> > Get technology previously reserved for billion-dollar > >>> corporations, FREE > >>> > > >>> > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > >>> > _______________________________________________ > >>> > Capstone-users mailing list > >>> > Cap...@li... > >>> <mailto:Cap...@li...> > >>> > <mailto:Cap...@li... > >>> <mailto:Cap...@li...>> > >>> > https://lists.sourceforge.net/lists/listinfo/capstone-users > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > ------------------------------------------------------------------------------ > >>> > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT > Server > >>> > from Actuate! Instantly Supercharge Your Business Reports and > >>> Dashboards > >>> > with Interactivity, Sharing, Native Excel Exports, App > >>> Integration & more > >>> > Get technology previously reserved for billion-dollar > >>> corporations, FREE > >>> > > >>> > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > >>> > > >>> > > >>> > > >>> > _______________________________________________ > >>> > Capstone-users mailing list > >>> > Cap...@li... > >>> <mailto:Cap...@li...> > >>> > https://lists.sourceforge.net/lists/listinfo/capstone-users > >>> > > >>> > >>> > >>> > ------------------------------------------------------------------------------ > >>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > >>> from Actuate! Instantly Supercharge Your Business Reports and > Dashboards > >>> with Interactivity, Sharing, Native Excel Exports, App Integration > & > >>> more > >>> Get technology previously reserved for billion-dollar > corporations, FREE > >>> > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > >>> _______________________________________________ > >>> Capstone-users mailing list > >>> Cap...@li... > >>> <mailto:Cap...@li...> > >>> https://lists.sourceforge.net/lists/listinfo/capstone-users > >>> > >>> > >>> > >>> > >>> > ------------------------------------------------------------------------------ > >>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > >>> from Actuate! Instantly Supercharge Your Business Reports and > Dashboards > >>> with Interactivity, Sharing, Native Excel Exports, App Integration & > more > >>> Get technology previously reserved for billion-dollar corporations, > FREE > >>> > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > >>> > >>> > >>> > >>> _______________________________________________ > >>> Capstone-users mailing list > >>> Cap...@li... > >>> https://lists.sourceforge.net/lists/listinfo/capstone-users > >>> > >> > >> > >> > ------------------------------------------------------------------------------ > >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > >> from Actuate! Instantly Supercharge Your Business Reports and Dashboards > >> with Interactivity, Sharing, Native Excel Exports, App Integration & > more > >> Get technology previously reserved for billion-dollar corporations, FREE > >> > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > >> _______________________________________________ > >> Capstone-users mailing list > >> Cap...@li... > >> https://lists.sourceforge.net/lists/listinfo/capstone-users > >> > > > > > > > ------------------------------------------------------------------------------ > > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > > with Interactivity, Sharing, Native Excel Exports, App Integration & more > > Get technology previously reserved for billion-dollar corporations, FREE > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > _______________________________________________ > > Capstone-users mailing list > > Cap...@li... > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & more > Get technology previously reserved for billion-dollar corporations, FREE > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > _______________________________________________ > Capstone-users mailing list > Cap...@li... > https://lists.sourceforge.net/lists/listinfo/capstone-users > |
|
From: Jan N. <jan...@ne...> - 2014-12-03 21:43:11
|
This is the python implementation of the disasm function (starting at
line 791):
def disasm(self, code, offset, count=0):
all_insn = ctypes.POINTER(_cs_insn)()
'''if not _python2:
print(code)
code = code.encode()
print(code)'''
res = _cs.cs_disasm(self.csh, code, len(code), offset, count,
ctypes.byref(all_insn))
if res > 0:
for i in range(res):
yield CsInsn(self, all_insn[i])
_cs.cs_free(all_insn, res)
else:
status = _cs.cs_errno(self.csh)
if status != CS_ERR_OK:
raise CsError(status)
return
yield
I'm really no python expert, but from what I see you apparently need to
free the instruction instances manually. However, if client code stops
enumeration over the instructions prematurely, then _cs.cs_free() is
never invoked, and thus memory is leaked, right?
On 03.12.2014 22:25, Jan Newger wrote:
> It seems the equivalent C implementation is not affected by the mem
> leak, which is to be expected, since the memory is explicitly freed
> anyways, and the group checking boils down to comparing an integer value.
>
> If I had to guess, I'd suspect that in the python case the group
> checking code introduces a spurious reference to the instruction
> instance(?) which cannot be claimed by the GC.
>
> On 03.12.2014 16:57, Jan Newger wrote:
>> No I haven't tried to reproduce the mem leak with C.
>> It already took me a considerable amount of time to come up with this
>> minimal example.
>>
>> On 12/03/2014 04:53 PM, Capstone Engine wrote:
>>>
>>>
>>> On Wed, Dec 3, 2014 at 10:32 PM, Jan Newger <jan...@ne...
>>> <mailto:jan...@ne...>> wrote:
>>>
>>>
>>> Yes, it's using the latest version.
>>> OS was win7 x64 running python 2.7 with 32bit libraries.
>>>
>>>
>>> this is interesting. have you tried to code the same program in C to see
>>> if the mem leak issue still happens?
>>>
>>>
>>> thanks.
>>>
>>>
>>>
>>>
>>> On 12/03/2014 04:31 PM, Nguyen Anh Quynh wrote:
>>> >
>>> >
>>> > On Wed, Dec 3, 2014 at 4:57 PM, Jan Newger <jan...@ne... <mailto:jan...@ne...>
>>> > <mailto:jan...@ne... <mailto:jan...@ne...>>> wrote:
>>> >
>>> > Hey,
>>> >
>>> > I was playing around with a few python scripts (using
>>> capstone among
>>> > other things) and always ran out of memory - and I have no
>>> freaking idea
>>> > why.
>>> >
>>> >
>>> > is this with the latest 3.0 version?
>>> >
>>> > thanks,
>>> > Q
>>> >
>>> >
>>> >
>>> >
>>> > The code is really short:
>>> >
>>> >
>>> > from capstone import Cs
>>> > from capstone import CS_ARCH_X86
>>> > from capstone import CS_MODE_32
>>> > from capstone import CS_GRP_JUMP
>>> > from capstone import CS_GRP_CALL
>>> > from capstone import CS_GRP_RET
>>> > from capstone.x86_const import X86_INS_JNE, X86_INS_JMP
>>> >
>>> > """
>>> > 0x401000: push ecx
>>> > 0x401001: pop ecx
>>> > 0x401002: mov eax, dword ptr [esp + 0x18]
>>> > 0x401006: mov eax, dword ptr [eax]
>>> > 0x401008: sar eax, 0
>>> > 0x40100b: xor edi, eax
>>> > 0x40100d: nop
>>> > 0x40100e: add dword ptr [esp + 0x18], 4
>>> > 0x401013: nop
>>> > 0x401014: dec word ptr [esp + 0x14]
>>> > 0x401019: shld edi, ecx, 0
>>> > 0x40101d: jne 0x401000
>>> > """
>>> > def get_code():
>>> > CODE =
>>> >
>>> "\x51\x59\x8B\x44\x24\x18\x8B\x00\xC1\xF8\x00\x33\xF8\x90\x83\x44\x24\x18\x04\x90\x66\xFF\x4C\x24\x14\x0F\xA4\xCF\x00\x75\xE1"
>>> >
>>> > return CODE
>>> >
>>> > def is_branch(instr):
>>> > for group in branch_groups:
>>> > if group in instr.groups:
>>> > return True
>>> > return False
>>> > #return False
>>> >
>>> > # Disassemble until we hit basic block end.
>>> > def disasm(code):
>>> > disasm = Cs(CS_ARCH_X86, CS_MODE_32)
>>> > disasm.detail = True
>>> > address = 0x401000
>>> > for instr in disasm.disasm(code, address):
>>> > print "0x%x:\t%s\t%s" % (instr.address, instr.mnemonic,
>>> > instr.op_str)
>>> > if is_branch(instr):
>>> > break
>>> >
>>> > branch_groups = [CS_GRP_JUMP, CS_GRP_CALL, CS_GRP_RET]
>>> > code = get_code()
>>> > while True:
>>> > disasm(code)
>>> >
>>> >
>>> > That code goes out of memory after a few seconds. The super
>>> weird thing
>>> > is, that if I change the implementation of "is_branch(instr)"
>>> to simply
>>> > return False all the time, then the program does not go out
>>> of memory!
>>> > Does anyone have an idea what's going on?
>>> >
>>> > Best
>>> > Jan
>>> >
>>> >
>>> ------------------------------------------------------------------------------
>>> > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>> > from Actuate! Instantly Supercharge Your Business Reports and
>>> Dashboards
>>> > with Interactivity, Sharing, Native Excel Exports, App
>>> Integration &
>>> > more
>>> > Get technology previously reserved for billion-dollar
>>> corporations, FREE
>>> >
>>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>>> > _______________________________________________
>>> > Capstone-users mailing list
>>> > Cap...@li...
>>> <mailto:Cap...@li...>
>>> > <mailto:Cap...@li...
>>> <mailto:Cap...@li...>>
>>> > https://lists.sourceforge.net/lists/listinfo/capstone-users
>>> >
>>> >
>>> >
>>> >
>>> >
>>> ------------------------------------------------------------------------------
>>> > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>> > from Actuate! Instantly Supercharge Your Business Reports and
>>> Dashboards
>>> > with Interactivity, Sharing, Native Excel Exports, App
>>> Integration & more
>>> > Get technology previously reserved for billion-dollar
>>> corporations, FREE
>>> >
>>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > Capstone-users mailing list
>>> > Cap...@li...
>>> <mailto:Cap...@li...>
>>> > https://lists.sourceforge.net/lists/listinfo/capstone-users
>>> >
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>>> with Interactivity, Sharing, Native Excel Exports, App Integration &
>>> more
>>> Get technology previously reserved for billion-dollar corporations, FREE
>>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>>> _______________________________________________
>>> Capstone-users mailing list
>>> Cap...@li...
>>> <mailto:Cap...@li...>
>>> https://lists.sourceforge.net/lists/listinfo/capstone-users
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>>> Get technology previously reserved for billion-dollar corporations, FREE
>>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>>>
>>>
>>>
>>> _______________________________________________
>>> Capstone-users mailing list
>>> Cap...@li...
>>> https://lists.sourceforge.net/lists/listinfo/capstone-users
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>> Get technology previously reserved for billion-dollar corporations, FREE
>> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Capstone-users mailing list
>> Cap...@li...
>> https://lists.sourceforge.net/lists/listinfo/capstone-users
>>
>
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
> _______________________________________________
> Capstone-users mailing list
> Cap...@li...
> https://lists.sourceforge.net/lists/listinfo/capstone-users
>
|
|
From: Jan N. <jan...@ne...> - 2014-12-03 21:25:33
|
It seems the equivalent C implementation is not affected by the mem leak, which is to be expected, since the memory is explicitly freed anyways, and the group checking boils down to comparing an integer value. If I had to guess, I'd suspect that in the python case the group checking code introduces a spurious reference to the instruction instance(?) which cannot be claimed by the GC. On 03.12.2014 16:57, Jan Newger wrote: > No I haven't tried to reproduce the mem leak with C. > It already took me a considerable amount of time to come up with this > minimal example. > > On 12/03/2014 04:53 PM, Capstone Engine wrote: >> >> >> On Wed, Dec 3, 2014 at 10:32 PM, Jan Newger <jan...@ne... >> <mailto:jan...@ne...>> wrote: >> >> >> Yes, it's using the latest version. >> OS was win7 x64 running python 2.7 with 32bit libraries. >> >> >> this is interesting. have you tried to code the same program in C to see >> if the mem leak issue still happens? >> >> >> thanks. >> >> >> >> >> On 12/03/2014 04:31 PM, Nguyen Anh Quynh wrote: >> > >> > >> > On Wed, Dec 3, 2014 at 4:57 PM, Jan Newger <jan...@ne... <mailto:jan...@ne...> >> > <mailto:jan...@ne... <mailto:jan...@ne...>>> wrote: >> > >> > Hey, >> > >> > I was playing around with a few python scripts (using >> capstone among >> > other things) and always ran out of memory - and I have no >> freaking idea >> > why. >> > >> > >> > is this with the latest 3.0 version? >> > >> > thanks, >> > Q >> > >> > >> > >> > >> > The code is really short: >> > >> > >> > from capstone import Cs >> > from capstone import CS_ARCH_X86 >> > from capstone import CS_MODE_32 >> > from capstone import CS_GRP_JUMP >> > from capstone import CS_GRP_CALL >> > from capstone import CS_GRP_RET >> > from capstone.x86_const import X86_INS_JNE, X86_INS_JMP >> > >> > """ >> > 0x401000: push ecx >> > 0x401001: pop ecx >> > 0x401002: mov eax, dword ptr [esp + 0x18] >> > 0x401006: mov eax, dword ptr [eax] >> > 0x401008: sar eax, 0 >> > 0x40100b: xor edi, eax >> > 0x40100d: nop >> > 0x40100e: add dword ptr [esp + 0x18], 4 >> > 0x401013: nop >> > 0x401014: dec word ptr [esp + 0x14] >> > 0x401019: shld edi, ecx, 0 >> > 0x40101d: jne 0x401000 >> > """ >> > def get_code(): >> > CODE = >> > >> "\x51\x59\x8B\x44\x24\x18\x8B\x00\xC1\xF8\x00\x33\xF8\x90\x83\x44\x24\x18\x04\x90\x66\xFF\x4C\x24\x14\x0F\xA4\xCF\x00\x75\xE1" >> > >> > return CODE >> > >> > def is_branch(instr): >> > for group in branch_groups: >> > if group in instr.groups: >> > return True >> > return False >> > #return False >> > >> > # Disassemble until we hit basic block end. >> > def disasm(code): >> > disasm = Cs(CS_ARCH_X86, CS_MODE_32) >> > disasm.detail = True >> > address = 0x401000 >> > for instr in disasm.disasm(code, address): >> > print "0x%x:\t%s\t%s" % (instr.address, instr.mnemonic, >> > instr.op_str) >> > if is_branch(instr): >> > break >> > >> > branch_groups = [CS_GRP_JUMP, CS_GRP_CALL, CS_GRP_RET] >> > code = get_code() >> > while True: >> > disasm(code) >> > >> > >> > That code goes out of memory after a few seconds. The super >> weird thing >> > is, that if I change the implementation of "is_branch(instr)" >> to simply >> > return False all the time, then the program does not go out >> of memory! >> > Does anyone have an idea what's going on? >> > >> > Best >> > Jan >> > >> > >> ------------------------------------------------------------------------------ >> > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server >> > from Actuate! Instantly Supercharge Your Business Reports and >> Dashboards >> > with Interactivity, Sharing, Native Excel Exports, App >> Integration & >> > more >> > Get technology previously reserved for billion-dollar >> corporations, FREE >> > >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> > _______________________________________________ >> > Capstone-users mailing list >> > Cap...@li... >> <mailto:Cap...@li...> >> > <mailto:Cap...@li... >> <mailto:Cap...@li...>> >> > https://lists.sourceforge.net/lists/listinfo/capstone-users >> > >> > >> > >> > >> > >> ------------------------------------------------------------------------------ >> > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server >> > from Actuate! Instantly Supercharge Your Business Reports and >> Dashboards >> > with Interactivity, Sharing, Native Excel Exports, App >> Integration & more >> > Get technology previously reserved for billion-dollar >> corporations, FREE >> > >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> > >> > >> > >> > _______________________________________________ >> > Capstone-users mailing list >> > Cap...@li... >> <mailto:Cap...@li...> >> > https://lists.sourceforge.net/lists/listinfo/capstone-users >> > >> >> >> ------------------------------------------------------------------------------ >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server >> from Actuate! Instantly Supercharge Your Business Reports and Dashboards >> with Interactivity, Sharing, Native Excel Exports, App Integration & >> more >> Get technology previously reserved for billion-dollar corporations, FREE >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> _______________________________________________ >> Capstone-users mailing list >> Cap...@li... >> <mailto:Cap...@li...> >> https://lists.sourceforge.net/lists/listinfo/capstone-users >> >> >> >> >> ------------------------------------------------------------------------------ >> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server >> from Actuate! Instantly Supercharge Your Business Reports and Dashboards >> with Interactivity, Sharing, Native Excel Exports, App Integration & more >> Get technology previously reserved for billion-dollar corporations, FREE >> http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk >> >> >> >> _______________________________________________ >> Capstone-users mailing list >> Cap...@li... >> https://lists.sourceforge.net/lists/listinfo/capstone-users >> > > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & more > Get technology previously reserved for billion-dollar corporations, FREE > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > _______________________________________________ > Capstone-users mailing list > Cap...@li... > https://lists.sourceforge.net/lists/listinfo/capstone-users > |
|
From: Jan N. <jan...@ne...> - 2014-12-03 15:57:54
|
No I haven't tried to reproduce the mem leak with C. It already took me a considerable amount of time to come up with this minimal example. On 12/03/2014 04:53 PM, Capstone Engine wrote: > > > On Wed, Dec 3, 2014 at 10:32 PM, Jan Newger <jan...@ne... > <mailto:jan...@ne...>> wrote: > > > Yes, it's using the latest version. > OS was win7 x64 running python 2.7 with 32bit libraries. > > > this is interesting. have you tried to code the same program in C to see > if the mem leak issue still happens? > > > thanks. > > > > > On 12/03/2014 04:31 PM, Nguyen Anh Quynh wrote: > > > > > > On Wed, Dec 3, 2014 at 4:57 PM, Jan Newger <jan...@ne... <mailto:jan...@ne...> > > <mailto:jan...@ne... <mailto:jan...@ne...>>> wrote: > > > > Hey, > > > > I was playing around with a few python scripts (using > capstone among > > other things) and always ran out of memory - and I have no > freaking idea > > why. > > > > > > is this with the latest 3.0 version? > > > > thanks, > > Q > > > > > > > > > > The code is really short: > > > > > > from capstone import Cs > > from capstone import CS_ARCH_X86 > > from capstone import CS_MODE_32 > > from capstone import CS_GRP_JUMP > > from capstone import CS_GRP_CALL > > from capstone import CS_GRP_RET > > from capstone.x86_const import X86_INS_JNE, X86_INS_JMP > > > > """ > > 0x401000: push ecx > > 0x401001: pop ecx > > 0x401002: mov eax, dword ptr [esp + 0x18] > > 0x401006: mov eax, dword ptr [eax] > > 0x401008: sar eax, 0 > > 0x40100b: xor edi, eax > > 0x40100d: nop > > 0x40100e: add dword ptr [esp + 0x18], 4 > > 0x401013: nop > > 0x401014: dec word ptr [esp + 0x14] > > 0x401019: shld edi, ecx, 0 > > 0x40101d: jne 0x401000 > > """ > > def get_code(): > > CODE = > > > "\x51\x59\x8B\x44\x24\x18\x8B\x00\xC1\xF8\x00\x33\xF8\x90\x83\x44\x24\x18\x04\x90\x66\xFF\x4C\x24\x14\x0F\xA4\xCF\x00\x75\xE1" > > > > return CODE > > > > def is_branch(instr): > > for group in branch_groups: > > if group in instr.groups: > > return True > > return False > > #return False > > > > # Disassemble until we hit basic block end. > > def disasm(code): > > disasm = Cs(CS_ARCH_X86, CS_MODE_32) > > disasm.detail = True > > address = 0x401000 > > for instr in disasm.disasm(code, address): > > print "0x%x:\t%s\t%s" % (instr.address, instr.mnemonic, > > instr.op_str) > > if is_branch(instr): > > break > > > > branch_groups = [CS_GRP_JUMP, CS_GRP_CALL, CS_GRP_RET] > > code = get_code() > > while True: > > disasm(code) > > > > > > That code goes out of memory after a few seconds. The super > weird thing > > is, that if I change the implementation of "is_branch(instr)" > to simply > > return False all the time, then the program does not go out > of memory! > > Does anyone have an idea what's going on? > > > > Best > > Jan > > > > > ------------------------------------------------------------------------------ > > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > > from Actuate! Instantly Supercharge Your Business Reports and > Dashboards > > with Interactivity, Sharing, Native Excel Exports, App > Integration & > > more > > Get technology previously reserved for billion-dollar > corporations, FREE > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > _______________________________________________ > > Capstone-users mailing list > > Cap...@li... > <mailto:Cap...@li...> > > <mailto:Cap...@li... > <mailto:Cap...@li...>> > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > > > > > > > ------------------------------------------------------------------------------ > > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > > from Actuate! Instantly Supercharge Your Business Reports and > Dashboards > > with Interactivity, Sharing, Native Excel Exports, App > Integration & more > > Get technology previously reserved for billion-dollar > corporations, FREE > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > > > > > > > _______________________________________________ > > Capstone-users mailing list > > Cap...@li... > <mailto:Cap...@li...> > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & > more > Get technology previously reserved for billion-dollar corporations, FREE > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > _______________________________________________ > Capstone-users mailing list > Cap...@li... > <mailto:Cap...@li...> > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & more > Get technology previously reserved for billion-dollar corporations, FREE > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > > > _______________________________________________ > Capstone-users mailing list > Cap...@li... > https://lists.sourceforge.net/lists/listinfo/capstone-users > |
|
From: Capstone E. <cap...@gm...> - 2014-12-03 15:54:03
|
On Wed, Dec 3, 2014 at 10:32 PM, Jan Newger <jan...@ne...> wrote: > > Yes, it's using the latest version. > OS was win7 x64 running python 2.7 with 32bit libraries. > this is interesting. have you tried to code the same program in C to see if the mem leak issue still happens? thanks. > > > On 12/03/2014 04:31 PM, Nguyen Anh Quynh wrote: > > > > > > On Wed, Dec 3, 2014 at 4:57 PM, Jan Newger <jan...@ne... > > <mailto:jan...@ne...>> wrote: > > > > Hey, > > > > I was playing around with a few python scripts (using capstone among > > other things) and always ran out of memory - and I have no freaking > idea > > why. > > > > > > is this with the latest 3.0 version? > > > > thanks, > > Q > > > > > > > > > > The code is really short: > > > > > > from capstone import Cs > > from capstone import CS_ARCH_X86 > > from capstone import CS_MODE_32 > > from capstone import CS_GRP_JUMP > > from capstone import CS_GRP_CALL > > from capstone import CS_GRP_RET > > from capstone.x86_const import X86_INS_JNE, X86_INS_JMP > > > > """ > > 0x401000: push ecx > > 0x401001: pop ecx > > 0x401002: mov eax, dword ptr [esp + 0x18] > > 0x401006: mov eax, dword ptr [eax] > > 0x401008: sar eax, 0 > > 0x40100b: xor edi, eax > > 0x40100d: nop > > 0x40100e: add dword ptr [esp + 0x18], 4 > > 0x401013: nop > > 0x401014: dec word ptr [esp + 0x14] > > 0x401019: shld edi, ecx, 0 > > 0x40101d: jne 0x401000 > > """ > > def get_code(): > > CODE = > > > "\x51\x59\x8B\x44\x24\x18\x8B\x00\xC1\xF8\x00\x33\xF8\x90\x83\x44\x24\x18\x04\x90\x66\xFF\x4C\x24\x14\x0F\xA4\xCF\x00\x75\xE1" > > > > return CODE > > > > def is_branch(instr): > > for group in branch_groups: > > if group in instr.groups: > > return True > > return False > > #return False > > > > # Disassemble until we hit basic block end. > > def disasm(code): > > disasm = Cs(CS_ARCH_X86, CS_MODE_32) > > disasm.detail = True > > address = 0x401000 > > for instr in disasm.disasm(code, address): > > print "0x%x:\t%s\t%s" % (instr.address, instr.mnemonic, > > instr.op_str) > > if is_branch(instr): > > break > > > > branch_groups = [CS_GRP_JUMP, CS_GRP_CALL, CS_GRP_RET] > > code = get_code() > > while True: > > disasm(code) > > > > > > That code goes out of memory after a few seconds. The super weird > thing > > is, that if I change the implementation of "is_branch(instr)" to > simply > > return False all the time, then the program does not go out of > memory! > > Does anyone have an idea what's going on? > > > > Best > > Jan > > > > > ------------------------------------------------------------------------------ > > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > > from Actuate! Instantly Supercharge Your Business Reports and > Dashboards > > with Interactivity, Sharing, Native Excel Exports, App Integration & > > more > > Get technology previously reserved for billion-dollar corporations, > FREE > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > _______________________________________________ > > Capstone-users mailing list > > Cap...@li... > > <mailto:Cap...@li...> > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > > > > > > > ------------------------------------------------------------------------------ > > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > > with Interactivity, Sharing, Native Excel Exports, App Integration & more > > Get technology previously reserved for billion-dollar corporations, FREE > > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > > > > > > > _______________________________________________ > > Capstone-users mailing list > > Cap...@li... > > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & more > Get technology previously reserved for billion-dollar corporations, FREE > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > _______________________________________________ > Capstone-users mailing list > Cap...@li... > https://lists.sourceforge.net/lists/listinfo/capstone-users > |
|
From: Jan N. <jan...@ne...> - 2014-12-03 15:33:17
|
Yes, it's using the latest version. OS was win7 x64 running python 2.7 with 32bit libraries. On 12/03/2014 04:31 PM, Nguyen Anh Quynh wrote: > > > On Wed, Dec 3, 2014 at 4:57 PM, Jan Newger <jan...@ne... > <mailto:jan...@ne...>> wrote: > > Hey, > > I was playing around with a few python scripts (using capstone among > other things) and always ran out of memory - and I have no freaking idea > why. > > > is this with the latest 3.0 version? > > thanks, > Q > > > > > The code is really short: > > > from capstone import Cs > from capstone import CS_ARCH_X86 > from capstone import CS_MODE_32 > from capstone import CS_GRP_JUMP > from capstone import CS_GRP_CALL > from capstone import CS_GRP_RET > from capstone.x86_const import X86_INS_JNE, X86_INS_JMP > > """ > 0x401000: push ecx > 0x401001: pop ecx > 0x401002: mov eax, dword ptr [esp + 0x18] > 0x401006: mov eax, dword ptr [eax] > 0x401008: sar eax, 0 > 0x40100b: xor edi, eax > 0x40100d: nop > 0x40100e: add dword ptr [esp + 0x18], 4 > 0x401013: nop > 0x401014: dec word ptr [esp + 0x14] > 0x401019: shld edi, ecx, 0 > 0x40101d: jne 0x401000 > """ > def get_code(): > CODE = > "\x51\x59\x8B\x44\x24\x18\x8B\x00\xC1\xF8\x00\x33\xF8\x90\x83\x44\x24\x18\x04\x90\x66\xFF\x4C\x24\x14\x0F\xA4\xCF\x00\x75\xE1" > > return CODE > > def is_branch(instr): > for group in branch_groups: > if group in instr.groups: > return True > return False > #return False > > # Disassemble until we hit basic block end. > def disasm(code): > disasm = Cs(CS_ARCH_X86, CS_MODE_32) > disasm.detail = True > address = 0x401000 > for instr in disasm.disasm(code, address): > print "0x%x:\t%s\t%s" % (instr.address, instr.mnemonic, > instr.op_str) > if is_branch(instr): > break > > branch_groups = [CS_GRP_JUMP, CS_GRP_CALL, CS_GRP_RET] > code = get_code() > while True: > disasm(code) > > > That code goes out of memory after a few seconds. The super weird thing > is, that if I change the implementation of "is_branch(instr)" to simply > return False all the time, then the program does not go out of memory! > Does anyone have an idea what's going on? > > Best > Jan > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & > more > Get technology previously reserved for billion-dollar corporations, FREE > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > _______________________________________________ > Capstone-users mailing list > Cap...@li... > <mailto:Cap...@li...> > https://lists.sourceforge.net/lists/listinfo/capstone-users > > > > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & more > Get technology previously reserved for billion-dollar corporations, FREE > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > > > > _______________________________________________ > Capstone-users mailing list > Cap...@li... > https://lists.sourceforge.net/lists/listinfo/capstone-users > |
|
From: Nguyen A. Q. <aq...@gm...> - 2014-12-03 15:31:43
|
On Wed, Dec 3, 2014 at 4:57 PM, Jan Newger <jan...@ne...> wrote: > Hey, > > I was playing around with a few python scripts (using capstone among > other things) and always ran out of memory - and I have no freaking idea > why. > is this with the latest 3.0 version? thanks, Q > The code is really short: > > > from capstone import Cs > from capstone import CS_ARCH_X86 > from capstone import CS_MODE_32 > from capstone import CS_GRP_JUMP > from capstone import CS_GRP_CALL > from capstone import CS_GRP_RET > from capstone.x86_const import X86_INS_JNE, X86_INS_JMP > > """ > 0x401000: push ecx > 0x401001: pop ecx > 0x401002: mov eax, dword ptr [esp + 0x18] > 0x401006: mov eax, dword ptr [eax] > 0x401008: sar eax, 0 > 0x40100b: xor edi, eax > 0x40100d: nop > 0x40100e: add dword ptr [esp + 0x18], 4 > 0x401013: nop > 0x401014: dec word ptr [esp + 0x14] > 0x401019: shld edi, ecx, 0 > 0x40101d: jne 0x401000 > """ > def get_code(): > CODE = > > "\x51\x59\x8B\x44\x24\x18\x8B\x00\xC1\xF8\x00\x33\xF8\x90\x83\x44\x24\x18\x04\x90\x66\xFF\x4C\x24\x14\x0F\xA4\xCF\x00\x75\xE1" > > return CODE > > def is_branch(instr): > for group in branch_groups: > if group in instr.groups: > return True > return False > #return False > > # Disassemble until we hit basic block end. > def disasm(code): > disasm = Cs(CS_ARCH_X86, CS_MODE_32) > disasm.detail = True > address = 0x401000 > for instr in disasm.disasm(code, address): > print "0x%x:\t%s\t%s" % (instr.address, instr.mnemonic, instr.op_str) > if is_branch(instr): > break > > branch_groups = [CS_GRP_JUMP, CS_GRP_CALL, CS_GRP_RET] > code = get_code() > while True: > disasm(code) > > > That code goes out of memory after a few seconds. The super weird thing > is, that if I change the implementation of "is_branch(instr)" to simply > return False all the time, then the program does not go out of memory! > Does anyone have an idea what's going on? > > Best > Jan > > > ------------------------------------------------------------------------------ > Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server > from Actuate! Instantly Supercharge Your Business Reports and Dashboards > with Interactivity, Sharing, Native Excel Exports, App Integration & more > Get technology previously reserved for billion-dollar corporations, FREE > > http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk > _______________________________________________ > Capstone-users mailing list > Cap...@li... > https://lists.sourceforge.net/lists/listinfo/capstone-users > |
|
From: Jan N. <jan...@ne...> - 2014-12-03 09:57:59
|
Hey,
I was playing around with a few python scripts (using capstone among
other things) and always ran out of memory - and I have no freaking idea
why.
The code is really short:
from capstone import Cs
from capstone import CS_ARCH_X86
from capstone import CS_MODE_32
from capstone import CS_GRP_JUMP
from capstone import CS_GRP_CALL
from capstone import CS_GRP_RET
from capstone.x86_const import X86_INS_JNE, X86_INS_JMP
"""
0x401000: push ecx
0x401001: pop ecx
0x401002: mov eax, dword ptr [esp + 0x18]
0x401006: mov eax, dword ptr [eax]
0x401008: sar eax, 0
0x40100b: xor edi, eax
0x40100d: nop
0x40100e: add dword ptr [esp + 0x18], 4
0x401013: nop
0x401014: dec word ptr [esp + 0x14]
0x401019: shld edi, ecx, 0
0x40101d: jne 0x401000
"""
def get_code():
CODE =
"\x51\x59\x8B\x44\x24\x18\x8B\x00\xC1\xF8\x00\x33\xF8\x90\x83\x44\x24\x18\x04\x90\x66\xFF\x4C\x24\x14\x0F\xA4\xCF\x00\x75\xE1"
return CODE
def is_branch(instr):
for group in branch_groups:
if group in instr.groups:
return True
return False
#return False
# Disassemble until we hit basic block end.
def disasm(code):
disasm = Cs(CS_ARCH_X86, CS_MODE_32)
disasm.detail = True
address = 0x401000
for instr in disasm.disasm(code, address):
print "0x%x:\t%s\t%s" % (instr.address, instr.mnemonic, instr.op_str)
if is_branch(instr):
break
branch_groups = [CS_GRP_JUMP, CS_GRP_CALL, CS_GRP_RET]
code = get_code()
while True:
disasm(code)
That code goes out of memory after a few seconds. The super weird thing
is, that if I change the implementation of "is_branch(instr)" to simply
return False all the time, then the program does not go out of memory!
Does anyone have an idea what's going on?
Best
Jan
|
|
From: Nguyen A. Q. <aq...@gm...> - 2014-11-19 13:33:10
|
Greetings,
We are happy & excited to release version 3.0 of Capstone disassembly
framework!
This major version brings three new architectures (Sparc, SystemZ & XCore),
together with a lot of bugfixes and important updates on Arm, Arm64, Mips,
PPC & X86. Find the link to source code, binaries & details on important
changes of this release at the link below:
http://capstone-engine.org/Version-3.0.html
Today is exactly one year since we put out the looking-for-beta-testers
announcement! Since then, Capstone has always been receiving incredible
supports from community. We would like to take this opportunity to thank
everybody who encouraged & got involved to push our little project this
far!
For those still looking for a disassembler your security tools, hopefully
our testimonials from many world-class experts & list of applications
already adopted Capstone can help you to decide:
http://capstone-engine.org/testimonial.html
http://capstone-engine.org/showcase.html
Thanks,
Quynh
|
|
From: Nguyen A. Q. <aq...@gm...> - 2014-11-02 00:30:33
|
Greetings, We are happy to announce the Release Candidate 3 of version 3.0 of Capstone disassembly framework! This would be the last RC before the final release 3.0 (very soon), so please help us to test it. The links to source code are available in http://capstone-engine.org/Version-3.0-RC3.html NOTE: Do use the bindings come with this version, as all the old bindings from previous version 3.0-RC2 or 2.x are incompatible and cannot be run with the *3.0-RC3 core*. For Java/Ocaml/Python bindings, see the respective README files under bindings/ directory in the source on how to do fresh-reinstall. Summary of the important changes since 3.0-RC2 (more detail): - Better support for cross-platform analysis: - Fix an buffer overflow bug in fill_insn() in cs.c. - Fixes & improvements for X86, ARM, Mips & PPC Thanks, Quynh |
|
From: Nguyen A. Q. <aq...@gm...> - 2014-10-16 13:47:47
|
Greetings, We are happy to announce the Release Candidate 2 of version 3.0 for Capstone disassembly framework! Find the source & information on important changes since RC1 release at http://capstone-engine.org/Version-3.0-RC2.html Please test & feedback. NOTE: - Do use the bindings come with this version, as all the old bindings from version 2.x are incompatible and cannot be used with the 3.0 core. - For Java/Ocaml/Python bindings, see respective README files under bindings/ directory in the source on how to do fresh-install. Thanks, Quynh |
|
From: Jason O. <ja...@ko...> - 2014-10-14 17:22:57
|
David, please do a hexdump -C on your binary. You'll find that storage order for your example instruction is little endian. E.g. 0x4F8010A4 is stored as \xA4\x10\x80\x4F
The data must provided to capstone in storage byte order to be treated as little endian.
Unless you can show that storage order is indeed \x4F\x80\x10\xA4 ... In which case you are building and executing big endian code. ;)
> On Oct 14, 2014, at 9:12, David Abdurachmanov <dav...@gm...> wrote:
>
>
>> On Oct 14, 2014, at 1:49 AM, Nguyen Anh Quynh wrote:
>>
>> hi David,
>>
>> (1) it is fine to use CS_MODE_ARM with Arm64, like below:
>>
>> md = Cs(CS_ARCH_ARM64, CS_MODE_ARM)
>>
>> (2) obviously you need Big Endian mode in your case, like:
>>
>> md = Cs(CS_ARCH_ARM64, CS_MODE_ARM + CS_MODE_BIG_ENDIAN)
>>
>> (3) on your machine, compiler produces Big Endian code for AArch64, and that is the reason why your code failed
>
> AFAIK, it's not or should not.
>
> $ rpm --eval='%{_host}'
> aarch64-redhat-linux-gnu
> $ rpm --eval='%{_build}'
> aarch64-redhat-linux-gnu
> $ rpm --eval='%{_target}'
> aarch64-linux
>
> https://gcc.gnu.org/onlinedocs/gcc/AArch64-Options.html
>
> -mbig-endian
> Generate big-endian code. This is the default when GCC is configured for an ‘aarch64_be-*-*’ target.
>
> $ lscpu
> Architecture: aarch64
> Byte Order: Little Endian
>
> I am running this code on little endian AArch64 silicon, the triplet is also little endian machine.
>
> 0x4F8010A4 == 0100'1111 1000'0000 0001'0000 1010'0100 (fmla v4.4s, v5.4s, v0.s[0])
>
> C7.3.108 from DDI0487A_c_armv8_arm.pdf
>
> FMLA by element vector encoding
>
> bits:
> 31 : 0
> 30 : 1 (Q)
> 29-23 : 0011111
> 22 : 0 (sz)
> 21 : 0 (L)
> 20 : 0 (M)
> 19-16 : 0001 (Rm)
>
> $ objdump -d inst.o
> 0000000000000000 <.text>:
> 0: 4f8010a4 fmla v4.4s, v5.4s, v0.s[0]
>
> $ od -t x1 -j 64 -N 4 inst.o
> 0000100 a4 10 80 4f
> 0000104
>
> B2.5.2 Instruction endianness
>
> In ARMv8-A, A64 instructions have a fixed length of 32 bits and are always little-endian.
>
> Let's test it.
>
> $ as -EL -o inst.o inst.s
> $ objdump -d inst.o
>
> inst.o: file format elf64-littleaarch64
>
>
> Disassembly of section .text:
>
> 0000000000000000 <.text>:
> 0: 4f8010a4 fmla v4.4s, v5.4s, v0.s[0]
>
> $ od -t x1 -j 64 -N 4 inst.o
> 0000100 a4 10 80 4f
> 0000104
>
> $ as -EB -o inst.o inst.s
> $ objdump -d inst.o
>
> inst.o: file format elf64-bigaarch64
>
>
> Disassembly of section .text:
>
> 0000000000000000 <.text>:
> 0: 4f8010a4 fmla v4.4s, v5.4s, v0.s[0]
>
> $ od -t x1 -j 64 -N 4 inst.o
> 0000100 a4 10 80 4f
> 0000104
>
> So, they are stored on disk as little endian (always, doesn't matter what is the mode). Objdump displays instruction as big endian. Well, it does make easier manually reading instructions.
>
> Let's look into x86_64.
>
> $ cat test.c
>
> void dummy(void) {
> int a = 0;
> int b = 1;
> int c = 3;
> if (a > b && b < c) {
> int d = a + b + c;
> }
> }
>
> 0000000000000000 <dummy>:
> 0: 55 push %rbp
> 1: 48 89 e5 mov %rsp,%rbp
> 4: c7 45 fc 00 00 00 00 movl $0x0,-0x4(%rbp)
> b: c7 45 f8 01 00 00 00 movl $0x1,-0x8(%rbp)
> 12: c7 45 f4 03 00 00 00 movl $0x3,-0xc(%rbp)
> 19: 8b 45 fc mov -0x4(%rbp),%eax
> 1c: 3b 45 f8 cmp -0x8(%rbp),%eax
> 1f: 7e 18 jle 39 <dummy+0x39>
> 21: 8b 45 f8 mov -0x8(%rbp),%eax
> 24: 3b 45 f4 cmp -0xc(%rbp),%eax
> 27: 7d 10 jge 39 <dummy+0x39>
> 29: 8b 45 f8 mov -0x8(%rbp),%eax
> 2c: 8b 55 fc mov -0x4(%rbp),%edx
> 2f: 01 c2 add %eax,%edx
> 31: 8b 45 f4 mov -0xc(%rbp),%eax
> 34: 01 d0 add %edx,%eax
> 36: 89 45 f0 mov %eax,-0x10(%rbp)
> 39: 5d pop %rbp
> 3a: c3 retq
>
> $ od -t x1 -j 68 -N 7 test.o
> 0000104 c7 45 fc 00 00 00 00
> 0000113
>
> Seems that objdump works differently for x86_64 and aarch64.
>
> david
> ------------------------------------------------------------------------------
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
> http://p.sf.net/sfu/Zoho
> _______________________________________________
> Capstone-users mailing list
> Cap...@li...
> https://lists.sourceforge.net/lists/listinfo/capstone-users
|
|
From: David A. <dav...@gm...> - 2014-10-14 16:13:02
|
On Oct 14, 2014, at 1:49 AM, Nguyen Anh Quynh wrote:
> hi David,
>
> (1) it is fine to use CS_MODE_ARM with Arm64, like below:
>
> md = Cs(CS_ARCH_ARM64, CS_MODE_ARM)
>
> (2) obviously you need Big Endian mode in your case, like:
>
> md = Cs(CS_ARCH_ARM64, CS_MODE_ARM + CS_MODE_BIG_ENDIAN)
>
> (3) on your machine, compiler produces Big Endian code for AArch64, and that is the reason why your code failed
AFAIK, it's not or should not.
$ rpm --eval='%{_host}'
aarch64-redhat-linux-gnu
$ rpm --eval='%{_build}'
aarch64-redhat-linux-gnu
$ rpm --eval='%{_target}'
aarch64-linux
https://gcc.gnu.org/onlinedocs/gcc/AArch64-Options.html
-mbig-endian
Generate big-endian code. This is the default when GCC is configured for an ‘aarch64_be-*-*’ target.
$ lscpu
Architecture: aarch64
Byte Order: Little Endian
I am running this code on little endian AArch64 silicon, the triplet is also little endian machine.
0x4F8010A4 == 0100'1111 1000'0000 0001'0000 1010'0100 (fmla v4.4s, v5.4s, v0.s[0])
C7.3.108 from DDI0487A_c_armv8_arm.pdf
FMLA by element vector encoding
bits:
31 : 0
30 : 1 (Q)
29-23 : 0011111
22 : 0 (sz)
21 : 0 (L)
20 : 0 (M)
19-16 : 0001 (Rm)
$ objdump -d inst.o
0000000000000000 <.text>:
0: 4f8010a4 fmla v4.4s, v5.4s, v0.s[0]
$ od -t x1 -j 64 -N 4 inst.o
0000100 a4 10 80 4f
0000104
B2.5.2 Instruction endianness
In ARMv8-A, A64 instructions have a fixed length of 32 bits and are always little-endian.
Let's test it.
$ as -EL -o inst.o inst.s
$ objdump -d inst.o
inst.o: file format elf64-littleaarch64
Disassembly of section .text:
0000000000000000 <.text>:
0: 4f8010a4 fmla v4.4s, v5.4s, v0.s[0]
$ od -t x1 -j 64 -N 4 inst.o
0000100 a4 10 80 4f
0000104
$ as -EB -o inst.o inst.s
$ objdump -d inst.o
inst.o: file format elf64-bigaarch64
Disassembly of section .text:
0000000000000000 <.text>:
0: 4f8010a4 fmla v4.4s, v5.4s, v0.s[0]
$ od -t x1 -j 64 -N 4 inst.o
0000100 a4 10 80 4f
0000104
So, they are stored on disk as little endian (always, doesn't matter what is the mode). Objdump displays instruction as big endian. Well, it does make easier manually reading instructions.
Let's look into x86_64.
$ cat test.c
void dummy(void) {
int a = 0;
int b = 1;
int c = 3;
if (a > b && b < c) {
int d = a + b + c;
}
}
0000000000000000 <dummy>:
0: 55 push %rbp
1: 48 89 e5 mov %rsp,%rbp
4: c7 45 fc 00 00 00 00 movl $0x0,-0x4(%rbp)
b: c7 45 f8 01 00 00 00 movl $0x1,-0x8(%rbp)
12: c7 45 f4 03 00 00 00 movl $0x3,-0xc(%rbp)
19: 8b 45 fc mov -0x4(%rbp),%eax
1c: 3b 45 f8 cmp -0x8(%rbp),%eax
1f: 7e 18 jle 39 <dummy+0x39>
21: 8b 45 f8 mov -0x8(%rbp),%eax
24: 3b 45 f4 cmp -0xc(%rbp),%eax
27: 7d 10 jge 39 <dummy+0x39>
29: 8b 45 f8 mov -0x8(%rbp),%eax
2c: 8b 55 fc mov -0x4(%rbp),%edx
2f: 01 c2 add %eax,%edx
31: 8b 45 f4 mov -0xc(%rbp),%eax
34: 01 d0 add %edx,%eax
36: 89 45 f0 mov %eax,-0x10(%rbp)
39: 5d pop %rbp
3a: c3 retq
$ od -t x1 -j 68 -N 7 test.o
0000104 c7 45 fc 00 00 00 00
0000113
Seems that objdump works differently for x86_64 and aarch64.
david
|
|
From: Nguyen A. Q. <aq...@gm...> - 2014-10-13 23:49:35
|
hi David,
(1) it is fine to use CS_MODE_ARM with Arm64, like below:
md = Cs(CS_ARCH_ARM64, CS_MODE_ARM)
(2) obviously you need Big Endian mode in your case, like:
md = Cs(CS_ARCH_ARM64, CS_MODE_ARM + CS_MODE_BIG_ENDIAN)
(3) on your machine, compiler produces Big Endian code for AArch64, and
that is the reason why your code failed
thanks,
Q
On Tue, Oct 14, 2014 at 6:41 AM, David Abdurachmanov <
dav...@gm...> wrote:
> Why is it that I can copy & paste objdump -d produced x86_64 machine code
> to
> www.cenigma.org and get the correct result, but I cannot do the same for
> aarch64.
>
> ## x86_64
>
> 45d4e4: 4c 8b ac 24 b0 00 00 mov 0xb0(%rsp),%r13
> 45d4eb: 00
>
>
> 0 4c8bac24b0000000 movq 0xb0(%rsp), %r13
>
> $ lscpu
> Architecture: x86_64
> CPU op-mode(s): 32-bit, 64-bit
> Byte Order: Little Endian
> [snip]
>
> ## aarch64
>
> 1588: bc6478a1 ldr s1, [x5,x4,lsl #2]
> 158c: bc2279a0 str s0, [x13,x2,lsl #2]
> 1590: bc6b79a0 ldr s0, [x13,x11,lsl #2]
>
> bc 64 78 a1
> bc 22 79 a0
> bc 6b 79 a0
>
> Error: Failed to disassemble! Invalid input?
>
> Setting "Big Endian" instead of "Little Endian" provides correct result:
>
> 0 bc6478a1 ldr s1, [x5, x4, lsl #2]
> 4 bc2279a0 str s0, [x13, x2, lsl #2]
> 8 bc6b79a0 ldr s0, [x13, x11, lsl #2]
>
> $ lscpu
> Architecture: aarch64
> Byte Order: Little Endian
> [snip]
>
> Both systems are Linux.
>
> Why is that it works different for little endian machines?
>
> david
>
>
> ------------------------------------------------------------------------------
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
> http://p.sf.net/sfu/Zoho
> _______________________________________________
> Capstone-users mailing list
> Cap...@li...
> https://lists.sourceforge.net/lists/listinfo/capstone-users
>
>
|
|
From: Jay O. <ja...@ko...> - 2014-10-13 23:02:08
|
Hi David, this is definitely an issue with the BIG-ENDIAN MODE flag not
being set correctly. By default, all architectures use little endian until
you tell it otherwise.
>>> from capstone import *
>>> CODE = "\x1e\x2f\x38\xb2"
>>> md = Cs(CS_ARCH_ARM64, CS_MODE_ARM | *CS_MODE_BIG_ENDIAN*)
>>> insn = next(md.disasm(CODE, 0x0))
>>> print("0x%x:\t%s\t%s" %(insn.address, insn.mnemonic, insn.op_str))
0x0: fsub s18, s5, s15
On Mon, Oct 13, 2014 at 12:05 PM, David Abdurachmanov <
dav...@gm...> wrote:
> Hi,
>
> I decided to try Capstone (3.0 RC1) on AArch64 machine code. From GDB on
> AArch64 machine:
>
> 0x7f330f29b8 <distce_+268>: fsub s18, s5, s15
>
> (gdb) x/x 0x7f330f29b8
> 0x7f330f29b8 <distce_+268>: 0x1e2f38b2
>
> >>> from capstone import *
> >>> CODE = "\x1e\x2f\x38\xb2"
> >>> md = Cs(CS_ARCH_ARM64, CS_MODE_ARM)
> >>> insn = next(md.disasm(CODE, 0x0))
> >>> print("0x%x:\t%s\t%s" %(insn.address, insn.mnemonic, insn.op_str))
> 0x0: orr x30, x24, #0xfff00000fff00
>
> Another example:
>
> 0x4F8010A4 (0100'1111 1000'0000 0001'0000 1010'0100)
>
> $ cat inst.s
> .text
> .inst 0x4F8010A4
>
> $ gas -o inst.o inst.s
>
> $ objdump -d inst.o
> [snip]
> 0000000000000000 <.text>:
> 0: 4f8010a4 fmla v4.4s, v5.4s, v0.s[0]
>
> >>> from capstone import *
> >>> CODE = "\x4F\x80\x10\xA4"
> >>> md = Cs(CS_ARCH_ARM64, CS_MODE_ARM)
> >>> insn = next(md.disasm(CODE, 0x0))
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> StopIteration
>
> I am running:
>
> Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53)
> [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on
> darwin
>
> Am I doing something wrong? Maybe CS_MODE_ARM means AArch32 on ARMv8
> silicon and AArch64 is not supported?
>
> Cheers,
> david
>
>
> ------------------------------------------------------------------------------
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
> http://p.sf.net/sfu/Zoho
> _______________________________________________
> Capstone-users mailing list
> Cap...@li...
> https://lists.sourceforge.net/lists/listinfo/capstone-users
>
>
|
|
From: David A. <dav...@gm...> - 2014-10-13 22:42:01
|
Why is it that I can copy & paste objdump -d produced x86_64 machine code to www.cenigma.org and get the correct result, but I cannot do the same for aarch64. ## x86_64 45d4e4: 4c 8b ac 24 b0 00 00 mov 0xb0(%rsp),%r13 45d4eb: 00 0 4c8bac24b0000000 movq 0xb0(%rsp), %r13 $ lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian [snip] ## aarch64 1588: bc6478a1 ldr s1, [x5,x4,lsl #2] 158c: bc2279a0 str s0, [x13,x2,lsl #2] 1590: bc6b79a0 ldr s0, [x13,x11,lsl #2] bc 64 78 a1 bc 22 79 a0 bc 6b 79 a0 Error: Failed to disassemble! Invalid input? Setting "Big Endian" instead of "Little Endian" provides correct result: 0 bc6478a1 ldr s1, [x5, x4, lsl #2] 4 bc2279a0 str s0, [x13, x2, lsl #2] 8 bc6b79a0 ldr s0, [x13, x11, lsl #2] $ lscpu Architecture: aarch64 Byte Order: Little Endian [snip] Both systems are Linux. Why is that it works different for little endian machines? david |
|
From: David A. <dav...@gm...> - 2014-10-13 19:05:29
|
Hi,
I decided to try Capstone (3.0 RC1) on AArch64 machine code. From GDB on AArch64 machine:
0x7f330f29b8 <distce_+268>: fsub s18, s5, s15
(gdb) x/x 0x7f330f29b8
0x7f330f29b8 <distce_+268>: 0x1e2f38b2
>>> from capstone import *
>>> CODE = "\x1e\x2f\x38\xb2"
>>> md = Cs(CS_ARCH_ARM64, CS_MODE_ARM)
>>> insn = next(md.disasm(CODE, 0x0))
>>> print("0x%x:\t%s\t%s" %(insn.address, insn.mnemonic, insn.op_str))
0x0: orr x30, x24, #0xfff00000fff00
Another example:
0x4F8010A4 (0100'1111 1000'0000 0001'0000 1010'0100)
$ cat inst.s
.text
.inst 0x4F8010A4
$ gas -o inst.o inst.s
$ objdump -d inst.o
[snip]
0000000000000000 <.text>:
0: 4f8010a4 fmla v4.4s, v5.4s, v0.s[0]
>>> from capstone import *
>>> CODE = "\x4F\x80\x10\xA4"
>>> md = Cs(CS_ARCH_ARM64, CS_MODE_ARM)
>>> insn = next(md.disasm(CODE, 0x0))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
I am running:
Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
Am I doing something wrong? Maybe CS_MODE_ARM means AArch32 on ARMv8 silicon and AArch64 is not supported?
Cheers,
david
|
|
From: Yegor D. <yeg...@gm...> - 2014-10-07 11:04:31
|
On Mon, Oct 06, 2014 at 09:06:41PM +0800, Nguyen Anh Quynh wrote: > this needs a new field named "subtracted" in cs_arm_op struct. i pushed a > commit to do this: > > https://github.com/aquynh/capstone/commit/8fb2eab4596e0c5920c86929b3a0cd47028db70e Thank you. It works! -- Yegor Derevenets |
|
From: Nguyen A. Q. <aq...@gm...> - 2014-10-06 13:07:10
|
On Mon, Oct 6, 2014 at 12:03 AM, Yegor Derevenets < yeg...@gm...> wrote: > On Sun, Oct 05, 2014 at 09:18:31PM +0800, Nguyen Anh Quynh wrote: > > - can you confirm that you are using the latest code from the "next" > > branch? > Yes. Confirm for fe4822. > > > - what is the input hexcode for these 2 instructions? are they both in > ARM > > mode + little endian? > > $ LANG=C /usr/arm-linux-gnueabi/bin/objdump -d postincrement.o > > postincrement.o: file format elf32-littlearm > > > Disassembly of section .text: > > 00000000 <.text>: > 0: 000080f4 strdeq r8, [r0], -r4 > 4: 008080f4 strdeq r8, [r0], r4 > this needs a new field named "subtracted" in cs_arm_op struct. i pushed a commit to do this: https://github.com/aquynh/capstone/commit/8fb2eab4596e0c5920c86929b3a0cd47028db70e find the test_arm.c in above commit to see how to extract this new field "subtracted". thanks, Q |
