Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

#4 exception.WindowsError: 'exception: access violation

closed-fixed
Thomas Heller
None
8
2008-09-02
2008-08-15
Mark Hirota
No

Getting a Python exception (access violation) after running back-to-back function calls with the exact same parameters. Using pdb, I've captured traces of both the calls for comparison.

Discussion

1 2 > >> (Page 1 of 2)
  • Mark Hirota
    Mark Hirota
    2008-08-15

     
    Attachments
  • Mark Hirota
    Mark Hirota
    2008-08-15

    Logged In: YES
    user_id=1375527
    Originator: YES

    File Added: run2.log

     
  • Mark Hirota
    Mark Hirota
    2008-08-15

     
    Attachments
  • Thomas Heller
    Thomas Heller
    2008-08-15

    Logged In: YES
    user_id=11105
    Originator: NO

    Mark, I'm afraid there is not enough information in the traces you have posted, but maybe I'm only too lazy to stare at them for a long time.
    It looks like that the freeing of some resources (safearrays? or strings contained in safearrays?) is causing the problem - I am not at all sure that the safearray code is bug free.

    Can you in any way make more information available?

     
  • Mark Hirota
    Mark Hirota
    2008-08-15

    Logged In: YES
    user_id=1375527
    Originator: YES

    Thomas, I'd be happy to provide more information -- can you tell me where to begin?

     
  • Mark Hirota
    Mark Hirota
    2008-08-19

    Logged In: YES
    user_id=1375527
    Originator: YES

    I've been going over the pdb trace and it occurred to me that perhaps the call to _free(self) on line 861 in __init__.py (class BSTR.__ctypes_from_outparam_) might be to blame?

    I commented this line out locally on my system and I'm no longer hitting the exception.

    Could this be the bug?

     
  • Mark Hirota
    Mark Hirota
    2008-08-19

    • assigned_to: nobody --> theller
     
  • Mark Hirota
    Mark Hirota
    2008-08-19

    • priority: 5 --> 8
     
  • Thomas Heller
    Thomas Heller
    2008-08-19

    Logged In: YES
    user_id=11105
    Originator: NO

    > Could this be the bug?

    No. Your code will probably work better when you make this patch, but BSTR strings received as [out] parameters will no longer be freed so there will be memory leaks.

    > Thomas, I'd be happy to provide more information -- can you tell me where to begin?

    Best would be if you can give me the code and the COM object that shows the problem so that I can reproduce and debug it on my own machine.

    If this is not possible, you should at least provide a code snippet that shows a little context around your method call, the IDL definition (for the method that you call) or the type library for the COM object. Plus any other information that might be useful.

    Something like that...

     
  • Mark Hirota
    Mark Hirota
    2008-08-19

    Logged In: YES
    user_id=1375527
    Originator: YES

    Here is the comtypes generated Python code for the method I'm calling:

    COMMETHOD([dispid(16), helpstring(u'method ReadState')], HRESULT, 'ReadState',
    ( ['in'], c_ulong, 'c_ulDeviceIndex' ),
    ( ['in'], BSTR, 'c_bstrStateSpec' ),
    ( ['out'], POINTER(BSTR), 'pbstrBitfieldFormat' ),
    ( ['out'], POINTER(VARIANT), 'pvData' )),

    Also, Thomas, could you please explain to me why it is necessary to call _free(self) twice inside of the BSTR class? (see below)

    class BSTR(_SimpleCData):
    "The windows BSTR data type"
    _type_ = "X"
    def __repr__(self):
    return "%s(%r)" % (self.__class__.__name__, self.value)

    def __ctypes_from_outparam__(self, _free=windll.oleaut32.SysFreeString):
    result = self.value
    _free(self) # <--- here
    return result

    def __del__(self, _free=windll.oleaut32.SysFreeString):
    """If we own the memory, call SysFreeString to free it."""
    if not self._b_base_:
    _free(self) # <--- and here

    Thanks,
    --Mark

     
  • Thomas Heller
    Thomas Heller
    2008-08-20

    Logged In: YES
    user_id=11105
    Originator: NO

    The intent of the free calls is this:

    - The _free call in the __del__ method only occurs when you create a standalone BSTR instance,
    _b_base_ is the container object that contains the instance and it is None in this case.

    - The _free call in __ctypes_from_outparam__ occurs when an output parameter value is retrieved
    from a POINTER(BSTR) instance, as in the COM method call you describe.

    Can you instrument the code to check if _free is really called twice on the same BSTR object?
    Maybe by setting a _been_freed instance variable in both places where _free(self) is called.

     
  • Mark Hirota
    Mark Hirota
    2008-08-20

    Logged In: YES
    user_id=1375527
    Originator: YES

    Okay, this may throw you for a loop:
    1. I've made some changes in my client code around the method call -- but not the method call itself -- and now I'm seeing the exception every time (the first time)
    2. After several iterations, I ended up instrumenting the code in this way (why I ended up here should be apparent after reading below):

    class BSTR(_SimpleCData):
    "The windows BSTR data type"
    _type_ = "X"
    def __repr__(self):
    return "%s(%r)" % (self.__class__.__name__, self.value)

    def __ctypes_from_outparam__(self, _free=windll.oleaut32.SysFreeString):
    result = self.value
    print hex(id(self)), "__ctypes_from_outparam__", len(str(self.value))
    _free(self)
    return result

    def __del__(self, _free=windll.oleaut32.SysFreeString):
    """If we own the memory, call SysFreeString to free it."""
    if not self._b_base_:
    print hex(id(self)), "__del__", len(str(self.value))
    _free(self)

    3. The attached traces show that it doesn't seem to matter that _free is called twice per BSTR instance... UNTIL you get a very large BSTR. I was able to control the size of the BSTR that is returned by the method call and narrowed it down to the 16k size threshold

    --> BSTR > 16k causes the exception
    --> BSTR < 16k avoids the exception
    (I wasn't able to hit it exactly at 16384)

    4. Where does this leave us? :D

     
  • Mark Hirota
    Mark Hirota
    2008-08-20

     
    Attachments
  • Mark Hirota
    Mark Hirota
    2008-08-20

    Logged In: YES
    user_id=1375527
    Originator: YES

    File Added: instrumented_fail.log

     
  • Mark Hirota
    Mark Hirota
    2008-08-20

     
    Attachments
  • Mark Hirota
    Mark Hirota
    2008-08-20

    Logged In: YES
    user_id=1375527
    Originator: YES

    File Added: instrumented_pass.log

     
  • Mark Hirota
    Mark Hirota
    2008-08-20

    Logged In: YES
    user_id=1375527
    Originator: YES

    Okay, this has got to be a bug in comtypes -- win32com is able to handle a BSTR > 16k

     
  • Thomas Heller
    Thomas Heller
    2008-08-25

    Logged In: YES
    user_id=11105
    Originator: NO

    I think this has something to do with caching of BSTRs that windows does by default. Maybe strings larger than 16kB are cached in a different way or not at all.

    I admit that there is a problem with comtypes.

    Setting the environment variable 'set OANOCACHE=1' (which will disable BSTR caching) and then running the testsuite also crashes.

    This 'hack' fixes the crashes (but the case needs to be investigated more deeply):

    Index: comtypes/__init__.py

    --- comtypes/__init__.py (revision 393)
    +++ comtypes/__init__.py (working copy)
    @@ -856,18 +856,23 @@
    class BSTR(_SimpleCData):
    "The windows BSTR data type"
    _type_ = "X"
    + _freed = False
    def __repr__(self):
    return "%s(%r)" % (self.__class__.__name__, self.value)

    def __ctypes_from_outparam__(self, _free=windll.oleaut32.SysFreeString):
    result = self.value
    - _free(self)
    + if not self._freed:
    + _free(self)
    + self._freed = True
    return result

    def __del__(self, _free=windll.oleaut32.SysFreeString):
    """If we own the memory, call SysFreeString to free it."""
    if not self._b_base_:
    - _free(self)
    + if not self._freed:
    + _free(self)
    + self._freed = True

    def from_param(cls, value):
    """Convert into a foreign function call parameter."""

     
  • Mark Hirota
    Mark Hirota
    2008-08-25

    Logged In: YES
    user_id=1375527
    Originator: YES

    I applied the change below, but I'm still getting the WindowsError: exception -- just without the Python window crashing.

    File "C:\Python25\lib\site-packages\comtypes\__init__.py", line 864, in __ctyp
    es_from_outparam__
    _free(self)
    WindowsError: exception: access violation reading 0x002AAC48

     
  • Logged In: NO

    Sorry, my mistake -- I didn't apply the patch below properly -- it is working now. I still don't understand why the _free() call must be invoked inside of __ctypes_from_outparam__() instead of just waiting for the __del__() to do the same thing?

    Is there a 0.5.2 release planned?

    Thanks,
    --Mark

     
  • Thomas Heller
    Thomas Heller
    2008-08-27

    Logged In: YES
    user_id=11105
    Originator: NO

    > Is there a 0.5.2 release planned?

    Sure, shouldn't take too long.

    > I still don't understand why the _free() call must be invoked
    > inside of __ctypes_from_outparam__() instead of just waiting for the
    > __del__() to do the same thing?

    Good question. There is probably no reason and this code is an artifact from the past. I have now changed the code (SVN revision 401) so that double-frees are safely avoided. Can you please test and report back?

    Here is the complete changed BSTR class:

    <snip>
    class BSTR(_SimpleCData):
    "The windows BSTR data type"
    _type_ = "X"
    _needsfree = False
    def __repr__(self):
    return "%s(%r)" % (self.__class__.__name__, self.value)

    def __ctypes_from_outparam__(self):
    self._needsfree = True
    return self.value

    def __del__(self, _free=windll.oleaut32.SysFreeString):
    # Free the string if self owns the memory
    # or if instructed by __ctypes_from_outparam__.
    if self._b_base_ is None \ or self._needsfree:
    _free(self)

    def from_param(cls, value):
    """Convert into a foreign function call parameter."""
    if isinstance(value, cls):
    return value
    # Although the builtin SimpleCData.from_param call does the
    # right thing, it doesn't ensure that SysFreeString is called
    # on destruction.
    return cls(value)
    from_param = classmethod(from_param)
    <snip>

     
  • Thomas Heller
    Thomas Heller
    2008-08-27

    • status: open --> open-fixed
     
  • Mark Hirota
    Mark Hirota
    2008-08-28

    Logged In: YES
    user_id=1375527
    Originator: YES

    This BSTR implementation is working as well.

     
  • Thomas Heller
    Thomas Heller
    2008-09-02

    • status: open-fixed --> closed-fixed
     
  • Thomas Heller
    Thomas Heller
    2008-09-02

    Logged In: YES
    user_id=11105
    Originator: NO

    Thanks for the confirmation.

     
1 2 > >> (Page 1 of 2)