PySAP and SAP R/3 with Unicode

  • Hi,
    after implementing a Python program that invokes a bunch of functions in SAP R/3 I was very happy with the module.
    Only one minor issue occured when using nested structures (SAP type 'u' in definition of the structure fields).

    But than the system was upgraded to use Unicode.
    Now structure fields are reported with double length (that is for CHAR fields). However the content is still single byte. So while a structure field is reported to be length 16 (that is 8 double byte characters), but content is only 8 chars (single byte). However the content of the subsequent fields is moved into the first one, as the alignment does not match.
    I tried several approaches, but was not able to convince SAP to report the single-byte character length without changing pySAP. If doing so (extra parameter in RFC_GET_STRUCTURE_DEFINITION_P (UCLEN) set to 0x01 the length is reported in single-byte chars, but the program fails with memory access errors.....


    Thanks in advance (additional details can be provided, if needed/helpful).

    • Hi,
      just found that in the CVS there are some changes that seam to relate to my issue mentioned above.
      Actually in the CVS version the UCLEN parameter is used. However the field length reported come back as Unicode version, although they should not....
      Can anyone verify if the retrieval of the structure definition works correctly for them with a Unicode R/3 system?
      SAP function called: RS_VARIANT_CONTENTS_RFC
      - Report: Any report you want
      - Variant: Any existing variant for the report
      - Move_Or_Write: 'W'
      In the resulting field list there is a structure RSPARAM which has only 'C' fields. However for me the field 'SELNAME' is reported to be C(16), while in a non-Unicode environment (UCLEN != 0x00) it should be C(8).

      Many thanks for any feedback

    • Hi again,
      was able to make it working. Basically there were two things that needed adjustment.
      a) memmove was used incorrectly in every single place. The parameter sequence is different to memcpy. SRC and DEST need to be switched.
      b) BCD handling required re-activation of the RFC-Lib BCD handling and change back to eval(vs) instead of float(vs), if no FixedPoint is used (at least for me...)

      If needed I can also create a diff to the CVS version. Plese post a request in this thread..


    • Hi,

      you managed to get Python3_refactoring branch to work? I started this branch a while ago to acomodate for changes in upcoming Python 3 (bytes/string types) but then decided to first clean code a bit and add support for newer version of ctypes. This means breaking Python 2.2 compatibility as newer versions of ctypes (> 0.6.x) do not support 2.2 anymore. One of those changes was switching libc memcpy for ctypes memmove. I probably just replaced _lib.memcpy with _c.memmove forgetting about different parameter sequence.
      I made some further changes to the module but I haven't committed them to the CVS until I fix the code and make the module work again. Unfortunately I'm rather busy lately (and in bed with nasty cold for last few days) but I hope I will be able to work on the module again in November.
      I checked the memmoves in the current version and they are still broken. A diff of your fixes against the CVS version would be most wellcome.


    • Hi Klavdij (and all others),
      just saw the response now and can create the div only on Monday, but will post it than.
      Just for your information I'll add the versions used cause I can't remember them right now.
      Is there be anything in addition you would like to know exepct for:
      - SAP version
      - Python version
      - ctypes version
      ??? Let me know and I'll include it.

      Hope your cold gets better quickly !

    • Hi Klavdij,
      found a way to do it now :-)
      Well partly at least. Can't give you the exact SAP release as it was upgraded during my recent holiday, but it's actually a SAP SolutionManager with Unicode active.
      I'm using Python 2.4.3 with ctypes 1.0.2. The diff is at the bottom of this post. As mentioned it's basically 3 things: Re-Activate old BCD handling, fix memmove's and switch back to old handling for BCD (eval() instead of new float() ).
      Probably these changes can be done in a better way (more pythonic), but this verison is working for me. Actually I would like to be able to exchange unicode data as well, but the quick hack to reference the librfc32u.lib did and to adjust e.g. the UCLEN parameter value did not work. Maybe usage of unicode could be detected by pysap (e.g. by asking SAP for some system information, there is a standard function mdolue for that). Also being able to overwrite the usage of unicode e.g. as parameter to the RfcConnection object would be great. Anyhow, right now it's working (for me).

      < ##_has_rfc_bcd=1
      < ##try:
      < ##    _char_to_bcd=librfc.RfcConvertCharToBcd
      < ##    _bcd_to_char=librfc.RfcConvertBcdToChar
      < ##except AttributeError:
      < ##    _has_rfc_bcd=0
      > _has_rfc_bcd=1
      > try:
      >     _char_to_bcd=librfc.RfcConvertCharToBcd
      >     _bcd_to_char=librfc.RfcConvertBcdToChar
      > except AttributeError:
      >     _has_rfc_bcd=0
      < _has_rfc_bcd=0   
      > #_has_rfc_bcd=0   
      <             else: return float(vs)
      >             #else: return float(vs)
      >             else: return eval(vs)
      <             if rez: _c.memmove(rez,self,self._length_)
      >             if rez: _c.memmove(self,rez,self._length_)
      <                     _c.memmove(v,self,lng)
      >                     _c.memmove(self,v,lng)
      <             _c.memmove(self,_c.byref(buff),self._length_)
      >             _c.memmove(_c.byref(buff),self,self._length_)
      <             _c.memmove(zeroes,self,length)
      >             _c.memmove(self,zeroes,length)
      <             _c.memmove(_c.byref(other),_c.byref(self),min(_c.sizeof(self),_c.sizeof(other)))
      >             _c.memmove(_c.byref(self),_c.byref(other),min(_c.sizeof(self),_c.sizeof(other)))
      <             _c.memmove(_c.byref(self),_c.byref(buff),lng)
      >             _c.memmove(_c.byref(buff),_c.byref(self),lng)
      <             _c.memmove(buff,_c.byref(self),min(_c.sizeof(self),len(str_val)))
      >             _c.memmove(_c.byref(self),buff,min(_c.sizeof(self),len(str_val)))

    • Hi,

      I've (finally) applied your patch to the current CVS version (Python3 branch).
      I'm must admit I never looked at Unicode data exchange as there wasn't a need for it (for me).  Some of our partners are using Unicode systems and here in Slovenia we use either ISO Latin-2 or Microsoft 1250 code page (they are similar but not the same as Microsoft decided to map couple of characters used in Slavic languages differently) but I usually managed to get correct encoding by sending CODEPAGE parameter with As far as I understand Unicode system should adapt when communicating with non-Unicode client (non-Unicode version of librfc) by encoding data send as specified by fore-mentioned CODEPAGE parameter.
      Using Unicode version of librfc would need adjustments to core pysap module (using correct ctypes type - either c_wchar or c_int for SAP CHAR).

      Regards and thank you,

    • Hi Klavdij,
      agree regarding the SAP Unicode with Non Unicode Python. SAP takes care of the conversion, if possible.
      I however do understand that SAP with Unicode would send UTF-16 (to be verified) which is why without the UCLEN parameter the CHAR,DATE,TIME,NUMC fields have duplicate length in the structure definition. But I have no clue about the ctypes bindings and requirements.
      In General I suspect that not always all characters can be converted and as such there might be issues.
      Unfortunatly I have not the time and python/ctypes knowledge to build Unicode support myself. However I would happily assist in testing, if that helps.