pyxser-users

[pyxser-users] Pyxser generates strange 'name' attribute values

[Max S. <ms...@gm...>]

Hi!
Thank you for your work on this module! I'm writing web-application
using Pylons and XSLT as a template system. That's why I really need
for something that can serialize python objects to XML.
Pyxser seems to be good for solving my problem, but when I've tried to
serialize SQLAlchemy-based object I got invalid XML with strange
values of atrributes named "name":

<pyxs:obj xmlns:pyxs="http://projects.coder.cl/pyxser/model/"
version="1.0" type="Company" module="unigineportal.model.company"
objid="id-33378896">
 <pyxs:col type="InstrumentedList" name="users">
   <pyxs:obj type="User" name="users"
module="unigineportal.model.user" objid="id-33410320">
     <pyxs:prop type="str"
name=";^A@Hpìû^A">sha1$4qQ1s$00f0c6fe031b3cd7900e158fb1ef8133dd4f5fdc</pyxs:prop>
     <pyxs:col type="InstrumentedList" name="roles"/>
     <pyxs:prop type="str" name="^Pat²Ðéû^A">User1</pyxs:prop>
     <pyxs:prop type="str" name="^[2=ñp^Kþ^A">UU</pyxs:prop>
     <pyxs:obj module="unigineportal.model.company" type="Company"
name="company" objref="id-33378896"/>
     <pyxs:prop type="long" name="vV^ð0çû^A">1</pyxs:prop>
     <pyxs:prop type="long" name="ÝQ<9f>Ûð^Lþ^A">1</pyxs:prop>
     <pyxs:prop type="str" name="Ê·Ûà@^Kþ^A">user1</pyxs:prop>
     <pyxs:prop type="str"
name="password">sha1$4qQ1s$00f0c6fe031b3cd7900e158fb1ef8133dd4f5fdc</pyxs:prop>
     <pyxs:prop type="str" name="'©Fü^P^Nþ^A">u1...@c1...</pyxs:prop>
     <pyxs:col type="list" name="permissions"/>
   </pyxs:obj>
   <pyxs:obj type="User" name="permissions"
module="unigineportal.model.user" objref="id-33410192"/>
 </pyxs:col>


Sometimes attribute's value is valid(<pyxs:col type="InstrumentedList"
name="roles"/>), sometimes not(<pyxs:prop type="str"
name=";^A@Hpìû^A">. I use 1.4.2r version. Do you have any ideas why it
happens and what to do with?

Re: [pyxser-users] Pyxser generates strange 'name' attribute values

[Daniel M. W. <dm...@co...>]

On Tuesday 04 May 2010,
Max Sinelnikov <ms...@gm...> wrote:

> Hi!

  Hello again! :)

> Thank you for your work on this module! I'm writing web-application
> using Pylons and XSLT as a template system. That's why I really need
> for something that can serialize python objects to XML.

  Thanks again for your feedback.

> Pyxser seems to be good for solving my problem, but when I've tried to
> serialize SQLAlchemy-based object I got invalid XML with strange
> values of atrributes named "name":

  OK, that problem is caused because dictionary keys are build on
top of unicode strings, so, I think that I've fixed it on the trunk
version of pyxser, if you can test it from the trunk revision, it
would be cool, here is the repository address:

https://pyxser.svn.sourceforge.net/svnroot/pyxser/trunk/pyxser

  Just do a checkout with:

  svn co repository_address pyxser

  In your favourite shell...

> 
> <pyxs:obj xmlns:pyxs="http://projects.coder.cl/pyxser/model/"
> version="1.0" type="Company" module="unigineportal.model.company"
> objid="id-33378896">
>  <pyxs:col type="InstrumentedList" name="users">
>    <pyxs:obj type="User" name="users"
> module="unigineportal.model.user" objid="id-33410320">
>      <pyxs:prop type="str"
> name=";^A@Hpìû^A">sha1$4qQ1s$00f0c6fe031b3cd7900e158fb1ef8133dd4f5fdc</py
> xs:prop> <pyxs:col type="InstrumentedList" name="roles"/>
>      <pyxs:prop type="str" name="^Pat²Ðéû^A">User1</pyxs:prop>
>      <pyxs:prop type="str" name="^[2=ñp^Kþ^A">UU</pyxs:prop>
>      <pyxs:obj module="unigineportal.model.company" type="Company"
> name="company" objref="id-33378896"/>
>      <pyxs:prop type="long" name="vV^ð0çû^A">1</pyxs:prop>
>      <pyxs:prop type="long" name="ÝQ<9f>Ûð^Lþ^A">1</pyxs:prop>
>      <pyxs:prop type="str" name="Ê·Ûà@^Kþ^A">user1</pyxs:prop>
>      <pyxs:prop type="str"
> name="password">sha1$4qQ1s$00f0c6fe031b3cd7900e158fb1ef8133dd4f5fdc</pyxs
> :prop> <pyxs:prop type="str" name="'©Fü^P^Nþ^A">u1...@c1...</pyxs:prop>
>  <pyxs:col type="list" name="permissions"/>
>    </pyxs:obj>
>    <pyxs:obj type="User" name="permissions"
> module="unigineportal.model.user" objref="id-33410192"/>
>  </pyxs:col>
> 
> 
> Sometimes attribute's value is valid(<pyxs:col type="InstrumentedList"
> name="roles"/>), sometimes not(<pyxs:prop type="str"
> name=";^A@Hpìû^A">. I use 1.4.2r version. Do you have any ideas why it
> happens and what to do with?

  OK, can you test it?, I've made some tests here, but if bring me your
feedback you will get a release this week with the proper solution ;)

> 
> [SNIP]

Best regards,
-- 
Daniel Molina Wegener <dmw [at] coder [dot] cl>
Software Architect, System Programmer & Web Developer
Phone: +56 (2) 979-0277 | Blog: http://coder.cl/

Re: [pyxser-users] Pyxser generates strange 'name' attribute values

[Max S. <ms...@gm...>]

Hi, Daniel.
I've tested version from svn trunk. It segfaults now when working with
unicode dictionaries keys.

This is a console session to reproduce an error:
>>> import pyxser
>>> import inspect
>>> selector = lambda o: dict(filter(lambda (name,value): not (name.startswith('_') or callable(value)), inspect.getmembers(o)))

# not unicode key and value
>>> class T:
...     def __init__(self, d):
...             self.d = d
...
>>> t = T({'hehe':'lala'})
>>> t.d
{'hehe': 'lala'}
>>> s = pyxser.serialize(obj=t,enc='utf-8',selector=selector)
>>> print s
<?xml version="1.0" encoding="utf-8"?>
<pyxs:obj xmlns:pyxs="http://projects.coder.cl/pyxser/model/"
version="1.0" type="T" module="pylons-admin" objid="id-33308472">
  <pyxs:col type="dict" name="d">
    <pyxs:prop type="str" name="hehe">lala</pyxs:prop>
  </pyxs:col>
</pyxs:obj>

Seems ok. Now with unicode string as a key:
>>> t = T({u'хехе': 'hehe'})   # Russian cyrillic in the dict key
>>> t.d
{u'\xd1\x85\xd0\xb5\xd1\x85\xd0\xb5': 'hehe'}
>>> s = pyxser.serialize(obj=t,enc='utf-8',selector=selector)
Segmentation Fault

Oops. But it works fine with non-unicode based dictionary keys and values:

>>> t = T({'хехе': 'ололо!'}) # Russian cyrillic in the key and value
>>> t.d
{'\xd1\x85\xd0\xb5\xd1\x85\xd0\xb5':
'\xd0\xbe\xd0\xbb\xd0\xbe\xd0\xbb\xd0\xbe!'}
>>> s = pyxser.serialize(obj=t,enc='utf-8',selector=selector)
>>> print s
<?xml version="1.0" encoding="utf-8"?>
<pyxs:obj xmlns:pyxs="http://projects.coder.cl/pyxser/model/"
version="1.0" type="T" module="pylons-admin" objid="id-33308400">
  <pyxs:col type="dict" name="d">
    <pyxs:prop type="str" name="хехе">ололо!</pyxs:prop>
  </pyxs:col>
</pyxs:obj>

It works with unicode-based dictionary values too. The problem is only
with unicode keys.
Best regards, Max Sinelnikov.

Re: [pyxser-users] Pyxser generates strange 'name' attribute values

[Max S. <ms...@gm...>]

May be it will be helpful: I'm running Debian stable (Lenny) 5.0 with
python2.5 and libxml2 2.6.32 and pyxser revision 139.
test-utf8.py passed, but when I'm adding unicode key python segfaults:

--- test-utf8.py	(revision 139)
+++ test-utf8.py	(working copy)
@@ -362,7 +362,7 @@
     another.dyn_prop1 = thisa
     test.dyn_prop1 = [u'holá', u'chaó', another]
     test.dyn_prop2 = (u'hol`', u'sïn', 'trip', other)
-    test.dyn_prop3 = {'sáludó1': u'hólà', 'sáludó2': u'chäó',
'sòludò4': 'goodbye', 'saludo5': thisc}
+    test.dyn_prop3 = {'sáludó1': u'hólà', 'sáludó2': u'chäó',
u'sòludò4': 'goodbye', 'saludo5': thisc}
     test.dyn_prop4 = u'sómé tèxtè ïñ Unicodè'
     test.dyn_prop5 = u'Añother Texé Iñ ÜnìcóDËc'
     test.dyn_prop6 = 1.5


Best regards, Max Sinelnikov.

Re: [pyxser-users] Pyxser generates strange 'name' attribute values

[Daniel M. W. <dm...@co...>]

On Wednesday 05 May 2010,
Max Sinelnikov <ms...@gm...> wrote:

> May be it will be helpful: I'm running Debian stable (Lenny) 5.0 with
> python2.5 and libxml2 2.6.32 and pyxser revision 139.
> test-utf8.py passed, but when I'm adding unicode key python segfaults:

  Hello Max, thanks for your feedback. I've fixed pyxser, so it will be
released this week, for a while you can try revision 140 (r140) which
is in trunk, it is quite stable, yet I need to do more tests, searching
for memory leaks and similar tasks, but almost is done.

> 
> --- test-utf8.py	(revision 139)
> +++ test-utf8.py	(working copy)
> @@ -362,7 +362,7 @@
>      another.dyn_prop1 = thisa
>      test.dyn_prop1 = [u'holá', u'chaó', another]
>      test.dyn_prop2 = (u'hol`', u'sïn', 'trip', other)
> -    test.dyn_prop3 = {'sáludó1': u'hólà', 'sáludó2': u'chäó',
> 'sòludò4': 'goodbye', 'saludo5': thisc}
> +    test.dyn_prop3 = {'sáludó1': u'hólà', 'sáludó2': u'chäó',
> u'sòludò4': 'goodbye', 'saludo5': thisc}
>      test.dyn_prop4 = u'sómé tèxtè ïñ Unicodè'
>      test.dyn_prop5 = u'Añother Texé Iñ ÜnìcóDËc'
>      test.dyn_prop6 = 1.5
> 
> 
> Best regards, Max Sinelnikov.
> 

Best regards,
-- 
Daniel Molina Wegener <dmw [at] coder [dot] cl>
Software Architect, System Programmer & Web Developer
Phone: +56 (2) 979-0277 | Blog: http://coder.cl/

Re: [pyxser-users] Pyxser generates strange 'name' attribute values

[Daniel M. W. <dm...@co...>]

On Friday 07 May 2010,
Max Sinelnikov <ms...@gm...> wrote:

> Oh, sorry, I clicked 'Reply' instead of 'Reply to all' and forgot to
> attache files.

  NP Max, I think that the problem is solved, please try the r145 on
trunk. Please notify me if that works, I've made some small test here,
but I need to be sure that it works with your application :)

  Thanks in advance...

> Original message:
> 
> 
> Hi, Daniel.
> 
> >  Thanks for your feedback. The code is fixed, pyxser now supports
> > 
> > unicode keys on dictionaries. I've refactored the code, so pyxser
> > now runs faster. I want to release pyxser-1.4.4r tomorrow, so if can
> > have your feedback about how is working the trunk branch of it, it
> > would be very helpful, since I don't have access to the modules
> > that you are using...
> 
> I'm happy with revision 143. All my objects successfully serialized(I
> don't need for deserializing now, so I haven't test this functionality
> yet, sorry-))
> But there is the same problem when I'm serializing one of my objects
> with depth=12 and more. I attached two files to this mail:
> depth-11.xml is a valid xml document I got with depth=11, and
> depth-12.xml is the same serialized object with depth=12. Problem is
> with Resource object objid="id33427536"(depth-12.xml, Line 58).
> 
> A few words about objects I have. I'm using sqlalchemy on top of
> mysql, there are several tables with relations. SQLAlchemy creates
> objects related to tables and, for exapmle, I have object named
> "company" with attribute "roles" which looks like list. Every item of
> company.roles is an company_role instance and so on. It's possible to
> 
> do something like this(as example):
> >>> r = Resource.get_by_id(1)
> >>> company.roles[1].permissions[0].pack.resources[0] is r
> 
> True
> 
> I'm trying to reproduce described issue with simple schema, but with
> no success. I thoungth that there may be a problem with nested
> dictionries, but things like t.a =
> {'hehe':{'ololo':{'haha':{u'хехе':{u'наан':{u'чочо':{u'хрхрх':11}}}}}}}
>  works fine. Will try again during this holidays.
> 
> Best regards, Max Sinelnikov.

Best regards,
-- 
Daniel Molina Wegener <dmw [at] coder [dot] cl>
Software Architect, System Programmer & Web Developer
Phone: +56 (2) 979-0277 | Blog: http://coder.cl/

Re: [pyxser-users] Pyxser generates strange 'name' attribute values

[Max S. <ms...@gm...>]

Hi, Daniel!
Thank you for your work! I've tested revision 145, it works fine.

But there is a really simple issue. When I'm trying to serialize any
sqlalchemy-based object, pyxser segaults.  So I looked at the sources
and added a simple check for a NULL pointer:


--- src/pyxser_collections.c	(revision 145)
+++ src/pyxser_collections.c	(working copy)
@@ -195,7 +195,9 @@
         args->rootNode = &csn;
         args->currentNode = &csn;
         newSerNode = pyxser_SerializeXml(args);
-        PYXSER_FREE_OBJECT(unic);
+        if (unic != NULL) {
+            PYXSER_FREE_OBJECT(unic);
+        }
         args->o = oold;
         args->item = oold;
         args->currentNode = currentNodeOld;

I don't know the C language enought to fix it in the right way, may be
there is a memory leak now.

All other tests I made generates valid xml. Now we have cool tool to
serialize python objects to xml-) Thank you!

Best regards, Max Sinelnikov.



On Mon, May 10, 2010 at 9:23 PM, Daniel Molina Wegener <dm...@co...> wrote:
>  NP Max, I think that the problem is solved, please try the r145 on
> trunk. Please notify me if that works, I've made some small test here,
> but I need to be sure that it works with your application :)
>
>  Thanks in advance...
>
>
> Best regards,
> --
> Daniel Molina Wegener <dmw [at] coder [dot] cl>
> Software Architect, System Programmer & Web Developer
> Phone: +56 (2) 979-0277 | Blog: http://coder.cl/
>

Re: [pyxser-users] Pyxser generates strange 'name' attribute values

[Daniel M. W. <dm...@co...>]

On Tuesday 04 May 2010,
Max Sinelnikov <ms...@gm...> wrote:

> Hi!

  Hello Max,

> Thank you for your work on this module! I'm writing web-application
> using Pylons and XSLT as a template system. That's why I really need
> for something that can serialize python objects to XML.

  OK, no problem with that.

> Pyxser seems to be good for solving my problem, but when I've tried to
> serialize SQLAlchemy-based object I got invalid XML with strange
> values of atrributes named "name":

  The 'name' attribute stands for the name of the object attributes. Seems 
that your application have some 'unicode' names in your attribute names, so 
pyxser is using them as normal python strings, and not unicode names, let me
check if those names are unicode for the next release.

  For example if your application have an object name obj with an attribute
names "hòlá", pyxser will not serialise the name properly since it handles
attribute names as normal strings, that's because you have all that garbage.

  This week I will release a new version of pyxser (1.4.4r), so you will get
this week a new revision of it ;)

> 
> <pyxs:obj xmlns:pyxs="http://projects.coder.cl/pyxser/model/"
> version="1.0" type="Company" module="unigineportal.model.company"
> objid="id-33378896">
> Â <pyxs:col type="InstrumentedList" name="users">
> Â  Â <pyxs:obj type="User" name="users"
> module="unigineportal.model.user" objid="id-33410320">
> Â  Â  Â <pyxs:prop type="str"
> name=";^A@Hpìû^A">sha1$4qQ1s$00f0c6fe031b3cd7900e158fb1ef8133dd4f5fdc</
> pyxs:prop> Â  Â  Â <pyxs:col type="InstrumentedList" name="roles"/>
>      <pyxs:prop type="str" name="^Pat²Ðéû^A">User1</pyxs:prop>
>      <pyxs:prop type="str" name="^[2=ñp^Kþ^A">UU</pyxs:prop>
> Â  Â  Â <pyxs:obj module="unigineportal.model.company" type="Company"
> name="company" objref="id-33378896"/>
>      <pyxs:prop type="long" name="vV^ð0çû^A">1</pyxs:prop>
>      <pyxs:prop type="long" name="ÝQ<9f>Ûð^Lþ^A">1</pyxs:prop>
>      <pyxs:prop type="str" name="Ê·Ûà @^Kþ^A">user1</pyxs:prop>
> Â  Â  Â <pyxs:prop type="str"
> name="password">sha1$4qQ1s$00f0c6fe031b3cd7900e158fb1ef8133dd4f5fdc</pyxs
> :prop> Â  Â  Â <pyxs:prop type="str"
>  name="'©Fü^P^Nþ^A">u1...@c1...</pyxs:prop>      <pyxs:col type="list"
>  name="permissions"/>
> Â  Â </pyxs:obj>
> Â  Â <pyxs:obj type="User" name="permissions"
> module="unigineportal.model.user" objref="id-33410192"/>
> Â </pyxs:col>
> 
> 
> Sometimes attribute's value is valid(<pyxs:col type="InstrumentedList"
> name="roles"/>), sometimes not(<pyxs:prop type="str"
> name=";^A@Hpìû^A">. I use 1.4.2r version. Do you have any ideas why it
> happens and what to do with?

  I see... let me try some examples like that here, and fix it. You will
get a new release this week ;)

  Thank you very much for your feedback...

> 
> [SNIP]
> 

Best regards,
-- 
Daniel Molina Wegener <dmw [at] coder [dot] cl>
Software Architect, System Programmer & Web Developer
Phone: +56 (2) 979-0277 | Blog: http://coder.cl/