YAML Ain't Markup Language

Hi Chris,

On Tue, May 02, 2006 at 04:52:28PM -0400, Chris S wrote:
> I have a few questions about how PySyck handles objects/instances. I
> subclassed Dumper and overwrote represent_dict() to ignore keys
> prefixed with '_v_', the aim being to temporarily store unserializable
> things like locks, threads, sockets, etc in large nested structures
> that I want to serialize.

The easiest way to make PySyck ignore some attributes is to define
the methods __getstate__ and __setstate__ in your classes. These
methods are parts of the pickle protocol. __getstate__ should return a
dictionary with keys and values to be serialized. In your case, it will
look like this (note: untested code):

    def __getstate__(self):
        state = {}
        for key in self.__dict__:
            if not key.startswith('_v_'):
                state[key] = self.__dict__[key]
        return state

> Btw, why are there two definitions of represent_object()? The later
> definition has a somewhat disquieting comment. Is it deprecated in
> favor of represent_instance?

represent_instance is for instances of old-style classes
(type(obj).__name__ == 'instance')

represent_object is for instances of new-style classes.

The code is complicated because it's supposed to fully support the pickle
protocol and produce more-or-less human readable representation.

-- 
xi

On 5/3/06, Kirill Simonov <xi@...> wrote:
> Hi Chris,
>
> On Tue, May 02, 2006 at 04:52:28PM -0400, Chris S wrote:
> > I have a few questions about how PySyck handles objects/instances. I
> > subclassed Dumper and overwrote represent_dict() to ignore keys
> > prefixed with '_v_', the aim being to temporarily store unserializable
> > things like locks, threads, sockets, etc in large nested structures
> > that I want to serialize.
>
> The easiest way to make PySyck ignore some attributes is to define
> the methods __getstate__ and __setstate__ in your classes. These
> methods are parts of the pickle protocol. __getstate__ should return a
> dictionary with keys and values to be serialized. In your case, it will
> look like this (note: untested code):
>
>     def __getstate__(self):
>         state =3D {}
>         for key in self.__dict__:
>             if not key.startswith('_v_'):
>                 state[key] =3D self.__dict__[key]
>         return state

Yes, I do do this in most cases. However, like I mentioned, I'm
serializing large nested structures (dicts containing lists containing
instances containing dicts, etc), so it's impractical to define a
custom __getstate__ for every class I might use. Since they would all
use the same code anyways, it's far easier to implement this directly
in the Dumper. And so far it works pretty reliably, except with a
classes __dict__, which isn't wrapped by represent_dict for some
reason. I could submit a small patch for this if you'd like.

Regards,
Chris Spencer

Hi Chris,

On Wed, May 03, 2006 at 02:43:41PM -0400, Chris S wrote:
> On 5/3/06, Kirill Simonov <xi@...> wrote:
> >    def __getstate__(self):
> >        state = {}
> >        for key in self.__dict__:
> >            if not key.startswith('_v_'):
> >                state[key] = self.__dict__[key]
> >        return state
> 
> Yes, I do do this in most cases. However, like I mentioned, I'm
> serializing large nested structures (dicts containing lists containing
> instances containing dicts, etc), so it's impractical to define a
> custom __getstate__ for every class I might use. Since they would all
> use the same code anyways, it's far easier to implement this directly
> in the Dumper. And so far it works pretty reliably, except with a
> classes __dict__, which isn't wrapped by represent_dict for some
> reason. I could submit a small patch for this if you'd like.

represent_dict converts dictionary objects to mapping nodes.

represent_instance and represent_object convert class instances to mapping
nodes. So represent_dict will never be called for class instances.

To do what you want, you may override the method 'represent' of Dumper.

class MyDumper(syck.Dumper):    # untested code
    def represent(self, object):
        node = super(MyDumper, self).represent(object)
        if isinstance(node.value, dict):
            for key in node.value.keys():
                if isinstance(key, str) and key.startswith('_v_'):
                    del node.value[key]
        return node

-- 
xi

On 5/3/06, Kirill Simonov <xi@...> wrote:
> Hi Chris,
>
> On Wed, May 03, 2006 at 02:43:41PM -0400, Chris S wrote:
> > On 5/3/06, Kirill Simonov <xi@...> wrote:
> > >    def __getstate__(self):
> > >        state =3D {}
> > >        for key in self.__dict__:
> > >            if not key.startswith('_v_'):
> > >                state[key] =3D self.__dict__[key]
> > >        return state
> >
> > Yes, I do do this in most cases. However, like I mentioned, I'm
> > serializing large nested structures (dicts containing lists containing
> > instances containing dicts, etc), so it's impractical to define a
> > custom __getstate__ for every class I might use. Since they would all
> > use the same code anyways, it's far easier to implement this directly
> > in the Dumper. And so far it works pretty reliably, except with a
> > classes __dict__, which isn't wrapped by represent_dict for some
> > reason. I could submit a small patch for this if you'd like.
>
> represent_dict converts dictionary objects to mapping nodes.
>
> represent_instance and represent_object convert class instances to mappin=
g
> nodes. So represent_dict will never be called for class instances.

I don't quite understand why this is. Wouldn't it be more intuitive
for an object's __dict__ to be handled by represent_dict? The calls to
_syck.Map are essentially the same in all three methods, so couldn't
we simplify things by replacing _syck.Map in represent_instance and
represent_object with a call to represent_dict?

Chris

Hi Chris,

> > represent_dict converts dictionary objects to mapping nodes.
> >
> > represent_instance and represent_object convert class instances to
> > mapping nodes. So represent_dict will never be called for class
> > instances.
>
> I don't quite understand why this is. Wouldn't it be more intuitive
> for an object's __dict__ to be handled by represent_dict? The calls to
> _syck.Map are essentially the same in all three methods, so couldn't
> we simplify things by replacing _syck.Map in represent_instance and
> represent_object with a call to represent_dict?

Well, not really.

represent_dict is called to convert a dict object to a mapping node.

represent_instance is called to convert a class instance to a mapping (or 
whatever node).

represent_dict creates a mapping node with key/value pairs taken from the 
dictionary tagged as 'tag:yaml.org,2002:dict'. That's what the call to 
_syck.Map does.

represent_instance creates a mapping node with key/value pairs taken from the 
instance state tagged as 'tag:python.yaml.org,2002:object:module.class'.

If you really want to override the creation of a mapping node, you need to set 
_syck.Map to your own implementation.

You may check PyYAML, it seems to work just like you want. Representer in 
PyYAML has three methods: represent_scalar, represent_sequence, and 
represent_mapping. They create a scalar, sequence, and mapping node 
correspondingly and both represent_dict and represent_instance call 
represent_mapping, which you may override.

-- 
Kirill