[ZooLib-dev] ZTuple changes

[Andrew G. <ag...@em...>]

I'm going to be checking in changes to ZTuple. This message explains 
the motivation for the changes, and a little of how to take advantage 
of them. In many situations no application changes will be necessary. 
If you use AddXXX, or GetXXX with an index, or if you have serialized 
tuples then you'll need to read this in more detail.

I put together ZTuple when I was living on Downey Street in San 
Francisco, so that must have been 1996. That version had a rather neat 
API that overloaded operator[] to return a value reference, which had 
assignment and conversion operators for each of the supported types. To 
exactly specify the type to be stored or retrieved one simply used 
appropriate casts. Very neat, but actually quite unpleasant to use 
because we'd end up using casts all the time just be sure of what was 
happening:
   theTuple["fred"] = 1;
is a bit ambiguous to the reader -- which of the standard int types 
will the compiler map an unadorned '1' to? To be sure we'd have to 
write:
   theTuple["fred"] = int32(1);
Ugh!

Porting ZooLib to BeOS a couple of years later inspired the revamping 
of ZTuple to use an API similar to BeOS's BMessage, which is the API we 
have now. It's less sophisticated, but much more predictable in day to 
day use. My original idea was that ZTuple/ZMessage would have a 
swappable underlying representation, and that on BeOS we'd be able to 
use BMessage as the actual storage medium (on the basis that the OS 
vendor will have optimized the performance more than I'm capable of). 
So having an exactly equivalent API was useful.

BeOS did not have a system-wide representation of lists of things, at 
least as far as BMessage and its supported types were concerned. So to 
allow the storage of lists of values under a single property name their 
API has an additional index parameter on getter methods. The normal 
setter method was AddXXX, which would add the passed in value to the 
list stored under a particular name, establishing the name if it wasn't 
already there. ReplaceXXX and RemoveXXX filled out the list-like API.

However, it's still an API that's not as rigorous as I'd like. There's 
no difference between a property with a single value and a property 
with a list of values containing a single item, they're represented 
identically. In some situations that's actually a nice thing. But 
there's no simple way to establish a property with an empty list of 
values, so code tends to treat the absence of a property as equivalent 
to an empty list, which may not always be what's wanted. For example if 
you want to communicate that something was looked for but nothing was 
found, vs not having looked for the thing at all.

In addition, calling AddXXX(theName, someXXX) on a tuple which already 
contains theName results in someXXX getting appended to the values 
already there. How can you be sure that what you're putting in place is 
what you'll later retrieve? Well, on BeOS, you have to mess about with 
code that invokes FindXXX, then calls ReplaceXXX or AddXXX 
appropriately. To ameliorate this particular weakness I enhanced the 
API with SetXXX methods that unconditionally overwrite whatever may 
already be stored under a name with a single value -- SetXXX guarantees 
that after it's called there will be a property with that single value. 
And that's the API I've tried to use unless I'm really expecting to be 
building a list. I've recently gone back through ZooLib and ensured 
that I'm not doing an AddXXX when I really mean SetXXX.

Although ZTuple can store and work with lists of values, *we* can't 
treat lists of values generically as we do with the other types ZTuple 
supports. So this:
   theTuple.SetValue("theName", otherTuple.GetValue("otherName"));
doesn't transfer every value stored by otherTuple under "otherName" to 
theTuple under "theName" -- in fact it copies only the first value to 
theTuple. The reason for this is that it's ZTuple that's doing the 
vector stuff rather than ZTupleValue, and ZTupleValue is what's 
returned by ZTuple::GetValue.

The original (circa 1996) ZTupleValue implementation *was* able to 
store various primitive types, strings, tuples and vectors of 
ZTupleValues. So treating lists polymorphically with other types was 
easy, and ZTuple didn't get involved.

With the impending profusion of ZTuple-based code it seemed time to 
clear up all these things that have been bugging me. So I've revamped 
ZTupleValue and ZTuple, maintaining the parts of the Get/Set API that's 
proven effective, having ZTupleValue do the work of storing vectors of 
other ZTupleValues, and losing the AddXXX API in favor of a more or 
less equivalent suite of AppendXXX methods.

AppendXXX will establish a name if it's not already present. If it is 
present and the stored value is already a vector then the passed in 
value is simply appended to it. If it's not a vector, or doesn't exist 
at all, then we establish a value which is a vector with a single 
entry, the value that was passed in. So where you might have had a loop:
   for (int x = 0; x < 10; ++x)
     theTuple.AddInt32("aName", x);

you can do this:
   for (int x = 0; x < 10; ++x)
     theTuple.AppendInt32("aName", x);

and get exactly the same effect, unless the property "aName" already 
had a non-vector value in it. In actual use this doesn't happen. Or at 
least it won't if the informal protocol by which tuples are build up 
and consumed is sufficiently well defined. It really should be the case 
that a particular property is expected to have multiple values, or its 
expected to have exactly one.

For pre-existing code that wants to be able to deal with vectors of 
values and single values identically, perhaps because the tuples are 
coming from permanent storage and contain vectors of values that just 
happened to have only a single value, we have a handful of GetXXXAt 
methods. We might need to expand them out to include all data types, 
but for now I've put in place only the ones I've seen used. The 
companion method to GetXXXAt is CountValuesAt:
   for (int x = 0; x < theTuple.CountValuesAt("aName"); ++x)
     {
     int32 anInt32 = theTuple.GetInt32At("aName", x);
     ...
     }

[Of course it's better to get the count once at the beginning, and to 
convert the name into a property number, but this is just for 
expositional purposes.]

The 'more modern' equivalent, where we're being more rigorous in 
constructing tuples and extracting data from them, we'd do something 
like this:
   theTuple.SetEmptyVector("aName");
   for (int x = 0; x < 10; ++x)
     theTuple.GetVectorMutable("aName").push_back(x);

or
   vector<ZTupleValue>& theVector = theTuple.SetEmptyVector("aName");
   for (int x = 0; x < 10; ++x)
     theVector.push_back(x);

or indeed any STL container-stuffing algorithm could be used to 
populate theVector.

To extract data:
   const vector<ZTupleValue>& theVector = theTuple.GetVector("aName");
   for (vector<ZTupleValue>::const_iterator i = theVector.begin(); i != 
theVector.end(); ++i)
     {
     int32 theInt = (*i).GetInt32();
     ...
     }

NOTE: GetVector and GetVectorMutable return const and non-const 
*references*, so you must not make mutating changes to the enclosing 
tuple that could invalidate the references.

To support reading of tuples and tuplevalues that are already in 
backing store ZTuple and ZTupleValue have FromStreamOld methods that 
will do the right thing with old tuples -- singleton values come in as 
singleton values, multiple values come in as vectors.

I've extended the ZType enum. The new values are eZType_Vector (of 
course), eZType_ID and eZType_Type. eZType_ID holds a 64 bit value, and 
can be distinguished from a regular 64 bit integer (necessary for 
tuplestores). eZType_Type is a confusing name, but is used to indicate 
that the content of a ZTupleValue is a ZType enum. With this we're able 
to describe the structure of a tuple using a tuple.

In summary, we've still got the same functionality as before, albeit 
with name changes to make sure existing code fails to compile and can 
be updated to use the new API if appropriate. We're also able to apply 
STL algorithms to tuple contents in a way that wasn't possible before. 
The specification of a tuple now more closely matches associative 
arrays in other languages, in particular Cocoa's dictionaries.

A+
-- 
Andrew Green                           mailto:ag...@em...
Electric Magic Co.           Vox/Fax: +1 (408) 907 2101