Sorry for this late reply, but I was very busy these latest days!
>>Maybe should we go a little further and build semantic tokens as
>>property lists=3F The token API would be simple and efficient
>>(basically wrappers to `plist-get' and `plist-put'), and very flexible
>>(no more depends on positional values, easy to add new properties).
>>Certain property names would be "standardized" like :name, :type,
>>:doc, :parts, :bounds, etc.. Also it would be possible to
>>"propertize" property symbols with useful flags like a 'mandatory'
>>flag, maybe a 'check' function, etc..
> That's an interesting idea. It would eliminate extra specifiers, and
> the need to know what details were specifiers, and which weren't.
> I still think it's useful to keep the name as the first element of
> the token, though. This way you can pass a stream into
> `try-completion' and it will work.
> I'm also attached to having the token type symbol second, though I
> don't have any good reasons for it.
There is no problem to do that ;-)
>>Here is a first basic idea, looking a bit like the widget API:
> We could also make all the tokens with `defstruct' (in CL) or with
> EIEIO. Converting to EIEIO would make token->text and other token
> querying functions into methods instead of our overload functions.
> (or we could do both.)
> The nice thing about plists is the simplicity of the code (as
> above). The nice thing with structs or classes is that they show a
> distinct structure, and individual languages can place named fields
> in them they consider popular. In particular, there is a syntax to
> declare the structure that can be read by other programmers.
> I'd be a little worried about speed for any of these solutions,
> however, particularly with name regex matching (the most common
> thing to do it seems) as the name will be in an unknown location.
> It might be worth using an API (as above) and hacking in a few
> back-ends to see what works well, in particular, just enough to do
> name or name regex matching so we can know for sure.
> If you have an API you think is good, I can create such a back end
> for EIEIO to compare with. If the difference is very small, I'd
> prefer declarative structure, otherwise speed is important.
Using CL `defstruct' or EIEIO `defclass' looks like a good idea too.
As you said that will probably increase the flexibility and robustness
of the implementation. But I suspect it will slow things down a lot
too :-( CL and EIEIO are sophisticated frameworks, their
implementations are not trivial and probably imply a relatively heavy
runtime overhead. On the contrary the property list API certainly
offers less possibilities, but is implemented in C, and, IMO, should
be fast enough to be a good compromise between flexibility and
performance. Nevertheless, I agree with you that some performance
tests are needed.
I worked a little more on my implementation of the token API based on
property lists, and updated some core functions based on it (see
[ Notice that I kept the token name and type (I prefer to use the term
token class) respectively as the first and second elements of the
token structure ;-) ]
IMO, it is worth trying a first approach based on plists. If it slows
things down too much, CL and EIEIO approaches will probably fail too!
The main task to do now is to migrate a representative part of the
Semantic code to use the new API! I did a first impact analysis, and
from what I saw, large parts of the code will probably need to be
If you think it is worth continuing, I am willing to work more on
that, more or less quickly depending on my spare time ;-)
Thanks for all!