Re: [Parser-devel] [token.c] A new functionality

[Jinesh K J <jin...@gm...>]

On 10/6/06, Noam Postavsky <npo...@uw...> wrote:
> OK, I think you mean should NOT be counted, that's what the test cases
> seem to indicate.
Yup! What I meant was 'NOT BE COUNTED'. Now our cutcomment() is
working perfectly.

> I've changed the function to pass the test cases, I think this is what
> you meant, if not, more thorough explanation is needed.
I think we have got it cleared. Good work!

So, one more functional change(I very well hope so) is all that we
need to merge these changes to the main tree. A behavioural change
with gettoken is needed. I don't really know how hard it would be for
you to make this change. It is as follows:

======begin========

char *gettoken(char *str, char **kv, char **end);

str - input string
*end - the next character after our extracted token
*kv - the value part of our token(if its a key/value pair) - malloc'ed

Returns ->  the key/value part  - malloc'ed

======end==========

Explanation:

Our str can contain tokens of two types:

  -  A normal token which we can call a 'value'
  -  A key/value pair which has a 'key' part and a 'value' part are
are joined by an '='.

Till now we've been worrying only about just a few words in a line.
The words could also be grouped by using quotes. Those rules are to
remain the same for our keys and values as well. The only change to be
made to gettoken() is to make it recognise a key/value pair and return
them.

For example, below are the examples for - str, function return value,
and *kv respectively. The function of *end remain the same.

=1=
[Hello World]    [Hello]   NULL

Since Hello and World does not form a key/value pair, *kv should be NULL.

=2=
[Hello=World]    [Hello]    [World]

This is a very simple and clear example of a key value pair.

=3=
[Hello  =   World]    [Hello]   [World]

There can be spaces between a key, value and its '='

=4=
["Hello"  =   "World"]    [Hello]   [World]

Keys and values can also be in tokens

=5=
["Hello=World"   =   "Parser=Library"]    [Hello=World]    [Parser=Library]

Rules for quotes remain the same - they are opaque objects

=6=
[Hello  ===   World]    [Hello]   [World]
[Hello  = = =  World]    [Hello]   [World]

More than one '=' can be joined together

=7=
[Hello  =   World   =   Parser   =    Library]    [Hello]   [World
Parser Library]

Cascading of '=' can cause the values to be joined. Can you think of a
better behaviour for this case? Should we have it this way:

[Hello  =   World   =   Parser   =    Library]    [Hello World Parser]
   [Library]

=8=
[Hello  =   ]    [Hello]   []



I think by now our requirements are almost clear and complete. What is
your opinion? Any suggestions?


Jinesh.