Re: [Roundup-devel] Patch for expressions for keywords searches

[Sascha L. T. <sas...@in...>]

Ralf,

Am 21.10.2010 08:45, schrieb Ralf Schlatterbeck:
> First of all, thanks Sascha and Bernhard for taking the initiative to
> extend the search capabilities of roundup. I really appreciate this, so
> please see the following as constructive criticism...
> 

We do. :-)

> On Wed, Oct 20, 2010 at 12:04:14PM +0200, Sascha L. Teichmann wrote:
>> Hi,
>>
>> Find a patch under keyword-expr-alldbm3.diff [1] attached
>> to [2] which adds the feature of define expressions of keywords
>> in the search dialog.
>>
>> Say, you want express
>>
>>     (keyword1 OR keyword2) AND NOT keyword3
>>
>> you can open a Javascript based expression editor which
>> allows you to enter this. Technically it is implemented by
>> using a reverse polish notation in the comma separated
>> list of arguments of the corresponding multilink field.
>> Operators are '-2' for NOT '-3' for AND and '-4' for OR.
>> If you look at the HTTP GET request generated by the search
>> dialog you will find something like:
>>
>>     keyword=42,64,-4,65,-2,-3
>>
>> 42, 64 and 65 being the ids of the keywords.
>> The numbers of the opcodes where choose because they
>> don't really interfere with the database ids. The
>> RPN was introduced to be compatible with the old
>> URL scheme and not to introduce a parser with
>> brackets and precedences.
> 
> Interesting, but this delegates the whole expression parsing to
> javascript. Why not introduce a (hmm: backward compatible) URL scheme
> that can express such expressions? wouldn't it be better to do the
> expression parsing in python so we can flexibly generate query code for
> the backends -- adding that step would not be much more work and we can
> probably keep the RPN notation as an intermediate format?

The JS solution is made to be useful with out more involving changes.
Sorry for saying this: The search UI needs some overhaul and don't
think should be part of this first step.

>>From the user perspective it's probably hard to require RPN and even
> harder if the operators are negative numbers if no javascript is
> available...

Here should be a better dialog in the HTML-UI.

> Can't we come up with an URL-scheme that is backward compatible and
> still human readable with the necessary expressiveness?
> 
> I'm thinking about a comma (now means 'or') separated list of expressions.
> Currently these are single numbers (or key names! see example below), we
> could extend these to expressions containing and, or, not. We already
> have a slightly more complex url schema for Date properties: These can
> contain "FROM", "TO" etc. instead of ';' as range separator. So this is
> not unprecedented...
> 
> I'm arguing for a single url-schema not as in your proposal where you
> switch parsers if negative numbers below -1 are seen...

Being with you but show me code! I've seen some discussion
about a new URL scheme on the devel-list. Is
there some existing code beyond abstract design discussions?

>> If the list of multilink items does not contain numbers
>> below -1 the original backend filter logic is triggered.
>> For real expressions I extend the different backend to
>> process the queries at the lowest level as possible to
>> perform well. I successfully tested the changes with
>> anydbm, SQLite3 and PostgreSQL. Other backends like
>> MySQL should work, too.
> 
> I've looked a little into the SQL generating code: Since I wrote the
> database part for transitive multilink expressions *without subqueries*
> at the time I wouldn't want to introduce subqueries now and I think we
> already have most of the necessary data structures to compile the
> expressions into SQL without subqueries, the engine now already creates
> the necessary outer joins. So it's just a matter of translating the
> expression into SQL. Have you looked at the data structures used by
> filter?

Yes, I have. There is exiting code in rdbms_common.py
(Database._subselect and ._filter) which use subselects already. Its
being overwritten in the MySQL backend. If this old code is not used
any more then it should be removed. If you can give me some more
concrete hints how to write my code not to use subselects I'm open
to hear it. Have You seen that I've already written a none
subselect fallback? Any kind of constructive refactoring is welcome.
:-)

> 
> We can now already do things like
> 
> db.issue.filter (None, {'msg.recipients.username' : 'admin'})
> 
> i.e. search for all issues that have a message where admin is the
> receiver. The necessary outer joins are already generated. The dot
> notation nicely integrates into the url scheme. So if we allow
> expressions instead of just lists of values we already can operate on
> the joined data.
> 
> Or didn't I understand this correctly that the main reason for the
> subqueries is the computation of the necessary joins?
> 
> Are you aware that currently Link and Multilink queries are not limited
> to numbers, you can use key-values (values of the key property), too,
> e.g.:
> 
> http://bee:8080/ttt/issue?@columns=id,category,title,area,kind,status,composed_of.id,part_of.id&@sort=category&@group=-effective_prio&@filter=status,responsible&@pagesize=50&@startwith=0&status=analyzing,open,feedback,escalated,testing&responsible=schlatterbeck
> 
> so we don't necessarily have numbers here.
> (thats a real example from an extended tracker I'm running for a
> customer)

Interesting. I need to test this if this possible in vanilla roundup.

> Ralf

Sascha