scidb-development — Development discussion

[Scidb-development] First Draft of eCQL - Player Search

[Gregor C. <re...@gm...>]

This is a continuation of the first draft of eCQL. In this part all the
functions/sub-functions for player search will be defined.

As usual the best view is with a monospaced font, and I've attached a text file
of this draft.

===================================================================

Player Search
~~~~~~~~~~~~~

The search for players will be performed with sub-functions of function 
?player.
Most of the function will be based on C/CIF standard
(http://ccif.sourceforge.net/cif-game-info-head-player.html).

eCQL                       CQL-S             CQL 5.1  Application    C/CIF
--------------------------------------------------------------------------
%player                    :whiteplayer      player   CB Scidb Scid    yes
&player.birth              :whitebirthyear            Scidb            yes
&player.death              :whitedeathyear                             yes
#player.age                                                            no
^player.clock                                                          yes
~player.federation         :whitecountry              CB Scidb Scid    yes
%player.forename                                      CB               yes
#player.frequency                                     Scidb            no
%player.gender             :whitegender               Scidb            yes
%player.identifier                                    Scidb            yes
%player.name               :whiteplayer      player   CB Scidb Scid    yes
~player.nation                                                         yes
%player.networkaddress                                                 yes
#player.ranking                                       Scidb            yes
#player.rating             :whiterating      elo      CB Scidb Scid    yes
#player.regioncode                                    Scidb            yes
^player.runningclock                                                   yes
%player.species            :whiteiscomputer           Scidb            yes
%player.surname                                       CB               yes
%player.team                                          CB               yes
%player.title              :whitetitle                CB Scidb Scid    yes
--------------------------------------------------------------------------
Per default the first level function %player is an abbreviation for the second
level function %player.name.

Note that a player query needs the color of the player as an argument, 
example:

   %player.name white

Some details:

#player.frequency returns the number of games with this player in searched
databases.

?player.name provides these third level functions:

   %player.name               := raw name (used in database)
   %player.name.ascii         := ASCII'fied name
	                              ("Réti" -> "Reti")
   %player.name.normalized    := normalized name
	                              ("Zunker,Reinhard,Dr." -> "Zunker, Reinhard")
   %player.name.standardized  := standardized name
	                              ("ZUNKER, REINHARD" -> "Zunker, Reinhard")
   %player.name.base          := base name from dictionary
	                              ("Aarland, St1" -> "Aarland, Stein")

Some sub-functions of ?player.name can be combined, example:

   %player.name.normalized.ascii

The specific federation will be tested with third level functions, some
examples:

   :player.federation.DE
   :player.federation.IT
   :player.federation.germany
   :player.federation.italy
   :player.nation.unknown

Nowadays the species of the player is an important information, but only Scidb
allows to define this attribute:

   :player.species.computer
   :player.species.human
   :player.species.unknown

The gender also contains the species 'computer' for convenience:

   :player.gender.computer
   :player.gender.man
   :player.gender.woman
   :player.gender.unknown

eCQL knows more rating types than only ELO as CQL 5.1:

   #player.rating.none
   #player.rating.acf
   #player.rating.blitz
   #player.rating.dwz
   #player.rating.ecf
   #player.rating.edo
   #player.rating.elo
   #player.rating.iccf
   #player.rating.rapid
   #player.rating.uscf

Note that the numeric functions for player attributes will provide the
additional fourth level functions "min", "max", "mid", "avg", and "abs".
This type of fourth level functions do not expect the color of the player.
Example:

   #player.rating.elo white   := ELO rating of white player
   #player.rating.elo black   := ELO rating of black player
   #player.rating.elo.min     := minimum of white and black ELO
   #player.rating.elo.max     := maximum of white and black ELO
   #player.rating.elo.mid     := 0 if no player has an ELO,
                                 average if both players have an ELO,
                                 otherwise value of single player with ELO
   #player.rating.elo.avg     := 0 if at least one of the players has no ELO,
                                 otherwise average of white and black ELO
   #player.rating.elo.abs     := absolute difference between white and black 
ELO

Moreover ?player.rating provides some selective functions:

   ?player.rating.best        := best rating of this player
   ?player.rating.actual      := most actual rating of this player

The title of the player is grouped by the organization:

   :player.title.none
   ----------------------
   :player.title.fide.cm
   :player.title.fide.fm
   :player.title.fide.gm
   :player.title.fide.im
   :player.title.fide.wcm
   :player.title.fide.wfm
   :player.title.fide.wgm
   :player.title.fide.wim
   ----------------------
   :player.title.iccf.gm
   :player.title.iccf.im
   :player.title.iccf.lgm
   :player.title.iccf.lim
   :player.title.iccf.sim
   ----------------------
   :player.title.uscf.nm
   :player.title.uscf.sm
   :player.title.uscf.lm
   ----------------------
   :player.title.bdf.njm
   :player.title.bdf.nm
   :player.title.bdf.nmb
   :player.title.bdf.nmc
   :player.title.bdf.nmg
   :player.title.bdf.nms
   :player.title.bdf.sm
   :player.title.bdf.wm
   :player.title.bdf.wsm
   ----------------------
   :player.title.cfc.nm
   :player.title.cfc.nwm
   ----------------------
   :player.title.ecf.nm
   ----------------------
   :player.title.nzcf.nm

The fourth level function "year" provides the year of the achieved title,
example:

   #player.title.fide.gm.year white := the year where white player has achieved
	                                    the GM title, is zero if unknown

[Scidb-development] First draft of eCQL - Event search

[Gregor C. <re...@gm...>]

This is a continuation of the first draft of eCQL. In this part sub-functions 
will be introduced, and all the functions/sub-functions for event search will 
be defined.

As usual the best view is with a monospaced font, and I've attached a text file 
of this draft.

===================================================================

Event Search
~~~~~~~~~~~~

For event search we use function ?event. Note that "?" is not denoting a type, 
we use this sign for the denotation of "any" type, this means that the type of 
?event is unspecified in this context, the real type will be specified with the 
use of this function. In this case the type depends on the sub-function. So we 
introduce sub-functions in eCQL. A sub-function may follow the function (or it 
may follow a sub-function), and will be delimited with a point. The 
introduction of sub-functions allows to group the functions into classes. The 
following specification of all sub-functions for ?event is demonstrating the 
practicability of grouping.

eCQL                       CQL-S             CQL 5.1  Application
-------------------------------------------------------------------
#event.boardpoints                                    CB
#event.category            :eventcategory             CB
:event.complete                                       CB
&event.date                :eventdate                 CB Scid Scidb
%event.mode                :eventmode                 CB Scidb
#event.rounds                                         CB
%event.site                :site             site     CB Scid/ Scidb
#event.site.country        :eventcountry              CB Scidb
%event.site.latitude                                  CB
%event.site.location       :site             site     CB Scid/ Scidb
%event.site.longitude                                 CB
%event.sponsor                                        CB
:event.threepointsforwin                              CB
%event.tiebreak                                       CB
%event.title               :event            event    CB Scid/ Scidb
%event.type                :eventtype                 CB Scidb

For a description of the sub-functions see <a 
href="http://ccif.sourceforge.net/cif-game-info-head-event.html">C/CIF</a> 
documentation. Note that all meta-data search will be based on C/CIF 
specification (C/CIF = Compressed/ Chess Interchange Format). Only some 
attributes (sub-functions) need more details:

Per default %event.site will return the location of the event site, this is 
%event.site.location.

The event mode (%event.mode) will return a string specifying the event mode 
("otb", "corr", etc.). But it's possible to query the specific event mode of a 
game. Also these types are based on <a href="http://ccif.sourceforge.net/cif-namebase-event.html">C/CIF</a> specification:

   :event.mode.otb            Over the board
   :event.mode.corr           Correspondence
   :event.mode.email          E-Mail
   :event.mode.internet       Internet Chess Server
   :event.mode.telecomm       Telecommunication (in general)
   :event.mode.analyis        Game Analysis
   :event.mode.composition    Composition

The event country #event.site.country returns the ISO 3166-1 country number. 
It is possible to query the specific event country, either the <a 
href="http://en.wikipedia.org/wiki/ISO_3166-1_alpha-2">ISO 3166-2</a> code, or 
the FIDE code (used in PGN standard). Two examples with ISO 3166-2:

   :event.country.DE          Germany
   :event.country.US          United States Of America

And one example with FIDE code:

   :event.country.ENG         England

But also the internationalized name of the country can be used for the query:

   :event.country.italy

eCQL also supports the query of the specific tie-break mode, see <a 
href="http://ccif.sourceforge.net/cif-namebase-event.html">C/CIF</a> 
documentation. Here are some examples:

   :event.tiebreak.buchholz
   :event.tiebreak.medianbuchholz
   :event.tiebreak.sonnebornberger

The event type can be queried with:

   :event.type.casual                    Casual Game
   :event.type.match                     Match
   :event.type.tournament                Tournament (of any type)
   :event.type.tournament.roundrobin     Round Robin Tournament (all play all)
   :event.type.tournament.swiss          Swiss-System Tournament
   :event.type.tournament.team           Team Tournament
   :event.type.tournament.knockout       Knockout Tournament
   :event.type.tournament.simultan       Simultaneous Tournament
   :event.type.tournament.scheveningen   Scheveningen-System Tournament

It is important to note that a search for ChessBase specific attributes like 
the event sponsor (#event.sponsor) is also useful in Scidb. But in this case 
the corresponding PGN tag will be queried. Normally this tag is available only 
if the original source of this game is ChessBase.

Now a simple example for meta-data search, in this case we are searching for 
games played in very specific events:

:cqlprematch {
   :event.type.match           ; only games played in a match
   :event.site.country.italy   ; this match has been played in Italy
   :or
      (:event.mode.otb)        ; only over-the-board matches
      (:event.mode.unknown)    ; if event mode is unknown we assume otb
   :match >=2000 (#event.date)  ; played in matches from this century
                               ; (remember that # is extracting the year part)
}

===================================================================

End of draft.

Gregor

[Scidb-development] First draft of eCQL

[Gregor C. <re...@gm...>]

This is the first draft of Scidb's (new) Chess Query Language. Because it's 
only a draft, the introduction/documentation is quite pragmatic. But hopefully 
the concept/syntax of this language is understandable. The set of functions 
will be introduced in a later draft.

Because it's a first draft the syntax may change with further development, but 
currently I'm optimistic that this draft is quite near to a final version.

Note: the best readability will be provided with a monospaced font (I've 
attached a text file containing this draft).

=================================================================

Scidb provides his own Chess Query Language, called eCQL, dedicated for the 
use in every chess database application. The letter "e" is for "extended", 
because eCQL is designed for chess database applications, but native CQL does 
not fulfill the requirements of chess database applications in general. We will 
not go the way of CQL 5.1, because:

1. The syntax of CQL 5.1 is a bit unsuccessful, an example like
   http://gadycosteff.com/cql/doc/examples/bishoppawnpinrelation.html
   is a bit hard to read. Moreover the new syntax still implies restrictions.

2. The author of CQL 5.1 has completely ignored the further development
   of prior CQL-S version (based on CQL 3.02), which contains extensions
   especially useful for searches in chess database applications. That's why
   even the newer version 5.1 is still not useful for Scidb.

3. The possibilities in native CQL are restricted to the features of PGN, but
   a chess database application in general offers more features.

eCQL is now loosely based on CQL 5.1 (http://gadycosteff.com/cql/) and CQL-S 
(prior CQL derivative of Scidb based on CQL 3.02), and has a stronger type 
based syntax. The syntax of eCQL provides good readability, is clean, and 
allows very flexible definitions, needed for searches in Scidb.

One simple example in eCQL which cannot be expressed in CQL 5.1 due to 
syntactical/functional limitations [is this statement right, or have I 
overseen a possibility in native CQL?]:

   :match (%player white) (%player black)

This expression is matching whether the name of the white and black players 
are equal.

In eCQL it is important that every expression results in a specific type. And 
each result type has a specific precedence, according to the following table:

   String, Date, Set  3
   Number             2
   Boolean            1

Note: type Set designates a set of squares on a chessboard.

It is also important that every type can be converted to a type with lower 
precedence - except type boolean which already has the lowest precedence.The 
conversions will be done based on the following rules ("iff" means "if and only 
if"):

   Date   -> Number: extract the year part
   String -> Number: number of characters in string
   Set    -> Number: number of squares in set

   Date   -> Boolean: "true" iff year is zero (date not set)
   String -> Boolean: "true" iff string is not empty
   Set    -> Boolean: "true" iff set is not empty
   Number -> Boolean: "true" iff not zero

eCQL provides two block types {...} and (...) with different behavior. Note 
that a block also counts as an expression. A block contains zero or more 
expressions.

The braced block {...} has the following behavior:

1. Any result (of an evaluated expression) inside this block will be
   converted to a boolean. A block is matching a position if all
   results will convert to "true" for this position, this means that
   a position is matching if all expressions are matching (the logical
   AND over all results has to be "true" for a match of this position).

2. The result type of a braced block is the number of positions matched
   by this block.

3. An empty braced block is matching any position.

Example:

   { [Pp] }

Any position with pawns will match this expression, because if a position 
contains pawns, then (and only then) the set will not be empty, and the 
conversion to a boolean results to "true".

A technical note: obviously an expression can abort the search as soon a 
position is matching, because in this example the number of matching positions 
is not used anywhere, so it's only relevant whether any positions in the game 
is matching, or if none of the positions in the game is matching. And the eCQL 
interpreter will do this for performance reasons. This means that the user 
must not think about the performance here.

The parenthesized block (...) has the following behavior:

1. Any result of type string, date, or number will be converted to a boolean
   type.

2. Any result will be converted to the type with the lowest precedence in
   this block.

3. The result type of a parenthesized block is the result type of all
   expressions after the required conversions have been performed.
   The result of the block will be computed by conjugation over all
   results. In case of boolean values this is the logical AND, and
   in case of sets it is nothing else than set intersection.

4. An empty parenthesized block is not allowed.

Example:

   ( [Bb] @check )

The first result is the set of squares with a bishop. The second expression 
delivers the set of all pieces giving check in this position. The intersection 
is the set of all bishops giving check.

Next example:

   ( [Bb] #check )

The expression #check is counting the number of squares in the set containing 
all squares giving check. Due to rule (1) this will be converted to a boolean, 
so the result is whether any piece is giving check. Due to rule(2) the set 
[Bb] will also be converted to a boolean, so the result is whether any bishop 
is contained in current position. This means that this block is matching (has 
result "true") when the current position contains any bishop which gives check 
at the same time. This is only a simple example for the behavior of 
parenthesized blocks, normally such an odd expressions will not be needed. 
Generally a parenthesized block will contain expressions returning the same 
type.

Note that every function has to be prefixed with a sign belonging to a specific 
type, examples:

   :matchstring  evaluates to a boolean
   #gamenumber   evaluates to a number
   @check        evaluates to a set
   %event        evaluates to a string
   &eventdate    evaluates to a date

The prefixes are improving the readability, especially with regard to the 
conversion rules.

It is possible to use a prefix which belongs to a type with a lower precedence, 
in this case an implicit conversion will be performed. For example @check will 
return the set of all squares containing a piece which is giving check. But 
#check will return the number of pieces giving check (the number of the 
squares in set), and :check will deliver whether any piece is giving check 
(which means: whether the side to move is in check). The use of prefix "%" for 
function "check" (this is %check) is not possible, because this function 
cannot return a string. The prefix cannot be omitted.

Also the special prefix "!" can be used. This prefix will convert to the negated 
boolean value. Example: !mate results to true if and only if the side to move 
is not mate.

In general an eCQL script consists of three sections:

:cqlinit {
   ...
}

:cqlprematch {
   ...
}

:cqlmatch {
   ...
}

Note that the order is mandatory. Every section is optional, but at least one 
of the sections :cqlinit, :cqlprematch, or :cqlmatch must be defined.

Only section :cqlmatch is matching positions, section :cqlprematch is matching 
meta-data (header data), and :cqlinit will be used for initialization and 
macro definitions. Only inside :cqlmatch a position match can be defined, and 
only this section is counting the number of matching positions (per game).

Section :cqlprematch is matching meta-data (game header data), and inside this 
section the number of matching games will be counted.

Only in section :cqlinit initializations can be performed. This section is 
only counting whether any initialization has failed - in this case an error 
will be thrown -, or if all initialization have been succeeded. Here the 
result of a block can only be zero (failed) or one (succeeded). In case of 
zero the search will not be performed, because an error will be thrown.

One simple script (compare this with the original example from CQL 5.1: 
http://gadycosteff.com/cql/doc/examples/most-king-moves.html):

-----------------------------------------------------------------------------
; Find games with at least 20 black king moves, sorted by the number
; of such moves

:cqlmatch {
   :sort ( #match >=20 { :movefrom k } )
}
-----------------------------------------------------------------------------

Note that #match is transparent, it returns the result from second argument, 
and this expression returns the number of matches in a game (due to the 
definition of braced blocks). #sort receives the number of matches, because the 
parenthesized block returns the result of #match. But consider that only games 
with >= 20 matching positions will match, because "#match >=20" excludes games 
with less matching positions.

Now an example for meta-data search (originally from 
http://gadycosteff.com/cql/doc/examples/gameinfo.html):

-----------------------------------------------------------------------------
; This eCQL file illustrates searching for certain game-level information.
; It only looks for games that Ivanchuk won against a player rated from
; 2000 to 2800.
; It sorts the result by opponent elo and year.

:cqlprematch {
   :flipcolor
   :player white "Ivanchuk"
   :result 1-0
   :sort ( #match 1900 2016 (#date) )
   :sort ( #match 2000 2800 (#rating ELO black) )
}
-----------------------------------------------------------------------------

eCQL is introducing a macro concept. A macro is a parameterized textual 
replacement.One example:

:cqlinit {
   :def GoodGame 'elo=2350 'dwz=2350 'ecf=220 'iccf=2350 'uscf=2375 {
      ; Only games with a normal time mode.
      :timemode normal corr
      ; Exclude simultaneous games
      !eventtype simul
      ; Exclude games with less than 16 half moves, and in case of drawn games
      ; even exclude games with less than 40 half moves.
      :cond
         (:result 1/2) (:match >=40)
         (:match >=16)
      ; Only games where one player has a major title, or a good rating score.
      :or
         ( :title IM GM WIM WGM CIM CGM )
         ( :match >='elo  (#rating ELO ) )
         ( :match >='dwz  (#rating DWZ ) )
         ( :match >='ecf  (#rating ECF ) )
         ( :match >='iccf (#rating ICCF) )
         ( :match >='uscf (#rating USCF) )
   }
}

BTW: this is a good example for demonstrating the stronger syntax of eCQL, the 
parentheses do not change the meaning when used for the :or expression.

A macro name always starts with an upper case letter. A macro parameter must 
start with a lower case letter, and will be prefixed with an apostrophe. The 
default value (after '=') is optional. For the use of a macro see the 
following examples:

:cqlprematch {
   'GoodGame
}

Here the macro will be applied (prefix ') with the use of the default values 
for the parameters. Note that a textual replacement will be done, therefore 
the macro will be applied without any conversion rule (based on a prefix). The 
example above is equivalent to this script:

:cqlprematch {
   ; Only games with a normal time mode.
   :timemode normal corr
   ; Exclude simultaneous games.
   !eventtype simul
   ; Exclude games with less than 16 half moves, and in case of drawn games
   ; even exclude games with less than 40 half moves.
   :cond
      (:result 1/2) (:match >=40)
      (:match >=16)
   ; Only games where one player has a major title, or a good rating score.
   :or
      ( :title IM GM WIM WGM CIM CGM )
      ( :match >=2350 (#rating ELO ) )
      ( :match >=2350 (#rating DWZ ) )
      ( :match >=220  (#rating ECF ) )
      ( :match >=2350 (#rating ICCF) )
      ( :match >=2375 (#rating USCF) )
}

Another example how to use macro GoodGame:

:cqlprematch {
   'GoodGame 'dwz=2450 'elo=2450 'uscf=2500 'iccf=2450
}

In this example only some of the parameters are bound to other values. It is 
not allowed to omit parameters which are not bound to a default value, with 
other words, unbound parameters are not possible.

:cqlprematch {
   'GoodGame 2350 2350 220 2350 2375
}

In this example the arguments are unnamed. In this case the number of 
arguments must match the number of parameters, and will be bound with given 
order.

But the other way, converting to a type with higher or same precedence, is not 
possible.

=================================================================

End of first draft.

Gregor

[Scidb-development] Test

[Gregor C. <re...@gm...>]

This is a test, for initialization of the new list.