DevelopersIntroduction
A Developer's Quick Introduction to Montezuma
(Based on the Ferret tutorial by Dave Balmain.)
The simplest way to use Montezuma is through the montezuma:index class.
Start by loading the Montezuma system:
(asdf:oos 'asdf:load-op '#:montezuma)
Creating an index
To create an in-memory index is very simple:
(defparameter *index* (make-instance 'montezuma:index))
To create a persistent index;
(defparameter *index* (make-instance 'montezuma:index
:path "/path/to/index"))
Both of these methods create new indexes with the standard-analyzer.
An analyzer is what you use to divide the input data up into tokens
which you can search for later. If you'd like to use a different analyzer
you can specify it here, eg;
(defparameter *index* (make-instance 'montezuma:index
:path "/path/to/index"
:analyzer (make-instance 'montezuma:whitespace-analyzer)))
For more options when creating an index refer to the montezumaindex class.
Adding Documents
To add a document you can simply add a string. This will store the
string in the "" (i.e. empty string) field (unless you specify the
default field when you create the index).
(montezuma:add-document-to-index *index* "This is a new document to be indexed")
But these are pretty simple documents. If this is all you want to
index you could probably just use simple-search. So let's give our
documents some fields:
(montezuma:add-document-to-index *index* '(("title" . "Programming Ruby")
("content" . "blah blah blah")))
(montezuma:add-document-to-index *index* '(("title" . "Programming Lisp")
("content" . "yada yada yada")))
Or if you are indexing data stored in a database, you'll probably want
to store the ID (this example assumes you have some kind of database
object in ROW, with ID, TITLE and DATE accessors):
(montezuma:add-document-to-index *index* `(("id" . ,(id row))
("title" . ,(title row))
("date" . ,(date row))))
The methods above will store all of the input data as well as tokenizing
and indexing it. Sometimes we won't want to tokenize (divide the
string into tokens) the data. For example, we might want to leave the
title as a complete string and only allow searches for that complete
string. Sometimes we won't want to store the data as it's already
stored in the database so it'll be a waste to store it in the
index. Or perhaps we are doing without a database and using Montezuma
to store all of our data, in which case we might not want to index
it. For example, if we are storing images in the index, we won't want
to index them. All of this can be done using Montezuma's document
class. e.g.,
;; Assume ROW is an object with reader methods ID, TITLE, DATA and
;; IMAGE.
(let ((doc (make-instance 'montezuma:document)))
(montezuma:add-field doc (montezuma:make-field
"id"
(id row)
:stored NIL
:index :untokenized))
(montezuma:add-field doc (montezuma:make-field
"title"
(title row)
:stored T
:index :untokenized))
(montezuma:add-field doc (montezuma:make-field
"data"
(data row)
:stored T
:index :tokenized))
(montezuma:add-field doc (montezuma:make-field
"image"
(image row)
:stored T
:index NIL))
(montezuma:add-document-to-index *index* doc))
You can also compress the data that you are storing or store term vectors with the data.
Read more about this in the documentation for Montezuma's field class.
Searching
Now that we have data in our index, how do we actually use this index to search the data?
The index class offers two search methods, search and search-each. The first method
returns a top-docs object. The second we'll show here. Lets say we wanted to find all
documents with the phrase "quick brown fox" in the content field. We'd write
(montezuma:search-each *index* "content:\"quick brown fox\""
#'(lambda (doc score)
(format T "~&Document ~S found with score of ~S." doc score)))
What if we want to find all documents entered on or after 5th of September, 2005
with the words "adipocere" or "gullet" in any field. We could type something like;
;; Doesn't work yet, spans haven't been implemented.
(montezuma:search-each *index* "date:(>=20050905) adipocere gullet"
#'(lambda (doc score)
(format T "~&Document ~S found with score of ~S." doc score)))
;; What you can do is this, which will find all documents from
;; September containing "adipocere" or "gullet":
(montezuma:search-each *index* "date:200509* adipocere gullet"
#'(lambda (doc score)
(format T "~&Document ~S found with score of ~S." doc score)))
Montezuma has quite a complex query language. To find out more about
Montezuma's query language, see the query-parser class.
Accessing Documents
You may have noticed that when we run a search we only get the
document number back. By itself this isn't much use to us. Getting the
data from the index is very straightforward. For example if we want
the "title" field from the 3rd document:
(montezuma:document-value (montezuma:get-document *index* 2) "title")
NOTE: documents are indexed from 0.
The default field is an empty string when you use the simple string
document so to access those strings you'll have type:
(montezuma:add-document-to-index *index* "This is a document")
(montezuma:document-value (montezuma:get-document *index* 0) "")
Let's go back to the database example above. If we store all of our
documents with an ID then we can access that field using the ID. As
long as we called our ID field "id" we can do this:
(let ((id "89721347"))
(montezuma:document-value (montezuma:get-document *index* id) "title"))
If however we called our id field "key" we'll have to do this;
(let ((id (make-term "key" "89721347")))
(document-value (get-document *index* id) "title"))
Pretty simple huh? You should note though that if there is more than
one document with the same id or key then only the first one will be
returned so it is probably better that you ensure the key is unique
somehow (Montezuma cannot do that for you).
Modifying and Deleting Documents
What if we want to change the data in the index. Montezuma doesn't actually let you
change the data once it is in the index. But you can delete documents so the standard
way to modify data is to delete it and re-add it again with the modifications made.
It is important to note that when doing this the documents will get a new document
number so you should be careful not to use a document number after the document has
been deleted. Here is an example of modifying a document;
;; Add a document to the index.
(montezuma:add-document-to-index *index* '(("title" . "Programming Lips")
("content" . "blah blah blah")))
;; Modify the document we just added.
(let ((doc-num nil))
(montezuma:search-each *index* "title:\"Programming Lips\""
#'(lambda (doc score)
(declare (ignore score))
(setf doc-num doc)))
(when doc-num
(let ((doc (montezuma:get-document *index* doc-num)))
(montezuma:delete-document *index* doc-num)
(setf (montezuma:document-value doc "title") "Programming Lisp")
(montezuma:add-document-to-index *index* doc))))
Note that if more than one document in the index has the title
"Programming Lips", only one of them will be updated.
Another way of doing the same thing:
;; Modify the document we just added.
(let ((doc-num nil))
(montezuma:search-each *index* "title:\"Programming Lips\""
#'(lambda (doc score)
(declare (ignore score))
(setf doc-num doc)))
(when doc-num
(montezuma:update *index* doc-num '(("title" . "Programming Lisp")))))
And another way, but one which causes all matching documents to be updated:
;; Modify the document we just added.
(montezuma:query-update *index* "title:\"Programming Lips\""
'(("title" . "Programming Lisp")))
And if we know the ID of the document we want to update, and it's
stored in the field named "id":
;; Modify the document with ID "123".
(montezuma:update *index* "123" '(("title" . "Programming Lisp")))
If the ID is stored in another field, say one named "key":
;; Modify the document with key "123".
(montezuma:update *index* (montezuma:make-term "key" "123") '(("title" . "Programming Lisp")))
Deleting documents is similar to updating them:
;; Deletes the document whose "id" field is "123".
(montezuma:delete-document *index* "123")
;; Deletes the document whose "key" field is "123".
(montezuma:delete-document *index* (montezuma:make-term "key" "123"))
;; Deletes the document with the specified misspelling in the title.
(montezuma:delete-document *index* "title:\"Programming Lips\"")
Specifying a Montezuma Index
While making a RAM based index or a minimal persistent Montezuma will work to get started with,
but a fully specified Index will assist in populating the index with documents and in the execution
of typed queries. A persistent index is created if a :PATH value is provided. The following
example of an Montezuma index illustrates the use of index metadata. The :DOCUMENT-ROOT is the
directory from which documents are to be added and, if fields are not stored, where fields may be
found (see text field path in the oanc-corpus index below). The :INDEX-KEY should uniquely identify
an Index in one Lisp session. The :DOCUMENT-KEY is an optional value that uniquely defines a document
in an Index of loaded documents. The :TITLE is a descriptive value of the Index for Montezuma users.
The :NAME is a possibly abbreviated name for the Index which also uniquely identifies the Index amongst
all the Indexes available but which is used to select an Index. The :URL may be specified to reference
where the documents may be obtained.
Fields are specified in :FIELD-DEFINITIONS which includes Montezuma directions for handling the
addition of fields to documents (:STORED, and :INDEX) and for parsing typed fields (Montezuma 1.4).
Each field definition is a list in which the first element is a quoted field name and which is
followed by a list of properties.
Generalised Regular Expression Preprocessor (REP)
Inspired by local-time date forms, the generalised Regular Expression Preprocessor (REP) may be used
to generate Common Lisp cl-ppcre scanners. REPs may be used wherever a regular expression is needed,
but not all of Perl's Regular Expression features are supported in REP (in this release: Montezuma 1.4.0).
Unlike cl-ppcre's regular expressions which are based on Perl's syntax, REP is designed to accept
an S-Expression consisting of lists and atoms. This makes it easier to enter in a Lisp IDE with its
built-in parenthesis matching. REP produces a regular expression string for use with cl-ppcre.
Strings generated by REP can be reviewed and modified for use anywhere they are needed. The impetus
for this developement arose from the difficulty of managing medium to high complexity regular
expressions in a Lisp environment which typically includes features facilitating Lisp code development.
A concise guide to REP
REP expressions consist of one or more subexpressions which begin with an optional register name
(which may be nil or t if no name is wanted).
Register names facilitate the processing of REP subexpression values. In the case of date forms,
REP is a good way to define the order, character patterns, and width of field data as found in
actual documents (see below for a description of REP date forms). Following the optional register
name, a character class may be specified, otherwise, the digit class is assumed.
A REP subexpressions may also specify a character class with an escaped character or a named
character class.
The following REP consists of three named registers: day, month, and year. It is converted
into a regular expression for use with cl-ppcre and cl-ppcre generates a compiled regular
expression with a call to create-scanner.
> (regular-expression '((day (1 2)) / (month (1 2)) / (year 4)))
< "(?<DAY>\\d{1,2})/(?<MONTH>\\d{1,2})/(?<YEAR>\\d{4})"
> (cl-ppcre:create-scanner (regular-expression '((day (1 2)) / (month (1 2)) / (year 4))))
< #<Closure 8 subfunction of CL-PPCRE::CREATE-SCANNER-AUX 2009C0AA>
< ("DAY" "MONTH" "YEAR")
(defparameter brown-corpus
(make-instance
'montezuma:index
:document-root (merge-pathnames "brown" corpus-root) ; where source documents are located
:path (merge-pathnames "brown/index" corpus-root) ; where index files are stored
:index-key "brown" ; unique identity of this index among all indexes in this session
:document-key "path" ; field whose values uniquely identify a document in the index
:title "Brown University Corpus" ; preferably unique description of this index
:url "https://en.wikipedia.org/wiki/Brown_Corpus/" ; where documents were obtained
:name "BROWN" ;
:field-definitions
'(("path" :stored t :index :untokenized) ; define path field; don't break into tokens
("name" :stored t :index :untokenized :query "name:cm04") ; extracted from path
("category"
:stored t :index :untokenized
:query "category:science_fiction") ; descriptive document category
("date" :stored t :index :tokenized ; dd/mm/yyyy possibly with trailing hh:mm
:type date
:form ((day (1 2)) / (month (1 2)) / (year 4) (? " " (hour 2) #\: (min 2)))
:query "date:8/8/2015")
("size" :stored t :index :tokenized :type int :query "size:KB")
("author" :stored t :index :tokenized :query "author:sommers")
("title" :stored t :index :tokenized :query "title:beautiful")
("source" :stored t :index :tokenized :query "source:research")
("text" :stored t :index :tokenized :abridge t :query "justice"))
:retrieved-fields '("name" "author" "title" "source" "date")
:default-search-field "text"))
The OANC index features a text field that is not stored in the index but whose values
may be read from the "path" field value.
(defparameter oanc-corpus
(make-instance
'montezuma:index
:path (merge-pathnames "oanc/index" corpus-root)
:document-root (merge-pathnames "oanc/" corpus-root)
:index-key "oanc"
:document-key "path"
:title "Open American National Corpus"
:url "http://www.anc.org/" ; download the OANC corpus from this address
:name "OANC"
:field-definitions
'(("path" :stored t :index :untokenized :query nil)
("name" :stored t :index :untokenized :query "name:Vallarta-WhatToDo")
("title" :stored t :index :tokenized :query "title:production")
("subject" :stored t :index :tokenized :query "subject:things to do")
("idno" :stored t :index :untokenized :query nil)
("author" :stored t :index :tokenized :query "author:david")
("date" :stored t :index :untokenized :type date
:form ((:year 4) - (:month 2) - (:day 2)) :query nil)
("publisher" :stored t :index :tokenized
:query "publisher:federal")
("pubdate" :stored t :index :untokenized :query nil)
("pubplace" :stored t :index :untokenized :query "pubplace:USA")
("source" :stored t :index :tokenized
:query "source:\"Cambridge University Press\"")
("extent" :stored t :index :tokenized :query "")
("audience" :stored t :index :untokenized :query "audience:adult")
("medium" :stored t :index :untokenized :query "medium:book")
("domain" :stored t :index :tokenized :query "domain:Leisure")
("subdomain" :stored t :index :tokenized :query "subdomain:Travel")
("size" :stored t :index :tokenized :query "size:35 KB")
("text"
:stored nil ; don't store text: it blows montezuma up
:index :tokenized
:path "path"
:abridged t
:query "first"))
:default-search-field "text"
:abridgement-threshold 800
:retrieved-fields '("path" "size" "title" "author" "date" "publisher"
"extent" "medium" "domain" "subdomain"))
An informal description of Montezuma Queries
This description of Montezuma Query Syntax is based on the implementation
of the Montezuma Query Parser grammar used to process search queries.
It is intended to be a guide for the construction of Montezuma queries, rather
than to be a rigourous BNF definition of Montezuma query syntax.
Queries
A Montezuma Query generally consists of one or more clauses, but not necessarily!
A query without clauses is equivalent to a wild-card search for all documents
regardless of document content but it is almost certainly more efficient than a
search using the match-all wild-card ('*').
The full range of sort orders may be used to express built-in document number
and score value sorts in addition to document fields in the index for sorting
in either ascending or descending order.
Boundary values can used to specify the first document and number of documents
to be retrieved.
Finally, the fields extracted from documents in search results can be used to
select specified document fields.
Clauses can be combined in a list or with logical operators ('and', 'or', 'not').
In addition, clauses may be prefixed by a field specifier, optionally using a
wild-card field specifier.
Queries may be given a boost factor to tune the weight of some clauses relative
to the others.
Range Queries consist of one or two limiting values with operators for inclusive
or exclusive limits.
Fuzzy Queries consist of a word value followed by a '~' and optionally a fuzzy
factor between 0 and 1.0 which are satisfied by field words with an closely
matching editing difference.
Support has been added for typed terms and range queries for: integer, float and dates.
Dates can be specified in a form based on ISO 8601: YYYYMMDDTHHmmSSssssss. A minimum
datetime value is a four digit year. Datetimes may be followed by a timezone offset:
Z, or a + or - followed by hhmm. For example, 2016 or 20151225. Note: dates with or
without times and timezone offsets must be given without either spaces or non-alphabetic
punctuatiom. Alternatively, datetime values may be specified in a quoted datetime phrase
which makes them easier to enter and read, e.g. '2015-12-25'.
Similarly, fields me be defined in a Montezuma Index as int or float in which case the
Query Parser generates typed queries for these fields.
Execution of typed terms and ranges is coordinated between field definitions in the
Montezuma Index and terms in queries.
~~~~~
<montezuma-query> ::= <clause> <sort-order> <boundaries> <field-list>
<boundaries> ::= ':' <first-document> ',' <number-of-documents> | ''
<first-document> ::= integer
<number-of-documents> ::= integer
<sort-order> ::= '/' <direction> <sorting> | ''
<direction> ::= '+' | '-' | ''
<sorting>> ::= 'document' | 'score' | <word> | '(' <sort-field-names> ')'
<sort-field-names> ::= <sort-field-name> | <sort-field-name> <sort-field-names>
<sort-field-name> ::= <direction> <word>
<field-list> ::= '[' <field-names> ']</field-names>' | ''
<field-names> ::= <field-name> | <field-name> <field-names>
<field-name> ::= <wild-word> | <word>
<logical-operator> ::= 'not' | 'and' | 'or' | ''
<field-specifier> ::= <field-name> ':' | ''
<simple-query> ::= <phrase-query> | <wild-query> | <parenthesised-query> |
<field-range-query> | <fuzzy-query> | <term-query>
<clause> ::= <logical-operator> <field-specifier> <simple-query> <boost> | ''
<boost> ::= '^' <number> | ''
<field-range-query> ::= <lower-bound> <word> 'to' <word> <upper-bound>
<lower-bound> ::= '[' | '{']
<upper-bound> ::= '</upper-bound>' | '}'
<parenthesised-query> ::= '(' <clause> ')'
<phrase> ::= '"' <word-list> '"' <proximity>
<word-list> ::= <word> | <word> <word-list>
<proximity> ::= '~' integer | ''
<term-query> ::= <word>
<fuzzy-query> ::= <word> '~' <fuzzy-factor>
<fuzzy-factor> ::= <number> | ''
<wild-query> ::= <wild-word>
<number> ::= integer | real
<letter> ::= <non-wild-letter> | <character-entity>
<word> ::= <letter> | <letter> <word>
<character-entity> ::= <decimal-entity> | <hexadecimal-entity> | <named-entity>
<decimal-entity> ::= '&#' integer ';'
<hexdecimal-entity> ::= '&#x' hexadecimal-number ';'
<named-entity> ::= '&' <word> ';'
<wild-word-constituent> ::= <any-letter> | <wild-letter> | <character-entity>
<wild-word> ::= non-wild-letter <wild-letter> <wild-word-constituent></wild-word-constituent></wild-letter>
<wild-letter> ::= '' | ''
<any-letter> ::= '[A-Z0-9-]'
<non-wild-letter> ::= [^*?]
~~~~</non-wild-letter></any-letter></wild-letter></wild-word></character-entity></wild-letter></any-letter></wild-word-constituent></word></named-entity></hexdecimal-entity></decimal-entity></named-entity></hexadecimal-entity></decimal-entity></character-entity></word></letter></letter></word></character-entity></non-wild-letter></letter></number></wild-word></wild-query></number></fuzzy-factor></fuzzy-factor></word></fuzzy-query></word></term-query></proximity></word-list></word></word></word-list></proximity></word-list></phrase></clause></parenthesised-query></lower-bound></upper-bound></word></word></lower-bound></field-range-query></number></boost></boost></simple-query></field-specifier></logical-operator></clause></term-query></fuzzy-query></field-range-query></parenthesised-query></wild-query></phrase-query></simple-query></field-name></field-specifier></logical-operator></word></wild-word></field-name></field-names></field-name></field-name></field-names></field-list></word></direction></sort-field-name></sort-field-names></sort-field-name></sort-field-name></sort-field-names></sort-field-names></word></sorting></direction></sorting></direction></sort-order></number-of-documents></first-document></number-of-documents></first-document></boundaries></field-list></boundaries></sort-order></clause></montezuma-query>
Onward
For a complete example of using Montezuma to index and retrieve real information, see the file tests/corpora/pastes-1000/paste-search.lisp.
Related
