https://sourceforge.net/p/montezuma/wiki/DevelopersIntroduction/Recent changes to DevelopersIntroductionenWed, 06 Jan 2016 19:37:45 -0000https://sourceforge.net/p/montezuma/wiki/DevelopersIntroduction/<div class="markdown_content"><pre>--- v17 +++ v18 @@ -73,7 +73,7 @@ ("date" . ,(date row)))) ~~~~ -The methods above will store all of the input data as well tokenizing +The methods above will store all of the input data as well as tokenizing and indexing it. Sometimes we won't want to tokenize (divide the string into tokens) the data. For example, we might want to leave the title as a complete string and only allow searches for that complete @@ -92,7 +92,8 @@ (montezuma:add-field doc (montezuma:make-field "id" (id row) - :stored NIL :index :untokenized)) + :stored NIL + :index :untokenized)) (montezuma:add-field doc (montezuma:make-field "title" (title row) @@ -111,18 +112,16 @@ (montezuma:add-document-to-index *index* doc)) ~~~~ -You can also compress the data that you are storing or store term -vectors with the data. Read more about this in the documentation for -Montezuma's `field` class. +You can also compress the data that you are storing or store term vectors with the data. +Read more about this in the documentation for Montezuma's `field` class. Searching ========= -Now that we have data in our index, how do we actually use this index -to search the data? The index class offers two search methods, `search` -and `search-each`. The first method returns a `top-docs` object. The -second we'll show here. Lets say we wanted to find all documents with -the phrase "quick brown fox" in the content field. We'd write +Now that we have data in our index, how do we actually use this index to search the data? +The index class offers two search methods, `search` and `search-each`. The first method +returns a `top-docs` object. The second we'll show here. Lets say we wanted to find all +documents with the phrase "quick brown fox" in the content field. We'd write ~~~~ (montezuma:search-each *index* "content:\"quick brown fox\"" @@ -130,9 +129,8 @@ (format T "~&amp;Document ~S found with score of ~S." doc score))) ~~~~ -What if we want to find all documents entered on or after 5th of -September, 2005 with the words "adipocere" or "gullet" in any -field. We could type something like; +What if we want to find all documents entered on or after 5th of September, 2005 +with the words "adipocere" or "gullet" in any field. We could type something like; ~~~~ ;; Doesn't work yet, spans haven't been implemented. @@ -198,13 +196,12 @@ Modifying and Deleting Documents -------------------------------- -What if we want to change the data in the index. Montezuma doesn't -actually let you change the data once it is in the index. But you can -delete documents so the standard way to modify data is to delete it -and re-add it again with the modifications made. It is important to -note that when doing this the documents will get a new document number -so you should be careful not to use a document number after the -document has been deleted. Here is an example of modifying a document; +What if we want to change the data in the index. Montezuma doesn't actually let you +change the data once it is in the index. But you can delete documents so the standard +way to modify data is to delete it and re-add it again with the modifications made. +It is important to note that when doing this the documents will get a new document +number so you should be careful not to use a document number after the document has +been deleted. Here is an example of modifying a document; ~~~~ ;; Add a document to the index. @@ -345,13 +342,15 @@ :path (merge-pathnames "brown/index" corpus-root) ; where index files are stored :index-key "brown" ; unique identity of this index among all indexes in this session :document-key "path" ; field whose values uniquely identify a document in the index - :title "Brown University Corpus" ; description of this index - preferably unique among other indexes - :url "https://en.wikipedia.org/wiki/Brown_Corpus/" ; web address from which raw documents may be obtained + :title "Brown University Corpus" ; preferably unique description of this index + :url "https://en.wikipedia.org/wiki/Brown_Corpus/" ; where documents were obtained :name "BROWN" ; :field-definitions - '(("path" :stored t :index :untokenized) ; define field named path and don't break it into indexed tokens - ("name" :stored t :index :untokenized :query "name:cm04") ; file name from path, also unique to document - ("category" :stored t :index :untokenized :query "category:science_fiction") ; descriptive document category + '(("path" :stored t :index :untokenized) ; define path field; don't break into tokens + ("name" :stored t :index :untokenized :query "name:cm04") ; extracted from path + ("category" + :stored t :index :untokenized + :query "category:science_fiction") ; descriptive document category ("date" :stored t :index :tokenized ; dd/mm/yyyy possibly with trailing hh:mm :type date :form ((day (1 2)) / (month (1 2)) / (year 4) (? " " (hour 2) #\: (min 2))) @@ -365,8 +364,8 @@ :default-search-field "text")) ~~~~ -The OANC index features a text field that is not stored in the index but whose -values may be read from the "path" field value. +The OANC index features a text field that is not stored in the index but whose values +may be read from the "path" field value. ~~~~ (defparameter oanc-corpus @@ -514,7 +513,7 @@ ~~~~ - -=== Onward === +Onward +------ For a complete example of using Montezuma to index and retrieve real information, see the file `tests/corpora/pastes-1000/paste-search.lisp`. </pre> </div>Roy Edward AndersonWed, 06 Jan 2016 19:37:45 -0000https://sourceforge.net75393946494d8cded1613a812264484ad51bfe4ahttps://sourceforge.net/p/montezuma/wiki/DevelopersIntroduction/<div class="markdown_content"><pre>--- v16 +++ v17 @@ -6,9 +6,9 @@ The simplest way to use Montezuma is through the `montezuma:index` class. Start by loading the Montezuma system: -~~~ +~~~~ (asdf:oos 'asdf:load-op '#:montezuma) -~~~ +~~~~ Creating an index </pre> </div>Roy Edward AndersonWed, 06 Jan 2016 18:28:58 -0000https://sourceforge.netbeb412654eeeb2e827871fc503840c168e035e93https://sourceforge.net/p/montezuma/wiki/DevelopersIntroduction/<div class="markdown_content"><pre>--- v15 +++ v16 @@ -1,7 +1,7 @@ A Developer's Quick Introduction to Montezuma ============================================= -(Based on the [http://ferret.davebalmain.com/api/files/TUTORIAL.html Ferret tutorial] by Dave Balmain.) +(Based on the [Ferret tutorial](http://ferret.davebalmain.com/api/files/TUTORIAL.html) by Dave Balmain.) The simplest way to use Montezuma is through the `montezuma:index` class. Start by loading the Montezuma system: </pre> </div>Roy Edward AndersonWed, 06 Jan 2016 18:17:01 -0000https://sourceforge.net51b685e6182105e3bd134298b022fe87fd10d6cchttps://sourceforge.net/p/montezuma/wiki/DevelopersIntroduction/<div class="markdown_content"><pre>--- v14 +++ v15 @@ -1,7 +1,7 @@ A Developer's Quick Introduction to Montezuma -==== - -(Based on the [Ferret tutorial](http://ferret.davebalmain.com/api/files/TUTORIAL.html) by Dave Balmain) +============================================= + +(Based on the [http://ferret.davebalmain.com/api/files/TUTORIAL.html Ferret tutorial] by Dave Balmain.) The simplest way to use Montezuma is through the `montezuma:index` class. Start by loading the Montezuma system: @@ -12,7 +12,7 @@ Creating an index ----- +----------------- To create an in-memory index is very simple: @@ -33,17 +33,16 @@ you can specify it here, eg; ~~~~ - (defparameter *index* (make-instance - 'montezuma:index - :path "/path/to/index" - :analyzer (make-instance 'montezuma:whitespace-analyzer))) + (defparameter *index* (make-instance 'montezuma:index + :path "/path/to/index" + :analyzer (make-instance 'montezuma:whitespace-analyzer))) ~~~~ For more options when creating an index refer to the `montezumaindex` class. Adding Documents ----- +---------------- To add a document you can simply add a string. This will store the string in the "" (i.e. empty string) field (unless you specify the @@ -53,15 +52,14 @@ (montezuma:add-document-to-index *index* "This is a new document to be indexed") ~~~~ -But these are pretty simple documents. If this is all you want to index you could probably just use -simple-search. So let's give our documents some fields: - -~~~~ - (montezuma:add-document-to-index *index* - '(("title" . "Programming Ruby") +But these are pretty simple documents. If this is all you want to +index you could probably just use simple-search. So let's give our +documents some fields: + +~~~~ + (montezuma:add-document-to-index *index* '(("title" . "Programming Ruby") ("content" . "blah blah blah"))) - (montezuma:add-document-to-index *index* - '(("title" . "Programming Lisp") + (montezuma:add-document-to-index *index* '(("title" . "Programming Lisp") ("content" . "yada yada yada"))) ~~~~ @@ -88,22 +86,26 @@ class. e.g., ~~~~ - ;; Assume ROW is an object with reader methods ID, TITLE, DATA and IMAGE. + ;; Assume ROW is an object with reader methods ID, TITLE, DATA and + ;; IMAGE. (let ((doc (make-instance 'montezuma:document))) - (montezuma:add-field doc (montezuma:make-field - "id" (id row) - :stored NIL - :index :untokenized)) - (montezuma:add-field doc (montezuma:make-field - "title" (title row) + (montezuma:add-field doc (montezuma:make-field + "id" + (id row) + :stored NIL :index :untokenized)) + (montezuma:add-field doc (montezuma:make-field + "title" + (title row) :stored T :index :untokenized)) - (montezuma:add-field doc (montezuma:make-field - "data" (data row) + (montezuma:add-field doc (montezuma:make-field + "data" + (data row) :stored T :index :tokenized)) (montezuma:add-field doc (montezuma:make-field - "image" (image row) + "image" + (image row) :stored T :index NIL)) (montezuma:add-document-to-index *index* doc)) @@ -113,9 +115,8 @@ vectors with the data. Read more about this in the documentation for Montezuma's `field` class. - Searching ----- +========= Now that we have data in our index, how do we actually use this index to search the data? The index class offers two search methods, `search` @@ -151,7 +152,7 @@ Accessing Documents ----- +------------------- You may have noticed that when we run a search we only get the document number back. By itself this isn't much use to us. Getting the @@ -195,7 +196,7 @@ Modifying and Deleting Documents ----- +-------------------------------- What if we want to change the data in the index. Montezuma doesn't actually let you change the data once it is in the index. But you can @@ -275,7 +276,245 @@ ~~~~ -Onward ----- +Specifying a Montezuma Index +============================ + +While making a RAM based index or a minimal persistent Montezuma will work to get started with, +but a fully specified Index will assist in populating the index with documents and in the execution +of typed queries. A persistent index is created if a :PATH value is provided. The following +example of an Montezuma index illustrates the use of index metadata. The :DOCUMENT-ROOT is the +directory from which documents are to be added and, if fields are not stored, where fields may be +found (see text field path in the oanc-corpus index below). The :INDEX-KEY should uniquely identify +an Index in one Lisp session. The :DOCUMENT-KEY is an optional value that uniquely defines a document +in an Index of loaded documents. The :TITLE is a descriptive value of the Index for Montezuma users. +The :NAME is a possibly abbreviated name for the Index which also uniquely identifies the Index amongst +all the Indexes available but which is used to select an Index. The :URL may be specified to reference +where the documents may be obtained. + +Fields are specified in :FIELD-DEFINITIONS which includes Montezuma directions for handling the +addition of fields to documents (:STORED, and :INDEX) and for parsing typed fields (Montezuma 1.4). +Each field definition is a list in which the first element is a quoted field name and which is +followed by a list of properties. + +Generalised Regular Expression Preprocessor (REP) +------------------------------------------------- + +Inspired by local-time date forms, the generalised Regular Expression Preprocessor (REP) may be used +to generate Common Lisp cl-ppcre scanners. REPs may be used wherever a regular expression is needed, +but not all of Perl's Regular Expression features are supported in REP (in this release: Montezuma 1.4.0). + +Unlike cl-ppcre's regular expressions which are based on Perl's syntax, REP is designed to accept +an S-Expression consisting of lists and atoms. This makes it easier to enter in a Lisp IDE with its +built-in parenthesis matching. REP produces a regular expression string for use with cl-ppcre. +Strings generated by REP can be reviewed and modified for use anywhere they are needed. The impetus +for this developement arose from the difficulty of managing medium to high complexity regular +expressions in a Lisp environment which typically includes features facilitating Lisp code development. + +A concise guide to REP +---------------------- + +REP expressions consist of one or more subexpressions which begin with an optional register name +(which may be nil or t if no name is wanted). + +Register names facilitate the processing of REP subexpression values. In the case of date forms, +REP is a good way to define the order, character patterns, and width of field data as found in +actual documents (see below for a description of REP date forms). Following the optional register +name, a character class may be specified, otherwise, the digit class is assumed. + +A REP subexpressions may also specify a character class with an escaped character or a named +character class. + +The following REP consists of three named registers: day, month, and year. It is converted +into a regular expression for use with cl-ppcre and cl-ppcre generates a compiled regular +expression with a call to create-scanner. + +~~~~ +&gt; (regular-expression '((day (1 2)) / (month (1 2)) / (year 4))) +&lt; "(?&lt;DAY&gt;\\d{1,2})/(?&lt;MONTH&gt;\\d{1,2})/(?&lt;YEAR&gt;\\d{4})" + +&gt; (cl-ppcre:create-scanner (regular-expression '((day (1 2)) / (month (1 2)) / (year 4)))) +&lt; #&lt;Closure 8="" subfunction="" of="" CL-PPCRE::CREATE-SCANNER-AUX="" 2009C0AA=""&gt; +&lt; ("DAY" "MONTH" "YEAR") +~~~~ + +~~~~ +(defparameter brown-corpus + (make-instance + 'montezuma:index + :document-root (merge-pathnames "brown" corpus-root) ; where source documents are located + :path (merge-pathnames "brown/index" corpus-root) ; where index files are stored + :index-key "brown" ; unique identity of this index among all indexes in this session + :document-key "path" ; field whose values uniquely identify a document in the index + :title "Brown University Corpus" ; description of this index - preferably unique among other indexes + :url "https://en.wikipedia.org/wiki/Brown_Corpus/" ; web address from which raw documents may be obtained + :name "BROWN" ; + :field-definitions + '(("path" :stored t :index :untokenized) ; define field named path and don't break it into indexed tokens + ("name" :stored t :index :untokenized :query "name:cm04") ; file name from path, also unique to document + ("category" :stored t :index :untokenized :query "category:science_fiction") ; descriptive document category + ("date" :stored t :index :tokenized ; dd/mm/yyyy possibly with trailing hh:mm + :type date + :form ((day (1 2)) / (month (1 2)) / (year 4) (? " " (hour 2) #\: (min 2))) + :query "date:8/8/2015") + ("size" :stored t :index :tokenized :type int :query "size:KB") + ("author" :stored t :index :tokenized :query "author:sommers") + ("title" :stored t :index :tokenized :query "title:beautiful") + ("source" :stored t :index :tokenized :query "source:research") + ("text" :stored t :index :tokenized :abridge t :query "justice")) + :retrieved-fields '("name" "author" "title" "source" "date") + :default-search-field "text")) +~~~~ + +The OANC index features a text field that is not stored in the index but whose +values may be read from the "path" field value. + +~~~~ +(defparameter oanc-corpus + (make-instance + 'montezuma:index + :path (merge-pathnames "oanc/index" corpus-root) + :document-root (merge-pathnames "oanc/" corpus-root) + :index-key "oanc" + :document-key "path" + :title "Open American National Corpus" + :url "http://www.anc.org/" ; download the OANC corpus from this address + :name "OANC" + :field-definitions + '(("path" :stored t :index :untokenized :query nil) + ("name" :stored t :index :untokenized :query "name:Vallarta-WhatToDo") + ("title" :stored t :index :tokenized :query "title:production") + ("subject" :stored t :index :tokenized :query "subject:things to do") + ("idno" :stored t :index :untokenized :query nil) + ("author" :stored t :index :tokenized :query "author:david") + ("date" :stored t :index :untokenized :type date + :form ((:year 4) - (:month 2) - (:day 2)) :query nil) + ("publisher" :stored t :index :tokenized + :query "publisher:federal") + ("pubdate" :stored t :index :untokenized :query nil) + ("pubplace" :stored t :index :untokenized :query "pubplace:USA") + ("source" :stored t :index :tokenized + :query "source:\"Cambridge University Press\"") + ("extent" :stored t :index :tokenized :query "") + ("audience" :stored t :index :untokenized :query "audience:adult") + ("medium" :stored t :index :untokenized :query "medium:book") + ("domain" :stored t :index :tokenized :query "domain:Leisure") + ("subdomain" :stored t :index :tokenized :query "subdomain:Travel") + ("size" :stored t :index :tokenized :query "size:35 KB") + ("text" + :stored nil ; don't store text: it blows montezuma up + :index :tokenized + :path "path" + :abridged t + :query "first")) + :default-search-field "text" + :abridgement-threshold 800 + :retrieved-fields '("path" "size" "title" "author" "date" "publisher" + "extent" "medium" "domain" "subdomain")) +~~~~ + +An informal description of Montezuma Queries +-------------------------------------------- + +This description of Montezuma Query Syntax is based on the implementation +of the Montezuma Query Parser grammar used to process search queries. +It is intended to be a guide for the construction of Montezuma queries, rather +than to be a rigourous BNF definition of Montezuma query syntax. + +Queries +------- + +A Montezuma Query generally consists of one or more clauses, but not necessarily! + +A query without clauses is equivalent to a wild-card search for all documents +regardless of document content but it is almost certainly more efficient than a +search using the match-all wild-card ('*'). + +The full range of sort orders may be used to express built-in document number +and score value sorts in addition to document fields in the index for sorting +in either ascending or descending order. + +Boundary values can used to specify the first document and number of documents +to be retrieved. + +Finally, the fields extracted from documents in search results can be used to +select specified document fields. + +Clauses can be combined in a list or with logical operators ('and', 'or', 'not'). + +In addition, clauses may be prefixed by a field specifier, optionally using a +wild-card field specifier. + +Queries may be given a boost factor to tune the weight of some clauses relative +to the others. + +Range Queries consist of one or two limiting values with operators for inclusive +or exclusive limits. + +Fuzzy Queries consist of a word value followed by a '~' and optionally a fuzzy +factor between 0 and 1.0 which are satisfied by field words with an closely +matching editing difference. + +Support has been added for typed terms and range queries for: integer, float and dates. + +Dates can be specified in a form based on ISO 8601: YYYYMMDDTHHmmSSssssss. A minimum +datetime value is a four digit year. Datetimes may be followed by a timezone offset: +Z, or a + or - followed by hhmm. For example, 2016 or 20151225. Note: dates with or +without times and timezone offsets must be given without either spaces or non-alphabetic +punctuatiom. Alternatively, datetime values may be specified in a quoted datetime phrase +which makes them easier to enter and read, e.g. '2015-12-25'. + +Similarly, fields me be defined in a Montezuma Index as int or float in which case the +Query Parser generates typed queries for these fields. + +Execution of typed terms and ranges is coordinated between field definitions in the +Montezuma Index and terms in queries. + +~~~~~ +&lt;montezuma-query&gt; ::= &lt;clause&gt; &lt;sort-order&gt; &lt;boundaries&gt; &lt;field-list&gt; +&lt;boundaries&gt; ::= ':' &lt;first-document&gt; ',' &lt;number-of-documents&gt; | '' +&lt;first-document&gt; ::= integer +&lt;number-of-documents&gt; ::= integer +&lt;sort-order&gt; ::= '/' &lt;direction&gt; &lt;sorting&gt; | '' +&lt;direction&gt; ::= '+' | '-' | '' +&lt;sorting&gt;&gt; ::= 'document' | 'score' | &lt;word&gt; | '(' &lt;sort-field-names&gt; ')' +&lt;sort-field-names&gt; ::= &lt;sort-field-name&gt; | &lt;sort-field-name&gt; &lt;sort-field-names&gt; +&lt;sort-field-name&gt; ::= &lt;direction&gt; &lt;word&gt; +&lt;field-list&gt; ::= '[' &lt;field-names&gt; ']' | '' +&lt;field-names&gt; ::= &lt;field-name&gt; | &lt;field-name&gt; &lt;field-names&gt; +&lt;field-name&gt; ::= &lt;wild-word&gt; | &lt;word&gt; +&lt;logical-operator&gt; ::= 'not' | 'and' | 'or' | '' +&lt;field-specifier&gt; ::= &lt;field-name&gt; ':' | '' +&lt;simple-query&gt; ::= &lt;phrase-query&gt; | &lt;wild-query&gt; | &lt;parenthesised-query&gt; | + &lt;field-range-query&gt; | &lt;fuzzy-query&gt; | &lt;term-query&gt; +&lt;clause&gt; ::= &lt;logical-operator&gt; &lt;field-specifier&gt; &lt;simple-query&gt; &lt;boost&gt; | '' +&lt;boost&gt; ::= '^' &lt;number&gt; | '' +&lt;field-range-query&gt; ::= &lt;lower-bound&gt; &lt;word&gt; 'to' &lt;word&gt; &lt;upper-bound&gt; +&lt;lower-bound&gt; ::= '[' | '{' +&lt;upper-bound&gt; ::= ']' | '}' +&lt;parenthesised-query&gt; ::= '(' &lt;clause&gt; ')' +&lt;phrase&gt; ::= '"' &lt;word-list&gt; '"' &lt;proximity&gt; +&lt;word-list&gt; ::= &lt;word&gt; | &lt;word&gt; &lt;word-list&gt; +&lt;proximity&gt; ::= '~' integer | '' +&lt;term-query&gt; ::= &lt;word&gt; +&lt;fuzzy-query&gt; ::= &lt;word&gt; '~' &lt;fuzzy-factor&gt; +&lt;fuzzy-factor&gt; ::= &lt;number&gt; | '' +&lt;wild-query&gt; ::= &lt;wild-word&gt; +&lt;number&gt; ::= integer | real +&lt;letter&gt; ::= &lt;non-wild-letter&gt; | &lt;character-entity&gt; +&lt;word&gt; ::= &lt;letter&gt; | &lt;letter&gt; &lt;word&gt; +&lt;character-entity&gt; ::= &lt;decimal-entity&gt; | &lt;hexadecimal-entity&gt; | &lt;named-entity&gt; +&lt;decimal-entity&gt; ::= '&amp;#' integer ';' +&lt;hexdecimal-entity&gt; ::= '&amp;#x' hexadecimal-number ';' +&lt;named-entity&gt; ::= '&amp;' &lt;word&gt; ';' +&lt;wild-word-constituent&gt; ::= &lt;any-letter&gt; | &lt;wild-letter&gt; | &lt;character-entity&gt; +&lt;wild-word&gt; ::= non-wild-letter* &lt;wild-letter&gt; &lt;wild-word-constituent&gt;* +&lt;wild-letter&gt; ::= '*' | '*' +&lt;any-letter&gt; ::= '[A-Z0-9-]' +&lt;non-wild-letter&gt; ::= [^*?] +~~~~ + + + +=== Onward === For a complete example of using Montezuma to index and retrieve real information, see the file `tests/corpora/pastes-1000/paste-search.lisp`. </pre> </div>Roy Edward AndersonWed, 06 Jan 2016 18:14:06 -0000https://sourceforge.net41850fede83f8689a4e123ab94d32ecd3c88ef99https://sourceforge.net/p/montezuma/wiki/DevelopersIntroduction/<div class="markdown_content"><pre>--- v13 +++ v14 @@ -1,4 +1,5 @@ -= A Developer's Quick Introduction to Montezuma = +A Developer's Quick Introduction to Montezuma +==== (Based on the [Ferret tutorial](http://ferret.davebalmain.com/api/files/TUTORIAL.html) by Dave Balmain) @@ -10,7 +11,8 @@ ~~~ -=== Creating an index === +Creating an index +---- To create an in-memory index is very simple: @@ -31,15 +33,17 @@ you can specify it here, eg; ~~~~ - (defparameter *index* (make-instance 'montezuma:index - :path "/path/to/index" - :analyzer (make-instance 'montezuma:whitespace-analyzer))) + (defparameter *index* (make-instance + 'montezuma:index + :path "/path/to/index" + :analyzer (make-instance 'montezuma:whitespace-analyzer))) ~~~~ For more options when creating an index refer to the `montezumaindex` class. -=== Adding Documents === +Adding Documents +---- To add a document you can simply add a string. This will store the string in the "" (i.e. empty string) field (unless you specify the @@ -49,14 +53,15 @@ (montezuma:add-document-to-index *index* "This is a new document to be indexed") ~~~~ -But these are pretty simple documents. If this is all you want to -index you could probably just use simple-search. So let's give our -documents some fields: - -~~~~ - (montezuma:add-document-to-index *index* '(("title" . "Programming Ruby") +But these are pretty simple documents. If this is all you want to index you could probably just use +simple-search. So let's give our documents some fields: + +~~~~ + (montezuma:add-document-to-index *index* + '(("title" . "Programming Ruby") ("content" . "blah blah blah"))) - (montezuma:add-document-to-index *index* '(("title" . "Programming Lisp") + (montezuma:add-document-to-index *index* + '(("title" . "Programming Lisp") ("content" . "yada yada yada"))) ~~~~ @@ -83,13 +88,24 @@ class. e.g., ~~~~ - ;; Assume ROW is an object with reader methods ID, TITLE, DATA and - ;; IMAGE. + ;; Assume ROW is an object with reader methods ID, TITLE, DATA and IMAGE. (let ((doc (make-instance 'montezuma:document))) - (montezuma:add-field doc (montezuma:make-field "id" (id row) :stored NIL :index :untokenized)) - (montezuma:add-field doc (montezuma:make-field "title" (title row) :stored T :index :untokenized)) - (montezuma:add-field doc (montezuma:make-field "data" (data row) :stored T :index :tokenized)) - (montezuma:add-field doc (montezuma:make-field "image" (image row) :stored T :index NIL)) + (montezuma:add-field doc (montezuma:make-field + "id" (id row) + :stored NIL + :index :untokenized)) + (montezuma:add-field doc (montezuma:make-field + "title" (title row) + :stored T + :index :untokenized)) + (montezuma:add-field doc (montezuma:make-field + "data" (data row) + :stored T + :index :tokenized)) + (montezuma:add-field doc (montezuma:make-field + "image" (image row) + :stored T + :index NIL)) (montezuma:add-document-to-index *index* doc)) ~~~~ @@ -98,7 +114,8 @@ Montezuma's `field` class. -=== Searching === +Searching +---- Now that we have data in our index, how do we actually use this index to search the data? The index class offers two search methods, `search` @@ -133,7 +150,8 @@ Montezuma's query language, see the `query-parser` class. -=== Accessing Documents === +Accessing Documents +---- You may have noticed that when we run a search we only get the document number back. By itself this isn't much use to us. Getting the @@ -176,7 +194,8 @@ somehow (Montezuma cannot do that for you). -=== Modifying and Deleting Documents === +Modifying and Deleting Documents +---- What if we want to change the data in the index. Montezuma doesn't actually let you change the data once it is in the index. But you can @@ -256,6 +275,7 @@ ~~~~ -=== Onward === +Onward +---- For a complete example of using Montezuma to index and retrieve real information, see the file `tests/corpora/pastes-1000/paste-search.lisp`. </pre> </div>Roy Edward AndersonWed, 06 Jan 2016 01:54:29 -0000https://sourceforge.net0425227c9b53d511b64c3bde082ba83d8a8ba090https://sourceforge.net/p/montezuma/wiki/DevelopersIntroduction/<div class="markdown_content"><pre>--- v12 +++ v13 @@ -1,6 +1,6 @@ = A Developer's Quick Introduction to Montezuma = -(Based on the [http://ferret.davebalmain.com/api/files/TUTORIAL.html Ferret tutorial] by Dave Balmain.) +(Based on the [Ferret tutorial](http://ferret.davebalmain.com/api/files/TUTORIAL.html) by Dave Balmain) The simplest way to use Montezuma is through the `montezuma:index` class. Start by loading the Montezuma system: </pre> </div>Roy Edward AndersonWed, 06 Jan 2016 00:55:02 -0000https://sourceforge.net21a9bdeeae7bc2a1f412ffe453406faf4c69e015https://sourceforge.net/p/montezuma/wiki/DevelopersIntroduction/<div class="markdown_content"><pre>--- v11 +++ v12 @@ -1,28 +1,34 @@ -#A Developer's Quick Introduction to Montezuma - -Based on the [Ferret tutorial](https://github.com/xing/ferret/blob/master/TUTORIAL) by Dave Balmain. += A Developer's Quick Introduction to Montezuma = + +(Based on the [http://ferret.davebalmain.com/api/files/TUTORIAL.html Ferret tutorial] by Dave Balmain.) The simplest way to use Montezuma is through the `montezuma:index` class. Start by loading the Montezuma system: -~~~~ +~~~ (asdf:oos 'asdf:load-op '#:montezuma) -~~~~ -##Creating an index -To create an in-memory index is simple: +~~~ + + +=== Creating an index === + +To create an in-memory index is very simple: ~~~~ (defparameter *index* (make-instance 'montezuma:index)) ~~~~ + To create a persistent index; + ~~~~ (defparameter *index* (make-instance 'montezuma:index :path "/path/to/index")) ~~~~ -Both of these methods create new indexes with the `standard-analyzer`. an analyzer is what you use to -divide the input data up into tokens which you can search for later. If you'd like to use a different analyzer you -can specify it here, eg; +Both of these methods create new indexes with the `standard-analyzer`. +An analyzer is what you use to divide the input data up into tokens +which you can search for later. If you'd like to use a different analyzer +you can specify it here, eg; ~~~~ (defparameter *index* (make-instance 'montezuma:index @@ -30,20 +36,22 @@ :analyzer (make-instance 'montezuma:whitespace-analyzer))) ~~~~ -For more options when creating an index refer to the `index` class. - - -##Adding Documents - -To add a document you can simply add a string. This will store the string in the "" (i.e. empty string) field -(unless you specify the default field when you create the index). +For more options when creating an index refer to the `montezumaindex` class. + + +=== Adding Documents === + +To add a document you can simply add a string. This will store the +string in the "" (i.e. empty string) field (unless you specify the +default field when you create the index). ~~~~ (montezuma:add-document-to-index *index* "This is a new document to be indexed") ~~~~ -But these are pretty simple documents. If this is all you want to index you could probably just use -simple-search. So let's give our documents some fields: +But these are pretty simple documents. If this is all you want to +index you could probably just use simple-search. So let's give our +documents some fields: ~~~~ (montezuma:add-document-to-index *index* '(("title" . "Programming Ruby") @@ -52,8 +60,9 @@ ("content" . "yada yada yada"))) ~~~~ -Or if you are indexing data stored in a database, you'll probably want to store the ID (this example -assumes you have some kind of database object in ROW, with ID, TITLE and DATE accessors): +Or if you are indexing data stored in a database, you'll probably want +to store the ID (this example assumes you have some kind of database +object in ROW, with ID, TITLE and DATE accessors): ~~~~ (montezuma:add-document-to-index *index* `(("id" . ,(id row)) @@ -61,12 +70,15 @@ ("date" . ,(date row)))) ~~~~ -The methods above will store all of the input data as well tokenizing and indexing it. Sometimes we -won't want to tokenize (divide the string into tokens) the data. For example, we might want to leave the -title as a complete string and only allow searches for that complete string. Sometimes we won't want to -store the data as it's already stored in the database so it'll be a waste to store it in the index. Or -perhaps we are doing without a database and using Montezuma to store all of our data, in which -case we might not want to index it. For example, if we are storing images in the index, we won't want +The methods above will store all of the input data as well tokenizing +and indexing it. Sometimes we won't want to tokenize (divide the +string into tokens) the data. For example, we might want to leave the +title as a complete string and only allow searches for that complete +string. Sometimes we won't want to store the data as it's already +stored in the database so it'll be a waste to store it in the +index. Or perhaps we are doing without a database and using Montezuma +to store all of our data, in which case we might not want to index +it. For example, if we are storing images in the index, we won't want to index them. All of this can be done using Montezuma's document class. e.g., @@ -81,16 +93,18 @@ (montezuma:add-document-to-index *index* doc)) ~~~~ -You can also compress the data that you are storing or store term vectors with the data. Read more about -this in the documentation for Montezuma's `field` class. - - -##Searching - -Now that we have data in our index, how do we actually use this index to search the data? -The index class offers two search methods, `search` and `search-each`. -The first method returns a `top-docs` object. The second we'll show here. -Lets say we wanted to find all documents with the phrase "quick brown fox" in the content field. We'd write +You can also compress the data that you are storing or store term +vectors with the data. Read more about this in the documentation for +Montezuma's `field` class. + + +=== Searching === + +Now that we have data in our index, how do we actually use this index +to search the data? The index class offers two search methods, `search` +and `search-each`. The first method returns a `top-docs` object. The +second we'll show here. Lets say we wanted to find all documents with +the phrase "quick brown fox" in the content field. We'd write ~~~~ (montezuma:search-each *index* "content:\"quick brown fox\"" @@ -98,61 +112,51 @@ (format T "~&amp;Document ~S found with score of ~S." doc score))) ~~~~ -What if we want to find all documents entered on or after 5th of September, 2005 with the -words "adipocere" or "gullet" in any field. We could type something like; - -~~~~ - ;; You could do this: - (montezuma:search-each *index* "date:[20050905 adipocere gullet" +What if we want to find all documents entered on or after 5th of +September, 2005 with the words "adipocere" or "gullet" in any +field. We could type something like; + +~~~~ + ;; Doesn't work yet, spans haven't been implemented. + (montezuma:search-each *index* "date:(&gt;=20050905) adipocere gullet" #'(lambda (doc score) (format T "~&amp;Document ~S found with score of ~S." doc score))) - ;; You can also do this, which will find all documents from + ;; What you can do is this, which will find all documents from ;; September containing "adipocere" or "gullet": (montezuma:search-each *index* "date:200509* adipocere gullet" #'(lambda (doc score) (format T "~&amp;Document ~S found with score of ~S." doc score))) ~~~~ -Montezuma has quite a complex query language. To find out more about Montezuma's -query language, see the `query-parser` class. - -**Highlighting and Annotating** - -Montezuma has an efficient highlighting method for search results. See montezuma's -hits module. Here is an example of how you would use it when printing results. - -~~~~ - (multiple-value-bind (docs total) - (annotated-search-results - index - #'(lambda(name) (and name "<b>")) - #'(lambda(name) (and name "</b>")) - query) - (values docs total)) -~~~~ -##Accessing Documents - -You may have noticed that when we run a search we only get the document number back. -By itself this isn't much use to us. Getting the data from the index is very straightforward. -For example if we want the "title" field from the 3rd document: +Montezuma has quite a complex query language. To find out more about +Montezuma's query language, see the `query-parser` class. + + +=== Accessing Documents === + +You may have noticed that when we run a search we only get the +document number back. By itself this isn't much use to us. Getting the +data from the index is very straightforward. For example if we want +the "title" field from the 3rd document: ~~~~ (montezuma:document-value (montezuma:get-document *index* 2) "title") ~~~~ -NOTE: **documents are indexed from 0.** - -The default field is an empty string when you use the simple string document so to access -those strings you'll have type: +NOTE: documents are indexed from 0. + +The default field is an empty string when you use the simple string +document so to access those strings you'll have type: ~~~~ (montezuma:add-document-to-index *index* "This is a document") (montezuma:document-value (montezuma:get-document *index* 0) "") ~~~~ -Let's go back to the database example above. If we store all of our documents with an ID -then we can access that field using the ID. As long as we called our ID field "id" we can do this: +Let's go back to the database example above. If we store all of our +documents with an ID then we can access that field using the ID. As +long as we called our ID field "id" we can do this: ~~~~ (let ((id "89721347")) @@ -166,18 +170,21 @@ (document-value (get-document *index* id) "title")) ~~~~ -Pretty simple huh? You should note though that if there is more than one document with -the same id or key then only the first one will be returned so it is probably better that you -ensure the key is unique somehow (Montezuma cannot do that for you). - - -##Modifying and Deleting Documents - -What if we want to change the data in the index. Montezuma doesn't actually let you change -the data once it is in the index. But you can delete documents so the standard way to modify data -is to delete it and re-add it again with the modifications made. It is important to note that when -doing this the documents will get a new document number so you should be careful not to use -a document number after the document has been deleted. Here is an example of modifying a document; +Pretty simple huh? You should note though that if there is more than +one document with the same id or key then only the first one will be +returned so it is probably better that you ensure the key is unique +somehow (Montezuma cannot do that for you). + + +=== Modifying and Deleting Documents === + +What if we want to change the data in the index. Montezuma doesn't +actually let you change the data once it is in the index. But you can +delete documents so the standard way to modify data is to delete it +and re-add it again with the modifications made. It is important to +note that when doing this the documents will get a new document number +so you should be careful not to use a document number after the +document has been deleted. Here is an example of modifying a document; ~~~~ ;; Add a document to the index. @@ -196,8 +203,8 @@ (montezuma:add-document-to-index *index* doc)))) ~~~~ -Note that if more than one document in the index has the title "Programming Lips", only one of them -will be updated. +Note that if more than one document in the index has the title +"Programming Lips", only one of them will be updated. Another way of doing the same thing: @@ -220,7 +227,8 @@ '(("title" . "Programming Lisp"))) ~~~~ -And if we know the ID of the document we want to update, and it's stored in the field named "id": +And if we know the ID of the document we want to update, and it's +stored in the field named "id": ~~~~ ;; Modify the document with ID "123". @@ -248,6 +256,6 @@ ~~~~ -##Onward +=== Onward === For a complete example of using Montezuma to index and retrieve real information, see the file `tests/corpora/pastes-1000/paste-search.lisp`. </pre> </div>Roy Edward AndersonWed, 06 Jan 2016 00:52:30 -0000https://sourceforge.net95e5c9a7c046feeb803f8c01554dd6bb71036e62https://sourceforge.net/p/montezuma/wiki/DevelopersIntroduction/<div class="markdown_content"><pre>--- v10 +++ v11 @@ -117,6 +117,20 @@ Montezuma has quite a complex query language. To find out more about Montezuma's query language, see the `query-parser` class. +**Highlighting and Annotating** + +Montezuma has an efficient highlighting method for search results. See montezuma's +hits module. Here is an example of how you would use it when printing results. + +~~~~ + (multiple-value-bind (docs total) + (annotated-search-results + index + #'(lambda(name) (and name "<b>")) + #'(lambda(name) (and name "</b>")) + query) + (values docs total)) +~~~~ ##Accessing Documents You may have noticed that when we run a search we only get the document number back. </pre> </div>Roy Edward AndersonThu, 19 Nov 2015 20:52:38 -0000https://sourceforge.net7b0b1b5da02b29d0d94e325584575a4c7c93dc12https://sourceforge.net/p/montezuma/wiki/DevelopersIntroduction/<div class="markdown_content"><pre>--- v9 +++ v10 @@ -1,6 +1,6 @@ #A Developer's Quick Introduction to Montezuma -Based on the [Ferret tutorial](http://ferret.davebalmain.com/api/files/TUTORIAL.html) by Dave Balmain. +Based on the [Ferret tutorial](https://github.com/xing/ferret/blob/master/TUTORIAL) by Dave Balmain. The simplest way to use Montezuma is through the `montezuma:index` class. Start by loading the Montezuma system: </pre> </div>Roy Edward AndersonThu, 19 Nov 2015 08:45:08 -0000https://sourceforge.netd733ac49d732c3860cf532ab1f24b9d6e285ed55https://sourceforge.net/p/montezuma/wiki/DevelopersIntroduction/<div class="markdown_content"><pre>--- v8 +++ v9 @@ -8,17 +8,13 @@ ~~~~ (asdf:oos 'asdf:load-op '#:montezuma) ~~~~ - ##Creating an index - To create an in-memory index is simple: ~~~~ (defparameter *index* (make-instance 'montezuma:index)) ~~~~ - To create a persistent index; - ~~~~ (defparameter *index* (make-instance 'montezuma:index :path "/path/to/index")) @@ -106,12 +102,12 @@ words "adipocere" or "gullet" in any field. We could type something like; ~~~~ - ;; Doesn't work yet, spans haven't been implemented. - (montezuma:search-each *index* "date:(&gt;=20050905) adipocere gullet" + ;; You could do this: + (montezuma:search-each *index* "date:[20050905 adipocere gullet" #'(lambda (doc score) (format T "~&amp;Document ~S found with score of ~S." doc score))) - ;; What you can do is this, which will find all documents from + ;; You can also do this, which will find all documents from ;; September containing "adipocere" or "gullet": (montezuma:search-each *index* "date:200509* adipocere gullet" #'(lambda (doc score) </pre> </div>Roy Edward AndersonThu, 19 Nov 2015 08:33:44 -0000https://sourceforge.netc55afe4940448e68530633124d72219cc86dd986