Home

Copyright 2006 John Wiseman jjwiseman@yahoo.com 7/13/2006
Copyright 2015 Roy Anderson reanz1959@gmail.com 2015-12-12

Updated 2016-01-17

** Welcome to Montezuma! ** A Common Lisp, open source system for indexing and searching large document collections.

For a Lisp developer, Montezuma provides a comprehensive solution for full-text index and search systems that works
in concert with other code systems. It also serves as a code base worthy of study including CLOS and a purpose built
query language. Montezuma is capable of large scale applications to corpus management, for example, the OANC
(Open American National Corpus). Unless you have an interest in a Common Lisp full text index library, Montezuma is probably not for you.

** Introduction **

Montezuma is a text search engine library for Common Lisp based on Ferret for Ruby which is itself based on the (Lucene library) for Java.

For a Lisp developer, Montezuma provides a comprehensive solution for full-text index and search systems that works in concert with other code systems. It also serves as a code base worthy of study including CLOS and a purpose built
query language. Montezuma is capable of large scale applications to corpus management, for example, indexing and searching of the OANC (Open American National Corpus) with more than 8000 documents.

** What's New? **

Montezuma 2.0.3 fixes a bug that caused query processing to fail: a list of field names was being passed in instead of an association list of field definitions.

Montezuma 2.0.2 fixes bugs in the regular expression query handling which caused them to be incorrectly treated as wildcard queries. Montezuma is being prepared for integration and release with Toronto, a collection of Common Lisp applets including Montezuma searches and a web server.

Montezuma 2.0.1 introduces improved handling of queries that cannot be parsed: instead of substituting a simple analysis of query tokens, Montezuma now raises an error condition and tries to identify the cause of the problem.

Montezuma now includes typed queries. More specifically, when a field definition in an index identifies a field type, the type guides the query parser's interpretation of target values. So far, field types include the following: date, int, and float. The date type is the most well developed: search values are parsed based upon the ISO 8601 standard, e.g. 'yyyy-mm-ddTHH:MM' for a query of all date-time values on that day and time.

Montezuma 1.2.0 brought a Listener based shell (Read Eval Print Loop) for exploring Montezuma including the query parser and search features. It also introduced Oropendola: a LispWorks GUI alternative to the Listener shell. To start the shell, copy the Montezuma source and dependencies and enter (in-package :montezuma) and then (shell). To start the Oropendola GUI, you may need to install LispWorks then enter (oropendola) in the Listener.

** Dependencies **

Montezuma requires the following systems, but they are incorporated into the Montezuma release in
the dependencies directory:

CL-PPCRE
ALEXANDRIA
TRIVIAL-FEATURES
BORDEAUX-THREADS
CL-FAD
BABEL
TRIVIAL-GRAY-STREAMS
LOCAL-TIME

** Installation Guide **

After downloading the Montezuma release file, and the packages it depends upon (included in the latest Montezuma releases from Montezuma SourceForge Files. Edit the montezuma.lisp file and change the root directory so that you can load montezuma. You will also need to set the index path in the shell/indexes file. It's a little cumbersome but it should work relatively well once you have configured the directories. When these changes have been made you should be able to load montezuma.lisp when you want to start up Montezuma. Occasionally the montezuma.lisp load fails to locate the asdf system; should this happen, try the compile and load option in the IDE File menu.

Montezuma has been tested with Lispworks 6.1.1 (Windows 10), SBCL 0.9.12 (OS X/PPC), SBCL 0.9.13
(Linux/x86) OpenMCL 1.0 (OS X/PPC) and ACL 8.0 (OS X/PPC). It has been extended in 2015 using Lispworks 6.1.1.

The only implementation-dependent code in Montezuma is in src/util/mop.lisp. To add support for another implementation may be as simple as adding one line to the definition of the CLASS-SLOTS function and one to SLOT-DEFINITION-NAME.

** Installation and Loading **

You can use ASDF-INSTALL to install Montezuma:

  (asdf-install:install '#:montezuma)

And ASDF to load it:

  (asdf:oos 'asdf:load-op '#:montezuma)

** Testing **

Once Montezuma has been loaded, you can run the unit tests if you like:

  (asdf:oos 'asdf:test-op '#:montezuma)

** Use **

See the Developers Inroduction file for more information on how to use Montezuma.

The Montezuma project page at https://sourceforge.net/projects/montezuma/ should have the latest information about Montezuma.

** Query Syntax **

See the docs/MontezumaQuerySyntax/MontezumaQueryParserSyntax.html for a description of the Montezuma Query Syntax.

** Acknowledgements **

Thanks to Dave Balmain, Gary King, Peter Seibel (for his META-inspired parser), Xach Beane (for the heap implementation from his TIMER library) and Franz. Inc. for their (Porter stemmer).

** Failures and Successes Adding Montezuma Documents **

REA While adding documents to Montezuma, every dozen or so additions would raise a Delete File or Rename File exception. I retried adding documents (for Rename exceptions) or restart the load from the last document added. This problem disappeared when I moved the index directory from a DropBox networked drive to a local drive. Not only did the exceptions disappear, but the load times improved from about 5 hours to 5 minutes. I could also remove the checkpoints and repeated index optimization without exceptions.

** Still interested?**

An informal description of Montezuma Query Syntax should provide enough information to leverage the power and elegance of the Montezuma/Lucene query language.

If you want to dig into the Montezuma library you might find the Developers Guide a useful resource.

Onward

For a complete example of using Montezuma to index and retrieve real information, see the file

contrib/pastes-1000/paste-search.lisp

There have been changes to the Montezuma Query Parser that makes some queries that include certain punctuation characters, for example - in a search term, behave differently than before. There are at least two ways to overcome this. The following paste-search query does not parse, but this may be solved by using either an '?';instead of a hyphen, or an equivalent regular expression.

+user:lemonodor +date:2006-04*

The closest query that produces the same intended result is the following:

+user:lemonodor +date:2006?04*

But, the recent extension to Montezuma providing regular expression searches unless a field definition is assigned a type such as date, int or float in which case type specific conversions are applied. But for the paste-search date field, which is untokenised and not stored in the index, the following query produces the same result with similar processing overhead.

+user:lemonodor +date:'2006-04.+'

Project Members:

• Roy Edward Anderson (admin)
• Tzaddi (admin)

[Roy Edward Anderson]

Montezuma has been quiescent for a while but I have spent quite a long time completing the implementation of Lucene Query Syntax including fuzzy queries and sorting. All the unit tests are succeeding. Instead of punctuation escaping, I have implemented numeric and named character entities.