Menu
▾
▴
xapian-devel — Mailing list for Xapian developers
You can subscribe to this list here.
| 2001 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(3) |
Nov
(79) |
Dec
(8) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2002 |
Jan
(105) |
Feb
(42) |
Mar
(37) |
Apr
(110) |
May
(275) |
Jun
(121) |
Jul
(127) |
Aug
(8) |
Sep
(49) |
Oct
(18) |
Nov
(3) |
Dec
(25) |
| 2003 |
Jan
(20) |
Feb
(37) |
Mar
(29) |
Apr
(25) |
May
(32) |
Jun
(29) |
Jul
(65) |
Aug
(15) |
Sep
|
Oct
(1) |
Nov
(27) |
Dec
(2) |
| 2004 |
Jan
(16) |
Feb
|
Mar
(3) |
Apr
(3) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: Franck M. <fra...@if...> - 2003-11-14 13:35:28
|
Hi List,
I just had a look at omega, and asked it to index some internal =
documentations. It works fine, but when i send it a query, it often =
propose me to add terms starting with an "R"...
I had a look at the documentation, and found informations about "B" =
terms, but not about "R" terms, so i get into the code and found that =
omindex add two postings for each words :
- the lower case stemmed word
- "R" + the lower case NON stemmed word (the "Raw" word)
By doing this, don't we break the wdf used to compute the weight of =
documents ?
Thank you for response, and keep the good job !
Franck |
|
From: Olly B. <ol...@su...> - 2003-10-08 13:13:06
|
On Wed, Oct 08, 2003 at 02:05:33PM +0100, Olly Betts wrote: > Log message: > backends/quartz/.cvsignore,tests/Makefile.am,testsuite/.cvsignore: > Updated to reflect move of libbtreecheck.la from backends/quartz > to testsuite. This problem prevents Xapian 0.7.4 passing "make check" (I failed to spot this as I had libbtreecheck.la in the old location in my tree, as did all the tinderbox machines). I don't think it justifies rushing a 0.7.5 release out immediately. If you want to run the testsuite, apply this patch to fix the problem: http://xapian.org/C?xapian/xapian-core/tests/Makefile.am?1.97?1.98 Cheers, Olly |
|
From: James A. <jam...@ta...> - 2003-08-27 09:47:59
|
On Wed, Aug 27, 2003 at 09:49:02AM +0200, Petter Egesund wrote:
> > Use xapian.open() - the Xapian::Auto namespace is flattened in the
> > bindings.
>
> That was what I thought, but when I import the library and enter
> "xapian.open();"
>
> The result is
>
> File <stdin>", line 1, in ?
> TypeError: No matching function for overloaded 'open'
>
> Any clues?
You need to actually give it something to open! eg:
# db = xapian.open("path/to/db")
# db2 = xapian.open("path/to/db", xapian.DB_CREATE_OR_OPEN)
Look at the documentation for Xapian::Auto::open() to figure out what
you can pass in. (Unlike C++, you need to qualify the second parameter
as xapian.WHATEVER rather than just WHATEVER.)
J
--
/--------------------------------------------------------------------------\
James Aylett xapian.org
ja...@ta... uncertaintydivision.org
|
|
From: Petter E. <pet...@ku...> - 2003-08-27 07:49:48
|
-----Opprinnelig melding----- Fra: Petter Egesund Sendt: 26. august 2003 16:00 Til: 'James Aylett' Emne: SV: [Xapian-devel] Python binding Hi and thanks very much for answering! That was what I thought, but when I import the library and enter "xapian.open();" The result is File <stdin>", line 1, in ? TypeError: No matching function for overloaded 'open' Any clues? Cheers, Petter Egesund -----Opprinnelig melding----- Fra: James Aylett [mailto:jam...@ta...] Sendt: 26. august 2003 15:43 Til: Petter Egesund Kopi: 'xap...@li...' Emne: Re: [Xapian-devel] Python binding On Thu, Aug 21, 2003 at 03:54:46PM +0200, Petter Egesund wrote: > I use the swig Python-binding but does not see how to create a new database. Use xapian.open() - the Xapian::Auto namespace is flattened in the bindings. J -- /--------------------------------------------------------------------------\ James Aylett xapian.org ja...@ta... uncertaintydivision.org ########################################### This message has been scanned by F-Secure Anti-Virus for Microsoft Exchange. For more information, connect to http://www.F-Secure.com/ |
|
From: James A. <jam...@ta...> - 2003-08-26 13:43:24
|
On Thu, Aug 21, 2003 at 03:54:46PM +0200, Petter Egesund wrote: > I use the swig Python-binding but does not see how to create a new database. Use xapian.open() - the Xapian::Auto namespace is flattened in the bindings. J -- /--------------------------------------------------------------------------\ James Aylett xapian.org ja...@ta... uncertaintydivision.org |
|
From: Petter E. <pet...@ku...> - 2003-08-22 21:40:00
|
Hi, I use the swig Python-binding but does not see how to create a new database. The tricky part seems to be this line (in C): Xapian::WritableDatabase database(Xapian::Quartz::open(argv[1], Xapian::DB_OPEN)); I can construct a new WritableDatabase but how do I call Xapian::Quarts::open from Python? Any clues? This looks very promising, and I would really try to test it with the Python-interface. (Perl interface does not build - I have perl 5.8.0, gcc 2.96, and Swig 1.3.19) Cheers, Petter Egesund Norway ########################################### This message has been scanned by F-Secure Anti-Virus for Microsoft Exchange. For more information, connect to http://www.F-Secure.com/ |
|
From: Olly B. <ol...@su...> - 2003-08-09 09:07:20
|
On Fri, Aug 08, 2003 at 09:05:11PM -0700, Nick Anderson wrote:
> I am still having problems compiling Xapian 0.7.2 under Solaris 8. I don't
> have Latex installed, is it absolutely necessary? If I can avoid using it in
> the build, how would I do this?
LaTeX should only be needed if you want to modify the source and rebuild
the documentation, but the bug in 0.7.2's build system makes it over
eager to do that.
Could you try the shiny new 0.7.3 instead? It should fix this problem.
Cheers,
Olly
|
|
From: Nick A. <mo...@ex...> - 2003-08-09 04:08:02
|
Hi there I am still having problems compiling Xapian 0.7.2 under Solaris 8. I don't have Latex installed, is it absolutely necessary? If I can avoid using it in the build, how would I do this? thanks -Nick |
|
From: Nicolay M. <ca...@fl...> - 2003-08-08 23:12:01
|
Hi,
I'm currently writing another PHP4 binding without Swig. It is not a
full 100% implementation of the whole API (I think this will be the Swig
implementation) but it supports the most important features.
Current function list (will be expanded a bit):
xapian_database_create("path");
$dbhandle=xapian_database_open("path",XAPIAN_OPEN_READ or
XAPIAN_OPEN_WRITE);
$docid=xapian_database_add_document($dbhandle,$dochandle)
$res=xapian_database_delete_document($dbhandle,$docid)
$res=xapian_database_replace_document($dbhandle,$docid,$dochandle)
$dochandle=xapian_database_get_document($dbhandle,$docid)
$num=xapian_database_doccount($dbhandle)
$dochandle=xapian_document();
$res=xapian_document_set_data($dochandle,"string")
$string=xapian_document_get_data($dochandle)
$res=xapian_document_add_posting($dochandle,"term",$position)
$stemhandle=xapian_stemmer("language");
$term=xapian_stemmer_stem_word($stemhandle,"term");
$qhandle=xapian_query_term("term");
$qhandle=xapian_query_op(XAPIAN_OP_...,$qhandle_left,$qhandle_right)
$result=xapian_search_execute($dbhandle,$qhandle,$from,$num,$withdata,$sorting,$sortkey,$collapse_key,$callback);
These functions are currently working well and most have been tested
with Xapian V 0.7 + PHP 4.3.2 under Linux.
I'm planning to release a first version of this extension under GNU in
the next week.
Nicolay
|
|
From: Olly B. <ol...@su...> - 2003-08-08 16:43:52
|
On Fri, Aug 08, 2003 at 09:20:15AM -0700, Nick Anderson wrote:
> I am trying to build Xapian 7.2
Actually, it'll be Xapian 0.7.2 unless you got your copy through a
timewarp...
> on a Sun 5.8 system. I do not have root
> access, so I am running configure with the
>
> --prefix =$HOME flag. That appears to work normally.
>
> However, when I follow this with "make", I get the following error:
>
> ./include/xapian/stem.h ../include/xapian/termiterator.h
> ../include/xapian/types.h ../include/xapian/valueiterator.h
> ../include/xapian/version.h -newer ./apidoc/html/index.html -print`" ; then
> \
> echo Not rebuilding apidoc/html/index.html despite make with crude VPATH
> ;\
> else \
> rm -rf apidoc ;\
> top_srcdir=.. ./doxygen doxygen_api_conf ;\
> fi
> bash: ./doxygen: No such file or directory
There's a bug in the build system - in some situations the documentation
is rebuilt if include/xapian/version.h changes, but that file is
generated by configure. This is fixed in CVS, and so will be fixed in
0.7.3 (which I'd actually hoped to upload today, though time is really
flying by...)
> Since I don't have root access, I downloaded the doxygen binary (1.3) and
> installed the binaries into my path. I have tried everything to get make to
> pick up the path for doxygen, but it doesen't seem to want to.
Did you rerun configure too? You may need to "make distclean" first to
stop configure using a cached value for the doxygen check.
Otherwise, try:
./configure --prefix=$HOME DOXYGEN=/path/to/doxygen
Cheers,
Olly
|
|
From: Nick A. <mo...@ex...> - 2003-08-08 16:23:07
|
Hi there
I am trying to build Xapian 7.2 on a Sun 5.8 system. I do not have root
access, so I am running configure with the
--prefix =$HOME flag. That appears to work normally.
However, when I follow this with "make", I get the following error:
./include/xapian/stem.h ../include/xapian/termiterator.h
../include/xapian/types.h ../include/xapian/valueiterator.h
../include/xapian/version.h -newer ./apidoc/html/index.html -print`" ; then
\
echo Not rebuilding apidoc/html/index.html despite make with crude VPATH
;\
else \
rm -rf apidoc ;\
top_srcdir=.. ./doxygen doxygen_api_conf ;\
fi
bash: ./doxygen: No such file or directory
*** Error code 1
make: Fatal error: Command failed for target `apidoc/html/index.html'
Current working directory
/home/students/andersni/Xapian7/xapian-core-0.7.2/docs
*** Error code 1
make: Fatal error: Command failed for target `all-recursive'
Current working directory /home/students/andersni/Xapian7/xapian-core-0.7.2
*** Error code 1
make: Fatal error: Command failed for target `all'
Since I don't have root access, I downloaded the doxygen binary (1.3) and
installed the binaries into my path. I have tried everything to get make to
pick up the path for doxygen, but it doesen't seem to want to.
Any ideas?
thanks -Nick
|
|
From: Olly B. <ol...@su...> - 2003-08-08 14:26:44
|
On Fri, Aug 08, 2003 at 02:48:02PM +0100, Richard Boulton wrote:
> We could just declare that bootstrap must be run from the top level
> directory, but since this could lead to confusion and the fix is easy, I
> put together a patch.
>
> Olly - if you agree with this, please commit it (or tell me to). I've
> not been keeping track of the build system stuff lately, so I don't want
> to stomp on someones work by mistake.
Erm, I'll need to look in more detail when I'm a bit less busy but the
patch seems plausible. The general idea is bootstrap should create a
build tree like you'd get from a dist tarball, and that would have
configure scripts in the src tree, and Makefiles in the build tree.
The only thing I wonder is what to do about the current behaviour of
removing the top-level configure on "make distclean" or "make
maintainer-clean". If you've several build trees you don't really want
the shared configure to be deleted when you clean up one of them...
Cheers,
Olly
|
|
From: Richard B. <ri...@ta...> - 2003-08-08 13:49:55
|
I just confused myself by running the toplevel bootstrap script from a directory other than the top level: xapian$ mkdir build-nodebug xapian$ cd build-nodebug xapian/build-nodebug$ ../bootstrap && ../configure Unfortunately, this results in an error, because the configure script is created in the build-nodebug directory. Since I already had a configure script in the top level, but it was out of date, it took a while to find the problem. We could just declare that bootstrap must be run from the top level directory, but since this could lead to confusion and the fix is easy, I put together a patch. Olly - if you agree with this, please commit it (or tell me to). I've not been keeping track of the build system stuff lately, so I don't want to stomp on someones work by mistake. -- Richard Boulton <ri...@ta...> |
|
From: Olly B. <ol...@su...> - 2003-08-08 13:16:49
|
On Fri, Aug 08, 2003 at 02:58:48PM +0200, Bart van Bragt wrote:
> It does do something more :D I can send you the complete output of
> the bootstrap script. A summary of the output:
>
> Bootstrapping `xapian-core'
[...]
> Makefile.am:5: require version 1.6.3, but have 1.4-p5
[...]
> FATAL ERROR: Autoconf version 2.54 or higher is required for this script
Hmm. There's rather a lot of noise, but it does at least report the
versions of autoconf and automake required, and building from CVS isn't
something we expect many people to do. I think that's probably better
than failing with incorrect syntax for autoreconf.
> [...]
> Bootstrapped successfully - now run "./configure" and "make"
Hmm, that's not good. I guess that's a side effect of running
autoreconf in a subshell.
> Thanks again! I'm going to play with Xapian and see if I can replace the
> SQL based system we're using in phpBB now (which scales pretty bad :D)).
Cool. Note that indexing doesn't scale as well as it should at present.
Fixing this is the next major thing on my todo list.
Cheers,
Olly
|
|
From: Bart v. B. <xa...@va...> - 2003-08-08 12:58:39
|
Wow, thanks for the extremely quick and acurate reply :D
Olly Betts wrote:
> This is a known problem, with a simple fix. Just add "#include
> <stdio.h>" near the start of the file (the exact location doesn't
> matter).
Thanks! Works like a charm..
> ${AUTORECONF-autoreconf} -i "$d"
>
> to:
>
> (cd "$d" && ${AUTORECONF-autoreconf} -i)
It does do something more :D I can send you the complete output of
the bootstrap script. A summary of the output:
Bootstrapping `xapian-core'
ls: aclocal.m4: No such file or directory
aclocal: configure.in: 318: macro `AM_CFLAGS' not found in library
aclocal: configure.in: 319: macro `AM_CXXFLAGS' not found in library
[snip]
Makefile.am:5: require version 1.6.3, but have 1.4-p5
include/Makefile.am:18: invalid variable `nodist_xapianinclude_HEADERS'
backends/Makefile.am:39: AC_PROG_MAKE_SET must be used in `configure.in'
automake: configure.in: AC_ARG_PROGRAM must be used in `configure.in'
automake: configure.in: AC_PROG_INSTALL must be used in `configure.in'
FATAL ERROR: Autoconf version 2.54 or higher is required for this script
FATAL ERROR: Autoconf version 2.54 or higher is required for this script
automake: configure.in: `PACKAGE' not defined in `configure.in'
automake: configure.in: `VERSION' not defined in `configure.in'
automake: configure.in: required file `./install-sh' not found
automake: configure.in: required file `./mkinstalldirs' not found
automake: configure.in: required file `./missing' not found
automake: no `Makefile.am' found or specified
autoconf: Undefined macros:
***BUG in Autoconf--please report*** AC_FD_MSG
***BUG in Autoconf--please report*** AC_FD_CC
***BUG in Autoconf--please report*** AC_FD_MSG
[snip]
tcl8/Makefile.am:11: variable `XAPIAN_LIBS' not defined
tcl8/Makefile.am:10: invalid unused variable name:
`nodist_xapianc_la_SOURCES'
automake: configure.in: AC_PROG_INSTALL must be used in `configure.in'
FATAL ERROR: Autoconf version 2.54 or higher is required for this script
FATAL ERROR: Autoconf version 2.54 or higher is required for this script
[snip]
Bootstrapped successfully - now run "./configure" and "make"
Thanks again! I'm going to play with Xapian and see if I can replace the
SQL based system we're using in phpBB now (which scales pretty bad :D)).
--
Bart van Bragt
Senior Developer phpBB (www.phpBB.com)
|
|
From: Olly B. <ol...@su...> - 2003-08-08 12:46:56
|
On Fri, Aug 08, 2003 at 02:31:19PM +0200, Bart van Bragt wrote:
> quartzcompact.cc: In function `int main(int, char **)':
> quartzcompact.cc:120: implicit declaration of function `int rename(...)'
This is a known problem, with a simple fix. Just add "#include
<stdio.h>" near the start of the file (the exact location doesn't
matter).
> The only thing out of the ordinary during make (on Debian 3.0 stable) was:
>
> checking for latex... /usr/bin/latex
> checking for doxygen... ${SHELL} /tmp/xapian-core-0.7.2/missing doxygen
> checking for dvips... /usr/bin/dvips
That's harmless - it just means configure didn't find doxygen, but you
only need it if you modify the source code and want to rebuild the
API documentation.
> Also tried to compile the CVS source but when running 'bootstrap' I get:
>
> Bootstrapping `xapian-core'
> Usage: autoreconf [-f] [-h] [--help] [-m dir] [--macrodir=dir]
> [-l dir] [--localdir=dir] [--force] [--verbose] [--version]
> [--cygnus] [--foreign] [--gnits] [--gnu] [-i] [--include-deps]
>
> It seems to fail on:
>
> autoreconf -i ./xapian-core
I'm guessing you have autoconf 2.13 installed (check the output of:
autoreconf --version). You'll need at least autoconf 2.54 to
successfully create xapian-core/configure etc, but it's not helpful
that the failure happens like this if you don't. I suspect we
ought to change:
${AUTORECONF-autoreconf} -i "$d"
to:
(cd "$d" && ${AUTORECONF-autoreconf} -i)
Could you try that and see if it fails in a more explanatory way?
The alternative would be to hardcode an autoconf version check into
bootstrap, which means another place to update if we raise the
required autoconf version...
Cheers,
Olly
|
|
From: Bart v. B. <xa...@va...> - 2003-08-08 12:31:09
|
Just tried to build xapian-core-0.7.2.tar.gz but I get:
creating quartzcheck
source='quartzcompact.cc' object='quartzcompact-quartzcompact.o'
libtool=no \
depfile='.deps/quartzcompact-quartzcompact.Po'
tmpdepfile='.deps/quartzcompact-quartzcompact.TPo' \
depmode=gcc /bin/sh ../depcomp \
g++ -DHAVE_CONFIG_H -I. -I. -I.. -I../common -I../include -I../include
-I../backends/quartz -g -O2 -c -o quartzcompact-quartzcompact.o `test
-f 'quartzcompact.cc' || echo './'`quartzcompact.cc
quartzcompact.cc: In function `int main(int, char **)':
quartzcompact.cc:120: implicit declaration of function `int rename(...)'
make[2]: *** [quartzcompact-quartzcompact.o] Error 1
make[2]: Leaving directory `/tmp/xapian-core-0.7.2/bin'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/tmp/xapian-core-0.7.2'
make: *** [all] Error 2
After doing './configure && make'
The only thing out of the ordinary during make (on Debian 3.0 stable) was:
checking for latex... /usr/bin/latex
checking for doxygen... ${SHELL} /tmp/xapian-core-0.7.2/missing doxygen
checking for dvips... /usr/bin/dvips
Also tried building on Redhat 7.3 and it also gives me an error with rename:
if g++ -DHAVE_CONFIG_H -I. -I. -I.. -I../common -I../include
-I../include -I../backends/quartz -g -O2 -MT
quartzcompact-quartzcompact.o -MD -MP -MF
".deps/quartzcompact-quartzcompact.Tpo" \
-c -o quartzcompact-quartzcompact.o `test -f 'quartzcompact.cc' ||
echo './'`quartzcompact.cc; \
then mv ".deps/quartzcompact-quartzcompact.Tpo"
".deps/quartzcompact-quartzcompact.Po"; \
else rm -f ".deps/quartzcompact-quartzcompact.Tpo"; exit 1; \
fi
quartzcompact.cc: In function `int main (int, char **)':
quartzcompact.cc:120: `rename' undeclared (first use this function)
quartzcompact.cc:120: (Each undeclared identifier is reported only once
for each function it appears in.)
make[2]: *** [quartzcompact-quartzcompact.o] Error 1
make[2]: Leaving directory
`/www/site/www.bokt.nl/xapian/xapian-core-0.7.2/bin'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/www/site/www.bokt.nl/xapian/xapian-core-0.7.2'
make: *** [all] Error 2
Also tried to compile the CVS source but when running 'bootstrap' I get:
Bootstrapping `xapian-core'
Usage: autoreconf [-f] [-h] [--help] [-m dir] [--macrodir=dir]
[-l dir] [--localdir=dir] [--force] [--verbose] [--version]
[--cygnus] [--foreign] [--gnits] [--gnu] [-i] [--include-deps]
It seems to fail on:
autoreconf -i ./xapian-core
gcc on Redhat:
gcc version 2.96 20000731 (Red Hat Linux 7.3 2.96-113)
gcc on Debian:
gcc version 2.95.4 20011002 (Debian prerelease)
--
Bart van Bragt
Senior Developer phpBB (www.phpBB.com)
|
|
From: Sam L. <sa...@li...> - 2003-07-26 21:06:19
|
Olly Betts wrote: >On Sat, Jul 19, 2003 at 11:09:07AM +0100, Sam Liddicott wrote: > > >>The generated php module source is meant to be a "standalone" php module >>to be compiled against the current installed xapian; though as james >>says there is no reason not to use the old (and still existing) swig php >>behaviour to try and link a php module; except it was too much of a pain >>for general use which is why I added phpize support. >> >> > >Do you think it would be possible to get a patch added to swig which >modified the generated config.m4 to cooperate with libtool if >appropriate? The generated file suggests it might need editting, >but that's rather unhelpful - it would be better if it could be >generated correctly in the first place if we can do it cleanly. > >That would solve our problems more neatly and probably help out other >swig users too. > > Sure, if you can send me a before and after config.m4 (not just a diff) then I can see what needs changing and will work out how to change this so it doesn't break normal module generation. Sam |
|
From: Olly B. <ol...@su...> - 2003-07-26 20:00:20
|
On Sat, Jul 19, 2003 at 11:09:07AM +0100, Sam Liddicott wrote:
> The generated php module source is meant to be a "standalone" php module
> to be compiled against the current installed xapian; though as james
> says there is no reason not to use the old (and still existing) swig php
> behaviour to try and link a php module; except it was too much of a pain
> for general use which is why I added phpize support.
Do you think it would be possible to get a patch added to swig which
modified the generated config.m4 to cooperate with libtool if
appropriate? The generated file suggests it might need editting,
but that's rather unhelpful - it would be better if it could be
generated correctly in the first place if we can do it cleanly.
That would solve our problems more neatly and probably help out other
swig users too.
Cheers,
Olly
|
|
From: Olly B. <ol...@su...> - 2003-07-26 19:48:40
|
On Thu, Jul 17, 2003 at 01:01:52PM +0100, Sam Liddicott wrote:
> "Olly Betts" <ol...@su...>:
> > Are you perhaps trying to run the top level configure before
> > bootstrapping? That won't work...
>
> That will have been it, then.
I've now modified things to avoid others falling into this trap. The
top level configure is generated by the top level bootstrap, so you
can't try to run it first (since it doesn't exist yet!)
Cheers,
Olly
|
|
From: Olly B. <ol...@su...> - 2003-07-26 18:21:24
|
On Wed, Jul 16, 2003 at 06:13:10PM +0100, James Aylett wrote:
> 1. simpleindex should create the database (easy these days)
Agreed - done.
> 2. either both ~index and ~search should stem, or neither
I've made simplesearch stem.
> 3. simpleindex (I think) doesn't index the last paragraph of stdin
It does for me, but while checking this I noticed it runs words together
between lines (so "hello\nworld" gives the term "helloworld" not "hello"
and "world"). I've fixed that.
Now that the Xapian library creates the directory if it doesn't exist
when creating a database, createdatabase seems to me to be rather a
pointless example. Should we remove it?
Cheers,
Olly
|
|
From: Olly B. <ol...@su...> - 2003-07-21 20:25:48
|
This discussion started from an off-list discussion between Martin
Porter and me, but Martin suggested it was worth opening to a wider
audience.
Martin:
> Olly:
> >what about allowing cursors in a writable btree so that quartz
> >doesn't have to have the btree open for reading and writing separately?
>
> I never quite understood this issue - it reflects the way Quartz is used in
> Xapian. As I wrote Quartz you have a cursor for writing, but only one, and
> it goes awry after each write. It is fairly easy to make it position to the
> item after the last thing written [with a small code extension] however.
At present, a quartz database has 5 btrees, and if the database is
opened for update, each btree is opened twice (once for reading and
once for writing). That's 10 file descriptors per writable database
which is rather excessive. Even 5 per database seems like rather a
lot to me.
One possiblity is for the reading and writing btrees to share a file
descriptor - that would work as we always seek before a read or write
(or use pread/pwrite which take a file offset).
After a quartz table has flushed to disk, it actually destroys and
recreates both the btree objects associated with it. I think that
means that the position of the cursor after a write to the btree is
irrelevant, but I'm not certain without looking into this more deeply
- it could be that during the flush the cursor is needed. It seems
excessive too - we certainly don't need to close the underlying file,
and can probably avoid some of the other overhead too.
I believe that multiple cursors are so that we can open multiple
posting lists, termlists, etc. I'm not sure we define the semantics
for the iterators which are built on these if the are active and the
database is updated, but even if we do we could probably reasonably
change it if that helps.
> If you have a batch of cursor
> during writing you need to chain them together and scan them after writes to
> make sure they are positioned correctly. DBMS systems like Oracle do this I
> believe. But that seemed like too much work for the anticipated use Quartz
> would get.
Hmm, I think I see - with two btrees open, the multiple cursors on the
read btree stay on the previous revision of the file, which we can't
sensibly do with just one btree...
Cheers,
Olly
|
|
From: Arjen v. d. M. <ar...@gl...> - 2003-07-21 10:26:30
|
> Olly Betts wrote: > > On Mon, Jul 21, 2003 at 09:30:19AM +0100, James Aylett wrote: > > On Mon, Jul 21, 2003 at 02:21:25AM +0100, Olly Betts wrote: > > > > > Some timed tests on a Linux 2.2.25 machine revealed that > > > paradoxically mmap is slower for searching by about 25-30%. > > > > Was this a multiprocessor machine? > > Yes, it was on ixion (James' colo box). > > > I read a while ago that mmap is > > better for uniprocessor, but read or similar are preferred for > > multiprocessor. > > Aha! I'll try it on a uniprocessor box and see how that compares. If you're interested I can also run a few tests on our dual-cpu (scsi-raid) box and a single-cpu (ide) box, both run very recent 2.4 Linux kernels. Regards, Arjen |
|
From: Olly B. <ol...@su...> - 2003-07-21 09:09:36
|
On Mon, Jul 21, 2003 at 09:30:19AM +0100, James Aylett wrote:
> On Mon, Jul 21, 2003 at 02:21:25AM +0100, Olly Betts wrote:
>
> > Some timed tests on a Linux 2.2.25 machine revealed that
> > paradoxically mmap is slower for searching by about 25-30%.
>
> Was this a multiprocessor machine?
Yes, it was on ixion (James' colo box).
> I read a while ago that mmap is
> better for uniprocessor, but read or similar are preferred for
> multiprocessor.
Aha! I'll try it on a uniprocessor box and see how that compares.
Cheers,
Olly
|
|
From: James A. <jam...@ta...> - 2003-07-21 08:31:23
|
On Mon, Jul 21, 2003 at 02:21:25AM +0100, Olly Betts wrote: > Some timed tests on a Linux 2.2.25 machine revealed that > paradoxically mmap is slower for searching by about 25-30%. Was this a multiprocessor machine? I read a while ago that mmap is better for uniprocessor, but read or similar are preferred for multiprocessor. J -- /--------------------------------------------------------------------------\ James Aylett xapian.org ja...@ta... uncertaintydivision.org |
4 messages has been excluded from this view by a project administrator.
