Menu
▾
▴
exist-development — eXist Developer's List
You can subscribe to this list here.
| 2009 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(22) |
Nov
(85) |
Dec
(20) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2010 |
Jan
(47) |
Feb
(127) |
Mar
(268) |
Apr
(78) |
May
(47) |
Jun
(38) |
Jul
(131) |
Aug
(221) |
Sep
(187) |
Oct
(54) |
Nov
(111) |
Dec
(84) |
| 2011 |
Jan
(152) |
Feb
(106) |
Mar
(94) |
Apr
(90) |
May
(53) |
Jun
(20) |
Jul
(24) |
Aug
(37) |
Sep
(32) |
Oct
(70) |
Nov
(22) |
Dec
(15) |
| 2012 |
Jan
(33) |
Feb
(110) |
Mar
(24) |
Apr
(1) |
May
(11) |
Jun
(8) |
Jul
(12) |
Aug
(37) |
Sep
(39) |
Oct
(81) |
Nov
(38) |
Dec
(50) |
| 2013 |
Jan
(23) |
Feb
(53) |
Mar
(23) |
Apr
(5) |
May
(19) |
Jun
(16) |
Jul
(16) |
Aug
(9) |
Sep
(21) |
Oct
(1) |
Nov
(2) |
Dec
(8) |
| 2014 |
Jan
(16) |
Feb
(6) |
Mar
(27) |
Apr
(1) |
May
(10) |
Jun
(1) |
Jul
(4) |
Aug
(10) |
Sep
(19) |
Oct
(22) |
Nov
(4) |
Dec
(6) |
| 2015 |
Jan
(3) |
Feb
(6) |
Mar
(9) |
Apr
|
May
(11) |
Jun
(23) |
Jul
(14) |
Aug
(10) |
Sep
(10) |
Oct
(9) |
Nov
(18) |
Dec
(4) |
| 2016 |
Jan
(5) |
Feb
(5) |
Mar
|
Apr
(2) |
May
(15) |
Jun
(2) |
Jul
(8) |
Aug
(2) |
Sep
(6) |
Oct
|
Nov
|
Dec
|
| 2017 |
Jan
(2) |
Feb
(12) |
Mar
(22) |
Apr
(6) |
May
|
Jun
|
Jul
(1) |
Aug
(1) |
Sep
(5) |
Oct
(2) |
Nov
|
Dec
|
| 2018 |
Jan
(2) |
Feb
|
Mar
|
Apr
|
May
|
Jun
(5) |
Jul
(3) |
Aug
|
Sep
(7) |
Oct
(19) |
Nov
|
Dec
|
| 2021 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(1) |
Nov
|
Dec
|
| 2022 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
(3) |
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: Loren C. <lor...@gm...> - 2010-07-15 18:59:35
|
I can implement format-number http://www.w3.org/TR/xslt#function-format-number Do we want this in util or in fn (http://www.w3.org/2005/xpath-functions)? All I need to do is remove the three functions (number-to-string, number-to-currency-string, and number-to-percent-string) and change number-to-formatted-string to format-number and add the third optional parameter. I just want to make sure that I place it in the correct function module. Loren On Jul 15, 2010, at 01:02 PM, Dmitriy Shabanov wrote: > Can it be xsl's function? > http://www.w3schools.com/XSL/func_formatnumber.asp > > (the module @ xsl extension) > > Can it be one function? > > On Thu, 2010-07-15 at 12:39 -0500, Loren Cahlander wrote: >> I am in the process of adding the formatting of a number to a >> formatted string. There are three general purpose functions that >> format the number to a general number, currency or percentage >> according to the optionally specified locale. The fourth function >> takes in a format string according to the format specification for the >> java.text.DecimalFormat Java class. >> >> >> Please supply me with your comments. Dan will be developing XQuery >> unit tests to validate the functions. You will find the function docs >> below. > > -- > Cheers, > > Dmitriy Shabanov |
|
From: Dmitriy S. <sha...@gm...> - 2010-07-15 18:02:02
|
Can it be xsl's function? http://www.w3schools.com/XSL/func_formatnumber.asp (the module @ xsl extension) Can it be one function? On Thu, 2010-07-15 at 12:39 -0500, Loren Cahlander wrote: > I am in the process of adding the formatting of a number to a > formatted string. There are three general purpose functions that > format the number to a general number, currency or percentage > according to the optionally specified locale. The fourth function > takes in a format string according to the format specification for the > java.text.DecimalFormat Java class. > > > Please supply me with your comments. Dan will be developing XQuery > unit tests to validate the functions. You will find the function docs > below. -- Cheers, Dmitriy Shabanov |
|
From: Loren C. <lor...@gm...> - 2010-07-15 17:39:09
|
I am in the process of adding the formatting of a number to a formatted string. There are three general purpose functions that format the number to a general number, currency or percentage according to the optionally specified locale. The fourth function takes in a format string according to the format specification for the java.text.DecimalFormat Java class. Please supply me with your comments. Dan will be developing XQuery unit tests to validate the functions. You will find the function docs below. Thank you, Loren number-to-currency-string util:number-to-currency-string($number as xs:double, $locale as xs:string?) xs:string? Returns a currency number formatted for the specified locale if specified (default otherwise) as an xs:string value. $number The number $locale two-letter ISO-639 code representing the locale Returns the formatted string number-to-formatted-string util:number-to-formatted-string($number as xs:double, $format as xs:string) xs:string? Returns the number formatted as an xs:string value. $number The number $format The format pattern string. Please see the JavaDoc for java.text.DecimalFormat to get the specifics of this format string. Returns the formatted string number-to-percent-string util:number-to-percent-string($number as xs:double, $locale as xs:string?) xs:string? Returns a percentage number formatted for the specified locale if specified (default otherwise) as an xs:string value. $number The number $locale two-letter ISO-639 code representing the locale Returns the formatted string number-to-string util:number-to-string($number as xs:double, $locale as xs:string?) xs:string? Returns a general-purpose number formatted for the specified locale if specified (default otherwise) as an xs:string value. $number The number $locale two-letter ISO-639 code representing the locale Returns the formatted string |
|
From: Adam R. <ad...@ex...> - 2010-07-15 09:21:12
|
Do you have the correct namespaces etc. Is that actually the correct
collection? Collection paths are typically expressed as
"/db/somecollection"
Also, what is your query and does it match your documents, perhaps you
could provide an example?
On 12 July 2010 23:59, Joe Wicentowski <jo...@gm...> wrote:
> Hi Patrick,
>
>> I created an lucene index on @value:
> ...
>> for $i in collection("tuscreen")//*[ft:query(@value, $query)]
>> return
>> $i
>>
>> Unfortunately it doesn't return a result?
>
> Strange. Just checking: did you reindex after adding the index definition?
>
> Cheers,
> Joe
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Sprint
> What will you do first with EVO, the first 4G phone?
> Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
> _______________________________________________
> Exist-development mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-development
>
--
Adam Retter
eXist Developer
{ United Kingdom }
ad...@ex...
irc://irc.freenode.net/existdb
|
|
From: Evgeny G. <gaz...@gm...> - 2010-07-14 17:08:17
|
2010/7/14 Dmitriy Shabanov <sha...@gm...>: > Hello, > > Can we move user/security related functions to separate module? > > -- +1 for separate "http://exist-db.org/xquery/security" module -- Evgeny |
|
From: Dmitriy S. <sha...@gm...> - 2010-07-14 15:30:21
|
Hello, Can we move user/security related functions to separate module? -- Cheers, Dmitriy Shabanov On Wed, 2010-07-14 at 00:31 +0000, ix...@us... wrote: > Revision: 11959 > http://exist.svn.sourceforge.net/exist/?rev=11959&view=rev > Author: ixitar > Date: 2010-07-14 00:31:14 +0000 (Wed, 14 Jul 2010) > > Log Message: > ----------- > [feature] Adding the function xmldb:get-current-user-attribute-names() > > Modified Paths: > -------------- > trunk/eXist/src/org/exist/xquery/functions/xmldb/XMLDBGetCurrentUserAttribute.java > trunk/eXist/src/org/exist/xquery/functions/xmldb/XMLDBModule.java > > Added Paths: > ----------- > trunk/eXist/src/org/exist/xquery/functions/xmldb/XMLDBGetCurrentUserAttributeNames.java > > Modified: trunk/eXist/src/org/exist/xquery/functions/xmldb/XMLDBGetCurrentUserAttribute.java > =================================================================== > --- trunk/eXist/src/org/exist/xquery/functions/xmldb/XMLDBGetCurrentUserAttribute.java 2010-07-13 12:34:12 UTC (rev 11958) > +++ trunk/eXist/src/org/exist/xquery/functions/xmldb/XMLDBGetCurrentUserAttribute.java 2010-07-14 00:31:14 UTC (rev 11959) > @@ -63,7 +63,7 @@ > public Sequence eval(Sequence args[], Sequence contextSequence) throws XPathException > { > if( args[0].isEmpty() ) { > - return BooleanValue.FALSE; > + return Sequence.EMPTY_SEQUENCE; > } > > String attributeName = args[0].getStringValue(); > > Added: trunk/eXist/src/org/exist/xquery/functions/xmldb/XMLDBGetCurrentUserAttributeNames.java > =================================================================== > --- trunk/eXist/src/org/exist/xquery/functions/xmldb/XMLDBGetCurrentUserAttributeNames.java (rev 0) > +++ trunk/eXist/src/org/exist/xquery/functions/xmldb/XMLDBGetCurrentUserAttributeNames.java 2010-07-14 00:31:14 UTC (rev 11959) > @@ -0,0 +1,64 @@ > +/* > + * eXist Open Source Native XML Database > + * Copyright (C) 2010 The eXist Project > + * http://exist-db.org > + * > + * This program is free software; you can redistribute it and/or > + * modify it under the terms of the GNU Lesser General Public License > + * as published by the Free Software Foundation; either version 2 > + * of the License, or (at your option) any later version. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU Lesser General Public License for more details. > + * > + * You should have received a copy of the GNU Lesser General Public License > + * along with this program; if not, write to the Free Software Foundation > + * Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. > + * > + * $Id $ > + */ > +package org.exist.xquery.functions.xmldb; > + > +import org.apache.log4j.Logger; > +import org.exist.dom.QName; > +import org.exist.xquery.*; > +import org.exist.xquery.value.*; > + > +import java.util.Set; > + > +/** > + * Created by IntelliJ IDEA. > + * User: lcahlander > + * Date: Jul 13, 2010 > + * Time: 1:51:11 PM > + * To change this template use File | Settings | File Templates. > + */ > +public class XMLDBGetCurrentUserAttributeNames extends BasicFunction { > + protected static final Logger logger = Logger.getLogger(XMLDBGetCurrentUserAttribute.class); > + > + public final static FunctionSignature signature = > + new FunctionSignature( > + new QName("get-current-user-attribute-names", XMLDBModule.NAMESPACE_URI, XMLDBModule.PREFIX), > + "Returns the names of the attributes of the current user from the xquery context.", > + null, > + new FunctionReturnSequenceType(Type.STRING, Cardinality.ZERO_OR_MORE, "the attribute names of the current user") > + ); > + > + public XMLDBGetCurrentUserAttributeNames(XQueryContext context, FunctionSignature signature) > + { > + super(context, signature); > + } > + > + public Sequence eval(Sequence args[], Sequence contextSequence) throws XPathException > + { > + Set<String> values = context.getUser().getAttributeNames(); > + Sequence retval = new ValueSequence(); > + for (String value : values) > + { > + retval.add(new StringValue(value)); > + } > + return retval; > + } > +} > > > Property changes on: trunk/eXist/src/org/exist/xquery/functions/xmldb/XMLDBGetCurrentUserAttributeNames.java > ___________________________________________________________________ > Added: svn:keywords > + Id > > Modified: trunk/eXist/src/org/exist/xquery/functions/xmldb/XMLDBModule.java > =================================================================== > --- trunk/eXist/src/org/exist/xquery/functions/xmldb/XMLDBModule.java 2010-07-13 12:34:12 UTC (rev 11958) > +++ trunk/eXist/src/org/exist/xquery/functions/xmldb/XMLDBModule.java 2010-07-14 00:31:14 UTC (rev 11959) > @@ -60,6 +60,7 @@ > new FunctionDef(XMLDBAuthenticate.loginSignatures[1], XMLDBAuthenticate.class), > new FunctionDef(XMLDBGetCurrentUser.signature, XMLDBGetCurrentUser.class), > new FunctionDef(XMLDBGetCurrentUserAttribute.signature, XMLDBGetCurrentUserAttribute.class), > + new FunctionDef(XMLDBGetCurrentUserAttributeNames.signature, XMLDBGetCurrentUserAttributeNames.class), > new FunctionDef(XMLDBXUpdate.signature, XMLDBXUpdate.class), > new FunctionDef(XMLDBCopy.signatures[0], XMLDBCopy.class), > new FunctionDef(XMLDBCopy.signatures[1], XMLDBCopy.class), |
|
From: Patrick S. <pat...@zi...> - 2010-07-13 08:14:33
|
Hi Joe,
>> for $i in collection("tuscreen")//*[ft:query(@value, $query)]
>> return
>> $i
>>
>> Unfortunately it doesn't return a result?
> Strange. Just checking: did you reindex after adding the index definition?
Yes, I did ;)
Patrick
|
|
From: Joe W. <jo...@gm...> - 2010-07-12 22:59:41
|
Hi Patrick,
> I created an lucene index on @value:
...
> for $i in collection("tuscreen")//*[ft:query(@value, $query)]
> return
> $i
>
> Unfortunately it doesn't return a result?
Strange. Just checking: did you reindex after adding the index definition?
Cheers,
Joe
|
|
From: Patrick S. <pat...@zi...> - 2010-07-12 09:24:11
|
Hi Wolfgang,
I tried the following query:
count(collection("tuscreen"))
and in fact this gives me an execution time of several seconds:
12 Jul 2010 11:16:30,390 [P1-9] DEBUG (XQuery.java [compile]:155) -
Query diagnostics:
count(collection("tuscreen"))
Query diagnostics:
collection("tuscreen")
12 Jul 2010 11:15:20,729 [P1-9] DEBUG (XQuery.java [compile]:161) -
Compilation took 1 ms
12 Jul 2010 11:15:22,818 [P1-9] DEBUG (XQuery.java [execute]:231) -
Execution took 2.089 ms
12 Jul 2010 11:15:30,800 [P1-9] DEBUG (LRDCache.java [cleanup]:153) -
totalReferences = 640001; maxReferences = 640000
12 Jul 2010 11:15:37,814 [P1-9] DEBUG (LRDCache.java [cleanup]:153) -
totalReferences = 640001; maxReferences = 640000
12 Jul 2010 11:15:44,559 [P1-9] DEBUG (LRDCache.java [cleanup]:153) -
totalReferences = 640001; maxReferences = 640000
12 Jul 2010 11:15:51,305 [P1-9] DEBUG (LRDCache.java [cleanup]:153) -
totalReferences = 640001; maxReferences = 640000
12 Jul 2010 11:15:51,828 [P1-9] DEBUG (HTTPUtils.java
[addLastModifiedHeader]:61) - mostRecentDocumentTime: 1278447949870
12 Jul 2010 11:15:51,992 [P1-9] DEBUG (HTTPUtils.java
[addLastModifiedHeader]:61) - mostRecentDocumentTime: 1278447949870
12 Jul 2010 11:15:51,992 [P1-9] INFO (RpcConnection.java [doQuery]:303)
- query took 31263ms.
and the second time it says:
12 Jul 2010 11:16:30,390 [P1-9] DEBUG (XQuery.java [compile]:161) -
Compilation took 1 ms
12 Jul 2010 11:16:33,475 [P1-9] DEBUG (XQuery.java [execute]:231) -
Execution took 3.085 ms
12 Jul 2010 11:16:33,475 [P1-9] DEBUG (HTTPUtils.java
[addLastModifiedHeader]:61) - mostRecentDocumentTime: 0
12 Jul 2010 11:16:33,475 [P1-9] DEBUG (HTTPUtils.java
[addLastModifiedHeader]:61) - mostRecentDocumentTime: 0
12 Jul 2010 11:16:33,476 [P1-9] INFO (RpcConnection.java [doQuery]:303)
- query took 3086ms.
12 Jul 2010 11:16:33,476 [P1-9] DEBUG (RpcConnection.java [queryP]:2434)
- found 1
I increased the conf.xml settings to:
<db-connection cacheSize="4000M" collectionCache="1000M" database="native"
files="webapp/WEB-INF/data" pageSize="4096" nodesBuffer="-1">
but this still results in several seconds:
12 Jul 2010 11:21:29,600 [P1-9] DEBUG (XQuery.java [execute]:231) -
Execution took 2.823 ms
12 Jul 2010 11:21:29,600 [P1-9] DEBUG (HTTPUtils.java
[addLastModifiedHeader]:61) - mostRecentDocumentTime: 0
12 Jul 2010 11:21:29,600 [P1-9] DEBUG (HTTPUtils.java
[addLastModifiedHeader]:61) - mostRecentDocumentTime: 0
12 Jul 2010 11:21:29,601 [P1-9] INFO (RpcConnection.java [doQuery]:303)
- query took 2824ms.
12 Jul 2010 11:21:29,601 [P1-9] DEBUG (RpcConnection.java [queryP]:2434)
- found 1
And memory usage is:
12 Jul 2010 11:22:23,519 [exist_QuartzScheduler_Worker-3] INFO
(NativeBroker.java [sync]:3191) - Memory: 1.595.904K total; 7.281.792K
max; 644.760K free
Patrick
>> 09 Jul 2010 15:11:55,784 [P1-9] DEBUG (XQuery.java [compile]:161) -
>> Compilation took 4 ms
>> 09 Jul 2010 15:11:58,119 [P1-9] TRACE (Optimize.java
>> [visitGeneralComparison]:219) - exist:optimize: found optimizable:
>> org.exist.xquery.GeneralComparison
> Most of the time seems to be spent after the query was compiled and
> before query execution starts. I would thus suspect that loading the
> metadata of your documents is the bottleneck, not the actual query.
> How long does it take to execute collection("tuscreen") on its own?
>
> The document loading should only have a negative effect the first time
> the query is executed. Subsequent queries will use the cached set.
>
> You may also check if increasing the collectionCache setting in
> conf.xml does help.
>
> Wolfgang
|
|
From: Patrick S. <pat...@zi...> - 2010-07-12 09:14:58
|
Hi Dmitriy, I get similar results for every repetition of the query: 12 Jul 2010 11:13:02,503 [P1-9] DEBUG (XQuery.java [execute]:231) - Execution took 3.780 ms 12 Jul 2010 11:13:02,504 [P1-9] DEBUG (HTTPUtils.java [addLastModifiedHeader]:61) - mostRecentDocumentTime: 0 12 Jul 2010 11:13:02,504 [P1-9] DEBUG (HTTPUtils.java [addLastModifiedHeader]:61) - mostRecentDocumentTime: 0 12 Jul 2010 11:13:02,504 [P1-9] INFO (RpcConnection.java [doQuery]:303) - query took 3781ms. 12 Jul 2010 11:13:02,504 [P1-9] DEBUG (RpcConnection.java [queryP]:2434) - found 1 12 Jul 2010 11:13:43,418 [P1-9] DEBUG (XQuery.java [execute]:231) - Execution took 3.779 ms 12 Jul 2010 11:13:43,418 [P1-9] DEBUG (HTTPUtils.java [addLastModifiedHeader]:61) - mostRecentDocumentTime: 0 12 Jul 2010 11:13:43,418 [P1-9] DEBUG (HTTPUtils.java [addLastModifiedHeader]:61) - mostRecentDocumentTime: 0 12 Jul 2010 11:13:43,418 [P1-9] INFO (RpcConnection.java [doQuery]:303) - query took 3779ms. 12 Jul 2010 11:13:59,979 [P1-9] DEBUG (XQuery.java [execute]:231) - Execution took 3.908 ms 12 Jul 2010 11:13:59,979 [P1-9] DEBUG (HTTPUtils.java [addLastModifiedHeader]:61) - mostRecentDocumentTime: 0 12 Jul 2010 11:13:59,980 [P1-9] DEBUG (HTTPUtils.java [addLastModifiedHeader]:61) - mostRecentDocumentTime: 0 12 Jul 2010 11:13:59,980 [P1-9] INFO (RpcConnection.java [doQuery]:303) - query took 3909ms. 12 Jul 2010 11:13:59,980 [P1-9] DEBUG (RpcConnection.java [queryP]:2434) - found 1 Patrick > On Fri, 2010-07-09 at 16:54 +0200, Patrick Schäfer wrote: >> 09 Jul 2010 16:52:50,788 [P1-9] TRACE (Optimize.java [eval]:160) - >> exist:optimize: inner expr took 91; found: 1 >> 09 Jul 2010 16:52:50,789 [P1-9] DEBUG (XQuery.java [execute]:231) - >> Execution took 3.710 ms > Is it first request, do you have same value for next one? > > As you can see the selection is quite fast. The problem is some where > else, IMHO. > |
|
From: Patrick S. <pat...@zi...> - 2010-07-12 09:08:07
|
Hi Joe,
I use the following settings:
conf.xml
<db-connection cacheSize="3000M" collectionCache="128M" database="native"
files="webapp/WEB-INF/data" pageSize="4096" nodesBuffer="-1">
eXist-settings.sh:
set_client_java_options() {
if [ -z "${CLIENT_JAVA_OPTIONS}" ]; then
CLIENT_JAVA_OPTIONS="-Xms128m -Xmx8000m -Dfile.encoding=UTF-8";
}
set_java_options() {
if [ -z "${JAVA_OPTIONS}" ]; then
JAVA_OPTIONS="-Xms128m -Xmx8000m -Dfile.encoding=UTF-8";
}
> This might take you on the wrong track, but you could try adding a
> Lucene index on @id / @value. Even though we usually think of using
> Lucene for fulltext search, it might be worth trying out the
> whitespace analyzer -- see http://exist-db.org/lucene.html#N102F8.
> Given all of the weird characters in your @value example, you should
> probably construct your query as XML rather than using the lucene
> search query parser, which could get thrown off -- see
> http://exist-db.org/lucene.html#N10352. It'd be interesting to know
> how this works compared to the range indexes.
I created an lucene index on @value:
<lucene>
<analyzer class="org.apache.lucene.analysis.standard.StandardAnalyzer"/>
<analyzer id="ws" class="org.apache.lucene.analysis.WhitespaceAnalyzer"/>
<text qname="@value" analyzer="ws"/>
</lucene>
and used the query:
let $query :=
<query>
<term>Cc1c2c(ncn(c2=O)C)sc1C(=O)NCc1ccc(cc1)CN1CCOCC1</term>
</query>
for $i in collection("tuscreen")//*[ft:query(@value, $query)]
return
$i
Unfortunately it doesn't return a result?
But the query still takes:
12 Jul 2010 11:05:44,920 [P1-9] DEBUG (XQuery.java [execute]:231) -
Execution took 5.456 ms
Patrick
|
|
From: Michael W. <wes...@ja...> - 2010-07-12 05:22:01
|
2010/7/8 Dmitriy Shabanov wrote:
It can be fixed if add (bold) to UntypedValueCheck
>
> private Item convert(Item item) throws XPathException {
> if (atomize || item.getType() == Type.UNTYPED_ATOMIC ||
> Type.subTypeOf(requiredType, Type.NUMBER) && Type.subTypeOf(item.getType(),
> Type.NUMBER)) {
> try {
> *if (Type.subTypeOf(item.getType(), requiredType))*
> * return item;*
>
> item = item.convertTo(requiredType);
> } catch (XPathException e) {
> error.addArgs(ExpressionDumper.dump(expression),
> Type.getTypeName(requiredType),
> Type.getTypeName(item.getType()));
> throw new XPathException(expression, error.toString());
> }
> }
> return item;
> }
>
This looks like a much better/simpler fix than mine. Can you submit a
potential patch for this so that I can confirm it?
Thank you.
--
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/
|
|
From: James F. <jam...@ex...> - 2010-07-11 11:45:17
|
Adam, can you re-post this question on the open list ... the eXist development list is for development issues/tasks, etc. can I ask also that you include a representative xpath expression and xml (if the xml u provide is not representative) and I will try to give you some help. James Fuller On 11 July 2010 13:39, Adam Hepner <Ada...@be...> wrote: > Hi > I have a (rather big - several hundred MB) XML of the following > structure stored in the db: > <krm:catalogue> > <krm:j id="1637192" pID="1637191" teryt="3716704" powiat="2183789" > wojewodztwo="2183783" > kategoria="1"> > <krm:k id="1637194"> > <krm:spec id="2205172"/> > <krm:spec id="2205241"/> > <krm:rk id="2205699"/> > <krm:rk id="2205588"/> > <krm:rk id="2205589"/> > <krm:rk id="2205358"/> > <krm:dsc id="2205171"/> > <krm:dsc id="2205239"/> > </krm:k> > <krm:k id="1637193"> > <krm:spec id="2205213"/> > <krm:rc id="2206812"/> > <krm:rk id="2205564"/> > <krm:dsc id="2205211"/> > <krm:sc id="2206811"/> > <krm:sc id="2206808"/> > <krm:sc id="2206804"/> > <krm:sc id="2206807"/> > <krm:sc id="2206806"/> > <krm:sc id="2206805"/> > <krm:sc id="2206810"/> > <krm:sc id="2206795"/> > <krm:sc id="2206798"/> > <krm:sc id="2206794"/> > <krm:sc id="2206791"/> > <krm:sc id="2206793"/> > <krm:sc id="2206799"/> > <krm:sc id="2206797"/> > <krm:sc id="2206802"/> > <krm:sc id="2206801"/> > <krm:sc id="2206800"/> > <krm:sc id="2206792"/> > <krm:sc id="2206796"/> > <krm:sc id="2206809"/> > </krm:k> > </krm:j> > </krm:catalogue> > > where there are many nodes of type krm:j, which have many nodes of > type krm:k. In a rather elaborate process, I filter those nodes, and > receive a certain subtree - which, therefore, still has the same > organization. And now I need to count occurrances of every @id in > every given node type (ie. I need to retrieve information that node > krm:sc with id 2206800 occurs 5 times, and krm:sc with id 2206801 > occurs 20 times). I found out that just performing a count() on a > nodeset i(XPath expression result) s surprisingly slow, and can take > my xquery from below 30 seconds to well over 10 minutes. I have > indices set up, which sped things up a lot in other contexts, but > still - counting occurences is a big pain in the lower back. I wonder > - is there any clever exist-only function which could help mi retrieve > the counts? > > -- > Best regards > Adam Hepner > > tel. 509 093 095 > e-mail: ada...@be... > http://AdamHepner.pl > > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by Sprint > What will you do first with EVO, the first 4G phone? > Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first > _______________________________________________ > Exist-development mailing list > Exi...@li... > https://lists.sourceforge.net/lists/listinfo/exist-development > |
|
From: Adam H. <Ada...@be...> - 2010-07-11 11:39:58
|
Hi
I have a (rather big - several hundred MB) XML of the following
structure stored in the db:
<krm:catalogue>
<krm:j id="1637192" pID="1637191" teryt="3716704" powiat="2183789"
wojewodztwo="2183783"
kategoria="1">
<krm:k id="1637194">
<krm:spec id="2205172"/>
<krm:spec id="2205241"/>
<krm:rk id="2205699"/>
<krm:rk id="2205588"/>
<krm:rk id="2205589"/>
<krm:rk id="2205358"/>
<krm:dsc id="2205171"/>
<krm:dsc id="2205239"/>
</krm:k>
<krm:k id="1637193">
<krm:spec id="2205213"/>
<krm:rc id="2206812"/>
<krm:rk id="2205564"/>
<krm:dsc id="2205211"/>
<krm:sc id="2206811"/>
<krm:sc id="2206808"/>
<krm:sc id="2206804"/>
<krm:sc id="2206807"/>
<krm:sc id="2206806"/>
<krm:sc id="2206805"/>
<krm:sc id="2206810"/>
<krm:sc id="2206795"/>
<krm:sc id="2206798"/>
<krm:sc id="2206794"/>
<krm:sc id="2206791"/>
<krm:sc id="2206793"/>
<krm:sc id="2206799"/>
<krm:sc id="2206797"/>
<krm:sc id="2206802"/>
<krm:sc id="2206801"/>
<krm:sc id="2206800"/>
<krm:sc id="2206792"/>
<krm:sc id="2206796"/>
<krm:sc id="2206809"/>
</krm:k>
</krm:j>
</krm:catalogue>
where there are many nodes of type krm:j, which have many nodes of
type krm:k. In a rather elaborate process, I filter those nodes, and
receive a certain subtree - which, therefore, still has the same
organization. And now I need to count occurrances of every @id in
every given node type (ie. I need to retrieve information that node
krm:sc with id 2206800 occurs 5 times, and krm:sc with id 2206801
occurs 20 times). I found out that just performing a count() on a
nodeset i(XPath expression result) s surprisingly slow, and can take
my xquery from below 30 seconds to well over 10 minutes. I have
indices set up, which sped things up a lot in other contexts, but
still - counting occurences is a big pain in the lower back. I wonder
- is there any clever exist-only function which could help mi retrieve
the counts?
--
Best regards
Adam Hepner
tel. 509 093 095
e-mail: ada...@be...
http://AdamHepner.pl
|
|
From: James F. <jam...@ex...> - 2010-07-10 14:54:08
|
FYI: I just committed some changes and looks like the build is breaking because I committed my build.properties, reverting *[16:42:10]:* *[prepare-svn] **echo*** *[16:42:10]:* *[echo] *Downloading libraries required by the subversion module *[16:42:10]:* *[prepare-svn] **echo*** *[16:42:10]:* *[echo] * ------------------------------------------------------- *[16:42:10]:* *[prepare-svn] **fetch*** *[16:42:10]:* *[fetch] *java.lang.NoSuchMethodError: org.apache.tools.ant.taskdefs.Get.setRetries(I)V *[16:42:10]:* *[ant] *The following error occurred while executing this line: /home/ljo/filer/buildAgent-1/work/8f01b9a44b69910a/extensions/modules/build.xml:218: java.lang.NoSuchMethodError: org.apache.tools.ant.taskdefs.Get.setRetries(I)V yes I owe a beer James Fuller |
|
From: James F. <jam...@ex...> - 2010-07-10 14:12:33
|
FYI: I commited a bunch of xquery level unit tests under test/src/xquery/util/util.xml for the util module which further demonstrates how developers should test their xquery functions using the xquery test library which for those who do not know, is now enabled by default in conf.xml (in trunk version) ala <module src="resource:org/exist/xquery/lib/test.xq" uri=" http://exist-db.org/xquery/testing"/> Its simple to use and provides us with an ability to test xquery consistently across core and extensions ... I think we probably need to have a discussion at some point about rationalizing the test directory (for example where should unit tests go for xquery extension modules). I know that this may seem a little strange (if not slightly masochistic) to add unit tests for existing functionality but I see this kind of activity as important to imparting robustness to eXist especially as we are constantly refactoring and adding/changing things. I would ask that existing extension developers consider adding xquery level unit tests to their work. Another consideration is if we are to run xquery unit tests as part of default core tests. Perhaps we can make a mention of using the xquery test library in the various developer / manifesto docs. James Fuller |
|
From: Wolfgang M. <wol...@ex...> - 2010-07-10 08:19:31
|
> 09 Jul 2010 15:11:55,784 [P1-9] DEBUG (XQuery.java [compile]:161) -
> Compilation took 4 ms
> 09 Jul 2010 15:11:58,119 [P1-9] TRACE (Optimize.java
> [visitGeneralComparison]:219) - exist:optimize: found optimizable:
> org.exist.xquery.GeneralComparison
Most of the time seems to be spent after the query was compiled and
before query execution starts. I would thus suspect that loading the
metadata of your documents is the bottleneck, not the actual query.
How long does it take to execute collection("tuscreen") on its own?
The document loading should only have a negative effect the first time
the query is executed. Subsequent queries will use the cached set.
You may also check if increasing the collectionCache setting in
conf.xml does help.
Wolfgang
|
|
From: Dmitriy S. <sha...@gm...> - 2010-07-10 03:53:11
|
On Fri, 2010-07-09 at 16:54 +0200, Patrick Schäfer wrote: > 09 Jul 2010 16:52:50,788 [P1-9] TRACE (Optimize.java [eval]:160) - > exist:optimize: inner expr took 91; found: 1 > 09 Jul 2010 16:52:50,789 [P1-9] DEBUG (XQuery.java [execute]:231) - > Execution took 3.710 ms Is it first request, do you have same value for next one? As you can see the selection is quite fast. The problem is some where else, IMHO. -- Cheers, Dmitriy Shabanov |
|
From: Joe W. <jo...@gm...> - 2010-07-09 21:52:41
|
Hi Patrick, What are your memory settings? Seeing this sometimes helps folks on the list troubleshoot. -Xmx, -Xms, and the "cacheSize" and "collectionCache" and other attributes of the <db-connection> element in your conf.xml file. This might take you on the wrong track, but you could try adding a Lucene index on @id / @value. Even though we usually think of using Lucene for fulltext search, it might be worth trying out the whitespace analyzer -- see http://exist-db.org/lucene.html#N102F8. Given all of the weird characters in your @value example, you should probably construct your query as XML rather than using the lucene search query parser, which could get thrown off -- see http://exist-db.org/lucene.html#N10352. It'd be interesting to know how this works compared to the range indexes. Joe |
|
From: James F. <jam...@ex...> - 2010-07-09 18:20:46
|
can anyone confirm that enabling exist:id, via conf.xml, in trunk works or doesn't work these days ? tia, J |
|
From: Patrick S. <pat...@zi...> - 2010-07-09 14:54:34
|
Hi Thomas, > What type index did you defined? my index definition is: <collection xmlns="http://exist-db.org/collection-config/1.0"> <index xmlns:atom="http://www.xml-cml.org/schema"> <fulltext default="none" attributes="no"/> ... <create qname="@id" type="xs:string"/> </index> </collection> > It may help to define a numeric type of range index and cast the value > to a number when you compare it. I changed the index to integer: <create qname="@id" type="xs:integer"/> but there is still no improvement 09 Jul 2010 16:52:47,078 [P1-9] DEBUG (XQuery.java [compile]:161) - Compilation took 5 ms 09 Jul 2010 16:52:49,177 [P1-9] TRACE (Optimize.java [visitGeneralComparison]:219) - exist:optimize: found optimizable: org.exist.xquery.GeneralComparison 09 Jul 2010 16:52:49,178 [P1-9] TRACE (Optimize.java [before]:240) - exist:optimize: context step: descendant-or-self::xq:molecule[attribute::id = 298038] 09 Jul 2010 16:52:49,178 [P1-9] TRACE (Optimize.java [before]:241) - exist:optimize: context var: null 09 Jul 2010 16:52:50,465 [P1-9] TRACE (GeneralComparison.java [preSelect]:266) - Using QName index on type xs:integer 09 Jul 2010 16:52:50,465 [P1-9] TRACE (GeneralComparison.java [preSelect]:298) - Using QName range index for key: 298038 09 Jul 2010 16:52:50,625 [P1-9] TRACE (Optimize.java [eval]:120) - exist:optimize: pre-selection: 1 09 Jul 2010 16:52:50,696 [P1-9] TRACE (Optimize.java [eval]:140) - Ancestor selection took 71 09 Jul 2010 16:52:50,697 [P1-9] TRACE (Optimize.java [eval]:141) - Found: 1 09 Jul 2010 16:52:50,697 [P1-9] TRACE (Optimize.java [eval]:152) - exist:optimize: context after optimize: 1 09 Jul 2010 16:52:50,788 [P1-9] TRACE (Optimize.java [eval]:160) - exist:optimize: inner expr took 91; found: 1 09 Jul 2010 16:52:50,789 [P1-9] DEBUG (XQuery.java [execute]:231) - Execution took 3.710 ms 09 Jul 2010 16:52:50,789 [P1-9] DEBUG (HTTPUtils.java [addLastModifiedHeader]:61) - mostRecentDocumentTime: 0 09 Jul 2010 16:52:50,789 [P1-9] DEBUG (HTTPUtils.java [addLastModifiedHeader]:61) - mostRecentDocumentTime: 0 09 Jul 2010 16:52:50,790 [P1-9] INFO (RpcConnection.java [doQuery]:303) - query took 3711ms. 09 Jul 2010 16:52:50,790 [P1-9] DEBUG (RpcConnection.java [queryP]:2434) - found 1 09 Jul 2010 16:52:50,814 [P1-9] DEBUG (RpcConnection.java [retrieveFirstChunk]:2991) - Writing to temporary file: eXistRPCC4083305984497633446.xml 09 Jul 2010 16:53:08,575 [exist_QuartzScheduler_Worker-2] DEBUG (NGramIndex.java [sync]:86) - SYNC NGRAM 09 Jul 2010 16:53:08,575 [exist_QuartzScheduler_Worker-2] INFO (NativeBroker.java [sync]:3191) - Memory: 3.448.768K total; 7.281.792K max; 2.636.440K free Thank you Patrick |
|
From: Patrick S. <pat...@zi...> - 2010-07-09 14:48:51
|
Hi Joe, thank you for the response. > Sounds like there are some theories about getting to the bottom of the > slowness, but another tack might be to adopt this approach from the > Performance Tuning article, "4.9. Use fn:id to lookup xml:id > attributes" - see http://exist-db.org/tuning.html#N10382. This would > require either using @xml:id instead of @id, or declaring @id as type > ID in a DTD - maybe more trouble than it's worth. But I have a hunch > that the id() function is faster than a range index; unfortunately, I > don't actually see that spelled out anywhere. That might solve this problem. Unfortunately the given query was just an example. The following query gives me similar execution times though I defined an index on "@value": declare namespace xq="http://www.xml-cml.org/schema"; for $i in collection("tuscreen")//*[@value="Cc1c2c(ncn(c2=O)C)sc1C(=O)NCc1ccc(cc1)CN1CCOCC1"] return $i 09 Jul 2010 16:47:17,746 [P1-9] DEBUG (XQuery.java [compile]:161) - Compilation took 3 ms 09 Jul 2010 16:47:19,971 [P1-9] TRACE (Optimize.java [visitGeneralComparison]:219) - exist:optimize: found optimizable: org.exist.xquery.GeneralComparison 09 Jul 2010 16:47:19,972 [P1-9] TRACE (Optimize.java [before]:240) - exist:optimize: context step: null 09 Jul 2010 16:47:19,972 [P1-9] TRACE (Optimize.java [before]:241) - exist:optimize: context var: null 09 Jul 2010 16:47:21,267 [P1-9] TRACE (GeneralComparison.java [preSelect]:266) - Using QName index on type xs:string 09 Jul 2010 16:47:21,267 [P1-9] TRACE (GeneralComparison.java [preSelect]:298) - Using QName range index for key: Cc1c2c(ncn(c2=O)C)sc1C(=O)NCc1ccc(cc1)CN1CCOCC1 09 Jul 2010 16:47:21,428 [P1-9] TRACE (Optimize.java [eval]:120) - exist:optimize: pre-selection: 1 09 Jul 2010 16:47:21,530 [P1-9] DEBUG (XQuery.java [execute]:231) - Execution took 3.784 ms 09 Jul 2010 16:47:21,530 [P1-9] DEBUG (HTTPUtils.java [addLastModifiedHeader]:61) - mostRecentDocumentTime: 1278439917815 09 Jul 2010 16:47:21,531 [P1-9] DEBUG (HTTPUtils.java [addLastModifiedHeader]:61) - mostRecentDocumentTime: 1278439917815 09 Jul 2010 16:47:21,531 [P1-9] INFO (RpcConnection.java [doQuery]:303) - query took 3785ms. 09 Jul 2010 16:47:21,531 [P1-9] DEBUG (RpcConnection.java [queryP]:2434) - found 1 Index definition is: <collection xmlns="http://exist-db.org/collection-config/1.0"> <index xmlns:atom="http://www.xml-cml.org/schema"> <fulltext default="none" attributes="no"/> <!-- Range indexes by qname --> <create qname="@id" type="xs:string"/> <create qname="@value" type="xs:string"/> </index> </collection> Patrick |
|
From: Joe W. <jo...@gm...> - 2010-07-09 14:42:53
|
p.s. I think this question/discussion actually belongs in exist-open, not exist-development. Joe |
|
From: Joe W. <jo...@gm...> - 2010-07-09 14:41:44
|
Hi Patrick,
> for $i in collection("tuscreen")//xq:molecule[@id="298038"]
...
> Is there a way to speed up queries?
Sounds like there are some theories about getting to the bottom of the
slowness, but another tack might be to adopt this approach from the
Performance Tuning article, "4.9. Use fn:id to lookup xml:id
attributes" - see http://exist-db.org/tuning.html#N10382. This would
require either using @xml:id instead of @id, or declaring @id as type
ID in a DTD - maybe more trouble than it's worth. But I have a hunch
that the id() function is faster than a range index; unfortunately, I
don't actually see that spelled out anywhere.
Cheers,
Joe
|
|
From: Patrick S. <pat...@zi...> - 2010-07-09 14:33:47
|
Hi,
thx for the response.
> If calculate execution time throw log, get 58,119 - 58,119 = 1.547 ms
> for collection("tuscreen")//xq:molecule[@id="298038"] Yes, it's slow,
> but the rest should be much faster ... strange ...
>
> What version do you use?
I am using the jar file eXist-setup-1.4.0-rev10440.jar ...
09 Jul 2010 15:22:53,534 [main] INFO (JettyStart.java [run]:101) -
[eXist Version : 1.4.0]
09 Jul 2010 15:22:53,534 [main] INFO (JettyStart.java [run]:103) -
[eXist Build : 20091111]
09 Jul 2010 15:22:53,535 [main] INFO (JettyStart.java [run]:105) -
[eXist Home : /usr/local/exist]
09 Jul 2010 15:22:53,535 [main] INFO (JettyStart.java [run]:107) - [SVN
Revision : 10440]
09 Jul 2010 15:22:53,535 [main] INFO (JettyStart.java [run]:115) -
[Operating System : Linux 2.6.31.12-0.2-default amd64]
Patrick
|
64 messages has been excluded from this view by a project administrator.
