exist-open — Open discussion

Re: [Exist-open] transform: Resource ... is a collection

[Joe W. <jo...@gm...>]

Hi Eduard,

When you say the XSLT "works via the command line", which command is it
that you use?

For the XQuery call to transform:transform, which version of eXist are you
using? And I assume you're using the stock version of Saxon included with
eXist and haven't replaced it with your own - but if I'm wrong, please
specify.

Joe

On Mon, Feb 1, 2021 at 5:38 AM Eduard Drenth <ed...@fr...>
wrote:

> Dear all,
>
> I have I xslt that works fine from the command line, it processes files in
> a directory:
>
> <xsl:for-each select="collection($dir)/*">
>
>
> Now I want to call it from xquery like this:
>
>
> transform:stream-transform((),$teidictjson:parallelxslt,
>     <parameters>
>         <param name="dir" value="xmldb:exist://{$config:data-root}"/>
>
>     </parameters>)
>
>
> This however produces an error:
>
>
> Exception while transforming node: I/O error reported by XML parser processing null: Resource /db/apps/teidictjson/data is a collection
>
>
> Tried a lot of things, no luck yet.
>
>
> Any idea's?
>
>
> Greetings, Eduard
>
>
> --
>
> Eduard Drenth, Software Architekt
>
> ed...@fr...
>
> Doelestrjitte 8
> 8911 DX  Ljouwert
> +31 58 234 30 47
> +31 62 094 34 28 (privé)
>
> skype: eduarddrenth
> https://github.com/eduarddrenth
> frisian.eu
> gpg: https://pgp.surfnet.nl/pks/lookup?search=eduarddrenth
>
>
> Op freed bin ik thús/wurkje ik minder
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

Re: [Exist-open] handling same path but multiple namespaces depending on data provider

[Joe W. <jo...@gm...>]

Hi Guillaume,

Short of normalizing the data, namespace wildcards are the most general
purpose solution for path expressions.

You can also use the local-name() function, e.g.,

  /*[local-name() eq "table"]/*[local-name() eq "tr"]/*[local-name() eq
"td"]

To really be sure you're not inadvertently including other namespaces, you
could declare all of the namespaces and explicitly select these in your
node tests with the union operator:


/(v12:table|v13:table|v14:table)/(v12:tr|v13:tr|v14:tr)/(v12:td/v13:td/v14:td)

It's hard to comment on performance concerns without knowing what kinds of
queries you need to run, but you might consider eXist 5's fields and facets
as part of your arsenal: you could construct a field with an expression
that selects the content you need, and then you could query the field
without worrying about which namespace the source came from. (I think.)

Joe

On Thu, Feb 4, 2021 at 11:26 AM Guillaume Mella <
gui...@un...> wrote:

> Hello,
>
>   I have a module which consumes VOTable (table/tr/td with meta data for
> Astronomy) using a standardised TableAccesProtocol. But depending on the
> data provider the returned document may get multiple namespaces depending
> on the version they implement, eg.:
>
> declare namespace votable="http://www.ivoa.net/xml/VOTable/v1.2"
> <http://www.ivoa.net/xml/VOTable/v1.2>; (: could be v1.3 v1.4 :)
>
> Most data part are the same in the various versions and I would like to
> support it dynamically without branching/duplication/copy.
>
> I tried with the wildcard namespace using *:tag but didn't investigate to
> know more about performance issues.
>
> Is there any best practices, hints or documentation pointers for such use
> case ?
>
>
> Cheers,
>
> --
> Guillaume Mella - Ingénieur CNRS
> OSUG UMS832 :  http://www.osug.fr  -  JMMC : http://www.jmmc.fr/~mellag
> OSUG-D, 122 rue de la piscine, 38 400 Saint Martin d’Hères, France
> ☎ +33 (0)4 76 63 58 22
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

[Exist-open] handling same path but multiple namespaces depending on data provider

[Guillaume M. <gui...@un...>]

Hello,

   I have a module which consumes VOTable (table/tr/td with meta data 
for Astronomy) using a standardised TableAccesProtocol. But depending on 
the data provider the returned document may get multiple namespaces 
depending on the version they implement, eg.:

declare namespace votable="http://www.ivoa.net/xml/VOTable/v1.2"; (: 
could be v1.3 v1.4 :)

Most data part are the same in the various versions and I would like to 
support it dynamically without branching/duplication/copy.

I tried with the wildcard namespace using *:tag but didn't investigate 
to know more about performance issues.

Is there any best practices, hints or documentation pointers for such 
use case ?


Cheers,

-- 
Guillaume Mella - Ingénieur CNRS
OSUG UMS832 :  http://www.osug.fr  -  JMMC : http://www.jmmc.fr/~mellag
OSUG-D, 122 rue de la piscine, 38 400 Saint Martin d’Hères, France
☎ +33 (0)4 76 63 58 22

Re: [Exist-open] can't get to eXide

[Ihe O. <ihe...@gm...>]

I had the database restarted and I can see eXide now. Thanks.

On Tue, Feb 2, 2021 at 2:23 PM Christian Achter <
chr...@ge...> wrote:

> How about you start it from this "Launcher" menu:
>
> http://yourdomain:8080/exist/apps/dashboard/admin
>
> Is there an icon for eXide?
>
>
> Am 01.02.21 um 13:10 schrieb Ihe Onwuka:
>
> No difference. :(
>
> On Mon, Feb 1, 2021 at 6:33 AM Michael Westbay <
> wes...@ja...> wrote:
>
>> Hi Ihe,
>>
>> Try:
>>
>> http://localhost:8080/exist/apps/eXide/index.html#
>>
>>
>> I haven't looked into it, but have found that the # at the end is needed
>> to get the web application started.
>>
>> Hope this helps.
>>
>> Take care.
>>
>>
>> 2021年2月1日(月) 18:24 Ihe Onwuka <ihe...@gm...>:
>>
>>> I had to unexpectedly switch laptops last week and have lost my exide
>>> "bookmark".
>>>
>>> I am running exist 5.2 on Amazon EC2.  Now
>>>
>>> ...:8080/exist/apps/dashboard/index.html gives me a blank Launcher
>>> screen.
>>>
>>> and :8080/exist/apps/eXide/index.html gives me
>>> <exception>
>>> <path>/db/apps/eXide/modules/view.xql</path>
>>> <message>exerr:ERROR An IO exception occurred: Could not load cache for
>>> class: org.exist.util.io.FileFilterInputStreamCache [at line 44, column 14]
>>> </message>
>>> </exception>
>>>
>>> Can anybody help.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Exist-open mailing list
>>> Exi...@li...
>>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>>
>>
>>
>> --
>> Michael Westbay
>> Writer/System Administrator
>> http://www.japanesebaseball.com/
>>
>
>
> _______________________________________________
> Exist-open mailing lis...@li...https://lists.sourceforge.net/lists/listinfo/exist-open
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

Re: [Exist-open] can't get to eXide

[Christian A. <chr...@ge...>]

How about you start it from this "Launcher" menu:

http://yourdomain:8080/exist/apps/dashboard/admin

Is there an icon for eXide?


Am 01.02.21 um 13:10 schrieb Ihe Onwuka:
> No difference. :(
>
> On Mon, Feb 1, 2021 at 6:33 AM Michael Westbay 
> <wes...@ja... 
> <mailto:wes...@ja...>> wrote:
>
>     Hi Ihe,
>
>     Try:
>
>         http://localhost:8080/exist/apps/eXide/index.html#
>         <http://localhost:8080/exist/apps/eXide/index.html#>
>
>
>     I haven't looked into it, but have found that the # at the end is
>     needed to get the web application started.
>
>     Hope this helps.
>
>     Take care.
>
>
>     2021年2月1日(月) 18:24 Ihe Onwuka <ihe...@gm...
>     <mailto:ihe...@gm...>>:
>
>         I had to unexpectedly switch laptops last week and have lost
>         my exide "bookmark".
>
>         I am running exist 5.2 on Amazon EC2.  Now
>
>         ...:8080/exist/apps/dashboard/index.html gives me a blank
>         Launcher screen.
>
>         and :8080/exist/apps/eXide/index.html gives me
>         <exception>
>         <path>/db/apps/eXide/modules/view.xql</path>
>         <message>exerr:ERROR An IO exception occurred: Could not load
>         cache for class: org.exist.util.io.FileFilterInputStreamCache
>         [at line 44, column 14]</message>
>         </exception>
>
>         Can anybody help.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>         _______________________________________________
>         Exist-open mailing list
>         Exi...@li...
>         <mailto:Exi...@li...>
>         https://lists.sourceforge.net/lists/listinfo/exist-open
>         <https://lists.sourceforge.net/lists/listinfo/exist-open>
>
>
>
>     -- 
>     Michael Westbay
>     Writer/System Administrator
>     http://www.japanesebaseball.com/ <http://www.japanesebaseball.com/>
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

[Exist-open] Partial backup

[Roy W. <gar...@ya...>]

Hi,
Is there a way to backup only .xql files from a db? 
Thanks in advance,Roy

Re: [Exist-open] can't get to eXide

[Ihe O. <ihe...@gm...>]

No difference. :(

On Mon, Feb 1, 2021 at 6:33 AM Michael Westbay <
wes...@ja...> wrote:

> Hi Ihe,
>
> Try:
>
> http://localhost:8080/exist/apps/eXide/index.html#
>
>
> I haven't looked into it, but have found that the # at the end is needed
> to get the web application started.
>
> Hope this helps.
>
> Take care.
>
>
> 2021年2月1日(月) 18:24 Ihe Onwuka <ihe...@gm...>:
>
>> I had to unexpectedly switch laptops last week and have lost my exide
>> "bookmark".
>>
>> I am running exist 5.2 on Amazon EC2.  Now
>>
>> ...:8080/exist/apps/dashboard/index.html gives me a blank Launcher screen.
>>
>> and :8080/exist/apps/eXide/index.html gives me
>> <exception>
>> <path>/db/apps/eXide/modules/view.xql</path>
>> <message>exerr:ERROR An IO exception occurred: Could not load cache for
>> class: org.exist.util.io.FileFilterInputStreamCache [at line 44, column 14]
>> </message>
>> </exception>
>>
>> Can anybody help.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>
>
>
> --
> Michael Westbay
> Writer/System Administrator
> http://www.japanesebaseball.com/
>

Re: [Exist-open] can't get to eXide

[Michael W. <wes...@ja...>]

Hi Ihe,

Try:

http://localhost:8080/exist/apps/eXide/index.html#


I haven't looked into it, but have found that the # at the end is needed to
get the web application started.

Hope this helps.

Take care.


2021年2月1日(月) 18:24 Ihe Onwuka <ihe...@gm...>:

> I had to unexpectedly switch laptops last week and have lost my exide
> "bookmark".
>
> I am running exist 5.2 on Amazon EC2.  Now
>
> ...:8080/exist/apps/dashboard/index.html gives me a blank Launcher screen.
>
> and :8080/exist/apps/eXide/index.html gives me
> <exception>
> <path>/db/apps/eXide/modules/view.xql</path>
> <message>exerr:ERROR An IO exception occurred: Could not load cache for
> class: org.exist.util.io.FileFilterInputStreamCache [at line 44, column 14]
> </message>
> </exception>
>
> Can anybody help.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>


-- 
Michael Westbay
Writer/System Administrator
http://www.japanesebaseball.com/

[Exist-open] transform: Resource ... is a collection

[Eduard D. <ed...@fr...>]

Dear all,

I have I xslt that works fine from the command line, it processes files in a directory:


<xsl:for-each select="collection($dir)/*">


Now I want to call it from xquery like this:


transform:stream-transform((),$teidictjson:parallelxslt,
    <parameters>
        <param name="dir" value="xmldb:exist://{$config:data-root}"/>

    </parameters>)


This however produces an error:


Exception while transforming node: I/O error reported by XML parser processing null: Resource /db/apps/teidictjson/data is a collection


Tried a lot of things, no luck yet.


Any idea's?


Greetings, Eduard


--

Eduard Drenth, Software Architekt

ed...@fr...<mailto:ed...@fr...>

Doelestrjitte 8
8911 DX  Ljouwert
+31 58 234 30 47
+31 62 094 34 28 (privé)

skype: eduarddrenth
https://github.com/eduarddrenth
frisian.eu
gpg: https://pgp.surfnet.nl/pks/lookup?search=eduarddrenth


Op freed bin ik thús/wurkje ik minder

[Exist-open] can't get to eXide

[Ihe O. <ihe...@gm...>]

I had to unexpectedly switch laptops last week and have lost my exide
"bookmark".

I am running exist 5.2 on Amazon EC2.  Now

...:8080/exist/apps/dashboard/index.html gives me a blank Launcher screen.

and :8080/exist/apps/eXide/index.html gives me
<exception>
<path>/db/apps/eXide/modules/view.xql</path>
<message>exerr:ERROR An IO exception occurred: Could not load cache for
class: org.exist.util.io.FileFilterInputStreamCache [at line 44, column 14]
</message>
</exception>

Can anybody help.

Re: [Exist-open] scientific notation

[Nick S. <nsi...@nu...>]

Nevermind. User error.

On 1/25/21 8:39 PM, Nick Sincaglia wrote:
> I am calling a web service which returns a number in scientific 
> notation. I need to convert it to an integer. This should be simple 
> but I am not able to find a solution.
>
> When I run the below code in eXide, I get the correct answer of 46622388.
> xquery version "3.1";
>
> let $x := <value>4.6622388E7</value>
> let $y := $x/text()
> return
>     xs:long(number($y))
>
> However, when I use the same conversion in my code, I get an error 
> complaining about xs:double('NaN') to xs:long
>
> Why is number($y) returning 'NaN'. What do I need to do to get convert 
> scientific notation to an integer?
>
> Nick
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

-- 
Nick Sincaglia
President/Founder
NueMeta, LLC
Digital Media & Technology
Phone: +1-630-303-7035
nsi...@nu...
http://www.nuemeta.com
Skype: nsincaglia

[Exist-open] scientific notation

[Nick S. <nsi...@nu...>]

I am calling a web service which returns a number in scientific 
notation. I need to convert it to an integer. This should be simple but 
I am not able to find a solution.

When I run the below code in eXide, I get the correct answer of 46622388.
xquery version "3.1";

let $x := <value>4.6622388E7</value>
let $y := $x/text()
return
     xs:long(number($y))

However, when I use the same conversion in my code, I get an error 
complaining about xs:double('NaN') to xs:long

Why is number($y) returning 'NaN'. What do I need to do to get convert 
scientific notation to an integer?

Nick

Re: [Exist-open] diacritic switch (Eduard Drenth)

[Eduard D. <ed...@fr...>]

ok, you sketch a more complex situation.

I would like to distinguish phases:

Phase 1) In this phase you can connect one or more (fields) terms to
original text. You are free to add to the index what you need, if
needed using custom analyzers/tokenizers

Phase 2) In this phase you can preprocess search terms and select (via
fields) an index to query


Phase 3) In this phase you (always I think) output original text.

With these possibilities you can achieve what you need I think? Or am I
missing something?

Regards, Eduard

-----Original Message-----
From: PIERRICK BRIHAYE <Pie...@br...h>
To: exi...@li... <exi...@li...>
Subject: Re: [Exist-open] diacritic switch (Eduard Drenth)
Date: Fri, 15 Jan 2021 15:34:13 +0000

Hello,

> Perhaps, but the fields option enables you to define multiple indexes
> on one qname already, see Joe's answer:
> collection.xconf:
> 
> <analyzer id="nostops"
> class="org.fryske_akademy.exist.lucene.InsensitiveWhitespaceAnalyzer"
> />
> <analyzer id="nostopssensitive"
> class="org.fryske_akademy.exist.lucene.SensitiveWhitespaceAnalyzer"/>

Yeap, you're right. I was rather speaking about query terms and
"serializers" in fact, as defined by Lucene standards... Sorry.
Here is an example :In german, "sprächt" could be indexed as "sprächt"
or "spracht" without the diacritic. Good to query with 2 indexes
(diacritics or no_diacritics), depending of the applied indexing
filter...My arabic indexer indexes it as "sprechen", i.e. the
infinitive form (the arabic root "SPRCH" in fact, but never mind...).
This is how arabic works and that is how every semitic language work
also. Arabic dictionnaries are ordered by "root", not by derivation
(noun, adjectives, verbs, adverbs...)...A simple (and funny ?) example
is : - singular "el Qasr" (which is the latin term "castrum", the
modern "castle" !!!)- plural "el-uqsur" (arabic "internal" plural :
specific vowels are dispatched around the QSR root) ! Doh !In fact,
everyboy know this term as... "Luxor" !I mean, there is such a
difference between singular and pural forms that we can not index terms
as they are...
For instance, to mimic arabic indexing in german, my analyzer would
index "sprächt", "spreche", "sprachen", "gesprochen", "gesprä̱chig" as
"sprechen", i.e. the infinitive form...
At query time, I have to analyze the query term. "spreche" for
instance, would in fact query "sprechen", that is the root form, 
So, first point, I would like to index the query term with a specific
analyzer. When I query "sprichst", I would like in fact it to query
"sprechen"...
At serialization time, and that is the main point IMHO, I need to
retrieve an original analyzed term most of the time... but not always.
I mean : index "6" and return "six" as a simple example...
As a conclusion, we need some filters at query time and serialization
time IMHO...
Cheers,
p.b._______________________________________________Exist-open mailing 
lis...@li...
https://lists.sourceforge.net/lists/listinfo/exist-open

-- 
Eduard Drenth, Software Architekt

ed...@fr...

Doelestrjitte 8
8911 DX  Ljouwert
+31 58 234 30 47
+31 62 094 34 28 (privé)

skype: eduarddrenth
https://github.com/eduarddrenth
frisian.eu
gpg: https://pgp.surfnet.nl/pks/lookup?search=eduarddrenth


Op freed bin ik thús/wurkje ik minder

Re: [Exist-open] Full backup at 4 a.m.

[Christian A. <chr...@ge...>]

That's a good thought for my next eXist instance. For now I stick with 
"Never change a running system" ;-)

Kind regards
Christian


Am 21.01.21 um 12:00 schrieb Roy Walter:
> I don't know if it's relevant to your use case but I find backups are 
> better controlled from cron. I create an xquery backup job and call it 
> from crontab:
> /usr/bin/curl http://localhost:8008/exist/rest/db/backup.xql 
> <http://localhost:8008/exist/rest/db/backup.xql>
> You could create full and incremental xquery jobs
>
> Just a thought.
>
> Regards,
> Roy
>
> On Thursday, 21 January 2021, 10:24:17 GMT, Christian Achter 
> <chr...@ge...> wrote:
>
>
> Dear all,
>
> thanks to your input I could solve my problem. As I also learnt 
> something new about backups in eXist and I want to share this 
> knowledge with you:
>
>  1. For my backups that are created every 4 hours (see configuration
>     below), I changed the value in /<parameter name="max" value="6"/>/
>     from 6 to 5.
>     *Fun fact:* This caused my full backup to "wander through the
>     day". On the first day the full backup was produced at 8 p.m.,
>     next day at 4 p.m., 3rd day at noon, 4th day at 8 a.m. and so on.
>     So the value="5" was less interpreted as "after 5 incremental
>     backups create a full backup" (as described in the O'Reilly book
>     and in the online documentation) but more like "every 5th backup
>     will be a full backup". So every 20th hour, a full backup was created.
>
>  2. So I changed back to /<parameter name="max" value="6"/>/. I still
>     could not force to have the full backup at 4 a.m., at least not
>     via the configuration in etc/conf.xml. The full backup was still
>     created at midnight.
>     But I found this solution: At 3 a.m. I deleted the last full
>     backup. So the newest backups were only around 7 or 8 or more
>     incremental backup. Thus at 4 a.m., when the next backup was
>     created, the system said: "Oh look, so many incremental backups -
>     let's create a full backup now!"
>     Since then, my full backup is always created at 4 a.m.
>
>  3. Additionally to 2. I found it interesting to see that the eXist
>     backup functionality probably checks the backup files in the
>     defined backup directory. I do not know if it checks the file
>     names or the dates of creation supplied by the OS but eXist
>     somehow knows how many incremental and full backups exist and how
>     old they are. That is why deleting the last full backup forced
>     eXist to create another full backup the next time backup scheduler
>     was run.
>
> I found that interesting, I hope you, too.
>
> Thanks again for all the input and kind regards
> Christian
>
>
> Am 15.01.21 um 20:53 schrieb Christian Achter:
>> Hello Florian.
>>
>> Thank you for getting back to me on this subject.
>>
>> What you wrote totally makes sense to me. So I changed my code for 
>> the backup to:
>>
>>         <job type="system" name="backup"
>> class="org.exist.storage.ConsistencyCheckTask"
>>                         cron-trigger="0 0 0/4 * * ?">
>>                 <parameter name="output" 
>> value="/opt/server/ediarum/exist-db_5.2.0/backup"/>
>>                 <parameter name="backup" value="yes"/>
>>                 <parameter name="incremental" value="yes"/>
>>                 <parameter name="incremental-check" value="no"/>
>>                 <parameter name="max" value="5"/>
>>         </job>
>>
>> Actually, I only changed the value of parameter "max" from 6 to 5. 
>> This will create a full backup at midnight 12 a.m. (instead of 4 
>> a.m.) but I am absolutely happy with that.
>>
>> Just one thing from our experience while having this eXist server up 
>> and running for several months: The full backup was actually not 
>> "wandering" in time, by 4 hours every day; it is always created at 8 
>> a.m. So maybe the "wandering" of the full backup only happens once, 
>> and the rule "0 0 0/4" might be more powerful than the parameter 
>> name="max" value="5". This might explain why the full backup 
>> "wandered" depending on the time that the eXist server was restarted.
>>
>> I am excited to see the results tomorrow morning.
>>
>> Thanks and kind regards
>> Christian
>>
>>
>> Am 15.01.21 um 18:32 schrieb Florian Schmitt:
>>> Hi Christian,
>>>
>>> in short: Based on the documentation, i don't see a way to make the
>>> backup start on 0 (12am = midnight), with the full backup at 4am.
>>>
>>> In your case, it seems that the combination of the increment of 4
>>> together with the max attribute of 6 leads to strange results. Setting
>>> max to 6 says: "Create one full backup after creating six incremental
>>> backups". But with an increment of 4 for [0-23], starting on 0, you
>>> get only five incremental backups per day. With max set to 6, the time
>>> of the full backup will "roam" over the entire day:
>>>
>>> On day 1, the backup run on 0 (12am = midnight) should be a full
>>> backup, since the first run is always a full backup. With max set to
>>> 6, the following six (!) backups will be incremental, so the sixth
>>> incremental backup is the first one of the following day (0 = 12am =
>>> midnight), with the next full backup on 4am of day 2. The third full
>>> backup again will take place after six incremental ones, thus on 8am
>>> of day 3, and so on.
>>>
>>> Thus, setting max to 5 should fix the problem of the different full
>>> backup times, but you won't get a full backup at 4am with that trigger
>>> definition; only the backup run on 0 (12am = midnight) will be a full
>>> backup; the following five are incremental, the next one (again at 0 =
>>> 12am = midnight) is full, and so on. In that case, you may even drop
>>> the max attribute completely; it doesn't have any special meaning in
>>> this case. It won't have any effect since there are only five
>>> incremental backups per day, with the first backup on next day as full
>>> backup.
>>>
>>>
>>> I assume that, if you want to have a full backup at 4am, you'll need
>>> to make 4am as first backup run, so the cron expression would be "0 0
>>> 4/4 * * ?". But this trigger will create only five backups per day,
>>> not six (no backup at 12am/midnight). Thus, i don't see a way to make
>>> the backup trigger start on 0 (12am = midnight), creating six backups
>>> per day, with the full backup on 4am.
>>>
>>>
>>> Kind regards
>>> Florian
>>>
>>>
>>> Am 15.01.21 um 15:01 schrieb Christian Achter:
>>>> Dear all,
>>>>
>>>> I have a backup scheduler running that creates 6 backups per day, 5
>>>> incremental and 1 full backup.
>>>>
>>>> The code in my etc/conf.xml looks like this:
>>>>
>>>>          <job type="system" name="backup"
>>>> class="org.exist.storage.ConsistencyCheckTask"
>>>>                          cron-trigger="0 0 0/4 * * ?">
>>>>                  <parameter name="output"
>>>> value="/opt/server/ediarum/exist-db_5.2.0/backup"/>
>>>>                  <parameter name="backup" value="yes"/>
>>>>                  <parameter name="incremental" value="yes"/>
>>>>                  <parameter name="incremental-check" value="no"/>
>>>>                  <parameter name="max" value="6"/>
>>>>          </job>
>>>>
>>>> Here is my problem:
>>>>
>>>> I want the full backup to be created at 4:00 a.m. I consulted the
>>>> eXist online documentation and the book and assumed that 0 0 0/4 would
>>>> be translated to: every 4 hours starting at midnight with the first
>>>> backup being a full backup.
>>>>
>>>> But in reality, the full backup is created at 8:00 a.m. My experience
>>>> during several months showed that this time even differs. Sometimes,
>>>> the full backup was created at 4 p.m. oder at noon - maybe depending
>>>> in the time when the eXist-db ws started.
>>>>
>>>> What am I doing wrong? How can I improve this scheduler to have a full
>>>> backup at 4 a.m. while keeping the 5 incremental backups during the 
>>>> day?
>>>>
>>>> Thanks in advance and kind regards
>>>>
>>>> Christian
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Exist-open mailing list
>>>> Exi...@li... 
>>>> <mailto:Exi...@li...>
>>>> https://lists.sourceforge.net/lists/listinfo/exist-open 
>>>> <https://lists.sourceforge.net/lists/listinfo/exist-open>
>>>
>>>
>>> _______________________________________________
>>> Exist-open mailing list
>>> Exi...@li... 
>>> <mailto:Exi...@li...>
>>> https://lists.sourceforge.net/lists/listinfo/exist-open 
>>> <https://lists.sourceforge.net/lists/listinfo/exist-open>
>>
>>
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li... 
>> <mailto:Exi...@li...>
>> https://lists.sourceforge.net/lists/listinfo/exist-open 
>> <https://lists.sourceforge.net/lists/listinfo/exist-open>
> _______________________________________________
> Exist-open mailing list
> Exi...@li... <mailto:Exi...@li...>
> https://lists.sourceforge.net/lists/listinfo/exist-open 
> <https://lists.sourceforge.net/lists/listinfo/exist-open>

Re: [Exist-open] Full backup at 4 a.m.

[Roy W. <gar...@ya...>]

 I don't know if it's relevant to your use case but I find backups are better controlled from cron. I create an xquery backup job and call it from crontab:/usr/bin/curl http://localhost:8008/exist/rest/db/backup.xqlYou could create full and incremental xquery jobs
Just a thought.
Regards,Roy
    On Thursday, 21 January 2021, 10:24:17 GMT, Christian Achter <chr...@ge...> wrote:  
 
  
Dear all,
 
thanks to your input I could solve my problem. As I also learnt something new about backups in eXist and I want to share this knowledge with you:
    
   - For my backups that are created every 4 hours (see configuration below), I changed the value in <parameter name="max" value="6"/> from 6 to 5.   
 Fun fact: This caused my full backup to "wander through the day". On the first day the full backup was produced at 8 p.m., next day at 4 p.m., 3rd day at noon, 4th day at 8 a.m. and so on.   
 So the value="5" was less interpreted as "after 5 incremental backups create a full backup" (as described in the O'Reilly book and in the online documentation) but more like "every 5th backup will be a full backup". So every 20th hour, a full backup was created.   
    
 
   - So I changed back to <parameter name="max" value="6"/>. I still could not force to have the full backup at 4 a.m., at least not via the configuration in etc/conf.xml. The full backup was still created at midnight.   
 But I found this solution: At 3 a.m. I deleted the last full backup. So the newest backups were only around 7 or 8 or more incremental backup. Thus at 4 a.m., when the next backup was created, the system said: "Oh look, so many incremental backups - let's create a full backup now!"   
 Since then, my full backup is always created at 4 a.m.   
    
 
   - Additionally to 2. I found it interesting to see that the eXist backup functionality probably checks the backup files in the defined backup directory. I do not know if it checks the file names or the dates of creation supplied by the OS but eXist somehow knows how many incremental and full backups exist and how old they are. That is why deleting the last full backup forced eXist to create another full backup the next time backup scheduler was run.   
 
 
I found that interesting, I hope you, too.
 
Thanks again for all the input and kind regards
 Christian
 

 
 Am 15.01.21 um 20:53 schrieb Christian Achter:
  
Hello Florian. 
 
 Thank you for getting back to me on this subject. 
 
 What you wrote totally makes sense to me. So I changed my code for the backup to: 
 
         <job type="system" name="backup" 
 class="org.exist.storage.ConsistencyCheckTask" 
                         cron-trigger="0 0 0/4 * * ?"> 
                 <parameter name="output" value="/opt/server/ediarum/exist-db_5.2.0/backup"/> 
                 <parameter name="backup" value="yes"/> 
                 <parameter name="incremental" value="yes"/> 
                 <parameter name="incremental-check" value="no"/> 
                 <parameter name="max" value="5"/> 
         </job> 
 
 Actually, I only changed the value of parameter "max" from 6 to 5. This will create a full backup at midnight 12 a.m. (instead of 4 a.m.) but I am absolutely happy with that. 
 
 Just one thing from our experience while having this eXist server up and running for several months: The full backup was actually not "wandering" in time, by 4 hours every day; it is always created at 8 a.m. So maybe the "wandering" of the full backup only happens once, and the rule "0 0 0/4" might be more powerful than the parameter name="max" value="5". This might explain why the full backup "wandered" depending on the time that the eXist server was restarted. 
 
 I am excited to see the results tomorrow morning. 
 
 Thanks and kind regards 
 Christian 
 
 
 Am 15.01.21 um 18:32 schrieb Florian Schmitt: 
 
Hi Christian, 
 
 in short: Based on the documentation, i don't see a way to make the 
 backup start on 0 (12am = midnight), with the full backup at 4am. 
 
 In your case, it seems that the combination of the increment of 4 
 together with the max attribute of 6 leads to strange results. Setting 
 max to 6 says: "Create one full backup after creating six incremental 
 backups". But with an increment of 4 for [0-23], starting on 0, you 
 get only five incremental backups per day. With max set to 6, the time 
 of the full backup will "roam" over the entire day: 
 
 On day 1, the backup run on 0 (12am = midnight) should be a full 
 backup, since the first run is always a full backup. With max set to 
 6, the following six (!) backups will be incremental, so the sixth 
 incremental backup is the first one of the following day (0 = 12am = 
 midnight), with the next full backup on 4am of day 2. The third full 
 backup again will take place after six incremental ones, thus on 8am 
 of day 3, and so on. 
 
 Thus, setting max to 5 should fix the problem of the different full 
 backup times, but you won't get a full backup at 4am with that trigger 
 definition; only the backup run on 0 (12am = midnight) will be a full 
 backup; the following five are incremental, the next one (again at 0 = 
 12am = midnight) is full, and so on. In that case, you may even drop 
 the max attribute completely; it doesn't have any special meaning in 
 this case. It won't have any effect since there are only five 
 incremental backups per day, with the first backup on next day as full 
 backup. 
 
 
 I assume that, if you want to have a full backup at 4am, you'll need 
 to make 4am as first backup run, so the cron expression would be "0 0 
 4/4 * * ?". But this trigger will create only five backups per day, 
 not six (no backup at 12am/midnight). Thus, i don't see a way to make 
 the backup trigger start on 0 (12am = midnight), creating six backups 
 per day, with the full backup on 4am. 
 
 
 Kind regards 
 Florian 
 
 
 Am 15.01.21 um 15:01 schrieb Christian Achter: 
 
Dear all, 
 
 I have a backup scheduler running that creates 6 backups per day, 5 
 incremental and 1 full backup. 
 
 The code in my etc/conf.xml looks like this: 
 
          <job type="system" name="backup" 
 class="org.exist.storage.ConsistencyCheckTask" 
                          cron-trigger="0 0 0/4 * * ?"> 
                  <parameter name="output" 
 value="/opt/server/ediarum/exist-db_5.2.0/backup"/> 
                  <parameter name="backup" value="yes"/> 
                  <parameter name="incremental" value="yes"/> 
                  <parameter name="incremental-check" value="no"/> 
                  <parameter name="max" value="6"/> 
          </job> 
 
 Here is my problem: 
 
 I want the full backup to be created at 4:00 a.m. I consulted the 
 eXist online documentation and the book and assumed that 0 0 0/4 would 
 be translated to: every 4 hours starting at midnight with the first 
 backup being a full backup. 
 
 But in reality, the full backup is created at 8:00 a.m. My experience 
 during several months showed that this time even differs. Sometimes, 
 the full backup was created at 4 p.m. oder at noon - maybe depending 
 in the time when the eXist-db ws started. 
 
 What am I doing wrong? How can I improve this scheduler to have a full 
 backup at 4 a.m. while keeping the 5 incremental backups during the day? 
 
 Thanks in advance and kind regards 
 
 Christian 
 
 
 
 
 _______________________________________________ 
 Exist-open mailing list 
 Exi...@li... 
 https://lists.sourceforge.net/lists/listinfo/exist-open 
 
 
 
 _______________________________________________ 
 Exist-open mailing list 
 Exi...@li... 
 https://lists.sourceforge.net/lists/listinfo/exist-open 
 
 
 
 _______________________________________________ 
 Exist-open mailing list 
 Exi...@li... 
 https://lists.sourceforge.net/lists/listinfo/exist-open 
 
 _______________________________________________
Exist-open mailing list
Exi...@li...
https://lists.sourceforge.net/lists/listinfo/exist-open
  

Re: [Exist-open] Full backup at 4 a.m.

[Christian A. <chr...@ge...>]

Dear all,

thanks to your input I could solve my problem. As I also learnt 
something new about backups in eXist and I want to share this knowledge 
with you:

 1. For my backups that are created every 4 hours (see configuration
    below), I changed the value in /<parameter name="max" value="6"/>/
    from 6 to 5.
    *Fun fact:* This caused my full backup to "wander through the day".
    On the first day the full backup was produced at 8 p.m., next day at
    4 p.m., 3rd day at noon, 4th day at 8 a.m. and so on.
    So the value="5" was less interpreted as "after 5 incremental
    backups create a full backup" (as described in the O'Reilly book and
    in the online documentation) but more like "every 5th backup will be
    a full backup". So every 20th hour, a full backup was created.

 2. So I changed back to /<parameter name="max" value="6"/>/. I still
    could not force to have the full backup at 4 a.m., at least not via
    the configuration in etc/conf.xml. The full backup was still created
    at midnight.
    But I found this solution: At 3 a.m. I deleted the last full backup.
    So the newest backups were only around 7 or 8 or more incremental
    backup. Thus at 4 a.m., when the next backup was created, the system
    said: "Oh look, so many incremental backups - let's create a full
    backup now!"
    Since then, my full backup is always created at 4 a.m.

 3. Additionally to 2. I found it interesting to see that the eXist
    backup functionality probably checks the backup files in the defined
    backup directory. I do not know if it checks the file names or the
    dates of creation supplied by the OS but eXist somehow knows how
    many incremental and full backups exist and how old they are. That
    is why deleting the last full backup forced eXist to create another
    full backup the next time backup scheduler was run.

I found that interesting, I hope you, too.

Thanks again for all the input and kind regards
Christian


Am 15.01.21 um 20:53 schrieb Christian Achter:
> Hello Florian.
>
> Thank you for getting back to me on this subject.
>
> What you wrote totally makes sense to me. So I changed my code for the 
> backup to:
>
>         <job type="system" name="backup"
> class="org.exist.storage.ConsistencyCheckTask"
>                         cron-trigger="0 0 0/4 * * ?">
>                 <parameter name="output" 
> value="/opt/server/ediarum/exist-db_5.2.0/backup"/>
>                 <parameter name="backup" value="yes"/>
>                 <parameter name="incremental" value="yes"/>
>                 <parameter name="incremental-check" value="no"/>
>                 <parameter name="max" value="5"/>
>         </job>
>
> Actually, I only changed the value of parameter "max" from 6 to 5. 
> This will create a full backup at midnight 12 a.m. (instead of 4 a.m.) 
> but I am absolutely happy with that.
>
> Just one thing from our experience while having this eXist server up 
> and running for several months: The full backup was actually not 
> "wandering" in time, by 4 hours every day; it is always created at 8 
> a.m. So maybe the "wandering" of the full backup only happens once, 
> and the rule "0 0 0/4" might be more powerful than the parameter 
> name="max" value="5". This might explain why the full backup 
> "wandered" depending on the time that the eXist server was restarted.
>
> I am excited to see the results tomorrow morning.
>
> Thanks and kind regards
> Christian
>
>
> Am 15.01.21 um 18:32 schrieb Florian Schmitt:
>> Hi Christian,
>>
>> in short: Based on the documentation, i don't see a way to make the
>> backup start on 0 (12am = midnight), with the full backup at 4am.
>>
>> In your case, it seems that the combination of the increment of 4
>> together with the max attribute of 6 leads to strange results. Setting
>> max to 6 says: "Create one full backup after creating six incremental
>> backups". But with an increment of 4 for [0-23], starting on 0, you
>> get only five incremental backups per day. With max set to 6, the time
>> of the full backup will "roam" over the entire day:
>>
>> On day 1, the backup run on 0 (12am = midnight) should be a full
>> backup, since the first run is always a full backup. With max set to
>> 6, the following six (!) backups will be incremental, so the sixth
>> incremental backup is the first one of the following day (0 = 12am =
>> midnight), with the next full backup on 4am of day 2. The third full
>> backup again will take place after six incremental ones, thus on 8am
>> of day 3, and so on.
>>
>> Thus, setting max to 5 should fix the problem of the different full
>> backup times, but you won't get a full backup at 4am with that trigger
>> definition; only the backup run on 0 (12am = midnight) will be a full
>> backup; the following five are incremental, the next one (again at 0 =
>> 12am = midnight) is full, and so on. In that case, you may even drop
>> the max attribute completely; it doesn't have any special meaning in
>> this case. It won't have any effect since there are only five
>> incremental backups per day, with the first backup on next day as full
>> backup.
>>
>>
>> I assume that, if you want to have a full backup at 4am, you'll need
>> to make 4am as first backup run, so the cron expression would be "0 0
>> 4/4 * * ?". But this trigger will create only five backups per day,
>> not six (no backup at 12am/midnight). Thus, i don't see a way to make
>> the backup trigger start on 0 (12am = midnight), creating six backups
>> per day, with the full backup on 4am.
>>
>>
>> Kind regards
>> Florian
>>
>>
>> Am 15.01.21 um 15:01 schrieb Christian Achter:
>>> Dear all,
>>>
>>> I have a backup scheduler running that creates 6 backups per day, 5
>>> incremental and 1 full backup.
>>>
>>> The code in my etc/conf.xml looks like this:
>>>
>>>          <job type="system" name="backup"
>>> class="org.exist.storage.ConsistencyCheckTask"
>>>                          cron-trigger="0 0 0/4 * * ?">
>>>                  <parameter name="output"
>>> value="/opt/server/ediarum/exist-db_5.2.0/backup"/>
>>>                  <parameter name="backup" value="yes"/>
>>>                  <parameter name="incremental" value="yes"/>
>>>                  <parameter name="incremental-check" value="no"/>
>>>                  <parameter name="max" value="6"/>
>>>          </job>
>>>
>>> Here is my problem:
>>>
>>> I want the full backup to be created at 4:00 a.m. I consulted the
>>> eXist online documentation and the book and assumed that 0 0 0/4 would
>>> be translated to: every 4 hours starting at midnight with the first
>>> backup being a full backup.
>>>
>>> But in reality, the full backup is created at 8:00 a.m. My experience
>>> during several months showed that this time even differs. Sometimes,
>>> the full backup was created at 4 p.m. oder at noon - maybe depending
>>> in the time when the eXist-db ws started.
>>>
>>> What am I doing wrong? How can I improve this scheduler to have a full
>>> backup at 4 a.m. while keeping the 5 incremental backups during the 
>>> day?
>>>
>>> Thanks in advance and kind regards
>>>
>>> Christian
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Exist-open mailing list
>>> Exi...@li...
>>> https://lists.sourceforge.net/lists/listinfo/exist-open
>>
>>
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-open] Full backup at 4 a.m.

[Christian A. <chr...@ge...>]

Hello Florian.

Thank you for getting back to me on this subject.

What you wrote totally makes sense to me. So I changed my code for the 
backup to:

         <job type="system" name="backup"
class="org.exist.storage.ConsistencyCheckTask"
                         cron-trigger="0 0 0/4 * * ?">
                 <parameter name="output" 
value="/opt/server/ediarum/exist-db_5.2.0/backup"/>
                 <parameter name="backup" value="yes"/>
                 <parameter name="incremental" value="yes"/>
                 <parameter name="incremental-check" value="no"/>
                 <parameter name="max" value="5"/>
         </job>

Actually, I only changed the value of parameter "max" from 6 to 5. This 
will create a full backup at midnight 12 a.m. (instead of 4 a.m.) but I 
am absolutely happy with that.

Just one thing from our experience while having this eXist server up and 
running for several months: The full backup was actually not "wandering" 
in time, by 4 hours every day; it is always created at 8 a.m. So maybe 
the "wandering" of the full backup only happens once, and the rule "0 0 
0/4" might be more powerful than the parameter name="max" value="5". 
This might explain why the full backup "wandered" depending on the time 
that the eXist server was restarted.

I am excited to see the results tomorrow morning.

Thanks and kind regards
Christian


Am 15.01.21 um 18:32 schrieb Florian Schmitt:
> Hi Christian,
>
> in short: Based on the documentation, i don't see a way to make the
> backup start on 0 (12am = midnight), with the full backup at 4am.
>
> In your case, it seems that the combination of the increment of 4
> together with the max attribute of 6 leads to strange results. Setting
> max to 6 says: "Create one full backup after creating six incremental
> backups". But with an increment of 4 for [0-23], starting on 0, you
> get only five incremental backups per day. With max set to 6, the time
> of the full backup will "roam" over the entire day:
>
> On day 1, the backup run on 0 (12am = midnight) should be a full
> backup, since the first run is always a full backup. With max set to
> 6, the following six (!) backups will be incremental, so the sixth
> incremental backup is the first one of the following day (0 = 12am =
> midnight), with the next full backup on 4am of day 2. The third full
> backup again will take place after six incremental ones, thus on 8am
> of day 3, and so on.
>
> Thus, setting max to 5 should fix the problem of the different full
> backup times, but you won't get a full backup at 4am with that trigger
> definition; only the backup run on 0 (12am = midnight) will be a full
> backup; the following five are incremental, the next one (again at 0 =
> 12am = midnight) is full, and so on. In that case, you may even drop
> the max attribute completely; it doesn't have any special meaning in
> this case. It won't have any effect since there are only five
> incremental backups per day, with the first backup on next day as full
> backup.
>
>
> I assume that, if you want to have a full backup at 4am, you'll need
> to make 4am as first backup run, so the cron expression would be "0 0
> 4/4 * * ?". But this trigger will create only five backups per day,
> not six (no backup at 12am/midnight). Thus, i don't see a way to make
> the backup trigger start on 0 (12am = midnight), creating six backups
> per day, with the full backup on 4am.
>
>
> Kind regards
> Florian
>
>
> Am 15.01.21 um 15:01 schrieb Christian Achter:
>> Dear all,
>>
>> I have a backup scheduler running that creates 6 backups per day, 5
>> incremental and 1 full backup.
>>
>> The code in my etc/conf.xml looks like this:
>>
>>          <job type="system" name="backup"
>> class="org.exist.storage.ConsistencyCheckTask"
>>                          cron-trigger="0 0 0/4 * * ?">
>>                  <parameter name="output"
>> value="/opt/server/ediarum/exist-db_5.2.0/backup"/>
>>                  <parameter name="backup" value="yes"/>
>>                  <parameter name="incremental" value="yes"/>
>>                  <parameter name="incremental-check" value="no"/>
>>                  <parameter name="max" value="6"/>
>>          </job>
>>
>> Here is my problem:
>>
>> I want the full backup to be created at 4:00 a.m. I consulted the
>> eXist online documentation and the book and assumed that 0 0 0/4 would
>> be translated to: every 4 hours starting at midnight with the first
>> backup being a full backup.
>>
>> But in reality, the full backup is created at 8:00 a.m. My experience
>> during several months showed that this time even differs. Sometimes,
>> the full backup was created at 4 p.m. oder at noon - maybe depending
>> in the time when the eXist-db ws started.
>>
>> What am I doing wrong? How can I improve this scheduler to have a full
>> backup at 4 a.m. while keeping the 5 incremental backups during the day?
>>
>> Thanks in advance and kind regards
>>
>> Christian
>>
>>
>>
>>
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-open] Full backup at 4 a.m.

[Christian A. <chr...@ge...>]

Thank you for getting back to me, Nick.

Our backups are pretty small, around 50 MB or less, and they are created 
by the server in less than 2 minutes during the past 3 years. So I think 
there is no problem here.

Kind regards
Christian


Am 15.01.21 um 18:15 schrieb Nick Sincaglia:
> Could it be that your backups are not yet completed before the next 
> backup is scheduled?
>
> Nick
>
> On 1/15/21 8:01 AM, Christian Achter wrote:
>> Dear all,
>>
>> I have a backup scheduler running that creates 6 backups per day, 5 
>> incremental and 1 full backup.
>>
>> The code in my etc/conf.xml looks like this:
>>
>>         <job type="system" name="backup"
>> class="org.exist.storage.ConsistencyCheckTask"
>>                         cron-trigger="0 0 0/4 * * ?">
>>                 <parameter name="output" 
>> value="/opt/server/ediarum/exist-db_5.2.0/backup"/>
>>                 <parameter name="backup" value="yes"/>
>>                 <parameter name="incremental" value="yes"/>
>>                 <parameter name="incremental-check" value="no"/>
>>                 <parameter name="max" value="6"/>
>>         </job>
>>
>> Here is my problem:
>>
>> I want the full backup to be created at 4:00 a.m. I consulted the 
>> eXist online documentation and the book and assumed that 0 0 0/4 
>> would be translated to: every 4 hours starting at midnight with the 
>> first backup being a full backup.
>>
>> But in reality, the full backup is created at 8:00 a.m. My experience 
>> during several months showed that this time even differs. Sometimes, 
>> the full backup was created at 4 p.m. oder at noon - maybe depending 
>> in the time when the eXist-db ws started.
>>
>> What am I doing wrong? How can I improve this scheduler to have a 
>> full backup at 4 a.m. while keeping the 5 incremental backups during 
>> the day?
>>
>> Thanks in advance and kind regards
>>
>> Christian
>>
>>
>>
>>
>> _______________________________________________
>> Exist-open mailing list
>> Exi...@li...
>> https://lists.sourceforge.net/lists/listinfo/exist-open
>
> -- 
> Nick Sincaglia
> President/Founder
> NueMeta, LLC
> Digital Media & Technology
> Phone: +1-630-303-7035
> nsi...@nu...  
> http://www.nuemeta.com
> Skype: nsincaglia
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-open] Full backup at 4 a.m.

[Florian S. <ml-...@fl...>]

Hi Christian,

in short: Based on the documentation, i don't see a way to make the
backup start on 0 (12am = midnight), with the full backup at 4am.

In your case, it seems that the combination of the increment of 4
together with the max attribute of 6 leads to strange results. Setting
max to 6 says: "Create one full backup after creating six incremental
backups". But with an increment of 4 for [0-23], starting on 0, you
get only five incremental backups per day. With max set to 6, the time
of the full backup will "roam" over the entire day:

On day 1, the backup run on 0 (12am = midnight) should be a full
backup, since the first run is always a full backup. With max set to
6, the following six (!) backups will be incremental, so the sixth
incremental backup is the first one of the following day (0 = 12am =
midnight), with the next full backup on 4am of day 2. The third full
backup again will take place after six incremental ones, thus on 8am
of day 3, and so on.

Thus, setting max to 5 should fix the problem of the different full
backup times, but you won't get a full backup at 4am with that trigger
definition; only the backup run on 0 (12am = midnight) will be a full
backup; the following five are incremental, the next one (again at 0 =
12am = midnight) is full, and so on. In that case, you may even drop
the max attribute completely; it doesn't have any special meaning in
this case. It won't have any effect since there are only five
incremental backups per day, with the first backup on next day as full
backup.


I assume that, if you want to have a full backup at 4am, you'll need
to make 4am as first backup run, so the cron expression would be "0 0
4/4 * * ?". But this trigger will create only five backups per day,
not six (no backup at 12am/midnight). Thus, i don't see a way to make
the backup trigger start on 0 (12am = midnight), creating six backups
per day, with the full backup on 4am.


Kind regards
Florian


Am 15.01.21 um 15:01 schrieb Christian Achter:
> Dear all,
> 
> I have a backup scheduler running that creates 6 backups per day, 5
> incremental and 1 full backup.
> 
> The code in my etc/conf.xml looks like this:
> 
>         <job type="system" name="backup"
> class="org.exist.storage.ConsistencyCheckTask"
>                         cron-trigger="0 0 0/4 * * ?">
>                 <parameter name="output"
> value="/opt/server/ediarum/exist-db_5.2.0/backup"/>
>                 <parameter name="backup" value="yes"/>
>                 <parameter name="incremental" value="yes"/>
>                 <parameter name="incremental-check" value="no"/>
>                 <parameter name="max" value="6"/>
>         </job>
> 
> Here is my problem:
> 
> I want the full backup to be created at 4:00 a.m. I consulted the
> eXist online documentation and the book and assumed that 0 0 0/4 would
> be translated to: every 4 hours starting at midnight with the first
> backup being a full backup.
> 
> But in reality, the full backup is created at 8:00 a.m. My experience
> during several months showed that this time even differs. Sometimes,
> the full backup was created at 4 p.m. oder at noon - maybe depending
> in the time when the eXist-db ws started.
> 
> What am I doing wrong? How can I improve this scheduler to have a full
> backup at 4 a.m. while keeping the 5 incremental backups during the day?
> 
> Thanks in advance and kind regards
> 
> Christian
> 
> 
> 
> 
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-open] Full backup at 4 a.m.

[Nick S. <nsi...@nu...>]

Could it be that your backups are not yet completed before the next 
backup is scheduled?

Nick

On 1/15/21 8:01 AM, Christian Achter wrote:
> Dear all,
>
> I have a backup scheduler running that creates 6 backups per day, 5 
> incremental and 1 full backup.
>
> The code in my etc/conf.xml looks like this:
>
>         <job type="system" name="backup"
> class="org.exist.storage.ConsistencyCheckTask"
>                         cron-trigger="0 0 0/4 * * ?">
>                 <parameter name="output" 
> value="/opt/server/ediarum/exist-db_5.2.0/backup"/>
>                 <parameter name="backup" value="yes"/>
>                 <parameter name="incremental" value="yes"/>
>                 <parameter name="incremental-check" value="no"/>
>                 <parameter name="max" value="6"/>
>         </job>
>
> Here is my problem:
>
> I want the full backup to be created at 4:00 a.m. I consulted the 
> eXist online documentation and the book and assumed that 0 0 0/4 would 
> be translated to: every 4 hours starting at midnight with the first 
> backup being a full backup.
>
> But in reality, the full backup is created at 8:00 a.m. My experience 
> during several months showed that this time even differs. Sometimes, 
> the full backup was created at 4 p.m. oder at noon - maybe depending 
> in the time when the eXist-db ws started.
>
> What am I doing wrong? How can I improve this scheduler to have a full 
> backup at 4 a.m. while keeping the 5 incremental backups during the day?
>
> Thanks in advance and kind regards
>
> Christian
>
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

-- 
Nick Sincaglia
President/Founder
NueMeta, LLC
Digital Media & Technology
Phone: +1-630-303-7035
nsi...@nu...
http://www.nuemeta.com
Skype: nsincaglia

Re: [Exist-open] diacritic switch (Eduard Drenth)

[PIERRICK B. <Pie...@br...>]

Hello,


> Perhaps, but the fields option enables you to define multiple indexes on one qname already, see Joe's answer:
> 
> collection.xconf:
>
>
> <analyzer id="nostops" class="org.fryske_akademy.exist.lucene.InsensitiveWhitespaceAnalyzer"/>
>
> <analyzer id="nostopssensitive" class="org.fryske_akademy.exist.lucene.SensitiveWhitespaceAnalyzer"/>


Yeap, you're right. I was rather speaking about query terms and "serializers" in fact, as defined by Lucene standards... Sorry.

Here is an example :
In german, "sprächt" could be indexed as "sprächt" or "spracht" without the diacritic. Good to query with 2 indexes (diacritics or no_diacritics), depending of the applied indexing filter...
My arabic indexer indexes it as "sprechen", i.e. the infinitive form (the arabic root "SPRCH" in fact, but never mind...). This is how arabic works and that is how every semitic language work also. Arabic dictionnaries are ordered by "root", not by derivation (noun, adjectives, verbs, adverbs...)...
A simple (and funny ?) example is : 
- singular "el Qasr" (which is the latin term "castrum", the modern "castle" !!!)
- plural "el-uqsur" (arabic "internal" plural : specific vowels are dispatched around the QSR root) ! Doh !
In fact, everyboy know this term as... "Luxor" !
I mean, there is such a difference between singular and pural forms that we can not index terms as they are...

For instance, to mimic arabic indexing in german, my analyzer would index "sprächt", "spreche", "sprachen", "gesprochen", "gesprä̱chig" as "sprechen", i.e. the infinitive form...

At query time, I have to analyze the query term. "spreche" for instance, would in fact query "sprechen", that is the root form, 

So, first point, I would like to index the query term with a specific analyzer. When I query "sprichst", I would like in fact it to query "sprechen"...

At serialization time, and that is the main point IMHO, I need to retrieve an original analyzed term most of the time... but not always.

I mean : index "6" and return "six" as a simple example...

As a conclusion, we need some filters at query time and serialization time IMHO...

Cheers,

p.b.

[Exist-open] Full backup at 4 a.m.

[Christian A. <chr...@ge...>]

Dear all,

I have a backup scheduler running that creates 6 backups per day, 5 
incremental and 1 full backup.

The code in my etc/conf.xml looks like this:

         <job type="system" name="backup"
class="org.exist.storage.ConsistencyCheckTask"
                         cron-trigger="0 0 0/4 * * ?">
                 <parameter name="output" 
value="/opt/server/ediarum/exist-db_5.2.0/backup"/>
                 <parameter name="backup" value="yes"/>
                 <parameter name="incremental" value="yes"/>
                 <parameter name="incremental-check" value="no"/>
                 <parameter name="max" value="6"/>
         </job>

Here is my problem:

I want the full backup to be created at 4:00 a.m. I consulted the eXist 
online documentation and the book and assumed that 0 0 0/4 would be 
translated to: every 4 hours starting at midnight with the first backup 
being a full backup.

But in reality, the full backup is created at 8:00 a.m. My experience 
during several months showed that this time even differs. Sometimes, the 
full backup was created at 4 p.m. oder at noon - maybe depending in the 
time when the eXist-db ws started.

What am I doing wrong? How can I improve this scheduler to have a full 
backup at 4 a.m. while keeping the 5 incremental backups during the day?

Thanks in advance and kind regards

Christian

Re: [Exist-open] diacritic switch (Eduard Drenth)

[Eduard D. <ed...@fr...>]

Perhaps, but the fields option enables you to define multiple indexes on one qname already, see Joe's answer:


collection.xconf:


<analyzer id="nostops" class="org.fryske_akademy.exist.lucene.InsensitiveWhitespaceAnalyzer"/>

<analyzer id="nostopssensitive" class="org.fryske_akademy.exist.lucene.SensitiveWhitespaceAnalyzer"/>


<text qname="tei:quote" analyzer="nostops">
    <field name="sensitive" analyzer="nostopssensitive"/>
</text>


xquery:


[ft:query(., teidictjson:searchTerm($searchterm))]


declare function teidictjson:searchTerm($term as xs:string) {
    if (teidictjson:findParam("sensitive","") = "") then
    $term else "sensitive:(" || $term || ")"
};




Regards, Eduard

-----Original Message-----
From: PIERRICK BRIHAYE <Pie...@br...h<mailto:PIERRICK%20BRIHAYE%20%3cP...@br...h%3e>>
To: Duncan Paterson <du...@ex...<mailto:Duncan%20Paterson%20%3cd...@ex...%3e>>, exist-open <exi...@li...<mailto:exist-open%20%3ce...@li...%3e>>
Subject: Re: [Exist-open] diacritic switch (Eduard Drenth)
Date: Tue, 12 Jan 2021 14:51:17 +0000


Hello,


When we introduced indexes in eXist some... 13 years ago , I planned a multi-indexing fixture, i.e. every text content would be indexed into several indexes. Unfortunately, this hasn't been developed yet, but I think it could be a very interesting feature to implement. It goes much further than "simply" treating diacritics IMHO.


I've made such an approach with my arabic analyzer

<http://www.nongnu.org/aramorph/english/lucene.html>

http://www.nongnu.org/aramorph/english/lucene.html

. You can query an arabic text in arabic or in english and get a result, sometimes totally inaccurate I must admit :-)


Cheers,


p.b.


________________________________________

De : Duncan Paterson <

<mailto:du...@ex...>

du...@ex...

>

Envoyé : mardi 12 janvier 2021 14:52

À : exist-open

Objet : Re: [Exist-open] diacritic switch (Eduard Drenth)


Dear Eduard,


wouldn’t unicode decomposition via

<http://www.xqueryfunctions.com/xq/fn_normalize-unicode.html>

http://www.xqueryfunctions.com/xq/fn_normalize-unicode.html

 be the easier option here? Is there something not working with that approach?


Greetings

Duncan

Ceterum censeo exist-db.org<

<http://exist-db.org>

http://exist-db.org

> esse conriganda




  1. diacritic switch (Eduard Drenth)

  2. Re: diacritic switch (Eduard Drenth)



----------------------------------------------------------------------


Message: 1

Date: Tue, 12 Jan 2021 07:08:50 +0000

From: Eduard Drenth <

<mailto:ed...@fr...>

ed...@fr...

<mailto:

<mailto:ed...@fr...>

ed...@fr...

>>

To: "

<mailto:exi...@li...>

exi...@li...

<mailto:

<mailto:exi...@li...>

exi...@li...

>"

<

<mailto:exi...@li...>

exi...@li...

<mailto:

<mailto:exi...@li...>

exi...@li...

>>

Subject: [Exist-open] diacritic switch

Message-ID:

<

<mailto:c21...@fr...>

c21...@fr...

<mailto:

<mailto:c21...@fr...>

c21...@fr...

>>

Content-Type: text/plain; charset="utf-8"


Dear all,


Can I programmatically query a specific index?


I want to offer my users a switch for case and diacritics sensitivity.


For this I want to define two indexes using different analyzers. Then I want to query one or the other depending on the switch.


Can this be done this way?


Bye, Eduard



--


Eduard Drenth, Software Architekt


<mailto:ed...@fr...>

ed...@fr...

<mailto:

<mailto:ed...@fr...>

ed...@fr...

><mailto:

<mailto:ed...@fr...>

ed...@fr...

>


Doelestrjitte 8

8911 DX  Ljouwert

+31 58 234 30 47

+31 62 094 34 28 (priv?)


skype: eduarddrenth

<https://github.com/eduarddrenth>

https://github.com/eduarddrenth


frisian.eu

gpg:

<https://pgp.surfnet.nl/pks/lookup?search=eduarddrenth>

https://pgp.surfnet.nl/pks/lookup?search=eduarddrenth




Op freed bin ik th?s/wurkje ik minder




-------------- next part --------------

An HTML attachment was scrubbed...


------------------------------


Message: 2

Date: Tue, 12 Jan 2021 09:27:56 +0000

From: Eduard Drenth <

<mailto:ed...@fr...>

ed...@fr...

>

To: "

<mailto:exi...@li...>

exi...@li...

"

<

<mailto:exi...@li...>

exi...@li...

>

Subject: Re: [Exist-open] diacritic switch

Message-ID:

<

<mailto:7a2...@fr...>

7a2...@fr...

>

Content-Type: text/plain; charset="utf-8"


If this isn't possible I will create a copy collection with a separate

collection.xconf


-----Original Message-----

From: Eduard Drenth <

<mailto:ed...@fr...>

ed...@fr...

>

To:

<mailto:exi...@li...>

exi...@li...

 <

<mailto:exi...@li...>

exi...@li...

>

Subject: [Exist-open] diacritic switch

Date: Tue, 12 Jan 2021 07:08:50 +0000






Dear all,






Can I programmatically query a specific index?






I want to offer my users a switch for case and diacritics sensitivity.






For this I want to define two indexes using different analyzers. Then I

want to query one or the other depending on the switch.






Can this be done this way?






Bye, Eduard







--


Eduard Drenth, Software Architekt




<mailto:ed...@fr...>

ed...@fr...





Doelestrjitte 8

8911 DX  Ljouwert

+31 58 234 30 47

+31 62 094 34 28 (priv?)




skype: eduarddrenth

<https://github.com/eduarddrenth>

https://github.com/eduarddrenth


frisian.eu

gpg:

<https://pgp.surfnet.nl/pks/lookup?search=eduarddrenth>

https://pgp.surfnet.nl/pks/lookup?search=eduarddrenth









Op freed bin ik th?s/wurkje ik minder














_______________________________________________Exist-open mailing

<mailto:lis...@li...>

lis...@li...


<https://lists.sourceforge.net/lists/listinfo/exist-open>

https://lists.sourceforge.net/lists/listinfo/exist-open



--

Eduard Drenth, Software Architekt


<mailto:ed...@fr...>

ed...@fr...



Doelestrjitte 8

8911 DX  Ljouwert

+31 58 234 30 47

+31 62 094 34 28 (priv?)


skype: eduarddrenth

<https://github.com/eduarddrenth>

https://github.com/eduarddrenth


frisian.eu

gpg:

<https://pgp.surfnet.nl/pks/lookup?search=eduarddrenth>

https://pgp.surfnet.nl/pks/lookup?search=eduarddrenth




Op freed bin ik th?s/wurkje ik minder





-------------- next part --------------

An HTML attachment was scrubbed...

-------------- next part --------------

A non-text attachment was scrubbed...

Name: signature.asc

Type: application/pgp-signature

Size: 488 bytes

Desc: This is a digitally signed message part


------------------------------




------------------------------


Subject: Digest Footer


_______________________________________________

Exist-open mailing list

<mailto:Exi...@li...>

Exi...@li...


<https://lists.sourceforge.net/lists/listinfo/exist-open>

https://lists.sourceforge.net/lists/listinfo/exist-open




------------------------------


End of Exist-open Digest, Vol 177, Issue 12

*******************************************




_______________________________________________

Exist-open mailing list

<mailto:Exi...@li...>

Exi...@li...


<https://lists.sourceforge.net/lists/listinfo/exist-open>

https://lists.sourceforge.net/lists/listinfo/exist-open


--

Eduard Drenth, Software Architekt

ed...@fr...<mailto:ed...@fr...>

Doelestrjitte 8
8911 DX  Ljouwert
+31 58 234 30 47
+31 62 094 34 28 (privé)

skype: eduarddrenth
https://github.com/eduarddrenth
frisian.eu
gpg: https://pgp.surfnet.nl/pks/lookup?search=eduarddrenth


Op freed bin ik thús/wurkje ik minder

Re: [Exist-open] diacritic switch

[Eduard D. <ed...@fr...>]

I can confirm that the fields approach works in combination with different analyzers.

Thanks!

-----Original Message-----
From: Joe Wicentowski <jo...@gm...<mailto:Joe%20Wicentowski%20%3cj...@gm...%3e>>
To: Eduard Drenth <ed...@fr...<mailto:Eduard%20Drenth%20%3ce...@fr...%3e>>
Cc: exi...@li... <exi...@li...<mailto:%22e...@li...%22%20%3ce...@li...%3e>>
Subject: Re: [Exist-open] diacritic switch
Date: Tue, 12 Jan 2021 09:10:14 -0500

Hi Eduard,

eXist 5's fields feature makes it possible to create different indexes on the same content and query a specific one. See the documentation:

  https://exist-db.org/exist/apps/doc/lucene#facets-and-fields

See especially the section, "Conditions." Building on this, you'll just need to find analyzers or expressions that are case and diacritics (in-)sensitive and hook these into your field definition.

Joe

On Tue, Jan 12, 2021 at 4:29 AM Eduard Drenth <ed...@fr...<mailto:ed...@fr...>> wrote:
If this isn't possible I will create a copy collection with a separate collection.xconf

-----Original Message-----
From: Eduard Drenth <ed...@fr...<mailto:Eduard%20Drenth%20%3ce...@fr...%3e>>
To: exi...@li...<mailto:exi...@li...> <exi...@li...<mailto:%22e...@li...%22%20%3ce...@li...%3e>>
Subject: [Exist-open] diacritic switch
Date: Tue, 12 Jan 2021 07:08:50 +0000

Dear all,

Can I programmatically query a specific index?

I want to offer my users a switch for case and diacritics sensitivity.

For this I want to define two indexes using different analyzers. Then I want to query one or the other depending on the switch.

Can this be done this way?

Bye, Eduard


--

Eduard Drenth, Software Architekt

ed...@fr...<mailto:ed...@fr...>

Doelestrjitte 8
8911 DX  Ljouwert
+31 58 234 30 47
+31 62 094 34 28 (privé)

skype: eduarddrenth
https://github.com/eduarddrenth
frisian.eu<http://frisian.eu>
gpg: https://pgp.surfnet.nl/pks/lookup?search=eduarddrenth


Op freed bin ik thús/wurkje ik minder




_______________________________________________

Exist-open mailing list

<mailto:Exi...@li...>

Exi...@li...

<https://lists.sourceforge.net/lists/listinfo/exist-open>

https://lists.sourceforge.net/lists/listinfo/exist-open


--

Eduard Drenth, Software Architekt

ed...@fr...<mailto:ed...@fr...>

Doelestrjitte 8
8911 DX  Ljouwert
+31 58 234 30 47
+31 62 094 34 28 (privé)

skype: eduarddrenth
https://github.com/eduarddrenth
frisian.eu<http://frisian.eu>
gpg: https://pgp.surfnet.nl/pks/lookup?search=eduarddrenth


Op freed bin ik thús/wurkje ik minder



_______________________________________________
Exist-open mailing list
Exi...@li...<mailto:Exi...@li...>
https://lists.sourceforge.net/lists/listinfo/exist-open

--

Eduard Drenth, Software Architekt

ed...@fr...<mailto:ed...@fr...>

Doelestrjitte 8
8911 DX  Ljouwert
+31 58 234 30 47
+31 62 094 34 28 (privé)

skype: eduarddrenth
https://github.com/eduarddrenth
frisian.eu
gpg: https://pgp.surfnet.nl/pks/lookup?search=eduarddrenth


Op freed bin ik thús/wurkje ik minder

Re: [Exist-open] diacritic switch

[Eduard D. <ed...@fr...>]

This might work yes, the analyzers I need are available. I don't
directly see the need for a condition, but I'll investigate this field 
possibility.
Regards, Eduard
-----Original Message-----From: Joe Wicentowski <jo...@gm...>To:
Eduard Drenth <ed...@fr...>Cc: 
exi...@li... <exi...@li...>Subj
ect: Re: [Exist-open] diacritic switchDate: Tue, 12 Jan 2021 09:10:14
-0500
Hi Eduard,
eXist 5's fields feature makes it possible to create different indexes
on the same content and query a specific one. See the documentation: 

  https://exist-db.org/exist/apps/doc/lucene#facets-and-fields

See especially the section, "Conditions." Building on this, you'll just
need to find analyzers or expressions that are case and diacritics (in-
)sensitive and hook these into your field definition.

Joe
On Tue, Jan 12, 2021 at 4:29 AM Eduard Drenth <
ed...@fr...> wrote:
> If this isn't possible I will create a copy collection with a
> separate collection.xconf
> 
> -----Original Message-----
> From: Eduard Drenth <ed...@fr...>
> To: exi...@li... <
> exi...@li...>
> Subject: [Exist-open] diacritic switch
> Date: Tue, 12 Jan 2021 07:08:50 +0000
> 
> 
> 
> 
> 
> Dear all,
> 
> 
> 
> 
> 
> Can I programmatically query a specific index?
> 
> 
> 
> 
> 
> I want to offer my users a switch for case and diacritics
> sensitivity.
> 
> 
> 
> 
> 
> For this I want to define two indexes using different analyzers. Then
> I want to query one or the other depending on the switch.
> 
> 
> 
> 
> 
> Can this be done this way?
> 
> 
> 
> 
> 
> Bye, Eduard
> 
> 
> 
> 
> 
> 
> -- 
> 
> Eduard Drenth, Software Architekt
> 
> 
> 
> ed...@fr...
> 
> 
> 
> Doelestrjitte 8
> 8911 DX  Ljouwert
> +31 58 234 30 47
> +31 62 094 34 28 (privé)
> 
> 
> 
> skype: eduarddrenth
> https://github.com/eduarddrenth
> frisian.eu
> gpg: https://pgp.surfnet.nl/pks/lookup?search=eduarddrenth
> 
> 
> 
> 
> 
> 
> 
> Op freed bin ik thús/wurkje ik minder
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________Exist-open mailing 
> lis...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
> -- 
> Eduard Drenth, Software Architekt
> 
> ed...@fr...
> 
> Doelestrjitte 8
> 8911 DX  Ljouwert
> +31 58 234 30 47
> +31 62 094 34 28 (privé)
> 
> skype: eduarddrenth
> https://github.com/eduarddrenth
> frisian.eu
> gpg: https://pgp.surfnet.nl/pks/lookup?search=eduarddrenth
> 
> 
> Op freed bin ik thús/wurkje ik minder
> 
> 
> 
> 
> _______________________________________________
> 
> Exist-open mailing list
> 
> Exi...@li...
> 
> https://lists.sourceforge.net/lists/listinfo/exist-open
> 


-- 
Eduard Drenth, Software Architekt

ed...@fr...

Doelestrjitte 8
8911 DX  Ljouwert
+31 58 234 30 47
+31 62 094 34 28 (privé)

skype: eduarddrenth
https://github.com/eduarddrenth
frisian.eu
gpg: https://pgp.surfnet.nl/pks/lookup?search=eduarddrenth


Op freed bin ik thús/wurkje ik minder