exist-open — Open discussion

[Exist-open] Incomplete shutdown of eXist 6.2 service

[Lizzi, V. <Vin...@ta...>]

Hello eXist community,

For a while now I've been getting an error message when trying to stop the Windows service for eXist-db. After upgrading to eXist-db version 6.2.0 this problem is still occurring. Here is what typically happens:


  1.  Stop the service for eXist-db using the Windows Services panel.
  2.  Wait for the progress bar. When the progress bar is nearly complete an error message appears saying that the service did not respond in a timely manner. The service status remains as "Stopping".
  3.  The log file exist.log contains messages "shutdown complete !", "Database shutdown: stopping server in 1sec", "Jetty server stopped"
  4.  Windows Task Manager shows that "Apache Commons Daemon Service Runner" is still running.
  5.  In eXist's "data" folder the dbx_dir.lck and journal.lck files are sometimes still present or sometimes have been removed. A *.log file is usually still present.

I have to manually kill the process using Task Manager before the service can be started again. If the *.lck files are still present I delete them manually.

There does not appear to be any indication in the logs about why the process is not exiting completely. This problem begins occurring after I import documents to eXist-db. On a completely fresh install of eXist-db the process shuts down completely.

Here are some details about the environment:

eXist Version:     6.2.0
eXist Build:          2023-02-04T22:42:29Z
Operating System:           Windows Server 2012 R2 6.3 amd64
Java Version:      11.0.18

Is there any way to resolve this problem or find out more about why it is happening?

Thanks,
Vincent

______________________________________________
Vincent M. Lizzi
Head of Information Standards | Taylor & Francis Group
530 Walnut St., Suite 850, Philadelphia, PA 19106
E-Mail: vin...@ta...<mailto:vin...@ta...>
Web: www.tandfonline.com<http://www.tandfonline.com>

Taylor & Francis is a trading name of Informa UK Limited,
registered in England under no. 1072954

"Everything should be made as simple as possible, but not simpler."



Information Classification: General

Re: [Exist-open] 504 error in relatively small query

[Ralph C. <ra...@ra...>]

Thank you all (apologies for not writing a specific thanks to @Lizzi, 
Vincent but also really useful).

However, @Jean-Paul Rehr you have also very correctly pointed out some 
xQuery issues, which have made a massive difference. It runs at the 
collection level now and returns the ~7000 orphaned records 0.17s. 
Amazing!

I've not come across the id() function before. Why does it operate so 
quickly?

Thank you again

On 13/03/2023 16:12, Jean-Paul Rehr wrote:

> I feel like your query could run much faster in eXist with some 
> adjustments.
> - I would avoid reopening the document in the where and put it in a 
> variable
> - not(exists()) can be replaced by empty() and make evaluation more 
> efficient
> -  /id() is faster than @xml:id =
> 
> let $listPerson := doc('/db/apps/****/data/listPerson.xml')
> for $ref in subsequence(collection('/db/apps/****/data')//ref[@type eq 
> '2'],1,####)
> where empty($listPerson/id($ref/@target))
> return
> $ref
> 
> Cheers,
> JPR
> 
> On Mon, Mar 13, 2023 at 3:18 PM Ralph Corrigan <ra...@ra...> 
> wrote:
> 
> Hi all, I was going to post this in an xQuery forum, but at small 
> numbers this query works fine and I feel confident that the issue here 
> is in my exist-db config.
> 
> I'm trying to return orphaned <ref> elements from a recent update that 
> was set up wrong. I'm using the following query to get the results:
> 
> for $ref in 
> subsequence(collection('/db/apps/****/data')//ref[@type='2'],1,####)
> 
> where 
> not(exists(doc('/db/apps/****/data/listPerson.xml')//person[@xml:id=$ref/@target]))
> 
> return
> $ref
> 
> This returns absolutely fine up to ~1200 items in the subsequence $ref 
> but by the time we get closer to 2000 (and more) this returns a 504 
> gateway time-out error in roughly 60s. This does not seem to me to be a 
> significantly onerous task (especially considering other things that 
> are handled well).
> 
> I've tried
> 
> declare option exist:timeout "600000" (which should be ten minutes)
> 
> to extend the timeout, but I'm getting the same error after ~60s.
> 
> Any ideas, as the total I'm trying to get to is ~12000 records of which 
> I have no idea how many are orphaned....
> 
> Cheers
> 
> Ralph
> 
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-open] 504 error in relatively small query

[Jean-Paul R. <re...@gm...>]

I feel like your query could run much faster in eXist with some adjustments.
- I would avoid reopening the document in the *where* and put it in a
variable
- not(exists()) can be replaced by empty() and make evaluation more
efficient
-  /id() is faster than @xml:id =

let $listPerson := doc('/db/apps/****/data/listPerson.xml')
for $ref in subsequence(collection('/db/apps/****/data')//ref[@type eq
'2'],1,####)
where empty($listPerson/id($ref/@target))
return
     $ref

Cheers,
JPR

On Mon, Mar 13, 2023 at 3:18 PM Ralph Corrigan <ra...@ra...> wrote:

> Hi all, I was going to post this in an xQuery forum, but at small numbers
> this query works fine and I feel confident that the issue here is in my
> exist-db config.
>
> I'm trying to return orphaned <ref> elements from a recent update that was
> set up wrong. I'm using the following query to get the results:
>
>
> for $ref in
> subsequence(collection('/db/apps/****/data')//ref[@type='2'],1,####)
>
> where
> not(exists(doc('/db/apps/****/data/listPerson.xml')//person[@xml:id=$ref/@target]))
>
> return
>  $ref
>
> This returns absolutely fine up to ~1200 items in the subsequence $ref but
> by the time we get closer to 2000 (and more) this returns a 504 gateway
> time-out error in roughly 60s. This does not seem to me to be a
> significantly onerous task (especially considering other things that are
> handled well).
>
> I've tried
>
> declare option exist:timeout "600000" (which should be ten minutes)
>
> to extend the timeout, but I'm getting the same error after ~60s.
>
> Any ideas, as the total I'm trying to get to is ~12000 records of which I
> have no idea how many are orphaned....
>
>
> Cheers
>
> Ralph
>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

Re: [Exist-open] 504 error in relatively small query

[Lizzi, V. <Vin...@ta...>]

Hi Ralph,

In addition to what Chris wrote about storing the results in the database, it may be helpful to create a separate collection for storing the query results to a avoid a write lock contention on the collection that holds the documents that are being read.  I tend to use a /db/temp collection for this purpose.

Another technique for long-running queries is to schedule the query to run as a background task with the Scheduler module, and then use Monex to see when the query finishes.

scheduler:schedule-xquery-periodic-job('/db/queries/name.xq', 0, 'name', (), 0, 0)

It can also be helpful to use util:log() to record messages about the progress or completion of the query in exist.log.

If you are getting errors about timeout or exceeding the maximum number of nodes, you might need to increase the timeout setting or output size limit setting, which can be done by declaring these options at the start of a query:

declare option exist:timeout "900000";
declare option exist:output-size-limit "1000000";

Most of this information is in the online documentation at exist-db.org but it can take a while to find and piece together what you need, so this forum is a helpful place to ask questions.

Cheers,
Vincent

_____________________________________________
Vincent M. Lizzi
Head of Information Standards | Taylor & Francis Group
vin...@ta...<mailto:vin...@ta...>

From: Ralph Corrigan <ra...@ra...>
Sent: Monday, March 13, 2023 11:37 AM
To: Chris Wallace <kit...@gm...>
Cc: ol...@ex...; Exi...@li...
Subject: Re: [Exist-open] 504 error in relatively small query


Thanks Chris, also useful to know.



On 13/03/2023 15:13, Chris Wallace wrote:
Doesn't the query keep running anyway?  I often have long jobs which timeout in the browser but run to completion (unless I terminate them in the Monitor).  When this is anticipated , I make sure the job writes output to a database file rather than (only) to return HTML

Chris Wallace

On Mon, Mar 13, 2023 at 3:00 PM Ralph Corrigan <ra...@ra...<mailto:ra...@ra...>> wrote:

Thanks Olaf, yes we are. I'll take a look at that...



Ralph



On 13/03/2023 14:35, ol...@ex...<mailto:ol...@ex...> wrote:
This returns absolutely fine up to ~1200 items in the subsequence $ref but
by the time we get closer to 2000 (and more) this returns a 504 gateway
time-out error in roughly 60s.

Are you running eXist behind a proxy like nginx?  I've had 504s created
from nginx frontend proxies.  In that case, you need to increase timeouts
in the nginx config.

Olaf


_______________________________________________
Exist-open mailing list
Exi...@li...<mailto:Exi...@li...>
https://lists.sourceforge.net/lists/listinfo/exist-open<https://lists.sourceforge.net/lists/listinfo/exist-open>
_______________________________________________
Exist-open mailing list
Exi...@li...<mailto:Exi...@li...>
https://lists.sourceforge.net/lists/listinfo/exist-open<https://lists.sourceforge.net/lists/listinfo/exist-open>


Information Classification: General

Re: [Exist-open] 504 error in relatively small query

[Ralph C. <ra...@ra...>]

Thanks Chris, also useful to know.

On 13/03/2023 15:13, Chris Wallace wrote:

> Doesn't the query keep running anyway?  I often have long jobs which 
> timeout in the browser but run to completion (unless I terminate them 
> in the Monitor).  When this is anticipated , I make sure the job writes 
> output to a database file rather than (only) to return HTML
> 
> Chris Wallace
> 
> On Mon, Mar 13, 2023 at 3:00 PM Ralph Corrigan <ra...@ra...> 
> wrote:
> 
> Thanks Olaf, yes we are. I'll take a look at that...
> 
> Ralph
> 
> On 13/03/2023 14:35, ol...@ex... wrote:
> This returns absolutely fine up to ~1200 items in the subsequence $ref 
> but
> by the time we get closer to 2000 (and more) this returns a 504 gateway
> time-out error in roughly 60s.
> Are you running eXist behind a proxy like nginx?  I've had 504s created
> from nginx frontend proxies.  In that case, you need to increase 
> timeouts
> in the nginx config.
> 
> Olaf
> 
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
  _______________________________________________
Exist-open mailing list
Exi...@li...
https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-open] 504 error in relatively small query

[Chris W. <kit...@gm...>]

Doesn't the query keep running anyway?  I often have long jobs which
timeout in the browser but run to completion (unless I terminate them in
the Monitor).  When this is anticipated , I make sure the job writes output
to a database file rather than (only) to return HTML

Chris Wallace

On Mon, Mar 13, 2023 at 3:00 PM Ralph Corrigan <ra...@ra...> wrote:

> Thanks Olaf, yes we are. I'll take a look at that...
>
>
> Ralph
>
>
> On 13/03/2023 14:35, ol...@ex... wrote:
>
> This returns absolutely fine up to ~1200 items in the subsequence $ref but
> by the time we get closer to 2000 (and more) this returns a 504 gateway
> time-out error in roughly 60s.
>
>
> Are you running eXist behind a proxy like nginx?  I've had 504s created
> from nginx frontend proxies.  In that case, you need to increase timeouts
> in the nginx config.
>
> Olaf
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

Re: [Exist-open] 504 error in relatively small query

[Ralph C. <ra...@ra...>]

Thanks Olaf, yes we are. I'll take a look at that...

Ralph

On 13/03/2023 14:35, ol...@ex... wrote:

>> This returns absolutely fine up to ~1200 items in the subsequence $ref 
>> but
>> by the time we get closer to 2000 (and more) this returns a 504 
>> gateway
>> time-out error in roughly 60s.
> 
> Are you running eXist behind a proxy like nginx?  I've had 504s created
> from nginx frontend proxies.  In that case, you need to increase 
> timeouts
> in the nginx config.
> 
> Olaf
> 
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-open] 504 error in relatively small query

[<ol...@ex...>]

> This returns absolutely fine up to ~1200 items in the subsequence $ref but
> by the time we get closer to 2000 (and more) this returns a 504 gateway
> time-out error in roughly 60s.

Are you running eXist behind a proxy like nginx?  I've had 504s created
from nginx frontend proxies.  In that case, you need to increase timeouts
in the nginx config.

Olaf

[Exist-open] 504 error in relatively small query

[Ralph C. <ra...@ra...>]

Hi all, I was going to post this in an xQuery forum, but at small 
numbers this query works fine and I feel confident that the issue here 
is in my exist-db config.

I'm trying to return orphaned <ref> elements from a recent update that 
was set up wrong. I'm using the following query to get the results:

> for $ref in 
> subsequence(collection('/db/apps/****/data')//ref[@type='2'],1,####)
> 
> where 
> not(exists(doc('/db/apps/****/data/listPerson.xml')//person[@xml:id=$ref/@target]))
> 
> return
> $ref

This returns absolutely fine up to ~1200 items in the subsequence $ref 
but by the time we get closer to 2000 (and more) this returns a 504 
gateway time-out error in roughly 60s. This does not seem to me to be a 
significantly onerous task (especially considering other things that are 
handled well).

I've tried

declare option exist:timeout "600000" (which should be ten minutes)

to extend the timeout, but I'm getting the same error after ~60s.

Any ideas, as the total I'm trying to get to is ~12000 records of which 
I have no idea how many are orphaned....

Cheers

Ralph

Re: [Exist-open] How to fix Security Database corruption?

[Dominic L. <dom...@gm...>]

Dear Vincent,

Thanks for raising this. I've had this problem for a while too. It's easy
enough to ignore but would be better to resolve. I'm on eXist 5.4.1.

With best wishes,
Dominic


On Thu, 2 Mar 2023 at 07:58, Lehtinen Outi (Kotus) <out...@ko...>
wrote:

> Hi,
>
>
>
> I don’t know if this helps, but I’ve seen this happen once or twice with
> eXist 5.2.0. But at the time I didn’t find anything relevant in the log
> files.
>
>
>
> -Outi
>
> --
>
> Outi Lehtinen, IT specialist
>
> The Institute for the Languages of Finland
>
> Hakaniemenranta 6, 00530 Helsinki
>
>
>
> *Lähettäjä:* Loren Cahlander <lor...@gm...>
> *Lähetetty:* keskiviikko 1. maaliskuuta 2023 23.20
> *Vastaanottaja:* Lizzi, Vincent <Vin...@ta...>
> *Kopio:* Exist-open <exi...@li...>
> *Aihe:* Re: [Exist-open] How to fix Security Database corruption?
>
>
>
> I found the following
>
>
>
> I found a flaw in the restore of the /db/system/security collection.  If
> I do a restore of the whole collection, the accounts are being restored
> first and then the groups do not get restored.  I fixed it by restoring
> /db/system/security/exist/groups and then restored /db/system/security.
> This is in 6.2.0
>
>
>
>
>
> I need to figure out how far back this is before I write a bug report.  I
> need to take the time to test in previous versions.
>
>
>
>
>
> On Mar 1, 2023, at 3:30 PM, Lizzi, Vincent <
> Vin...@ta...> wrote:
>
>
>
> Hello eXist community,
>
>
>
> I have had several instances where an eXist database’s user accounts and
> groups revert to the defaults, wiping out the user accounts and groups that
> I’ve created and resetting the admin password to the default (blank). This
> just happened again. There is no indication in the logs about what happened
> except that the database shut down at the time of a planned operating
> system restart, started up again, and then there are errors about login
> attempts failing. In User Manager the accounts and groups are missing. The
> collection /db/system/security/exist still contains an XML document for
> each of the user accounts and groups, however attempts to login fail. In
> previous times this problem occurred the log file exist.log contained error
> messages such as this:
>
>
>
> 2023-01-12 18:07:24,049 [qtp897892154-30] FATAL (UnixStylePermission.java
> [getGroup]:175) - Detected a Security Database corruption. Could not find
> group for id: 14.
>
> 2023-01-12 18:07:24,049 [qtp897892154-30] FATAL (UnixStylePermission.java
> [getOwner]:93) - Detected a Security Database corruption. Could not find
> account for id: 14.
>
>
>
> To repair the problem, I recreate the users and groups. However, the
> problem just occurs again.
>
>
>
> Here are some details about the environment:
>
>
>
> eXist Version:     6.0.1
>
> eXist Build:          20220209234135
>
> Operating System:           Windows Server 2012 R2 6.3 amd64
>
> Java Version:      11.0.16.1
>
>
>
> Is there any way to resolve this problem?
>
>
>
> Thanks,
>
> Vincent
>
>
>
> ______________________________________________
>
> *Vincent M. Lizzi*
>
> Head of Information Standards | Taylor & Francis Group
>
> 530 Walnut St., Suite 850, Philadelphia, PA 19106
>
> E-Mail: vin...@ta...
>
> Web: www.tandfonline.com
> <https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.tandfonline.com%2F&data=05%7C01%7Couti.lehtinen%40kotus.fi%7Cd11e72d80e65419bae0908db1a9acc25%7C7c14dfa4c0fc47259f0476a443deb095%7C0%7C0%7C638133024374611103%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MUWBuxZ9jxg6ur3FV%2BPrRsFUDHTqVPJh87hXv3Ds1lA%3D&reserved=0>
>
>
>
> Taylor & Francis is a trading name of Informa UK Limited,
>
> registered in England under no. 1072954
>
>
>
> "Everything should be made as simple as possible, but not simpler."
>
>
>
>
>
> Information Classification: General
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
> <https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fexist-open&data=05%7C01%7Couti.lehtinen%40kotus.fi%7Cd11e72d80e65419bae0908db1a9acc25%7C7c14dfa4c0fc47259f0476a443deb095%7C0%7C0%7C638133024374611103%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=TyGIi5R5SS8U19tD3%2BVAy03jjGqD0XxYBQ7Vxv9OYrs%3D&reserved=0>
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

Re: [Exist-open] How to fix Security Database corruption?

[Lehtinen O. (Kotus) <out...@ko...>]

Hi,

I don't know if this helps, but I've seen this happen once or twice with eXist 5.2.0. But at the time I didn't find anything relevant in the log files.

-Outi
--
Outi Lehtinen, IT specialist
The Institute for the Languages of Finland
Hakaniemenranta 6, 00530 Helsinki

Lähettäjä: Loren Cahlander <lor...@gm...>
Lähetetty: keskiviikko 1. maaliskuuta 2023 23.20
Vastaanottaja: Lizzi, Vincent <Vin...@ta...>
Kopio: Exist-open <exi...@li...>
Aihe: Re: [Exist-open] How to fix Security Database corruption?

I found the following

I found a flaw in the restore of the /db/system/security collection.  If I do a restore of the whole collection, the accounts are being restored first and then the groups do not get restored.  I fixed it by restoring /db/system/security/exist/groups and then restored /db/system/security.  This is in 6.2.0



I need to figure out how far back this is before I write a bug report.  I need to take the time to test in previous versions.


On Mar 1, 2023, at 3:30 PM, Lizzi, Vincent <Vin...@ta...<mailto:Vin...@ta...>> wrote:

Hello eXist community,

I have had several instances where an eXist database's user accounts and groups revert to the defaults, wiping out the user accounts and groups that I've created and resetting the admin password to the default (blank). This just happened again. There is no indication in the logs about what happened except that the database shut down at the time of a planned operating system restart, started up again, and then there are errors about login attempts failing. In User Manager the accounts and groups are missing. The collection /db/system/security/exist still contains an XML document for each of the user accounts and groups, however attempts to login fail. In previous times this problem occurred the log file exist.log contained error messages such as this:

2023-01-12 18:07:24,049 [qtp897892154-30] FATAL (UnixStylePermission.java [getGroup]:175) - Detected a Security Database corruption. Could not find group for id: 14.
2023-01-12 18:07:24,049 [qtp897892154-30] FATAL (UnixStylePermission.java [getOwner]:93) - Detected a Security Database corruption. Could not find account for id: 14.

To repair the problem, I recreate the users and groups. However, the problem just occurs again.

Here are some details about the environment:

eXist Version:     6.0.1
eXist Build:          20220209234135
Operating System:           Windows Server 2012 R2 6.3 amd64
Java Version:      11.0.16.1

Is there any way to resolve this problem?

Thanks,
Vincent

______________________________________________
Vincent M. Lizzi
Head of Information Standards | Taylor & Francis Group
530 Walnut St., Suite 850, Philadelphia, PA 19106
E-Mail: vin...@ta...<mailto:vin...@ta...>
Web: www.tandfonline.com<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.tandfonline.com%2F&data=05%7C01%7Couti.lehtinen%40kotus.fi%7Cd11e72d80e65419bae0908db1a9acc25%7C7c14dfa4c0fc47259f0476a443deb095%7C0%7C0%7C638133024374611103%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=MUWBuxZ9jxg6ur3FV%2BPrRsFUDHTqVPJh87hXv3Ds1lA%3D&reserved=0>

Taylor & Francis is a trading name of Informa UK Limited,
registered in England under no. 1072954

"Everything should be made as simple as possible, but not simpler."




Information Classification: General
_______________________________________________
Exist-open mailing list
Exi...@li...<mailto:Exi...@li...>
https://lists.sourceforge.net/lists/listinfo/exist-open<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.sourceforge.net%2Flists%2Flistinfo%2Fexist-open&data=05%7C01%7Couti.lehtinen%40kotus.fi%7Cd11e72d80e65419bae0908db1a9acc25%7C7c14dfa4c0fc47259f0476a443deb095%7C0%7C0%7C638133024374611103%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=TyGIi5R5SS8U19tD3%2BVAy03jjGqD0XxYBQ7Vxv9OYrs%3D&reserved=0>

Re: [Exist-open] How to fix Security Database corruption?

[Loren C. <lor...@gm...>]

I found the following

I found a flaw in the restore of the /db/system/security collection.  If I do a restore of the whole collection, the accounts are being restored first and then the groups do not get restored.  I fixed it by restoring /db/system/security/exist/groups and then restored /db/system/security.  This is in 6.2.0


I need to figure out how far back this is before I write a bug report.  I need to take the time to test in previous versions.


On Mar 1, 2023, at 3:30 PM, Lizzi, Vincent <Vin...@ta...> wrote:

Hello eXist community,
 
I have had several instances where an eXist database’s user accounts and groups revert to the defaults, wiping out the user accounts and groups that I’ve created and resetting the admin password to the default (blank). This just happened again. There is no indication in the logs about what happened except that the database shut down at the time of a planned operating system restart, started up again, and then there are errors about login attempts failing. In User Manager the accounts and groups are missing. The collection /db/system/security/exist still contains an XML document for each of the user accounts and groups, however attempts to login fail. In previous times this problem occurred the log file exist.log contained error messages such as this:
 
2023-01-12 18:07:24,049 [qtp897892154-30] FATAL (UnixStylePermission.java [getGroup]:175) - Detected a Security Database corruption. Could not find group for id: 14.
2023-01-12 18:07:24,049 [qtp897892154-30] FATAL (UnixStylePermission.java [getOwner]:93) - Detected a Security Database corruption. Could not find account for id: 14.
 
To repair the problem, I recreate the users and groups. However, the problem just occurs again.
 
Here are some details about the environment:
 
eXist Version:     6.0.1
eXist Build:          20220209234135
Operating System:           Windows Server 2012 R2 6.3 amd64
Java Version:      11.0.16.1
 
Is there any way to resolve this problem?
 
Thanks,
Vincent
 
______________________________________________
Vincent M. Lizzi
Head of Information Standards | Taylor & Francis Group
530 Walnut St., Suite 850, Philadelphia, PA 19106
E-Mail: vin...@ta... <mailto:vin...@ta...>
Web: www.tandfonline.com <http://www.tandfonline.com/>
 
Taylor & Francis is a trading name of Informa UK Limited,
registered in England under no. 1072954
 
"Everything should be made as simple as possible, but not simpler."
 

Information Classification: General

_______________________________________________
Exist-open mailing list
Exi...@li... <mailto:Exi...@li...>
https://lists.sourceforge.net/lists/listinfo/exist-open

[Exist-open] How to fix Security Database corruption?

[Lizzi, V. <Vin...@ta...>]

Hello eXist community,

I have had several instances where an eXist database's user accounts and groups revert to the defaults, wiping out the user accounts and groups that I've created and resetting the admin password to the default (blank). This just happened again. There is no indication in the logs about what happened except that the database shut down at the time of a planned operating system restart, started up again, and then there are errors about login attempts failing. In User Manager the accounts and groups are missing. The collection /db/system/security/exist still contains an XML document for each of the user accounts and groups, however attempts to login fail. In previous times this problem occurred the log file exist.log contained error messages such as this:

2023-01-12 18:07:24,049 [qtp897892154-30] FATAL (UnixStylePermission.java [getGroup]:175) - Detected a Security Database corruption. Could not find group for id: 14.
2023-01-12 18:07:24,049 [qtp897892154-30] FATAL (UnixStylePermission.java [getOwner]:93) - Detected a Security Database corruption. Could not find account for id: 14.

To repair the problem, I recreate the users and groups. However, the problem just occurs again.

Here are some details about the environment:

eXist Version:     6.0.1
eXist Build:          20220209234135
Operating System:           Windows Server 2012 R2 6.3 amd64
Java Version:      11.0.16.1

Is there any way to resolve this problem?

Thanks,
Vincent

______________________________________________
Vincent M. Lizzi
Head of Information Standards | Taylor & Francis Group
530 Walnut St., Suite 850, Philadelphia, PA 19106
E-Mail: vin...@ta...<mailto:vin...@ta...>
Web: www.tandfonline.com<http://www.tandfonline.com>

Taylor & Francis is a trading name of Informa UK Limited,
registered in England under no. 1072954

"Everything should be made as simple as possible, but not simpler."



Information Classification: General

Re: [Exist-open] ft:query facets and options

[Gus R. <gu...@gm...>]

Thanks, Boris!

That is exactly what I was looking for. I have something like this and it
works:

$query_options := map {
"facets": map {
"script": ($scriptParam),
"condition": ($conditionParam),
},
"fields": ('place', 'material'),
"leading-wildcard": "yes"
}

On Fri, Feb 24, 2023 at 4:11 PM Boris Lehečka <bo...@da...> wrote:

>     Hello Gus,
>
>     take a look at
> https://github.com/eeditiones/tei-publisher-app/blob/master/modules/query.xql,
> TEI Publushers uses json format for options, as you can see here
>
>     declare variable $query:QUERY_OPTIONS := map {
>     "leading-wildcard": "yes",
>     "filter-rewrite": "yes"
>     };
>
>     and here:
>
>     declare function query:options($sortBy as xs:string*) {
>     map:merge((
>         $query:QUERY_OPTIONS,
>         map {
>             "facets":
>                 map:merge((
>                     for $param in
> request:get-parameter-names()[starts-with(., 'facet-')]
>                     let $dimension := substring-after($param, 'facet-')
>                     return
>                         map {
>                             $dimension: request:get-parameter($param, ())
>                         }
>                 ))
>         },
>         if ($sortBy) then
>             map { "fields": ($sortBy, $config:default-fields) }
>         else
>             map { "fields": $config:default-fields }
>         ))
>     };
>
>    I would also prefer the <option> in XML, maybe someone else can help.
>
>     Boris
> Dne 24.02.2023 v 15:11 Gus Riva napsal(a):
>
> Hi!
>
> Hope someone can help me. I am trying to do a full text query using the
> lucene index with facets and ft:query. The documentation says that the 3°
> parameter in the query can be the options as an xml fragment. For example:
> let $options := <options> <query-analyzer-id>ws</query-analyzer-id> <
> default-operator>and</default-operator> <phrase-slop>1</phrase-slop> <
> leading-wildcard>no</leading-wildcard> <filter-rewrite>yes</filter-rewrite
> > <lowercase-expanded-terms>yes</lowercase-expanded-terms> </options>
>
> Described here:
>
> https://exist-db.org/exist/apps/fundocs/view.html?uri=http://exist-db.org/xquery/lucene&location=java:org.exist.xquery.modules.lucene.LuceneModule#query.3
>
> And towards the end of:
> https://exist-db.org/exist/apps/doc/lucene
>
> This is however not working for me.
> The 3° parameter of ft:query can also be used with a map for facets and
> fields, described also in the documentation. Like this:
> let $options := map { "facets": map { "keyword": ("indexing", "facets"),
> "date": [("2018", "06"), ("2018", "05")] } }
>
> This works for me.
>
> Ideally, I would like to do both: use the facets and fields argument, and
> change the options. Is there a way to do that? Has the options function
> been deprecated?
>
> Thanks for your help!
>
> Best,
> Gustavo Fernández Riva
> Universität Heidelberg
>
>
> _______________________________________________
> Exist-open mailing lis...@li...://lists.sourceforge.net/lists/listinfo/exist-open
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

Re: [Exist-open] ft:query facets and options

[Boris L. <bo...@da...>]

     Hello Gus,

     take a look at 
https://github.com/eeditiones/tei-publisher-app/blob/master/modules/query.xql, 
TEI Publushers uses json format for options, as you can see here

     declare variable $query:QUERY_OPTIONS := map {
     "leading-wildcard": "yes",
     "filter-rewrite": "yes"
     };

     and here:

     declare function query:options($sortBy as xs:string*) {
     map:merge((
         $query:QUERY_OPTIONS,
         map {
             "facets":
                 map:merge((
                     for $param in 
request:get-parameter-names()[starts-with(., 'facet-')]
                     let $dimension := substring-after($param, 'facet-')
                     return
                         map {
                             $dimension: request:get-parameter($param, ())
                         }
                 ))
         },
         if ($sortBy) then
             map { "fields": ($sortBy, $config:default-fields) }
         else
             map { "fields": $config:default-fields }
         ))
     };

    I would also prefer the <option> in XML, maybe someone else can help.

     Boris

Dne 24.02.2023 v 15:11 Gus Riva napsal(a):
> Hi!
>
> Hope someone can help me. I am trying to do a full text query using 
> the lucene index with facets and ft:query. The documentation says that 
> the 3° parameter in the query can be the options as an xml fragment. 
> For example:
> let$options:= <options> 
> <query-analyzer-id>ws</query-analyzer-id><default-operator>and</default-operator><phrase-slop>1</phrase-slop><leading-wildcard>no</leading-wildcard><filter-rewrite>yes</filter-rewrite><lowercase-expanded-terms>yes</lowercase-expanded-terms></options>
>
> Described here:
> https://exist-db.org/exist/apps/fundocs/view.html?uri=http://exist-db.org/xquery/lucene&location=java:org.exist.xquery.modules.lucene.LuceneModule#query.3 
> <https://exist-db.org/exist/apps/fundocs/view.html?uri=http://exist-db.org/xquery/lucene&location=java:org.exist.xquery.modules.lucene.LuceneModule#query.3>
>
> And towards the end of:
> https://exist-db.org/exist/apps/doc/lucene
>
> This is however not working for me.
> The 3° parameter of ft:query can also be used with a map for facets 
> and fields, described also in the documentation. Like this:
> let$options:= map{ "facets": map{ "keyword": ("indexing", "facets"), 
> "date": [("2018", "06"), ("2018", "05")] } }
>
> This works for me.
>
> Ideally, I would like to do both: use the facets and fields argument, 
> and change the options. Is there a way to do that? Has the options 
> function been deprecated?
>
> Thanks for your help!
>
> Best,
> Gustavo Fernández Riva
> Universität Heidelberg
>
>
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open

[Exist-open] ft:query facets and options

[Gus R. <gu...@gm...>]

Hi!

Hope someone can help me. I am trying to do a full text query using the
lucene index with facets and ft:query. The documentation says that the 3°
parameter in the query can be the options as an xml fragment. For example:
let $options := <options> <query-analyzer-id>ws</query-analyzer-id> <
default-operator>and</default-operator> <phrase-slop>1</phrase-slop> <
leading-wildcard>no</leading-wildcard> <filter-rewrite>yes</filter-rewrite>
<lowercase-expanded-terms>yes</lowercase-expanded-terms> </options>

Described here:
https://exist-db.org/exist/apps/fundocs/view.html?uri=http://exist-db.org/xquery/lucene&location=java:org.exist.xquery.modules.lucene.LuceneModule#query.3

And towards the end of:
https://exist-db.org/exist/apps/doc/lucene

This is however not working for me.
The 3° parameter of ft:query can also be used with a map for facets and
fields, described also in the documentation. Like this:
let $options := map { "facets": map { "keyword": ("indexing", "facets"),
"date": [("2018", "06"), ("2018", "05")] } }

This works for me.

Ideally, I would like to do both: use the facets and fields argument, and
change the options. Is there a way to do that? Has the options function
been deprecated?

Thanks for your help!

Best,
Gustavo Fernández Riva
Universität Heidelberg

Re: [Exist-open] implicit validation

[Eduard D. <ed...@fr...>]

rng validation (via catalog setup) isn't working in exide:

[cid:cfb3c00a-e3b1-4f67-9946-8c21aca8b828]

Not too big of a problem, just to let you know


Eduard Drenth, Software Architekt


ed...@fr...


Doelestrjitte 8

8911 DX  Ljouwert

(058) 213 14 14

+31 62 094 34 28

https://www.fryske-akademy.nl/
https://frysker.nl/
https://frisian.eu/
[cid:b750b829-1537-4088-9d43-82260d6e4167]

gpg: pgp.key-server.io/pks/lookup?search=eduarddrenth

________________________________
From: Eduard Drenth <ed...@fr...>
Sent: Wednesday, February 22, 2023 1:21 PM
To: Dannes Wessels <di...@ex...>
Cc: Exist-open <exi...@li...>; Joe Wicentowski <jo...@gm...>
Subject: Re: [Exist-open] implicit validation

Sorry, did not see rng working in exide, xsd still used n my test.

answer to my question: schematron rules aren't applied when using validation:jing-report




Eduard Drenth, Software Architekt


ed...@fr...


Doelestrjitte 8

8911 DX  Ljouwert

(058) 213 14 14

+31 62 094 34 28

https://www.fryske-akademy.nl/
https://frysker.nl/
https://frisian.eu/
[cid:3d23ab6d-f5de-485a-b5d1-25f700329013]

gpg: pgp.key-server.io/pks/lookup?search=eduarddrenth

________________________________
From: Eduard Drenth <ed...@fr...>
Sent: Wednesday, February 22, 2023 1:02 PM
To: Dannes Wessels <di...@ex...>
Cc: Joe Wicentowski <jo...@gm...>; Exist-open <exi...@li...>
Subject: Re: [Exist-open] implicit validation

Thanks, rng works in exide, embedded schematron rules aren't applied so to see?



Eduard Drenth, Software Architekt


ed...@fr...


Doelestrjitte 8

8911 DX  Ljouwert

(058) 213 14 14

+31 62 094 34 28

https://www.fryske-akademy.nl/
https://frysker.nl/
https://frisian.eu/
[cid:80248d0e-b139-4bcb-aea9-b63724b39ee2]

gpg: pgp.key-server.io/pks/lookup?search=eduarddrenth

________________________________
From: Dannes Wessels <di...@ex...>
Sent: Tuesday, February 21, 2023 1:10 PM
To: Eduard Drenth <ed...@fr...>
Cc: Joe Wicentowski <jo...@gm...>; Exist-open <exi...@li...>
Subject: Re: [Exist-open] implicit validation

For explicit validation yes, implicit no

Regards

Dannes

On 21 Feb 2023, at 08:38, Eduard Drenth <ed...@fr...> wrote:


Thanks, looks like I can use .rng in a catalog?




Eduard Drenth,

[Exist-open] seems that xquery can't deal with some kind of PI

[Jens T. <tis...@do...>]

Hello everybody,

I'm quite new at exist-open. Until today I just followed the 
conversations here, it's quite interesting for me.

But now I hope that someone is able to give me a hint, what is goring 
wrong here:

Among others I try to convert CALS tables to HTML, and it worked for years.

But now I got documents with PIs in there, ...

... and my code ...

... gets null pointer exception, when reaching PIs. As far as I know is 
node type 7 the  PI when SAX parsing.

The problem is solved, if I deleted the PIs:

Even a little example, I wrote, is still able to cope with PIs and their 
name():

But not the table example above.

Has anybody experience?

I'm using eXist-db 5.1.1 on Linux.

Many thanks in advance.

Regards
Jens


-- 
__________________________________________________________________________
<> DOSCO Document Systems Consulting GmbH
Jens Tischler

phone +49 162 4664334
      (+49 6221 1486 25)
mai...@do...
webhttp://www.dosco.de

       DOSCO Document Systems Consulting GmbH
       Mannheimer Strasse 1
       69115 Heidelberg, GERMANY

       Handelsregister:  Mannheim HRB 335122
       Geschäftsführung: Robert Erfle

Re: [Exist-open] implicit validation

[Eduard D. <ed...@fr...>]

Sorry, did not see rng working in exide, xsd still used n my test.

answer to my question: schematron rules aren't applied when using validation:jing-report




Eduard Drenth, Software Architekt


ed...@fr...


Doelestrjitte 8

8911 DX  Ljouwert

(058) 213 14 14

+31 62 094 34 28

https://www.fryske-akademy.nl/
https://frysker.nl/
https://frisian.eu/
[cid:3d23ab6d-f5de-485a-b5d1-25f700329013]

gpg: pgp.key-server.io/pks/lookup?search=eduarddrenth

________________________________
From: Eduard Drenth <ed...@fr...>
Sent: Wednesday, February 22, 2023 1:02 PM
To: Dannes Wessels <di...@ex...>
Cc: Joe Wicentowski <jo...@gm...>; Exist-open <exi...@li...>
Subject: Re: [Exist-open] implicit validation

Thanks, rng works in exide, embedded schematron rules aren't applied so to see?



Eduard Drenth, Software Architekt


ed...@fr...


Doelestrjitte 8

8911 DX  Ljouwert

(058) 213 14 14

+31 62 094 34 28

https://www.fryske-akademy.nl/
https://frysker.nl/
https://frisian.eu/
[cid:80248d0e-b139-4bcb-aea9-b63724b39ee2]

gpg: pgp.key-server.io/pks/lookup?search=eduarddrenth

________________________________
From: Dannes Wessels <di...@ex...>
Sent: Tuesday, February 21, 2023 1:10 PM
To: Eduard Drenth <ed...@fr...>
Cc: Joe Wicentowski <jo...@gm...>; Exist-open <exi...@li...>
Subject: Re: [Exist-open] implicit validation

For explicit validation yes, implicit no

Regards

Dannes

On 21 Feb 2023, at 08:38, Eduard Drenth <ed...@fr...> wrote:


Thanks, looks like I can use .rng in a catalog?




Eduard Drenth,

Re: [Exist-open] implicit validation

[Eduard D. <ed...@fr...>]

Thanks, rng works in exide, embedded schematron rules aren't applied so to see?



Eduard Drenth, Software Architekt


ed...@fr...


Doelestrjitte 8

8911 DX  Ljouwert

(058) 213 14 14

+31 62 094 34 28

https://www.fryske-akademy.nl/
https://frysker.nl/
https://frisian.eu/
[cid:80248d0e-b139-4bcb-aea9-b63724b39ee2]

gpg: pgp.key-server.io/pks/lookup?search=eduarddrenth

________________________________
From: Dannes Wessels <di...@ex...>
Sent: Tuesday, February 21, 2023 1:10 PM
To: Eduard Drenth <ed...@fr...>
Cc: Joe Wicentowski <jo...@gm...>; Exist-open <exi...@li...>
Subject: Re: [Exist-open] implicit validation

For explicit validation yes, implicit no

Regards

Dannes

On 21 Feb 2023, at 08:38, Eduard Drenth <ed...@fr...> wrote:


Thanks, looks like I can use .rng in a catalog?




Eduard Drenth,

Re: [Exist-open] implicit validation

[Dannes W. <di...@ex...>]

For explicit validation yes, implicit no

Regards

Dannes 

> On 21 Feb 2023, at 08:38, Eduard Drenth <ed...@fr...> wrote:
> 
> 
> Thanks, looks like I can use .rng in a catalog?
> 
> 
> 
> Eduard Drenth,

Re: [Exist-open] implicit validation

[Eduard D. <ed...@fr...>]

Thanks, looks like I can use .rng in a catalog?




Eduard Drenth, Software Architekt


ed...@fr...


Doelestrjitte 8

8911 DX  Ljouwert

(058) 213 14 14

+31 62 094 34 28

https://www.fryske-akademy.nl/
https://frysker.nl/
https://frisian.eu/
[cid:45ab8689-c29e-49f8-a81e-21989d2ee00c]

gpg: pgp.key-server.io/pks/lookup?search=eduarddrenth

________________________________
From: Joe Wicentowski <jo...@gm...>
Sent: Monday, February 20, 2023 11:00 PM
To: Eduard Drenth <ed...@fr...>
Cc: Exist-open <exi...@li...>
Subject: Re: [Exist-open] implicit validation

Hi Eduard,

eXist rejects attempts to store only non-well-formed documents. Failure to adhere to a schema will not trigger a rejection.

eXide - at the application level - performs some limited schema checking - of .xconf files. See:

- Validation code: https://github.com/eXist-db/eXide/blob/develop/modules/validate-xml.xq
- Catalog: https://github.com/eXist-db/eXide/blob/develop/resources/schema/catalog.xml

Joe

On Mon, Feb 20, 2023 at 10:50 AM Eduard Drenth <ed...@fr...<mailto:ed...@fr...>> wrote:
Dear all,

When validation is configured uploading invalid documents (via eXide) seems possible. Is that correct and by design?

Can someone give an overview of when documents are (not) validated?

Regards,


Eduard Drenth, Software Architekt


ed...@fr...<mailto:ed...@fr...>


Doelestrjitte 8

8911 DX  Ljouwert

(058) 213 14 14

+31 62 094 34 28

https://www.fryske-akademy.nl/
https://frysker.nl/
https://frisian.eu/
[cid:18670d3c07833a855901]

gpg: pgp.key-server.io/pks/lookup?search=eduarddrenth<http://pgp.key-server.io/pks/lookup?search=eduarddrenth>

_______________________________________________
Exist-open mailing list
Exi...@li...<mailto:Exi...@li...>
https://lists.sourceforge.net/lists/listinfo/exist-open

Re: [Exist-open] implicit validation

[Joe W. <jo...@gm...>]

Hi Eduard,

eXist rejects attempts to store only non-well-formed documents. Failure to
adhere to a schema will not trigger a rejection.

eXide - at the application level - performs some limited schema checking -
of .xconf files. See:

- Validation code:
https://github.com/eXist-db/eXide/blob/develop/modules/validate-xml.xq
- Catalog:
https://github.com/eXist-db/eXide/blob/develop/resources/schema/catalog.xml

Joe

On Mon, Feb 20, 2023 at 10:50 AM Eduard Drenth <ed...@fr...>
wrote:

> Dear all,
>
> When validation is configured uploading invalid documents (via eXide)
> seems possible. Is that correct and by design?
>
> Can someone give an overview of when documents are (not) validated?
>
> Regards,
>
> Eduard Drenth, Software Architekt
>
>
> ed...@fr...
>
>
> Doelestrjitte 8
>
> 8911 DX  Ljouwert
> (058) 213 14 14
>
> +31 62 094 34 28
> https://www.fryske-akademy.nl/
> https://frysker.nl/
> https://frisian.eu/
>
> gpg: pgp.key-server.io/pks/lookup?search=eduarddrenth
> _______________________________________________
> Exist-open mailing list
> Exi...@li...
> https://lists.sourceforge.net/lists/listinfo/exist-open
>

[Exist-open] implicit validation

[Eduard D. <ed...@fr...>]

Dear all,

When validation is configured uploading invalid documents (via eXide) seems possible. Is that correct and by design?

Can someone give an overview of when documents are (not) validated?

Regards,


Eduard Drenth, Software Architekt


ed...@fr...


Doelestrjitte 8

8911 DX  Ljouwert

(058) 213 14 14

+31 62 094 34 28

https://www.fryske-akademy.nl/
https://frysker.nl/
https://frisian.eu/
[cid:bc5c924b-d605-442b-b3c5-266b8fc6e65a]

gpg: pgp.key-server.io/pks/lookup?search=eduarddrenth

[Exist-open] trigger validation

[Eduard D. <ed...@fr...>]

Dear all,

I want to implement validating a document via a before-update-document trigger.

When validation fails I want the update to fail and I want to provide the caller with feedback.

I wrote this function:


declare function local:validate($uri as xs:string) {
    let $report := validation:jing-report(xs:anyURI($uri), doc("xmldb:exist:///db/apps/corpora/schematron/corpora_linguistics.sch"))
    return if ($report/status='invalid') then
        (
            util:log('warn',$report/message),
            error(QName('http://corpora.fa','invalid'))
        )
    else ()
};

It does the job sort of, but:

  1.  the currently stored document is validated, not the user input
  2.  I found no option to get $report/message back to the user (in eXide)

I could perhaps remember the current document in before-update-document and when validation fails in after-update-document restore the documment, but that seems a bit tricky perhaps.

Does any of you have experience with such a scenario?


Eduard Drenth, Software Architekt


ed...@fr...


Doelestrjitte 8

8911 DX  Ljouwert

(058) 213 14 14

+31 62 094 34 28

https://www.fryske-akademy.nl/
https://frysker.nl/
https://frisian.eu/
[cid:b16880ff-4d10-476b-8d0b-2082bf54e33f]

gpg: pgp.key-server.io/pks/lookup?search=eduarddrenth