Re: [Exist-open] dom.dbx blowing up in size.

[Jason S. <js...@in...>]

More information.  This appears to be happening in spurts.  We think it's probably the result of some particular user operation.  Why this is making the dom.dbx file grow, though, I don't understand.  We're trying to correlate the growth in size to something in our logs, so maybe that will prove fruitful.

(0) forgot to ask if you were using lucene as well

Yes, we are using Lucene.  This is the 1.4.0 release version of eXist.

I'm getting ready to look more closely at the indexing configuration.  I'll send that along with conf.xml if I find anything interesting.

(1) Is there any cyclical processing going on that correlates e.g. do you
have a volatile db or growing db ?

The db does not appear to be growing, and we aren't adding much new data.  

(2) Do you use the default conf.xml ?

I'll send it in a separate email.  It's pretty close to the default, but the data has been located outside the webapp.

(3) Is this machine public e.g. could there be some DOS type activity ?

No, eXist is part of the backend.  And pretty well isolated.  And the actual data in the database isn't growing, at least if you believe the backup.

(4)  Do you have any custom java integration working with db ?

Yes.  We've got an alternate RESTful services layer installed into the eXist webapp.  It makes direct calls to the eXist API.

(5) Can you comment on your environment.

Currently running on Windows, under Tomcat 5.5.  


Jason Smith
________________________________________
From: James Fuller [jam...@gm...]
Sent: Thursday, June 17, 2010 12:32 PM
To: Jason Smith
Cc: ml
Subject: Re: [Exist-open] dom.dbx blowing up in size.

On Thu, Jun 17, 2010 at 8:15 PM, Jason Smith <js...@in...> wrote:
> Is this a known problem?  We're currently attempting to quantify this, and
> any information would be appreciated.
>
> We have a dom.dbx file that normally runs in the 150MB range.  Since
> upgrading to 1.4.0, we've seen it blow up in size, in a short (so far
> indeterminate) amount of time to 4GB, 7GB, and 19GB.  When this happens, the
> JVM appears to be maxed out in memory usage (around 1.7GB in this case), and
> performance slows way down.
>
> In one case, we were able to do a live backup of the data, and the total
> size of the backup was only around 20MB (which is consistent with a 150MB
> dom.dbx).  Upon restore, the database resumes normal operation.
>
> This looks like there is no actual data being added by any direct means when
> this happens, but again, we are still quantifying the problem.

a few questions:

Is there any cyclical processing going on that correlates e.g. do you
have a volatile db or growing db ?

Do you use the default conf.xml ?

What kind of indexes have you setup ?

Is this machine public e.g. could there be some DOS type activity ?

Do you have any custom java integration working with db ?

Can you comment on your environment.

will comment further once I know the answers to the above.

thx, James Fuller