Re: [Geoserver-users] Image Mosaic REST harvesting - failure after time passes

[Mike G. <d.m...@gm...>]

Ben,

Same machine; localhost connection.

So, I'm not sure where the idle connection is getting severed, if it even
is.  Everything seems to be suggesting that it is ... though ... with the
passage of time seeming to cause the problem and database (JDBC) issues in
the error logs.

Thanks,

Mike



On Mon, May 4, 2015 at 12:18 AM, Ben Caradoc-Davies <be...@tr...>
wrote:

> Mike,
>
> are tomcat and postgres on the same machine or is there a firewall between
> them?
>
> I have seen Cisco(?) firewalls silently sever idle connections after an
> hour or so; the situation could not be recovered by connection validation.
> Note that this was app-schema WFS output and I have not used imagemosaic,
> but if this is a firewall problem a similar workaround for the database
> connection might help. Here is a JNDI Context that shuts down idle
> connections after 5 to 10 minutes:
>
> http://docs.geoserver.org/latest/en/user/data/app-schema/data-stores.html#jndi
>
> "Firewall timeouts can silently sever idle connections to the database and
> cause GeoServer to hang. If there is a firewall between GeoServer and the
> database, a connection pool configured to shut down idle connections before
> the firewall can drop them will prevent GeoServer from hanging. This JNDI
> connection pool is configured to shut down idle connections after 5 to 10
> minutes."
>
> Kind regards,
> Ben.
>
>
> On 01/05/15 13:40, Mike Grogan wrote:
>
>> (GeoServer 2.6.2, Postgres 9.3.6; Ubuntu 14.04; JNDI pools via Tomcat)
>>
>> I am experiencing another hard to track down issue with harvesting /
>> creating new image mosaics via REST ... that seems to be occurring after
>> time passes.
>>
>> What I mean is that harvesting & creating new mosaics works fine most of
>> the time.  However, it seems to fail after some amount of time passes.
>> For
>> instance, if I have not created a new mosaic via REST for a day or 2, and
>> I
>> try to create one, it will fail.  If I then try to create several more via
>> REST, again, they all fail.
>>
>> If I restart Tomcat or postgres and then run the very same REST call (via
>> gsconfig in python) against the very same data, voila, it is successful.
>> After restart, I can then go on creating multiple new mosaics via REST
>> over
>> the course of many hours.
>>
>> Let some time pass, though, and I'll start seeing failures again.
>>
>> Yes, this is a troubleshooting nightmare.
>>
>> My initial thoughts have been this has something to do with stale
>> connections via JNDI ... but I have validation turned on, both in JNDI
>> Tomcat Resource and in datastore.properties.  I have tried leaving in the
>> eviction settings (shown in some of the JNDI Image Mosaic examples) and
>> removing them ... but this still doesn't help.
>>
>> What's odd is that I have no problems accessing existing mosaic indexes
>> backed via JNDI.  It just seems the ability to create new ones goes bad
>> after some time passes (usually 24 hours or so).
>>
>> Again, JNDI has been a suspicion.  I kind of also am asking myself if
>> something is somehow getting garbage collected that shouldn't be (I know
>> that sounds strange) ... or something else is somehow disappearing over
>> time that would prevent successful mosaic creation via REST.
>>
>> Does this sound familiar to anyone?  Any pointers?
>>
>> The stack trace from when this happens is below.
>>
>> Thanks!
>>
>
> --
> Ben Caradoc-Davies <be...@tr...>
> Director
> Transient Software Limited <http://transient.nz/>
> New Zealand
>