Re: [Wrapper-user] EXCEPTION _ACCESS_VIOLATION

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Mark,
So it looks like once it restarts once, it crashes consistently
between 4 and 4.5 minutes after the new JVM has started
again.

If you look at the task manager, does it look like the memory
usage of the Java process is growing?   One possibility is a
memory leak.  In native C code, it is common to allocate memory
and assume that the returned pointer is valid.  If the system runs
out of memory and the allocation fails however, NULL will be
returned and you will get an ACCESS_VIOLATION if that NULL
pointer is then used.

The following two properties will cause the Wrapper to output
the memory usage of the Wrapper and Java processes once per
second.  Try adding this in addition to the debug output and
then reproduce your problem.   It will show if there are any
memory leaks before the JVM crashes.

wrapper.memory_output=true
wrapper.memory_output.interval=1
wraper.debug=true

If your second JVM or another process is the one consuming all
the memory then this may not give you the information you need
as this process could be running out of memory even though its
own memory usage has not grown that much.  Looking at the
task manager would shed light on this.

You said that this application is encoding video files.   Is it possible
that the system is running fine over time because is it encoding say
files A, B, C, D.  But then when it hits file E it is encountering a
problem where that leads to the crash.  If the system works like a
queue then file E would be retried after a restart so the next crash
would be more immediate.

Cheers,
Leif

no spam wrote:
> Here is what the log looks like.  I will enable debug and try and get 
> more information:
>
> Can you give me details on how the ping works?
>
>
> ERROR | wrapper | 2007/09/25 03:25:31 | The JVM process terminated due 
> to an uncaught exception: EXCEPTION_ACCESS_VIOLATION (0xc0000005)
> ERROR | wrapper | 2007/09/25 03:25:31 | JVM exited unexpectedly.
> STATUS | wrapper | 2007/09/25 03:25:35 | Launching a JVM...
> INFO | jvm 2 | 2007/09/25 03:25:36 | Wrapper (Version 3.2.3) 
> http://wrapper.tanukisoftware.org
> INFO | jvm 2 | 2007/09/25 03:25:36 | Copyright 1999-2006 Tanuki 
> Software, Inc. All Rights Reserved.
> INFO | jvm 2 | 2007/09/25 03:25:36 |
> INFO | jvm 2 | 2007/09/25 03:25:38 |
> INFO | jvm 2 | 2007/09/25 03:25:38 |
> INFO | jvm 2 | 2007/09/25 03:25:38 | J-Integra 2.5 Copyright (C) 
> 1998-2005 Intrinsyc Software International, Inc. 
> http://j-integra.intrinsyc.com/ <http://j-integra.intrinsyc.com/>
> INFO | jvm 2 | 2007/09/25 03:25:38 |
> INFO | jvm 2 | 2007/09/25 03:25:38 | 
> ***************************************************
> INFO | jvm 2 | 2007/09/25 03:25:38 |
> INFO | jvm 2 | 2007/09/25 03:25:38 | Thank you for choosing J-Integra.
> INFO | jvm 2 | 2007/09/25 03:25:38 | Please visit 
> http://j-integra.intrinsyc.com/ for
> INFO | jvm 2 | 2007/09/25 03:25:38 | documentation, support, and 
> purchasing information.
> INFO | jvm 2 | 2007/09/25 03:25:38 |
> INFO | jvm 2 | 2007/09/25 03:25:38 | 
> ***************************************************
> INFO | jvm 2 | 2007/09/25 03:25:38 |
> INFO | jvm 2 | 2007/09/25 03:25:38 |
> ERROR | wrapper | 2007/09/25 03:29:36 | The JVM process terminated due 
> to an uncaught exception: EXCEPTION_ACCESS_VIOLATION (0xc0000005)
> ERROR | wrapper | 2007/09/25 03:29:36 | JVM exited unexpectedly.
> STATUS | wrapper | 2007/09/25 03:29:41 | Launching a JVM...
> INFO | jvm 3 | 2007/09/25 03:29:41 | Wrapper (Version 3.2.3) 
> http://wrapper.tanukisoftware.org
> INFO | jvm 3 | 2007/09/25 03:29:41 | Copyright 1999-2006 Tanuki 
> Software, Inc. All Rights Reserved.
> INFO | jvm 3 | 2007/09/25 03:29:41 |
> INFO | jvm 3 | 2007/09/25 03:29:48 |
> INFO | jvm 3 | 2007/09/25 03:29:48 |
> INFO | jvm 3 | 2007/09/25 03:29:48 | J-Integra 2.5 Copyright (C) 
> 1998-2005 Intrinsyc Software International, Inc. 
> http://j-integra.intrinsyc.com/
> INFO | jvm 3 | 2007/09/25 03:29:48 |
> INFO | jvm 3 | 2007/09/25 03:29:48 | 
> ***************************************************
> INFO | jvm 3 | 2007/09/25 03:29:48 |
> INFO | jvm 3 | 2007/09/25 03:29:48 | Thank you for choosing J-Integra.
> INFO | jvm 3 | 2007/09/25 03:29:48 | Please visit 
> http://j-integra.intrinsyc.com/ for
> INFO | jvm 3 | 2007/09/25 03:29:48 | documentation, support, and 
> purchasing information.
> INFO | jvm 3 | 2007/09/25 03:29:48 |
> INFO | jvm 3 | 2007/09/25 03:29:48 | 
> ***************************************************
> INFO | jvm 3 | 2007/09/25 03:29:48 |
> INFO | jvm 3 | 2007/09/25 03:29:48 |
> ERROR | wrapper | 2007/09/25 03:34:00 | The JVM process terminated due 
> to an uncaught exception: EXCEPTION_ACCESS_VIOLATION (0xc0000005)
> ERROR | wrapper | 2007/09/25 03:34:00 | JVM exited unexpectedly.
> STATUS | wrapper | 2007/09/25 03:34:04 | Launching a JVM...
> INFO | jvm 4 | 2007/09/25 03:34:04 | Wrapper (Version 3.2.3) 
> http://wrapper.tanukisoftware.org
> INFO | jvm 4 | 2007/09/25 03:34:04 | Copyright 1999-2006 Tanuki 
> Software, Inc. All Rights Reserved.
> INFO | jvm 4 | 2007/09/25 03:34:04 |
> INFO | jvm 4 | 2007/09/25 03:34:33 |
> INFO | jvm 4 | 2007/09/25 03:34:33 |
> INFO | jvm 4 | 2007/09/25 03:34:33 | J-Integra 2.5 Copyright (C) 
> 1998-2005 Intrinsyc Software International, Inc. 
> http://j-integra.intrinsyc.com/ <http://j-integra.intrinsyc.com/>
> INFO | jvm 4 | 2007/09/25 03:34:33 |
> INFO | jvm 4 | 2007/09/25 03:34:33 | 
> ***************************************************
> INFO | jvm 4 | 2007/09/25 03:34:33 |
> INFO | jvm 4 | 2007/09/25 03:34:33 | Thank you for choosing J-Integra.
> INFO | jvm 4 | 2007/09/25 03:34:33 | Please visit 
> http://j-integra.intrinsyc.com/ for
> INFO | jvm 4 | 2007/09/25 03:34:33 | documentation, support, and 
> purchasing information.
> INFO | jvm 4 | 2007/09/25 03:34:33 |
> INFO | jvm 4 | 2007/09/25 03:34:33 | 
> ***************************************************
> INFO | jvm 4 | 2007/09/25 03:34:33 |
> INFO | jvm 4 | 2007/09/25 03:34:33 |
> ERROR | wrapper | 2007/09/25 03:38:54 | The JVM process terminated due 
> to an uncaught exception: EXCEPTION_ACCESS_VIOLATION (0xc0000005)
> ERROR | wrapper | 2007/09/25 03:38:54 | JVM exited unexpectedly.
> STATUS | wrapper | 2007/09/25 03:38:58 | Launching a JVM...
> INFO | jvm 5 | 2007/09/25 03:38:58 | Wrapper (Version 3.2.3) 
> http://wrapper.tanukisoftware.org
> INFO | jvm 5 | 2007/09/25 03:38:58 | Copyright 1999-2006 Tanuki 
> Software, Inc. All Rights Reserved.
> INFO | jvm 5 | 2007/09/25 03:38:58 |
> INFO | jvm 5 | 2007/09/25 03:39:01 |
> INFO | jvm 5 | 2007/09/25 03:39:01 |
> INFO | jvm 5 | 2007/09/25 03:39:01 | J-Integra 2.5 Copyright (C) 
> 1998-2005 Intrinsyc Software International, Inc. 
> http://j-integra.intrinsyc.com/
> INFO | jvm 5 | 2007/09/25 03:39:01 |
> INFO | jvm 5 | 2007/09/25 03:39:01 | 
> ***************************************************
> INFO | jvm 5 | 2007/09/25 03:39:01 |
> INFO | jvm 5 | 2007/09/25 03:39:01 | Thank you for choosing J-Integra.
> INFO | jvm 5 | 2007/09/25 03:39:01 | Please visit 
> http://j-integra.intrinsyc.com/ for
> INFO | jvm 5 | 2007/09/25 03:39:01 | documentation, support, and 
> purchasing information.
> INFO | jvm 5 | 2007/09/25 03:39:01 |
> INFO | jvm 5 | 2007/09/25 03:39:01 | 
> ***************************************************
> INFO | jvm 5 | 2007/09/25 03:39:01 |
> INFO | jvm 5 | 2007/09/25 03:39:01 |
> ERROR | wrapper | 2007/09/25 03:42:40 | The JVM process terminated due 
> to an uncaught exception: EXCEPTION_ACCESS_VIOLATION (0xc0000005)
> ERROR | wrapper | 2007/09/25 03:42:40 | JVM exited unexpectedly.
> STATUS | wrapper | 2007/09/25 03:42:44 | Launching a JVM...
> INFO | jvm 6 | 2007/09/25 03:42:45 | Wrapper (Version 3.2.3) 
> http://wrapper.tanukisoftware.org
> INFO | jvm 6 | 2007/09/25 03:42:45 | Copyright 1999-2006 Tanuki 
> Software, Inc. All Rights Reserved.
> INFO | jvm 6 | 2007/09/25 03:42:45 |
> INFO | jvm 6 | 2007/09/25 03:43:48 |
> INFO | jvm 6 | 2007/09/25 03:43:48 |
> INFO | jvm 6 | 2007/09/25 03:43:48 | J-Integra 2.5 Copyright (C) 
> 1998-2005 Intrinsyc Software International, Inc. 
> http://j-integra.intrinsyc.com/ <http://j-integra.intrinsyc.com/>
> INFO | jvm 6 | 2007/09/25 03:43:48 |
> INFO | jvm 6 | 2007/09/25 03:43:48 | 
> ***************************************************
> INFO | jvm 6 | 2007/09/25 03:43:48 |
> INFO | jvm 6 | 2007/09/25 03:43:48 | Thank you for choosing J-Integra.
> INFO | jvm 6 | 2007/09/25 03:43:48 | Please visit 
> http://j-integra.intrinsyc.com/ for
> INFO | jvm 6 | 2007/09/25 03:43:48 | documentation, support, and 
> purchasing information.
> INFO | jvm 6 | 2007/09/25 03:43:48 |
> INFO | jvm 6 | 2007/09/25 03:43:48 | 
> ***************************************************
> INFO | jvm 6 | 2007/09/25 03:43:48 |
> INFO | jvm 6 | 2007/09/25 03:43:48 |
> ERROR | wrapper | 2007/09/25 03:47:24 | The JVM process terminated due 
> to an uncaught exception: EXCEPTION_ACCESS_VIOLATION (0xc0000005)
> ERROR | wrapper | 2007/09/25 03:47:24 | JVM exited unexpectedly.
> FATAL | wrapper | 2007/09/25 03:47:24 | There were 5 failed launches 
> in a row, each lasting less than 300 seconds. Giving up.
> FATAL | wrapper | 2007/09/25 03:47:24 | There may be a configuration 
> problem: please check the logs.
> STATUS | wrapper | 2007/09/25 03:47:24 | <-- Wrapper Stopped
> STATUS | wrapper | 2007/09/25 21:30:48 | --> Wrapper Started as Service
> STATUS | wrapper | 2007/09/25 21:30:49 | Launching a JVM...
> INFO | jvm 1 | 2007/09/25 21:30:50 | Wrapper (Version 3.2.3) 
> http://wrapper.tanukisoftware.org
> INFO | jvm 1 | 2007/09/25 21:30:50 | Copyright 1999-2006 Tanuki 
> Software, Inc. All Rights Reserved.
> INFO | jvm 1 | 2007/09/25 21:30:50 |
> INFO | jvm 1 | 2007/09/25 21:31:06 |
> INFO | jvm 1 | 2007/09/25 21:31:06 |
> INFO | jvm 1 | 2007/09/25 21:31:06 | J-Integra 2.5 Copyright (C) 
> 1998-2005 Intrinsyc Software International, Inc. 
> http://j-integra.intrinsyc.com/
> INFO | jvm 1 | 2007/09/25 21:31:06 |
> INFO | jvm 1 | 2007/09/25 21:31:06 | 
> ***************************************************
> INFO | jvm 1 | 2007/09/25 21:31:06 |
> INFO | jvm 1 | 2007/09/25 21:31:06 | Thank you for choosing J-Integra.
> INFO | jvm 1 | 2007/09/25 21:31:06 | Please visit 
> http://j-integra.intrinsyc.com/ for
> INFO | jvm 1 | 2007/09/25 21:31:06 | documentation, support, and 
> purchasing information.
> INFO | jvm 1 | 2007/09/25 21:31:06 |
> INFO | jvm 1 | 2007/09/25 21:31:06 | 
> ***************************************************
> INFO | jvm 1 | 2007/09/25 21:31:06 |
> INFO | jvm 1 | 2007/09/25 21:31:06 |
>
>
>
> On 10/1/07, *Leif Mortenson* < le...@ta... 
> <mailto:le...@ta...>> wrote:
>
>     Mark,
>     What does the wrapper.log look like just before the crash?  I am
>     wondering if the Wrapper is thinking that the JVM is frozen and
>     attempting to restart it.  If the native code is not written correctly
>     it could be leading to a crash as the JVM tries to shutdown.
>     I ask because you mentioned that making the ping timeout longer
>     helped.
>
>     Could you enable wrapper.debug=true and try this again.  Then
>     post back or send the resulting log files (from a couple minutes
>     before
>     the crash) to the list or to me directly if it has private data.
>
>     Cheers,
>     Leif
>
>     no spam wrote:
>     > Hi Leif,
>     >
>     > The wrapper process seems to have problems restarting the jvm
>     > process.  In fact after this crash there will be several other
>     crashes
>     > noticeable in the event viewer and also in the wrapper log.
>     >
>     > Right the Dr Watson output is very confusing.  I've been running a
>     > test since last Thursday without the wrapper and everything
>     seems to
>     > be running great.  I believe that my last test ran better when I
>     upped
>     > the ping time in the wrapper config.  This process runs in parallel
>     > with a second JVM process that hovers around 90-95% cpu, could
>     this be
>     > an issue?
>     >
>     > Yes I'm using JIntegra to talk to another vendor's objects (IPV).  I
>     > should know a bit more in a few days if things keep
>     running.  This is
>     > really some interesting software here at CNN.  This code
>     controls all
>     > the low resolution encoders for the newsroom.   It has to be 100%
>     > stable which is why I have been load testing for the past few weeks.
>     >
>     > Thanks for your prompt reply.  I really want to be able to run this
>     > via the wrapper as it's so convenient.
>     >
>     > Mark
>