Re: [Wrapper-user] HP-UX 11i 64 bit:log level DEBUG crashes application

[Leif M. <le...@ta...>]

Venkatesh,
 From the logs you sent me I now understand what the problem is.  The 
thing is, I
am not sure who you are encountering it.

In both the case where I accept the socket from the JVM and when I read 
a packet,
there is a chance that there is no socket or no data.  In such cases, I 
have the sockets
configured to be in nonblocking mode so the lack of data does not hang 
things up.

In such cases, both the accept and recv functions set errno to 
EWOULDBLK.  See
the following code where I test the error after attempt to accept a socket.

    if (sd == INVALID_SOCKET) {
        rc = wrapperGetLastError();
        if (rc == EWOULDBLOCK) {
            /* There are no incomming sockets right now. */
            return;
        } else {
            if (wrapperData->isDebugging) {
                log_printf(WRAPPER_SOURCE_PROTOCOL, LEVEL_DEBUG,
                    "socket creation failed. (%d)", rc);
            }
            return;
        }
    }

To avoid the debug error messages you are seeing on startup, I am 
special casing
the EWOULDBLOCK code so that it silently fails.  Any other error will 
display a
debug error message.

Now the strange part.   On your system, accept is setting errno to a 
value of 11
when a socket is not available.   On Linux and every other system the 
Wrapper
works on to date, EWOULDBLOCK = 11.

Since the above code is failing on your system, EWOULDBLOCK is obviously
not = 11 with your compiler.    Could you place the following lines of 
code just
before the above code.

log_printf(WRAPPER_SOURCE_PROTOCOL, LEVEL_DEBUG,
    "errno 11 = %s", strerror(11));
log_printf(WRAPPER_SOURCE_PROTOCOL, LEVEL_DEBUG,
    "errno EWOULDBLOCK = %d = %s", EWOULDBLOCK, strerror(EWOULDBLOCK));

On linux, this prints out the following:
errno 11 = Resource temporarily unavailable
errno EWOULDBLOCK = 11 = Resource temporarily unavailable

Now on with the explanation...
In the case of accept, the above problem is simply causing an extra 
debug message
to be printed out.   But after a socket is connected, the Wrapper reads 
a packet
from the JVM immediately after it connects, but when it calls recv again 
to see if
there are any more packets, recv returns that there is no data and sets 
errno to 11.
Pretty much the same code fails to match this to EWOULDBLOCK and so
assumes that it is an unexpected socket error.   This leads to the 
socket being
closed and the JVM being restarted as you are seeing.

So the cause of all your problems is the value of the EWOULDBLOCK constant
on your system.

Could you try recompiling the c source and make sure that there are no 
warning
messages about this constant being undefined?  If the constant just has 
a different
value on your system, then recv and accept should still be using the 
constant and
returning that other value...

Cheers,
Leif

Leif Mortenson wrote:

> Venkatesh,
>    Sorry about the hassle getting your hands on a  64-bit build.   It 
> sounds like there are
> actually several such users out there.  Not having access to any 
> hardware however, I
> must rely on users willing to donate the time to maintain and build 
> such releases.  We
> have someone for the 32-bit version, but not the 64-bit version yet.  
> I have been
> hoping that there is a way to build both the 32-bit .so and 64-bit .sl 
> file on the same
> machine so that they can both be shipped with the HP-UX distribution.
>
>    See the answers to your questions below.
>
> Venkatesh Sellappa wrote:
>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Please read the disclaimer at the bottom of this e-mail.
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>> Hi All,
>>
>> Congratulations on building an excellent product.
>> It fills a very important need in the Java market.
>>
>> For our production environment we are using the following:
>> 0.HP-UX 11i ( 64 bit )
>> 1.Java 1.4.2_03
>> 2.Wrapper version 3.1.0
>>
>> In the absence of a 64-bit binary library and hitting the problems 
>> discussed earlier in the mailing list,
>> with the 32-bit library.
>> Re: 
>> http://sourceforge.net/mailarchive/forum.php?thread_id=3751203&forum_id=11948 
>>
>>
>> I used the cc,make and ld native to HP-UX to build the libwrapper.sl 
>> library from source.(GNU utilities not being allowed).
>> The source files for building the library were 
>> wrapperjni_unix.c,wrapperinfo.c,wrapperjni.c.
>>
>> On changing the properties
>> wrapper.console.loglevel=DEBUG
>> wrapper.logfile.loglevel=DEBUG
>> from INFO to DEBUG
>>
>> I hit the following,
>> DEBUG OUTPUT
>> ------------
>> STATUS | wrapper  | 2004/06/29 13:20:34 | --> Wrapper Started as Console
>> DEBUG  | wrapperp | 2004/06/29 13:20:34 | server listening on port 
>> 32000.
>> ..
>> DEBUG  | wrapperp | 2004/06/29 13:20:35 | socket creation failed. (11)
>> INFO   | jvm 1    | 2004/06/29 13:20:35 | Initializing...
>> DEBUG  | wrapperp | 2004/06/29 13:20:35 | socket creation failed. (11)
>> ..
>> INFO   | jvm 1    | 2004/06/29 13:20:35 | Loaded native library: 
>> libwrapper.so
>> INFO   | jvm 1    | 2004/06/29 13:20:35 | Calling native 
>> initialization method.
>> INFO   | jvm 1    | 2004/06/29 13:20:35 | Inside native 
>> WrapperManager initialization method
>> ..
>> INFO   | jvm 1    | 2004/06/29 13:20:35 | 
>> WrapperManager.start(org.tanukisoftware.wrapper.test.Main@a981ca, 
>> args[]) called by thread: main
>> ..
>> INFO   | jvm 1    | 2004/06/29 13:20:35 | Open socket to wrapper...
>> INFO   | jvm 1    | 2004/06/29 13:20:35 | Opened Socket
>> INFO   | jvm 1    | 2004/06/29 13:20:35 | Send a packet KEY : 
>> HjjwQG4LdL3g_kNV
>> INFO   | jvm 1    | 2004/06/29 13:20:35 | 
>> handleSocket(Socket[addr=/127.0.0.1,port=32000,localport=57670])
>> DEBUG  | wrapperp | 2004/06/29 13:20:35 | accepted a socket from 
>> 127.0.0.1 on port 57670
>> ..
>> DEBUG  | wrapper  | 2004/06/29 13:20:35 | Start Application.
>> DEBUG  | wrapperp | 2004/06/29 13:20:35 | socket creation failed. (11)
>> DEBUG  | wrapperp | 2004/06/29 13:20:35 | socket not open, so packet 
>> not sent START : start
>> ERROR  | wrapper  | 2004/06/29 13:20:35 | Unable to send the start 
>> command to the JVM.
>>  
>>
> Strange.  The JVM is opening its socket to the JVM.  The Wrapper 
> appears to be
> correctly accepting the new socket, and trading keys correctly?(this 
> part is clipped?).
> But then when the  wrapper attempts to tell the Java side app to 
> start, it finds that the
> socket is no longer open.    I am wondering what happened in the 
> region that is cut
> out.
>
>> ERROR  | wrapper  | 2004/06/29 13:20:35 | Sending the JVM process a 
>> SIGTERM
>> ..
>> ERROR  | wrapper  | 2004/06/29 13:20:37 | Critical error: wait for 
>> JVM process failed (No child processes)
>>
>> END DEBUG OUTPUT
>> ----------------
>> Note 1:It still says libwrapper.so loaded, even though there is no 
>> libwrapper.so in the path, its libwrapper.sl
>>  
>>
> The actual location and loading of the library is being handled by the 
> JVM.   All I am
> able to do is to tell whether or not it was loaded.  Currently I 
> always display a message
> that libwrapper.so was loaded, with the exceptions of Wrapper.dll for 
> Windows and
> libwrapper.jnilib for OSX.   Until HP-UX, this has been correct for 
> all other platforms.
>
> If you know of any System property values which I can use to 
> distinguish between
> 32 and 64 bit versions then I can add a special case to correct this 
> message.
> On Solaris, the same so file seems to be working for both 32 and 64 
> bit versions of
> the OS.
>
>> Note 2:On changing the log level back to INFO from DEBUG it works fine
>>  
>>
> This could be one of two things.  A timing issue.  The startup would 
> be slightly faster
> without the debug output.   Or a corruption/synchronization problem 
> caused by the
> debug output itself.
>
> You have debug output enabled both the console and file.  What happens 
> if you only
> do one or the other?
>
>> Note 3:Using method 3 i integrated my application with the wrapper 
>> and hit the exact same problem, if i keep the wrapper loglevel at 
>> INFO and my application loglevel at DEBUG( using log4j )it works fine.
>> Note 4:The debug output has been truncated to make it a bit more 
>> readable, i can send the full log if required.
>>  
>>
> Could you send the full log file to me directly?   Keep all other 
> messages on
> list please.  Also send along your wrapper.conf file.   tar.gz them 
> both up
> before sending them off to make sure that the text attachments don't get
> reformated or anything.
>
> Recreate the log using the following:
>
> wrapper.console.loglevel=DEBUG
> wrapper.logfile.loglevel=DEBUG
> wrapper.state_output=TRUE
>
> The state output causes the main event loop to dump a message for each 
> cycle.  The
> log output goes way up, but it may help pit down the exact timing of 
> what is
> happening.
>
>> What other diagnostics can i run to narrow down the problem ?
>>  
>>
> Let me take a look at the log and I may have some more questions.
>
> Cheers,
> Leif