Re: [tcljava-user] debugging "assertion`tsdPtr->initialized' failed"

[Mo D. <mo...@mo...>]

I would not start messing around with the internal implementations of these libraries. The issue with integrating with Tcl Blend is that when you start calling C code from different threads, you have to make sure it is done in a thread safe way. That is what the setup and checking that Tcl Blend is doing actually does. The code is setting up a threaded version of Tcl Blend that can then be called from multiple threads in a safe way.

But, the problem with the Java libs is that they are doing some fancy thread pooling stuff to get max performance with networking, and this approach is at odds with what Tcl Blend expects. You don't want to have to init a Tcl Blend interp in every thread in the thread pool. The Java code is assuming that one thread in the pool is the same as another. But, once you start linking thread specific data at the C layer, that assumption is no longer valid.

What I would do if I were you is rethink the basic approach. Have 1 thread that is the Tcl Blend "processing" thread. This thread can wait for a request to do something that comes from one of these Java threads in the thread pool. The Tcl Blend thread can then be setup properly with an interp created in that thread. The Java network thread puts a request in queue, then waits for it to be done. The Tcl Blend thread just processes one request at a time and delivers the result back to the waiting Java thread.

That is about as simple as it gets, and it means that your Java part of the solution will not depend on threaded interactions at the C layer.

cheers
Mo DeJong


On Feb 20, 2012, at 2:17 PM, Russell Roy wrote:

> Mo,
> 
> Ok, so thanks for that explanation.
> 
> I've attached a log w/ TCBLEND_DEBUG enabled.   I must have a different version of tclblend than what you quoted below (I'd pulled the latest sources directly from  the tree at sourceforge ...as opposed to the prepackaged.   I don't have sources for tclblend.jar.)   as I don't see the explicit "Calling Tcl_CreateInterp) ... but, no matter   I do see ouput related to JavaSetupJava success and we do get off the ground properly.  The point you make is that I need to see *two* of these, when in fact I only see one.
> 
> I hope this is not getting into too much detail but HttpClient gets run on a separate thread.  It hasA SocketConnector or a SelectChannelConnector.   These in turn simulate multithreading within a single  java thread by  means of a synchronous dispatcher   ("dispatch") - one for each HttpClient connection attempt:
>  
> [rroy@deepwater client]$ pwd
> /home/rroy/jetty-all-8.1.0.v20120127-sources/org/eclipse/jetty/client
> [rroy@deepwater client]$ grep "run(" *
> HttpClient.java:            public void run()
> SocketConnector.java:            public void run()
> [rroy@deepwater client]$
> [rroy@deepwater client]$ grep dispatch *
> HttpClient.java:        _threadPool.dispatch(new Runnable()
> SelectConnector.java:        public boolean dispatch(Runnable task)
> SelectConnector.java:            return _httpClient._threadPool.dispatch(task);
> SocketConnector.java:        _httpClient.getThreadPool().dispatch(new Runnable()
> [rroy@deepwater client]$
> 
> Obviously, being third party code, they know nothing about tclblend.
> 
> Now, as it happens, this (Jetty) is open source so I could, in theory, get in there and start mucking about to insure that the appropriate per-thread tclblend init takes place (I have no idea the implications of "dispatch")  but this seems pretty ugly to me.      And what would one do with a closed third-party package for which threading was an issue?  Is it too strong a statement to say that tclblend has limited utility when it comes to java threading?
> 
> Whereas I know the javclsh has not experienced this crash for my application  I guess I'd be more inclined to try to make that solution work for me.   My problem there though is I need to be able to pull in some other Tcl packages like Expect, Itcl, etc and it wasn't clear to me whether jaclsh was capable of loading any legal Tcl package in the same manner as the "normal" tcl interpreter (I notice you use  XpUtils::iload rather than load .... i'm a bit fuzzy on that.)
> 
> Anywho ... I think we're coming to the end of the road on this issue.    I appreciate your advice.
> 
> Russell
> On Sun, Feb 19, 2012 at 10:48 PM, Mo DeJong <mo...@mo...> wrote:
> So, I am almost 100% sure that the issue is with the fact that Tcl Blend is not getting initialized in that secondary thread that your Java library is creating.
> 
> Here is the related code that gets invoked when Interp() is called from Java code.
> 
> jlong JNICALL
> Java_tcl_lang_Interp_create(
>     JNIEnv *env,                /* Java environment. */
>     jobject interpObj)          /* Handle to Interp object. */
> {
>     jlong lvalue;
>     Tcl_Interp *interp;
> 
> #ifdef TCLBLEND_DEBUG
>     fprintf(stderr, "TCLBLEND_DEBUG: Calling Tcl_CreateInterp()\n");
> #endif /* TCLBLEND_DEBUG */
> 
>     interp = Tcl_CreateInterp();
>     if (JavaSetupJava(env, interp) != TCL_OK) {
>         jclass err = (*env)->FindClass(env, "tcl/lang/TclRuntimeError");
>         if (err) {
>             (*env)->ThrowNew(env, err, Tcl_GetStringResult(interp));
>             (*env)->DeleteLocalRef(env, err);
>         }
>         Tcl_DeleteInterp(interp);
>         lvalue = 0;
>     } else {
>         lvalue = 0;
>         *(Tcl_Interp**)&lvalue = interp;
>     }
>     return lvalue;
> }
> 
> That call to JavaSetupJava() will init the thread specific data that Tcl Blend needs. In your case, this init is not getting done. My guess would be that what is going on is that you created the Interp Java object in one thread, but then you pass it to another thread and try to invoke methods in that other thread. That will not work for a Tcl Blend interp because the interp object needs to be created in the same thread where it will be used used. This would typically work with a Java object in Jacl, but Tcl Blend is much more strict about how objects are used from threads. Could you take a look at your code and make sure the interp logic is getting invoked from the same thread that originally created the specific Interp object.
> 
> Mo
> 
> On Feb 18, 2012, at 2:05 PM, Russell Roy wrote:
> 
>> Mo,
>> 
>> Thanks for taking the time to respond.
>> 
>> I'd already taken the step of turning on the TCLBLEND_DEBUG.  Unfortunately, the last debug output prior to the crash provided no new information - just that JavaCmdProc had been called.
>> 
>> I will poke around a bit more in the open source Jetty HttpClient code to see if I can discern anything there.   Yes, there is a new thread kicked off in there.   I'd gone so far as to put some println's in that code to see if I could narrow the focus.   That seems to indicate we're making it all the way through the HttpClient start method in the main thread and crash upon return into tclblend...but i may be misinterpreting (i don't pretend to know what i'm doing when it comes to the threaded world.)   The most solid info I got was running it in gdb which showed the underlying cause of the failed tclblend assert as sigsev while switching threads.
>> 
>> One other clue:   I'm running the exact same script in jtclsh and jaclsh.    What is it telling us that this crash occurs in jtclsh instance but runs cleanly in jaclsh?
>> 
>> Any hope that the problem might just 'go away' were I to find a way to do my tclblend in tcl8.5 (where the thread implementation is part of the tcl core) as opposed to the current external thread pkg?
>> 
>> Russell
>> 
>> On Fri, Feb 17, 2012 at 2:09 PM, Mo DeJong <mo...@mo...> wrote:
>> Hi Russell
>> 
>> So, this has something to do with threads. The specific assert you are running into is in JavaGetCache()
>> 
>> http://tcljava.cvs.sourceforge.net/viewvc/tcljava/tcljava/src/native/javaCmd.c?revision=1.27&view=markup
>> 
>> See line 398:
>> 
>> 
>> 393     TCLBLEND_EXTERN JavaInfo*
>> 394     JavaGetCache()
>> 395     {
>> 396     ThreadSpecificData *tsdPtr = TCL_TSD_INIT(&dataKey);
>> 397
>> 398     assert(tsdPtr->initialized);
>> 399
>> 400     return &(tsdPtr->jcache);
>> 401     }
>> 
>> 
>> So, this assert is being hit because the Java code calls into Tcl Blend and then Tcl Blend checks to see if the thread specific data has been initialized in this thread. That assert is failing for unknown reasons.
>> 
>> My first thought would be to check to see if the Java code in your HttpClient is actually creating another background thread and then invoking Tcl Blend methods in that other background thread. That would be the most likely cause of the assert. It is quite honestly hard to track down Tcl Blend problems in gdb, it is possible, but not easy because Java calls into C and vice versa. It is possible to set breakpoints in the Tcl Blend C code, but it takes some gdb mojo.
>> 
>> You can also enable the TCLBLEND_DEBUG macro and create a build that will print debug and init messages as it is loaded. This is a lot easier way to debug these issues compared to using gdb.
>> 
>> As far as the class loading issue goes, it does seem very odd. What I would do is try to "exec javap CLASS" and see if you can invoke the JVM with the specific classpath you are using. If you can't get the JDK tools to read from that classpath, then that is the source of your problem. Also, make sure the classpath being used in Jacl is exactly the same as the one being used in Tcl Blend (aside from jacl.jar vs tclblend.jar).
>> 
>> Mo
>> 
>> On Feb 16, 2012, at 12:20 PM, Russell Roy wrote:
>> 
>> > I have a simple script.  It works w/o error in jaclsh.  It core dumps in jtclsh.
>> >
>> > Here's the script:
>> >
>> > 1 set env(TCL_CLASSPATH) "...some-big-long-classpath..."
>> > 2 package require java
>> > 3 set httpClient [java::new org.eclipse.jetty.client.HttpClient]
>> > 4 $httpClient start
>> >
>> > The script crashes in jtclsh on line 4 with this message:
>> >
>> >        tclsh8.4: /home/rroy/tcljava/src/native/javaCmd.c:398: JavaGetCache: Assertion `tsdPtr->initialized' failed.
>> > Aborted (core dumped)
>> >
>> > That's the whole of the output when the thing dies.
>> >
>> >
>> > I was hoping to be able to do some java debugging by running the script in jaclsh but, as I said, the thing works fine there - no problem, no crash.   I really need this to work in jctlsh.
>> >
>> >
>> > I'm using tclBlend 1.4.1 pulled from latest at sourceforge, and thread2.6.5, tcl 8.4.19  (I also tried various permutations and combinations of thread2.6.3, tcl8.4.13 and earlier versions of tclBlend, all w/o any luck.)
>> >
>> > I've rebuilt and and re-installed tclBlend cleanly with each change so as to ensure consistency and jaclsh and jtclsh are setting up a correct environment but the problem persists.
>> >
>> > I stuck the core image in gdb but I there's was nothing discernibly useful, to me anyways (I believe the failed assertion is forcing the core dump.)
>> >
>> >
>> > I've run out of ideas on how to debug this.
>> >
>> >
>> > Russ
>> >
>> > PS:   For what it's worth ... The HttpClient of above comes in two flavors - CONNECTOR_SOCKET and CONNECTOR_SELECT_CHANNEL (the default).   I happened to discover through trial and error that if I configure the client as CONNECTOR_SOCKET before I do the start then it runs w/o error in jtclsh.   That's interesting but it doesn't help me because I need to run in it as select channel.
>> >
>> >
>> >
>> > ------------------------------------------------------------------------------
>> > Virtualization & Cloud Management Using Capacity Planning
>> > Cloud computing makes use of virtualization - but cloud computing
>> > also focuses on allowing computing to be delivered as a service.
>> > http://www.accelacomm.com/jaw/sfnl/114/51521223/_______________________________________________
>> > tcljava-user mailing list
>> > tcl...@li...
>> > https://lists.sourceforge.net/lists/listinfo/tcljava-user
>> 
>> 
> 
> 
> <tclblend.log>