Re: [tcljava-user] debugging "assertion`tsdPtr->initialized' failed"
Brought to you by:
mdejong
Menu
▾
▴
Re: [tcljava-user] debugging "assertion`tsdPtr->initialized' failed"
|
From: Russell R. <rus...@gm...> - 2012-02-23 15:43:15
|
Mo,
I tried to parse your suggestion about how to proceed but I'm afraid you
lost me.
Tclblend is a powerful and elegant marriage of tcl and java. I'd really
hoped to make it work for my application because it would result in a clean
and simple implementation. Unfortunately I seem to have run into a corner
case issue with this crash. I think I may be getting in over my head w/
this and with you my only resource for resolving roadblocks it places
unreasonable expectations on you and is risky for me going forward. (As it
is I'm probably looking at a more loosely coupled solution with a separate
pure-Java process to handle my http/cometd stuff with some ipc mechanism
back to tcl land.)
Thank you for the time and effort you invested in helping me and bringing
the investigation this far. Hopefully other will benefit from the
discussion.
Best regards,
Russell
On Mon, Feb 20, 2012 at 6:41 PM, Mo DeJong <mo...@mo...> wrote:
> I would not start messing around with the internal implementations of
> these libraries. The issue with integrating with Tcl Blend is that when you
> start calling C code from different threads, you have to make sure it is
> done in a thread safe way. That is what the setup and checking that Tcl
> Blend is doing actually does. The code is setting up a threaded version of
> Tcl Blend that can then be called from multiple threads in a safe way.
>
> But, the problem with the Java libs is that they are doing some fancy
> thread pooling stuff to get max performance with networking, and this
> approach is at odds with what Tcl Blend expects. You don't want to have to
> init a Tcl Blend interp in every thread in the thread pool. The Java code
> is assuming that one thread in the pool is the same as another. But, once
> you start linking thread specific data at the C layer, that assumption is
> no longer valid.
>
> What I would do if I were you is rethink the basic approach. Have 1 thread
> that is the Tcl Blend "processing" thread. This thread can wait for a
> request to do something that comes from one of these Java threads in the
> thread pool. The Tcl Blend thread can then be setup properly with an interp
> created in that thread. The Java network thread puts a request in queue,
> then waits for it to be done. The Tcl Blend thread just processes one
> request at a time and delivers the result back to the waiting Java thread.
>
> That is about as simple as it gets, and it means that your Java part of
> the solution will not depend on threaded interactions at the C layer.
>
> cheers
> Mo DeJong
>
>
> On Feb 20, 2012, at 2:17 PM, Russell Roy wrote:
>
> Mo,
>
> Ok, so thanks for that explanation.
>
> I've attached a log w/ TCBLEND_DEBUG enabled. I must have a different
> version of tclblend than what you quoted below (I'd pulled the latest
> sources directly from the tree at sourceforge ...as opposed to the
> prepackaged. I don't have sources for tclblend.jar.) as I don't see the
> explicit "Calling Tcl_CreateInterp) ... but, no matter I do see ouput
> related to JavaSetupJava success and we do get off the ground properly.
> The point you make is that I need to see *two* of these, when in fact I
> only see one.
>
> I hope this is not getting into too much detail but HttpClient gets run on
> a separate thread. It *hasA* SocketConnector or a
> SelectChannelConnector. These in turn simulate multithreading within a
> single java thread by means of a synchronous dispatcher ("dispatch") -
> one for each HttpClient connection attempt:
>
> [rroy@deepwater client]$ pwd
> /home/rroy/jetty-all-8.1.0.v20120127-sources/org/eclipse/jetty/client
> [rroy@deepwater client]$ grep "run(" *
> HttpClient.java: public void run()
> SocketConnector.java: public void run()
> [rroy@deepwater client]$
> [rroy@deepwater client]$ grep dispatch *
> HttpClient.java: _threadPool.dispatch(new Runnable()
> SelectConnector.java: public boolean dispatch(Runnable task)
> SelectConnector.java: return
> _httpClient._threadPool.dispatch(task);
> SocketConnector.java: _httpClient.getThreadPool().dispatch(new
> Runnable()
> [rroy@deepwater client]$
>
> Obviously, being third party code, they know nothing about tclblend.
>
> Now, as it happens, this (Jetty) is *open source* so I could, in theory,
> get in there and start mucking about to insure that the appropriate
> per-thread tclblend init takes place (I have no idea the implications of
> "dispatch") but this seems pretty ugly to me. And what would one do
> with a *closed* third-party package for which threading was an issue? Is
> it too strong a statement to say that tclblend has limited utility when it
> comes to java threading?
>
> Whereas I know the javclsh has not experienced this crash for my
> application I guess I'd be more inclined to try to make that solution work
> for me. My problem there though is I need to be able to pull in some
> other Tcl packages like Expect, Itcl, etc and it wasn't clear to me whether
> jaclsh was capable of loading any legal Tcl package in the same manner as
> the "normal" tcl interpreter (I notice you use XpUtils::iload rather than
> load .... i'm a bit fuzzy on that.)
>
> Anywho ... I think we're coming to the end of the road on this issue. I
> appreciate your advice.
>
> Russell
> On Sun, Feb 19, 2012 at 10:48 PM, Mo DeJong <mo...@mo...> wrote:
>
>> So, I am almost 100% sure that the issue is with the fact that Tcl Blend
>> is not getting initialized in that secondary thread that your Java library
>> is creating.
>>
>> Here is the related code that gets invoked when Interp() is called from
>> Java code.
>>
>> jlong JNICALL
>> Java_tcl_lang_Interp_create(
>> JNIEnv *env, /* Java environment. */
>> jobject interpObj) /* Handle to Interp object. */
>> {
>> jlong lvalue;
>> Tcl_Interp *interp;
>>
>> #ifdef TCLBLEND_DEBUG
>> fprintf(stderr, "TCLBLEND_DEBUG: Calling Tcl_CreateInterp()\n");
>> #endif /* TCLBLEND_DEBUG */
>>
>> interp = Tcl_CreateInterp();
>> if (JavaSetupJava(env, interp) != TCL_OK) {
>> jclass err = (*env)->FindClass(env, "tcl/lang/TclRuntimeError");
>> if (err) {
>> (*env)->ThrowNew(env, err, Tcl_GetStringResult(interp));
>> (*env)->DeleteLocalRef(env, err);
>> }
>> Tcl_DeleteInterp(interp);
>> lvalue = 0;
>> } else {
>> lvalue = 0;
>> *(Tcl_Interp**)&lvalue = interp;
>> }
>> return lvalue;
>> }
>>
>> That call to JavaSetupJava() will init the thread specific data that Tcl
>> Blend needs. In your case, this init is not getting done. My guess would be
>> that what is going on is that you created the Interp Java object in one
>> thread, but then you pass it to another thread and try to invoke methods in
>> that other thread. That will not work for a Tcl Blend interp because the
>> interp object needs to be created in the same thread where it will be used
>> used. This would typically work with a Java object in Jacl, but Tcl Blend
>> is much more strict about how objects are used from threads. Could you take
>> a look at your code and make sure the interp logic is getting invoked from
>> the same thread that originally created the specific Interp object.
>>
>> Mo
>>
>> On Feb 18, 2012, at 2:05 PM, Russell Roy wrote:
>>
>> Mo,
>>
>> Thanks for taking the time to respond.
>>
>> I'd already taken the step of turning on the TCLBLEND_DEBUG.
>> Unfortunately, the last debug output prior to the crash provided no new
>> information - just that JavaCmdProc had been called.
>>
>> I will poke around a bit more in the open source Jetty HttpClient code to
>> see if I can discern anything there. Yes, there is a new thread kicked
>> off in there. I'd gone so far as to put some println's in that code to
>> see if I could narrow the focus. That seems to indicate we're making it
>> all the way through the HttpClient start method in the main thread and
>> crash upon return into tclblend...but i may be misinterpreting (i don't
>> pretend to know what i'm doing when it comes to the threaded world.) The
>> most solid info I got was running it in gdb which showed the underlying
>> cause of the failed tclblend assert as sigsev while switching threads.
>>
>> One other clue: I'm running the exact same script in jtclsh and jaclsh.
>> What is it telling us that this crash occurs in jtclsh instance but runs
>> cleanly in jaclsh?
>>
>> Any hope that the problem might just 'go away' were I to find a way to do
>> my tclblend in tcl8.5 (where the thread implementation is part of the tcl
>> core) as opposed to the current external thread pkg?
>>
>> Russell
>>
>> On Fri, Feb 17, 2012 at 2:09 PM, Mo DeJong <mo...@mo...> wrote:
>>
>>> Hi Russell
>>>
>>> So, this has something to do with threads. The specific assert you are
>>> running into is in JavaGetCache()
>>>
>>>
>>> http://tcljava.cvs.sourceforge.net/viewvc/tcljava/tcljava/src/native/javaCmd.c?revision=1.27&view=markup
>>>
>>> See line 398:
>>>
>>>
>>> 393 TCLBLEND_EXTERN JavaInfo*
>>> 394 JavaGetCache()
>>> 395 {
>>> 396 ThreadSpecificData *tsdPtr = TCL_TSD_INIT(&dataKey);
>>> 397
>>> 398 assert(tsdPtr->initialized);
>>> 399
>>> 400 return &(tsdPtr->jcache);
>>> 401 }
>>>
>>>
>>> So, this assert is being hit because the Java code calls into Tcl Blend
>>> and then Tcl Blend checks to see if the thread specific data has been
>>> initialized in this thread. That assert is failing for unknown reasons.
>>>
>>> My first thought would be to check to see if the Java code in your
>>> HttpClient is actually creating another background thread and then invoking
>>> Tcl Blend methods in that other background thread. That would be the most
>>> likely cause of the assert. It is quite honestly hard to track down Tcl
>>> Blend problems in gdb, it is possible, but not easy because Java calls into
>>> C and vice versa. It is possible to set breakpoints in the Tcl Blend C
>>> code, but it takes some gdb mojo.
>>>
>>> You can also enable the TCLBLEND_DEBUG macro and create a build that
>>> will print debug and init messages as it is loaded. This is a lot easier
>>> way to debug these issues compared to using gdb.
>>>
>>> As far as the class loading issue goes, it does seem very odd. What I
>>> would do is try to "exec javap CLASS" and see if you can invoke the JVM
>>> with the specific classpath you are using. If you can't get the JDK tools
>>> to read from that classpath, then that is the source of your problem. Also,
>>> make sure the classpath being used in Jacl is exactly the same as the one
>>> being used in Tcl Blend (aside from jacl.jar vs tclblend.jar).
>>>
>>> Mo
>>>
>>> On Feb 16, 2012, at 12:20 PM, Russell Roy wrote:
>>>
>>> > I have a simple script. It works w/o error in jaclsh. It core dumps
>>> in jtclsh.
>>> >
>>> > Here's the script:
>>> >
>>> > 1 set env(TCL_CLASSPATH) "...some-big-long-classpath..."
>>> > 2 package require java
>>> > 3 set httpClient [java::new org.eclipse.jetty.client.HttpClient]
>>> > 4 $httpClient start
>>> >
>>> > The script crashes in jtclsh on line 4 with this message:
>>> >
>>> > tclsh8.4: /home/rroy/tcljava/src/native/javaCmd.c:398:
>>> JavaGetCache: Assertion `tsdPtr->initialized' failed.
>>> > Aborted (core dumped)
>>> >
>>> > That's the whole of the output when the thing dies.
>>> >
>>> >
>>> > I was hoping to be able to do some java debugging by running the
>>> script in jaclsh but, as I said, the thing works fine there - no problem,
>>> no crash. I really need this to work in jctlsh.
>>> >
>>> >
>>> > I'm using tclBlend 1.4.1 pulled from latest at sourceforge, and
>>> thread2.6.5, tcl 8.4.19 (I also tried various permutations and
>>> combinations of thread2.6.3, tcl8.4.13 and earlier versions of tclBlend,
>>> all w/o any luck.)
>>> >
>>> > I've rebuilt and and re-installed tclBlend cleanly with each change so
>>> as to ensure consistency and jaclsh and jtclsh are setting up a correct
>>> environment but the problem persists.
>>> >
>>> > I stuck the core image in gdb but I there's was nothing discernibly
>>> useful, to me anyways (I believe the failed assertion is forcing the core
>>> dump.)
>>> >
>>> >
>>> > I've run out of ideas on how to debug this.
>>> >
>>> >
>>> > Russ
>>> >
>>> > PS: For what it's worth ... The HttpClient of above comes in two
>>> flavors - CONNECTOR_SOCKET and CONNECTOR_SELECT_CHANNEL (the default). I
>>> happened to discover through trial and error that if I configure the client
>>> as CONNECTOR_SOCKET before I do the start then it runs w/o error in jtclsh.
>>> That's interesting but it doesn't help me because I need to run in it as
>>> select channel.
>>> >
>>> >
>>> >
>>> >
>>> ------------------------------------------------------------------------------
>>> > Virtualization & Cloud Management Using Capacity Planning
>>> > Cloud computing makes use of virtualization - but cloud computing
>>> > also focuses on allowing computing to be delivered as a service.
>>> >
>>> http://www.accelacomm.com/jaw/sfnl/114/51521223/_______________________________________________
>>> > tcljava-user mailing list
>>> > tcl...@li...
>>> > https://lists.sourceforge.net/lists/listinfo/tcljava-user
>>>
>>>
>>
>>
> <tclblend.log>
>
>
>
|
