From: <no...@so...> - 2002-08-08 19:02:09
|
Feature Requests item #577014, was opened at 2002-07-03 17:35 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=360894&aid=577014&group_id=10894 Category: 80. Thread Package Group: None Status: Open Resolution: None Priority: 5 Submitted By: Markus Elfring (elfring) Assigned to: Zoran Vasiljevic (vasiljevic) Summary: documentation; wait on thread exits Initial Comment: 1. When will the documentation for the "Thread package 2.2" be available? 2. When will it be possible to wait for the end of several threads like the shell command "wait" does it for background processes? (See "man wait") ---------------------------------------------------------------------- >Comment By: Markus Elfring (elfring) Date: 2002-08-08 21:02 Message: Logged In: YES user_id=572001 Thanks for your response. Please answer question 8 (2002-08-08 01:10). ---------------------------------------------------------------------- Comment By: Zoran Vasiljevic (vasiljevic) Date: 2002-08-08 11:45 Message: Logged In: YES user_id=95086 1. No 7. Unfortunately. at the moment no. The "waiting queue" is really the event loop and Tcl does not export any functions about interrogating the event loop entries, the size of it, etc. 9. I'd say not more than handling processes. Actually, they are much simpler to handle than processes, IMO. 2. Mo did update the thread.n file in CVS directly and is now not in sync with html again. Idea is to have thread.tmml and generate both doc format automatically. I will fix this. 2.2.1. Depends on the underlying threads library. Normaly what is saved is the exit code which is usually and integer. But this may vary from implementation to implementation. You better join threads explicitly if you created them as joinable, otherwise you're searching for trouble. 2.2.2 Now I understand. Ok, the library has create/destroy things for mutexes and condition variables. It also implicitly sets shared arrays. Those live independently of threads and are never automatically destroyed, unless you explicitly say so. Even on the app exit (there are good reasons for that). What is not so clear is thread creation/destroy. This is somehow shadowed. The thread creation is straightforward. The termination not. You have thread::unwind, thread::exit and thread::release for this. All serve different purposes and it is not always feasible nor possible to maintain the create/destroy paradigm in this case. Concerning docs, many of useful hints and ideas are found in this dialog. I will compile them and extend the docs. Thanks for you remarks and inputs. ---------------------------------------------------------------------- Comment By: Markus Elfring (elfring) Date: 2002-08-08 07:53 Message: Logged In: YES user_id=572001 1. Can joinable threads be emulated with "vwait" for earlier TCL versions? 7. Would it be useful to allow a thread to query the "waiting queue" of incoming scripts? I'm thinking about some statistics to generate a thread listing that is similar to what the shell command "top" performs for processes. Can you offer such a tool? It would help to get an overwiev of thread usage. 9. Threads can be a threat and scare if they are not handled properly. ;-) ---------------------------------------------------------------------- Comment By: Markus Elfring (elfring) Date: 2002-08-08 01:10 Message: Logged In: YES user_id=572001 8. Can it be that the thread package expects that the TCL interpreter was compiled with threads enabled(http://www.tcl.tk/man/tcl8.3/TclCmd/tclvars.htm#M36)? It seems that users must be careful about this setting because it is not enabled by default. ---------------------------------------------------------------------- Comment By: Markus Elfring (elfring) Date: 2002-08-08 00:27 Message: Logged In: YES user_id=572001 2. Is it possible that the HTML version of the documentation is not as up to date as your other file? 2.2.1 How much resources are still allocated if an identifier does not go out of scope because the thread won't be joined? 2.2.2 Bookkeeping of identifiers for threads, mutexes or variables ... The identifier handling for threads seems to be difficult because there is no clear pair of create/destroy functions. Please add your answers to the documentation for the questions 1, 2.2.1 and 7. ---------------------------------------------------------------------- Comment By: Zoran Vasiljevic (vasiljevic) Date: 2002-08-07 14:43 Message: Logged In: YES user_id=95086 1. Which parts of the package are usable with earlier TCL versions? Threading extension support Tcl8.3.2+ on all commands except: thread::join (and the thread::create -joinable) thread::transfer 7. What does happen if several threads send scripts to the same target Scripts will be entered in the event loop of the target thread and executed one after another on FIFO basis. 1. What are the support calls that are needed by the thread package Eh, the Tcl API ones. Grep all source files for Tcl_* and you'll get the list. I do not think that those need to be referenced in the docs. 2. http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/tcl/thread/doc/man/thread.n?rev=1.6&content-type=text/vnd.viewcvs-markup 2.2.1 Is the identifier still valid if a thread was terminated? Can the thread ID be reused? Identifier is valid until thread::join for threads created with "-joinable" flag. Identifier cannot be reused. It is generated by the system for each new thread. Identifier goes out of scope when the thread is joined. Thread is joined by the Tcl API (implicitly) or by user (using the thread::join command) explicitly. 2.2.2 How much "bookkeeping" is performed by your package Tough question. Define "bookkeeping". 2.2.3 Can a finished thread be restarted or brought it to wait for new incoming commands No. 3.1 Ok. 3.2. I plan to do pools on C-level, because it is so common and very useful. Other threading paradigms like workpile and others can be implemented on Tcl level, which is out of the scope of the threading extension per-se. 4. thread::create ?-preserved? ?-joinable? ?script? 5. A good point. 6. Another good point :) ---------------------------------------------------------------------- Comment By: Markus Elfring (elfring) Date: 2002-08-07 13:35 Message: Logged In: YES user_id=572001 1. Which parts of the package are usable with earlier TCL versions? ---------------------------------------------------------------------- Comment By: Markus Elfring (elfring) Date: 2002-08-07 13:30 Message: Logged In: YES user_id=572001 7. What does happen if several threads send scripts to the same target thread? ---------------------------------------------------------------------- Comment By: Markus Elfring (elfring) Date: 2002-08-07 13:03 Message: Logged In: YES user_id=572001 1. What are the support calls that are needed by the thread package? Do you want to add such detailed dependencies to the documentation? 2. Sorry, I don't see it in "http://cvs.sourceforge.net/cgi- bin/viewcvs.cgi/*checkout*/tcl/thread/doc/html/". 2.2.1 Is the identifier still valid if a thread was terminated? Can the thread ID be reused? When is the identifier really destroyed? 2.2.2 How much "bookkeeping" is performed by your package? 2.2.3 Can a finished thread be restarted or brought it to wait for new incoming commands? 3.1 Or the thread halts until a script is received... Please add this "until" cases to the function description "thread::wait". 3.2 "Thread pools" is a good thing. How do you think about the suggestion to provide more "well known situation handling" by the package? 4. Do you think about a function like "create -preserve ..."? 5. Can these answers pe put into the documentation, too? 6. I thank you also for your answers about your plans. I suggest that you put your plans and ideas for next releases into a document "Road Map" that the SourceForge audience can read. I think that it would save you some questions (from me and others). ---------------------------------------------------------------------- Comment By: Zoran Vasiljevic (vasiljevic) Date: 2002-08-06 16:13 Message: Logged In: YES user_id=95086 1. Because Tcl API < 8.4 misses neccessary support calls. 2. Documentation defines "thread::create" and thread::join" commands. It is not clear that these two are only for 8.4 core but this is corrected in the CVS. 2.1 Thread may exit as and when it wishes. The thread::join assures that the caller gets its return code (0-ok, 1-error) and that that caller knows exactly that the thread has exited. 2.2 The exiting thread removes itself from the list. If it is created as joinable, its status can be collected with the thread::join". 3.1 Yes. The script halts until the thread does the thread::unwind or the caller does the thread::release. After that, all commands *after* the thread.:wait will be executed and at the end of the script, the thread will exit. One can do an thread::exit instead of thead::unwind. This causes the thread to exit immediately. This is not advised, though (see the docs about thread::exit and thread::unwind). 3.2 The function which creates a thread and waits for its exit is already in place. Just do a thread::create "..." where "..." is a Tcl script. Thread will be created, it will run the script and exit when done. This will trigger one thread only. I'm planning to implement thread pools where one can create several threads, not just one, and feed them with work. 4. thread::create creates a non-reserved thread. This is just for backward compatiblity. I wanted to make and option for thread::create where one can create and reserve the thread in one atomic operation. This will be included in next release of the extension. 5.1 All Tcl variables are bound to a thread. You can't share Tcl variables between threads. You can share their values by putting them in thread shared arrays. Only means of exchanging data between threads is over script-passing or thread shared arrays. 5.2. tsv::names is an empty list on thread start. All thread shared variables are created by user action. ---------------------------------------------------------------------- Comment By: Markus Elfring (elfring) Date: 2002-08-06 15:33 Message: Logged In: YES user_id=572001 1. Why does this work only with "Tcl8.4+" and not with earlier TCL versions? 2. I need additional information to understand your code example because I have not found it in the documentation. The key part is the option "joinable". 2.1 What does happen if a thread exits before the function "join" is called? I assume that the thread object and its corresponding identifier are destroyed automatically. 2.2 Or does it work like this? The thread's execution is stopped or has reached its end. But this is not the same as an exit which terminates the object. - The "$tid" is still valid because the finished thread remains stored in a list and the function call "join" removes just a list element. 3.1 A thread that called "thread::wait" is waiting for incoming TCL scripts. Is this waiting stopped if a script should be executed by the function call "thread::send"? Should the target thread repeat the "thread::wait" call? 3.2 I suggest a function that can be offered in the package to wait for more well known situations. Your example can be one "application" for this. How do you think about a (little) function that would combine thread creations and the wait for their exit? What other "well known situations" do you know? 4. PLease clarify the relation between the functions "thread::create" and "thread::preserve". What is the reference counter's value for a new thread? Can a new thread be "preserved" automatically? 5.1 How do you work with "global" variables (for example: TCL globals or variables that are global inside a namespace) in the thread package? Do they belong to the first thread? 5.2 Does an initial set of shared variables exist? What is the result of a function call "[tsv::names]" after process start? ---------------------------------------------------------------------- Comment By: Zoran Vasiljevic (vasiljevic) Date: 2002-08-06 12:45 Message: Logged In: YES user_id=95086 Consider: foreach script {{set a 1} {set b 2} {set c 3}} { lappend tids [thread::create -joinable $script] } foreach tid $tids { thread::join $tid } This works for Tcl8.4+. The 8.4 is really best choice for threads. Out of this, you might be able to construct your own wait function in Tcl. The "thread::wait" is meant to put the thread in the event loop. It is not designed to wait for the thread exit or such. ---------------------------------------------------------------------- Comment By: Markus Elfring (elfring) Date: 2002-08-06 02:56 Message: Logged In: YES user_id=572001 Well, I've got another idea. When we talk about shell background processes, then we talk also about the "special" case that there is only one foreground process. I think that the shell command "wait" does nothing more than to wait until one process is left running again. How can we map this situation to threads? I think that it's unusual to talk about foreground and background threads because all threads are executed at the same level. We divide threads by their responsibility. But it's more common to call them manager and worker threads. Several manager threads that delegate jobs to the worker threads are started at the beginning of a process. My goal is to achieve a situation where the set of running threads is the same after some worker threads were executed. I would like the function "thread::wait" to be adjusted so that it does an other wait instead of an "vwait". I suggest the following function calls. # Store current thread situation set foreground [thread::names] # Create some "background"/worker threads foreach t {function1 function2 function3} {thread::create $t} # Wait until the additional threads finished execution by waiting that the list of running threats will be "restored". thread::wait -on IDs $foreground ---------------------------------------------------------------------- Comment By: Zoran Vasiljevic (vasiljevic) Date: 2002-08-06 01:11 Message: Logged In: YES user_id=95086 Well, nobody ever claimed so :) To wait for a thread exit, one should create a joinable thread and then use thread::join. This is the proper way to wait for a thread to exit. This, however works only for Tcl8.4+. Another way is to do a "thread::release -wait" which will work for 8.3 as well. To "vwait" on a variable which is set by the remote thread is just an asynchronous message passing between the target thread and the caller thread. This is done in order to simplify interaction between threads where one (the caller) is doing something in the event loop, wants to delegate some work to target thread and "vwait" on the result to come back without blocking his event loop. So, these two things are entirey different. Now, what was your *exact* question? ---------------------------------------------------------------------- Comment By: Markus Elfring (elfring) Date: 2002-08-05 23:43 Message: Logged In: YES user_id=572001 I'm not sure if a wait for a thread exit is the same as a wait for a variable (function "vwait") that is set by a thread. ---------------------------------------------------------------------- Comment By: Zoran Vasiljevic (vasiljevic) Date: 2002-07-24 20:34 Message: Logged In: YES user_id=95086 6. What is your answer for the missing parameter lists? They are missing on most functions. I will fix this sometimes end of this week. 7.2 I will add the tutorial-style docs where such (and alike) examples should be included and explained. I expect to have the tutorial for the next 2.5 release. 10. You're right. I think I have mentioned the "pair" function in the corresponding function explanation. If not, I'll add that. ---------------------------------------------------------------------- Comment By: Markus Elfring (elfring) Date: 2002-07-23 20:59 Message: Logged In: YES user_id=572001 6. What is your answer for the missing parameter lists? 7.2 Please put this important information with some explanation into the documentation. 9. It will support the solution for discussion topic 7.1. How many iterations are needed in the for loop to call the vwait function? 10. It is clear that several functions build a pair that must be called one after the other. - create / destroy - preserve / release - lock / unlock This leads to the concept "allocator" in object oriented programming. (https://sourceforge.net/tracker/?func=detail&aid=581355&group_id=7586&atid=357586) 12. Thanks for your offer to send mail to you directly. I prefer our discussion this way at the moment. I think that a part of the SourceForge audience is interested in these improvements, too. They can follow it by this feature request. ---------------------------------------------------------------------- Comment By: Zoran Vasiljevic (vasiljevic) Date: 2002-07-23 16:24 Message: Logged In: YES user_id=95086 7.1 Hey, this little bastard is not there? I'll add that, of course. 7.2 Must a "vwait" as often be called as a thread will store the execution result in the specified variable? Yes. Did you call it twice in your example because you created two threads with the same variable "result"? Yes. 8. What will happen in a "thread::send" if an error occurs or the TCL function "return" is called before the execution result is stored in the specified variable? Result of the script will ALWAYS be stored in the "result" variable, no matter what happes. Event if the target thread exits in the middle of processing. 9. I will consider this. This introspective capability should be good for statistics and such. I agree. Some of suggested functionality is quite easy to do, but some is really impossible (mutex is_locked, most notably). 10. What functions you're referring to? 11. Possible (in some situations) but not so simple as it might appear on the first glance. Doing a unlock of a chain of mutexes of notify of a chain of cond vars is *very* tricky and I'd say, impossible in most cases. Preserve/release should be easier, though. ---- If we continue at this pace, we'll blow the SF storage ;-) You can always send me an email directly: zo...@ar... ---------------------------------------------------------------------- Comment By: Markus Elfring (elfring) Date: 2002-07-23 15:54 Message: Logged In: YES user_id=572001 1. I would say: This is still missing. 6. The parameter lists are missing after all tsv and some thread function names, too. (http://cvs.sourceforge.net/cgi- bin/viewcvs.cgi/*checkout*/tcl/thread/doc/html/thread.html?rev=1.3) The parameters for a few thread functions are displayed in an italic font. 7.1 I miss a separate description for the function "vwait". It is referenced at severall places in the file "thread.html". Are going to add an documentation entry "thread::vwait"? 7.2 Must a "vwait" as often be called as a thread will store the execution result in the specified variable? Did you call it twice in your example because you created two threads with the same variable "result"? 8. What will happen in a "thread::send" if an error occurs or the TCL function "return" is called before the execution result is stored in the specified variable? 9. I would like to get additional infomation about the thread usage by the library. 9.1.1 thread::mutex names This should do the same for mutexes as "thread::names" does for threads. 9.1.2 thread::mutex is_locked Query the state of a single mutex or a list of mutexes. 9.1.3 thread::mutex locked Get a list of mutex IDs that are used by locked threads. 9.1.4 thread::mutex locked_threads Get a list of thread IDs that are locked at the moment. 9.2.1 thread::cond names This should do the same for condition variable as "thread::names" does for threads. 9.2.1 thread::cond waiting Get a list of thread IDs that are waiting on condition variables. The list should be configurable to limit the size of the returned data. 9.2.2 thread::cond is_waiting Query the state of a condition variable or a list of condition variables. 10. Can you offer pictures that show the depencies of the functions? The sequence for function calls are clear for some of them but not for all. Please show a call order or a workflow. 11. Some functions seem to expect only a single ID. Does it make sense to pass a list of IDs for the functions "preserve", "release", "destroy", "lock", "unlock" or "notify"? ---------------------------------------------------------------------- Comment By: Zoran Vasiljevic (vasiljevic) Date: 2002-07-23 14:34 Message: Logged In: YES user_id=95086 4. If you think that locked script evaluation belongs to the TCL core, will you ask a "core" developer? Or is it worth to turn it into a TCL feature request? To put the functionality in the core, one ought to write a TIP. After being discussed by the TCT, it may eventually go into the core. If you propose a TIP, you better also give the reference implementation, otherwise it will slip under the carpet. I see this functionality as a convenience of avoiding a mutex around some script code. It is just too much of work to get it in the core and it really adds no substantial functionality. But you might try writing a TIP for this. ---------------------------------------------------------------------- Comment By: Markus Elfring (elfring) Date: 2002-07-23 14:21 Message: Logged In: YES user_id=572001 3. I've submitted two feature requests. https://sourceforge.net/tracker/index.php?func=detail&aid=585358&group_id=16386&atid=366386 https://sourceforge.net/tracker/index.php?func=detail&aid=585369&group_id=16386&atid=366386 4. If you think that locked script evaluation belongs to the TCL core, will you ask a "core" developer? Or is it worth to turn it into a TCL feature request? 5. Interesting development ... ---------------------------------------------------------------------- Comment By: Zoran Vasiljevic (vasiljevic) Date: 2002-07-23 09:47 Message: Logged In: YES user_id=95086 5. It seems that you duplicate some TCL functions (e. g. for lists and arrays) to make them an "atomic operation". Are you going to continue this way with functions like "string", "regexp" or "keyed list" from the TclX extension? The keyed-list is already done, but I did not include it in the current release since it needs some more testing. I did not look into others because I had no need for them. Mainly because strings are strings and you're not gaining too much. For all Tcl datatypes which *have* internal object representation, this can be a big gain. For all others it is just a simple convenience of saving yourself one mutex. 1. I would like to read more about error handling. Are any errors or exceptions thrown by the thread and tsv functions? Generally, syntax errors. thread::send reports error when the posted script results in error. Most tsv::* functions report error when they are not able to find the array or the element, but I think this should be already documented. If not, please report and I'll correct it. 2. Is it possible to divide the function list into two sections "thread" and "tsv"? Good point. I did not like it this way either in the first place. But one has to have some room for future improvements ;-) 3. Can each entry in the synopsis link to the function description? Unfortunately, this is TMML and I must play by the rules. TMML does not provide such funtionality AFAIK. But you can always post a question to TMML group (on SF). 4. You offer protection for thread shared variables. Does it make sense to extend this technique for TCL scripts? You can to this with a simple mutex arround the script. The "locked" script evaluation should really be the feature of the Tcl core, IMHO. But, this is tricky stuff and might leed to recursive locks, reader/writer locks and all other sort of things. If you were the developer, what path would you take? Just curious... ---------------------------------------------------------------------- Comment By: Markus Elfring (elfring) Date: 2002-07-22 23:02 Message: Logged In: YES user_id=572001 5. It seems that you duplicate some TCL functions (e. g. for lists and arrays) to make them an "atomic operation". Are you going to continue this way with functions like "string", "regexp" or "keyed list" from the TclX extension? ---------------------------------------------------------------------- Comment By: Markus Elfring (elfring) Date: 2002-07-22 22:42 Message: Logged In: YES user_id=572001 I've read the file "thread.html" from the CVS interface. 1. I would like to read more about error handling. Are any errors or exceptions thrown by the thread and tsv functions? 2. Is it possible to divide the function list into two sections "thread" and "tsv"? 3. Can each entry in the synopsis link to the function description? 4. You offer protection for thread shared variables. Does it make sense to extend this technique for TCL scripts? It would be nice to reduce calls into the thread library if all the locking and unlocking to execute a script is done as a service. How do you create a kind of "thread shared function"? ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2002-07-22 21:08 Message: Logged In: NO 1.1 I think that it can happen in your code example that the second threads exits while the for loop is still waiting for the first thread to exit. Or is this a wrong assumption from me? Is it enough to wait for the change of the variable? Is this the only "wrapping" that needs to be done The example waits for 2 triggers on the "result" variable. The variable will be triggered twice, each time when a thread is finished with processing the script. Since we have passed 2 scripts to two threads, it will work. Remember that the events accumulate in the caller's thread event loop, so there is no race condition, you have anticipated. Yes, this is the only really wrapping needed. One can of course do much much more there, but this is out of the scope of this little example. 1.2 Are you going to provide thread pooling in the library? I play with the thought to abstract thread-pools on the C-level because it is such a common construct. I can't say when I'll be attending this. You might jump in if you like and join the show :) 3. Will an example for this application in the documentation? As I said, 2.5 will have a tutorial section where I plan to present some common usage patterns. 4. Please send a link to the documentation for the 2.4 version. You can check-out the docs from CVS on SF (here). It is under Tcl project, thread/doc directory. But I'll be wrapping the release of 2.4 now, so you'll be able to get the whole tarball (zipball) from the SF download area. BTW, you might switch to c.l.t and start a thread so everybody else also gets involved. ---------------------------------------------------------------------- Comment By: Markus Elfring (elfring) Date: 2002-07-22 20:24 Message: Logged In: YES user_id=572001 Thanks for your answers. 1.1 I think that it can happen in your code example that the second threads exits while the for loop is still waiting for the first thread to exit. Or is this a wrong assumption from me? Is it enough to wait for the change of the variable? Is this the only "wrapping" that needs to be done? 1.2 Are you going to provide thread pooling in the library? 3. Will an example for this application in the documentation? 4. Please send a link to the documentation for the 2.4 version. ---------------------------------------------------------------------- Comment By: Zoran Vasiljevic (vasiljevic) Date: 2002-07-22 19:53 Message: Logged In: YES user_id=95086 1. I've read that the function "vwait" can only wait for the exit of a single thread. The "vwait" waits on the variable. The variable can be set from one or more threads, so by some simple tcl wrap functions you can implement pretty powerful thread pool environment. 2. Are you going to provide the "clever wrapping" for all users of the TCL thread library? Yes, I do plan to make some tutorial docs with some simple examples, which could be used as a starting point for your own development. This is scheduled for 2.5 version which I'd like to release in 2-3 months. The 2.4 version is going to be released today or tomorrow since I've solved all pending doc and MinGW issues :-) 3. Can a running thread put to "sleep" and wake up automatically if one or more other threads have done their processing? Of course. The best way is to use thread::mutex and thread::cond synchronization primitives which are built into the extension since 2.2 version. ---------------------------------------------------------------------- Comment By: Markus Elfring (elfring) Date: 2002-07-22 19:30 Message: Logged In: YES user_id=572001 1. I've read that the function "vwait" can only wait for the exit of a single thread. 2. Are you going to provide the "clever wrapping" for all users of the TCL thread library? 3. Can a running thread put to "sleep" and wake up automatically if one or more other threads have done their processing? ---------------------------------------------------------------------- Comment By: Zoran Vasiljevic (vasiljevic) Date: 2002-07-03 18:47 Message: Logged In: YES user_id=95086 1. In couple of days the 2.4 version of the extension is going to be released. This will include complete documentation in both html and unix man formats. 2. With the 2.4 version, it is possible to post scripts to threads asynchronously and wait for results in the event loop of the poster. This way, thread-pools can be implemented. Consider the following example: set t1 [thread::create] set t2 [thread::create] thread::send -async $t1 "set a 1" result thread::send -async $t2 "set b 2" result for {set i 0} {$i < 2} {incr i} { vwait result } With some clever wrapping, you can implement the desired functionality on the script level very easily. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=360894&aid=577014&group_id=10894 |