tclhttpd-users — Discussion and general Q&A for TclHttpd

[Tclhttpd-users] core dump on x86_64

[Wart <wa...@ko...>]

The crypt extension causes a core dump on Fedora x86_64.

Steps to reproduce:

1) Build and install tclhttpd on a Fedora x86_64 machine.
2) Start tcl and load the crypt extension
bash# tclsh
% package require crypt
1.0
% crypt foo ba
Segmentation fault (core dumped)

A compilation warning points to the problem:

src/crypt.c: In function `Crypt_Cmd':
src/crypt.c:63: warning: cast to pointer from integer of different size

It seems that the #include "unistd.h" is working, but isn't providing
the real declaration for crypt, as the compiler still thinks that it
returns an int, not char *.  Adding #include <crypt.h> fixes the
problem.  Perhaps we should be including crypt.h on systems where it's
available?

--Mike

[Tclhttpd-users] I just installed an implemented the 3.5.1 release -- and

[Shawn K. <sha...@ea...>]

It was virtually painless upgrading from my old tclhttpd version on my win2k
server.  All of the flaws were mine, so far  .. . I have found no bugs.

Excellently done.

Thanks

Shawn Kielty
http://www.shawnkielty.com
(480)636-1524
(510) 828-9411

Re: [Tclhttpd-users] Problem with form data in threaded tclhttpd

[David G. <dav...@po...>]

George Harvey <fr...@di...> wrote:

>The failure occurs when a script does not get back
>the page it expects and can happen anytime from a few minutes to a
>couple of hours into the test.

Does the server sockets die?  You can try replacing the socket channel
implementation entirely with IOCPSOCK (http://sf.net/projects/iocpsock).

The current core implemention for sockets has trouble on windows and tends to
drop accepts (meaning some are ACK'd to the client by AFD, but at the
application level no FD_ACCEPT is sent by winsock, making tclhttpd not know a
new client has connected).  IOCPSOCK fixes this and numerous other performance
problems with server sockets on windows NT.
-- 
David Gravereaux <dav...@po...>
[species: human; planet: earth,milkyway(western spiral arm),alpha sector]

[Tclhttpd-users] Performance

[Colin M. <co...@ch...>]

Would tclhttpd performance be improved by replacing this 'if [...]' and
this 'if ![...]' with their bracketed equivalents?
-- 
Colin McCormack <co...@ch...>

[Tclhttpd-users] Re: I need your advice, Cookies - global state - interrupted pages

[Colin M. <co...@ch...>]

On Tue, 2004-10-19 at 11:59, Brent Welch wrote:
> The Doc modules and template handlers make the assumption that
> page processing runs to completion, so they use global variables instead
> of per-connection state.  This has obvious problems, and I apologize
> for that state of affairs.

It doesn't seem all that dire to me - page and Cookies are the only
globals I can spot right away which do that.

Httpd_Suspend and _Resume make sure page is stashed away and restored,
Cookies() shouldn't be being used for storing temp cookies at all.

> But, for simple template processing with
> a non-threaded interpreter and no calls to vwait, it has worked
> reliably for me.

Yes.  I hadn't noticed any problems either.

I wonder where unexpected vwaits could arise from a naive programmer
using tcllib modules and such.  For example, calling out to http will
cause grief, but I don't think many people realise it - I know I didn't
(not that I've been bitten, that I know of.)

> If you use the threaded interpreter, then the connection state is
> copied into the interp of the worker thread, and so that model should
> continue to work OK.

> However, anything that relies on the env() global variable could get into trouble.

Nothing I write depends on env(), but I note it's set up for templates. 
If the data in it can't be relied upon, then it should (IMHO) be
deprecated.

> The ncgi module, however,
> is explicitly initialized by TclHttpd and that doesn't use the env()
> array.  If you are calling ncgi::reset yourself, then you should
> dig out the per-connection query string instead of letting ncgi
> fish it out of the environment.  Look at the existing calls to
> ncgi::reset to see what I mean.

> The Thread layer in TclHttpd doesn't explicitly set up the ncgi
> module, so if you did that yourself, you might have missed this
> subtle point.
> 
> About cookies, they have a similar issue of no easily obtainable
> connection state, hence the evil global.

Isn't the current socket always available by [Httpd_CurrentSock] or
something?  In that case it's easy enough - if not particularly
efficient - to discover the connection, thence the connection state.

Since there's no necessary dependence on global state, this raises the
question of the best interface to cookies.

There seem to be at least two ways to do it, and they seem to be
radically inconsistent - this matters because you can't write a module
of code which uses cookies without running the risk that it won't work
when someone uses them differently.

When you have a minute can you outline the history of the Cookie_* and
the Httpd_Cookie* interfaces?  What problems was Cookie_* trying to
solve?  It's trying to store stuff temporarily before shuffling it out
via Httpd_Cookie*, but the same could be done without a global.

> I really think the best way to deal with this is if you are
> doing requests that get interrupted and so violate the
> "run-to-completion" event semantics, then you move those
> into worker threads using the threaded version.

Another wrinkle is the virtual host stuff, which creates per-host slave
interpreters with copies of some of the tclhttpd code.

> Then you can
> use a per-thread interpreter and not worry about sharing
> globals.  The only global you must avoid is env().

I'm tending to move toward a per-session interpreter and keeping the
event-driven semantics, 

-- 
Colin McCormack <co...@ch...>

RE: [Tclhttpd-users] Re: I need your advice, Cookies - global state - interrupted pages

[Shawn K. <sha...@ea...>]

Hi Folks --

What I would like to be able to do is to deliver an image file though a
request to it's URL (say /image.jpg) and have a template process the
comments in the file into a web page, along with copyright and header
information, and the original image -- Probably this is possible -- but the
problem I foresee is that generically processing a jpg file into a web page
is going to interfere with the requests to use it in a page.

What I really want is to assure that any request for a single image on my
site -- is presented with a specific title and description, credits and
copyright, and perhaps some keywords, and the anticipated header and footer,
with the real benefit being that adding an image to my site would be as
simple as adding the file in an appropriate directory - with comments.

Does anyone have any suggestions about good ways to go about this?

Thanks

Shawn

-----Original Message-----
From: tcl...@li...
[mailto:tcl...@li...]On Behalf Of Brent
Welch
Sent: Monday, October 18, 2004 7:00 PM
To: Colin McCormack
Cc: tcl...@li...
Subject: [Tclhttpd-users] Re: I need your advice, Cookies - global state
- interrupted pages


The Doc modules and template handlers make the assumption that
page processing runs to completion, so they use global variables instead
of per-connection state.  This has obvious problems, and I apologize
for that state of affairs.  But, for simple template processing with
a non-threaded interpreter and no calls to vwait, it has worked
reliably for me.

If you use the threaded interpreter, then the connection state is
copied into the interp of the worker thread, and so that model should
continue to work OK.  However, anything that relies on the env()
global variable could get into trouble.  The ncgi module, however,
is explicitly initialized by TclHttpd and that doesn't use the env()
array.  If you are calling ncgi::reset yourself, then you should
dig out the per-connection query string instead of letting ncgi
fish it out of the environment.  Look at the existing calls to
ncgi::reset to see what I mean.

The Thread layer in TclHttpd doesn't explicitly set up the ncgi
module, so if you did that yourself, you might have missed this
subtle point.

About cookies, they have a similar issue of no easily obtainable
connection state, hence the evil global.

I really think the best way to deal with this is if you are
doing requests that get interrupted and so violate the
"run-to-completion" event semantics, then you move those
into worker threads using the threaded version.  Then you can
use a per-thread interpreter and not worry about sharing
globals.  The only global you must avoid is env().

>>>Colin McCormack said:
 > Hi Brent,
 >
 > Some code in cookie.tcl relies on the global Cookie array to stage
 > cookie data before passing it to Httpd_ files which actually send the
 > cookies.
 >
 > The procs in question are called by template and direct modules.
 >
 > I think that the intention is to allow data to be passed back to the
 > main interpreter when it has been set in a slave interpreter, because
 > that's the only reason I can imagine for them to be set up and called
 > the way they are.
 >
 > There's a bug in this approach, because the data can be corrupted by a
 > page interrupting another page.
 >
 > (Incidentally, there's another bug hiding in the way we use global page
 > to pass back the dynamic status of generated templates ... although this
 > is probably less dangerous than cookies being overwritten by interloping
 > pages.)
 >
 > Perhaps this is the cause of "[760802] data(mime,cookies) - Cookie_Get
 > out of sync" at http://snipurl.com/9t7n ... I don't know.
 >
 > Can you help me by telling me what you understand to be the purpose of
 > the Cookie_* setting and getting commands, so I can try to change the
 > implementation to avoid the race, by using the data array to store
 > cookies.  I would have done this already, but I noticed that the Httpd_
 > cookie stuff already does it, and am puzzled by there being two ways to
 > do (almost) the same thing.
 >
 > Also, should one of the interfaces be deprecated, or is there some
 > subtle difference between them, which eludes me?  This is quite
 > possible.
 >
 > More generally ... would it be worth adding an optional quasi-semaphore
 > to the Httpd protocol stack which just increments a global on receipt of
 > a protocol request and decrements it on completion of processing, to
 > detect interleaving of requests?  It'd have to take account of
 > Suspend/Resume too.  If the counter ever had a value other than 0 or 1,
 > it would signal a potential race, which could be logged.
 >
 > Such a facility could be used for early detection of some of the race
 > conditions we're seeing reported.
 > --
 > Colin McCormack <co...@ch...>
 >

--
Brent Welch
Software Architect, Panasas Inc
Delivering the premier storage system for scalable Linux clusters

www.panasas.com
we...@pa...




-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl
_______________________________________________
TclHttpd-users mailing list
Tcl...@li...
https://lists.sourceforge.net/lists/listinfo/tclhttpd-users

[Tclhttpd-users] Re: I need your advice, Cookies - global state - interrupted pages

[Brent W. <we...@pa...>]

The Doc modules and template handlers make the assumption that
page processing runs to completion, so they use global variables instead
of per-connection state.  This has obvious problems, and I apologize
for that state of affairs.  But, for simple template processing with
a non-threaded interpreter and no calls to vwait, it has worked
reliably for me.

If you use the threaded interpreter, then the connection state is
copied into the interp of the worker thread, and so that model should
continue to work OK.  However, anything that relies on the env()
global variable could get into trouble.  The ncgi module, however,
is explicitly initialized by TclHttpd and that doesn't use the env()
array.  If you are calling ncgi::reset yourself, then you should
dig out the per-connection query string instead of letting ncgi
fish it out of the environment.  Look at the existing calls to
ncgi::reset to see what I mean.

The Thread layer in TclHttpd doesn't explicitly set up the ncgi
module, so if you did that yourself, you might have missed this
subtle point.

About cookies, they have a similar issue of no easily obtainable
connection state, hence the evil global.

I really think the best way to deal with this is if you are
doing requests that get interrupted and so violate the
"run-to-completion" event semantics, then you move those
into worker threads using the threaded version.  Then you can
use a per-thread interpreter and not worry about sharing
globals.  The only global you must avoid is env().

>>>Colin McCormack said:
 > Hi Brent,
 > 
 > Some code in cookie.tcl relies on the global Cookie array to stage
 > cookie data before passing it to Httpd_ files which actually send the
 > cookies.
 > 
 > The procs in question are called by template and direct modules.
 > 
 > I think that the intention is to allow data to be passed back to the
 > main interpreter when it has been set in a slave interpreter, because
 > that's the only reason I can imagine for them to be set up and called
 > the way they are.
 > 
 > There's a bug in this approach, because the data can be corrupted by a
 > page interrupting another page.
 > 
 > (Incidentally, there's another bug hiding in the way we use global page
 > to pass back the dynamic status of generated templates ... although this
 > is probably less dangerous than cookies being overwritten by interloping
 > pages.)
 > 
 > Perhaps this is the cause of "[760802] data(mime,cookies) - Cookie_Get
 > out of sync" at http://snipurl.com/9t7n ... I don't know.
 > 
 > Can you help me by telling me what you understand to be the purpose of
 > the Cookie_* setting and getting commands, so I can try to change the
 > implementation to avoid the race, by using the data array to store
 > cookies.  I would have done this already, but I noticed that the Httpd_
 > cookie stuff already does it, and am puzzled by there being two ways to
 > do (almost) the same thing.
 > 
 > Also, should one of the interfaces be deprecated, or is there some
 > subtle difference between them, which eludes me?  This is quite
 > possible.
 > 
 > More generally ... would it be worth adding an optional quasi-semaphore
 > to the Httpd protocol stack which just increments a global on receipt of
 > a protocol request and decrements it on completion of processing, to
 > detect interleaving of requests?  It'd have to take account of
 > Suspend/Resume too.  If the counter ever had a value other than 0 or 1,
 > it would signal a potential race, which could be logged.
 > 
 > Such a facility could be used for early detection of some of the race
 > conditions we're seeing reported.
 > -- 
 > Colin McCormack <co...@ch...>
 > 

--
Brent Welch
Software Architect, Panasas Inc
Delivering the premier storage system for scalable Linux clusters

www.panasas.com
we...@pa...

Re: [Tclhttpd-users] env array pitfalls with threaded tclhttpd

[Brent W. <we...@pa...>]

Tcl goes to great lengths to share the env() array among all threads,
so it is now not that safe to use :-(
We really need to be using, instead, the nsarr stuff, the atomically
shared array-like values that are provided by the latest versions
of the Thread extension.

>>>George Harvey said:
 > Hi,
 > 
 > Following on from my cookie problem last week, I'm now seeing other
 > failures caused by one thread updating the env() array while another
 > thread is using it. In particular, I get an occasional 'no such
 > variable' error while reading an env entry even though the code
 > explicitly checks that it exists before using it. This is most likely
 > because Cgi_SetEnvAll unsets most of the array before re-populating it
 > from the socket data. It isn't just my code that breaks, the env array
 > is also used by some of the standard library modules and I caught one
 > failure when msgcat couldn't read env(LANG).
 > 
 > I can see solutions for some of these issues, for example adding LANG to
 > the 'env-pass' list should prevent it from being unset and I can access
 > the CGI data directly from the socket data, but I was just wondering if
 > there is a more general solution to accessing env() data in a threaded
 > environment.
 > 
 > Regards,
 > George
 > 
 > 
 > -------------------------------------------------------
 > This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
 > Use IT products in your business? Tell us what you think of them. Give us
 > Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out 
more
 > http://productguide.itmanagersjournal.com/guidepromo.tmpl
 > _______________________________________________
 > TclHttpd-users mailing list
 > Tcl...@li...
 > https://lists.sourceforge.net/lists/listinfo/tclhttpd-users
 > 

--
Brent Welch
Software Architect, Panasas Inc
Delivering the premier storage system for scalable Linux clusters

www.panasas.com
we...@pa...

[Tclhttpd-users] env array pitfalls with threaded tclhttpd

[George H. <fr...@di...>]

Hi,

Following on from my cookie problem last week, I'm now seeing other
failures caused by one thread updating the env() array while another
thread is using it. In particular, I get an occasional 'no such
variable' error while reading an env entry even though the code
explicitly checks that it exists before using it. This is most likely
because Cgi_SetEnvAll unsets most of the array before re-populating it
from the socket data. It isn't just my code that breaks, the env array
is also used by some of the standard library modules and I caught one
failure when msgcat couldn't read env(LANG).

I can see solutions for some of these issues, for example adding LANG to
the 'env-pass' list should prevent it from being unset and I can access
the CGI data directly from the socket data, but I was just wondering if
there is a more general solution to accessing env() data in a threaded
environment.

Regards,
George

Re: [Tclhttpd-users] Problem with form data in threaded tclhttpd

[George H. <fr...@di...>]

On Sun, 24 Oct 2004 01:17:56 -0400
Angel Sosa <xt...@am...> wrote:

> tcl...@li... wrote:
> 
> >
> >Today's Topics:
> >
> >   1. Problem with form data in threaded tclhttpd (George Harvey)
> >
> >--__--__--
> >
> >Message: 1
> >Date: Tue, 12 Oct 2004 21:01:04 +0100
> >From: George Harvey <fr...@di...>
> >To: tcl...@li...
> >Subject: [Tclhttpd-users] Problem with form data in threaded tclhttpd
> >
> >Hi,
> >
> >I'm conducting some stress testing of an application, built on
> >threaded tclhttpd, and I'm seeing some intermittent failures which
> >appear to be related to the handling of form and cookie data returned
> >to the server. I'm still trying to narrow down the problem but I'd
> >like to describe the symptoms to see if anyone else has seen (or even
> >fixed!) anything similar.
> >
[ snip ]  
> >
> 1. What OS are you running the web server on?
> 
> 2. If possible please take some small packet traces.
> 
> 3. You may want to try and place some delays with in your code.
> Increase the delays untiil your custom code works correctly. This step
> is just a way of seeing how much of a delay needs to be introduced.
> The results can then be used for adjusting tclhttpd code if necessary.

I did resort to packet traces at one point in my investigation but the
problem turned out to be cookie data getting over-written on the server
when two requests arrived close together. I have replaced a call to
'ncgi::cookie' with one to 'Cookie_GetSock' and haven't seen any more
mis-matches between my cookie and form data.

Regards,
George

[Tclhttpd-users] I need your advice, Cookies - global state - interrupted pages

[Colin M. <co...@ch...>]

Hi Brent,

Some code in cookie.tcl relies on the global Cookie array to stage
cookie data before passing it to Httpd_ files which actually send the
cookies.

The procs in question are called by template and direct modules.

I think that the intention is to allow data to be passed back to the
main interpreter when it has been set in a slave interpreter, because
that's the only reason I can imagine for them to be set up and called
the way they are.

There's a bug in this approach, because the data can be corrupted by a
page interrupting another page.

(Incidentally, there's another bug hiding in the way we use global page
to pass back the dynamic status of generated templates ... although this
is probably less dangerous than cookies being overwritten by interloping
pages.)

Perhaps this is the cause of "[760802] data(mime,cookies) - Cookie_Get
out of sync" at http://snipurl.com/9t7n ... I don't know.

Can you help me by telling me what you understand to be the purpose of
the Cookie_* setting and getting commands, so I can try to change the
implementation to avoid the race, by using the data array to store
cookies.  I would have done this already, but I noticed that the Httpd_
cookie stuff already does it, and am puzzled by there being two ways to
do (almost) the same thing.

Also, should one of the interfaces be deprecated, or is there some
subtle difference between them, which eludes me?  This is quite
possible.

More generally ... would it be worth adding an optional quasi-semaphore
to the Httpd protocol stack which just increments a global on receipt of
a protocol request and decrements it on completion of processing, to
detect interleaving of requests?  It'd have to take account of
Suspend/Resume too.  If the counter ever had a value other than 0 or 1,
it would signal a potential race, which could be logged.

Such a facility could be used for early detection of some of the race
conditions we're seeing reported.
-- 
Colin McCormack <co...@ch...>

[Tclhttpd-users] TclHttpd-users digest, Vol 1 #534 - 1 msg

[Angel S. <xt...@am...>]

tcl...@li... wrote:

>Send TclHttpd-users mailing list submissions to
>	tcl...@li...
>
>To subscribe or unsubscribe via the World Wide Web, visit
>	https://lists.sourceforge.net/lists/listinfo/tclhttpd-users
>or, via email, send a message with subject or body 'help' to
>	tcl...@li...
>
>You can reach the person managing the list at
>	tcl...@li...
>
>When replying, please edit your Subject line so it is more specific
>than "Re: Contents of TclHttpd-users digest..."
>
>
>Today's Topics:
>
>   1. Problem with form data in threaded tclhttpd (George Harvey)
>
>--__--__--
>
>Message: 1
>Date: Tue, 12 Oct 2004 21:01:04 +0100
>From: George Harvey <fr...@di...>
>To: tcl...@li...
>Subject: [Tclhttpd-users] Problem with form data in threaded tclhttpd
>
>Hi,
>
>I'm conducting some stress testing of an application, built on threaded
>tclhttpd, and I'm seeing some intermittent failures which appear to be
>related to the handling of form and cookie data returned to the server.
>I'm still trying to narrow down the problem but I'd like to describe the
>symptoms to see if anyone else has seen (or even fixed!) anything
>similar.
>
>The application is written in Tcl, loaded from the custom directory and
>accessed as an Application Direct URL. To test it, I have written a
>script (using tclwebtest) which will select all the menus, fill in
>forms, follow links etc. To stress test it, I run multiple copies of the
>test script on different machines.
>
>What I'm seeing is that a single test script runs OK but multiple copies
>will eventually fail. The failure occurs when a script does not get back
>the page it expects and can happen anytime from a few minutes to a
>couple of hours into the test. I'm using a session ID number, stored in
>a cookie, to keep track of session state and when a failure occurs the
>server log shows a mis-match between the session ID and the expected
>query data. What appears to be happening is that the query data for one
>session is somehow getting passed to a different session.
>
>The intermittent nature of the failures suggest that they are timing
>dependent, most likely needing a second request to arrive within some
>critical time window during processing of the first request. I've been
>looking at the tclhttpd code and can see that all the header and query
>data is held in a per-socket array in the master thread before being
>copied to the worker thread so it is difficult to see how it could get
>mixed up. I'm still trying to narrow this down.
>
>Anyone seen anything like this?
>
>For reference, I'm running Tcl 8.4.7, tclhttpd 3.5.1, tcllib 1.6.1 and
>thread 2.5.2. Tclhttpd is configured to run 5 threads and I have checked
>that worker threads are being created as expected. The server platform
>is Slackware 9.1.
>
>Regards,
>George
>
>
>
>--__--__--
>
>_______________________________________________
>TclHttpd-users mailing list
>Tcl...@li...
>https://lists.sourceforge.net/lists/listinfo/tclhttpd-users
>
>
>End of TclHttpd-users Digest
>
>  
>
1. What OS are you running the web server on?

2. If possible please take some small packet traces.

3. You may want to try and place some delays with in your code. Increase 
the delays untiil your custom code works correctly. This step is just a 
way of seeing how much of a delay needs to be introduced. The results 
can then be used for adjusting tclhttpd code if necessary.

4. Just make sure that as you test and introduce delays, you have been 
taking packet traces.

5. If it's ok with you, I can diagnose traces for you!!

Good Luck

    From: xt...@am...

Re: [Tclhttpd-users] Problem with form data in threaded tclhttpd

[George H. <fr...@di...>]

On Wed, 13 Oct 2004 23:47:42 -0500
"Gerald W. Lester" <Ger...@co...> wrote:

> Colin McCormack wrote:
> 
> >On Thu, 2004-10-14 at 10:59, Jeff Smith wrote:
> >
> >  
> >
> >>I have seen some similar behaviour when I open
> >>multiple forms in a browser and that have the session
> >>ID stored in a cookie. I modified my application not
> >>to use cookies to store the session ID but instead
> >>stored the session ID in the form data and everything
> >>worked as expected.
> >>    
> >>
> >
> >Cookie seems to be a common thread ... is this session-cookie mapping
> >something you rolled yourself, or is it using the Session_Cookies
> >stuff?

It is my own code.

> >In any case, Cookies are stored in a global variable - from
> >discussions with Brent, it seems possible that globals can be
> >corrupted if tclhttpd enters the event loop via a (nested?) vwait and
> >another incoming request is serviced.
> >
> >A solution might be to store cookies per-socket.  Needs more feedback
> >and investigation.
> >  
> >
> Cookies *are* the problem.
> 
> You need to ensure you use the Cookie calls that read them from the 
> socket and not the uploaded directory.

Yes, I was using ncgi::cookie, which reads env(HTTP_COOKIE), which is
bad because the env() array is shared across all Tcl threads. I have now
modified my code to use Cookie_GetSock instead and the tests run for
much longer (5 hours) before failing. My debug log shows correct session
state for the new failures so it looks like the cookie problem is fixed
but I have a new, and more elusive, bug to track down :(

Regards,
George

Re: [Tclhttpd-users] Problem with form data in threaded tclhttpd

[Colin M. <co...@ch...>]

On Thu, 2004-10-14 at 15:03, Colin McCormack wrote:
> On Thu, 2004-10-14 at 14:47, Gerald W. Lester wrote:

> > You need to ensure you use the Cookie calls that read them from the 
> > socket and not the uploaded directory.

> What about setting Cookies?  Cookie_Set is in error, because it stores
> cookies in a global.  This is definitely a bug.  It should store them in
> per-socket data.  I'll fix this now.

> Bottom line: if you're using Cookie_Set, the cookie is going to be
> corrupted sooner or later.

Looking at the code more closely, it seems that there are two interfaces
to set cookies - the ones in cookie.tcl (Cookie_GetSock, Cookie_Set and
Cookie_Save) and the ones in httpd.tcl (Httpd_SetCookie and
Httpd_RemoveCookies).

The second interface stores cookies per-socket, and whatever cookies are
stored with Httpd_SetCookie are inevitably sent to the browser.

The first interface stores cookies in the evil global variable Cookie,
and transfers them to the per-socket structure for sending only when
Cookie_Save is called.

The reason for this seems to be that some evaluation occurs in a slave
interpreter (evaluation of virtual host interactions, template
substitution, etc) and variables set there are slave-local, and so have
to be retrieved by the main interpreter before they can be used by the
Httpd_*Cookie versions.

At least, this seems to be the intent of the calls to Cookie_Save in
template.tcl and direct.tcl.

So, you seemingly can't (directly) use the Httpd_*Cookie interface from
a slave interpreter (template, direct, virtual?) and the code which
marshals the cookies to the browser can't know whether there's some
interpreter somewhere with values it should send back up the pipe to the
browser.

The more I look at this, the less clear it is :)
-- 
Colin McCormack <co...@ch...>

Re: [Tclhttpd-users] Problem with form data in threaded tclhttpd

[Colin M. <co...@ch...>]

On Thu, 2004-10-14 at 14:47, Gerald W. Lester wrote:
> Colin McCormack wrote:
> 
> >On Thu, 2004-10-14 at 10:59, Jeff Smith wrote:
> >
> >  
> >
> >>I have seen some similar behaviour when I open
> >>multiple forms in a browser and that have the session
> >>ID stored in a cookie. I modified my application not
> >>to use cookies to store the session ID but instead
> >>stored the session ID in the form data and everything
> >>worked as expected.
> >>    
> >>
> >
> >Cookie seems to be a common thread ... is this session-cookie mapping
> >something you rolled yourself, or is it using the Session_Cookies stuff?
> >
> >In any case, Cookies are stored in a global variable - from discussions
> >with Brent, it seems possible that globals can be corrupted if tclhttpd
> >enters the event loop via a (nested?) vwait and another incoming request
> >is serviced.
> >
> >A solution might be to store cookies per-socket.  Needs more feedback
> >and investigation.

> Cookies *are* the problem.
> 
> You need to ensure you use the Cookie calls that read them from the 
> socket and not the uploaded directory.

What about setting Cookies?  Cookie_Set is in error, because it stores
cookies in a global.  This is definitely a bug.  It should store them in
per-socket data.  I'll fix this now.

Bottom line: if you're using Cookie_Set, the cookie is going to be
corrupted sooner or later.

> Since you are using sessions, I'd strongly suggest not using cookies, 
> but rather a hidden field on the form with the session ID and the 
> Session_Match API.

I like the Session_Cookie interface.

> If you really want cookies, repost next week (the Tcl Conference is 
> eating most of my time right now).  BTW -- this was covered in the 
> "Secrets of TclHttpd" tutorial Monday morning at the Conference.

Shame I missed it.
-- 
Colin McCormack <co...@ch...>

Re: [Tclhttpd-users] Problem with form data in threaded tclhttpd

[Gerald W. L. <Ger...@co...>]

Colin McCormack wrote:

>On Thu, 2004-10-14 at 10:59, Jeff Smith wrote:
>
>  
>
>>I have seen some similar behaviour when I open
>>multiple forms in a browser and that have the session
>>ID stored in a cookie. I modified my application not
>>to use cookies to store the session ID but instead
>>stored the session ID in the form data and everything
>>worked as expected.
>>    
>>
>
>Cookie seems to be a common thread ... is this session-cookie mapping
>something you rolled yourself, or is it using the Session_Cookies stuff?
>
>In any case, Cookies are stored in a global variable - from discussions
>with Brent, it seems possible that globals can be corrupted if tclhttpd
>enters the event loop via a (nested?) vwait and another incoming request
>is serviced.
>
>A solution might be to store cookies per-socket.  Needs more feedback
>and investigation.
>  
>
Cookies *are* the problem.

You need to ensure you use the Cookie calls that read them from the 
socket and not the uploaded directory.

Since you are using sessions, I'd strongly suggest not using cookies, 
but rather a hidden field on the form with the session ID and the 
Session_Match API.

If you really want cookies, repost next week (the Tcl Conference is 
eating most of my time right now).  BTW -- this was covered in the 
"Secrets of TclHttpd" tutorial Monday morning at the Conference.

-- 
+--------------------------------+---------------------------------------+
| Gerald W. Lester               | "The man who fights for his ideals is |
| Ger...@co...          |  the man who is alive." -- Cervantes  |
+--------------------------------+---------------------------------------+

Re: [Tclhttpd-users] Problem with form data in threaded tclhttpd

[Colin M. <co...@ch...>]

On Thu, 2004-10-14 at 10:59, Jeff Smith wrote:

> I have seen some similar behaviour when I open
> multiple forms in a browser and that have the session
> ID stored in a cookie. I modified my application not
> to use cookies to store the session ID but instead
> stored the session ID in the form data and everything
> worked as expected.

Cookie seems to be a common thread ... is this session-cookie mapping
something you rolled yourself, or is it using the Session_Cookies stuff?

In any case, Cookies are stored in a global variable - from discussions
with Brent, it seems possible that globals can be corrupted if tclhttpd
enters the event loop via a (nested?) vwait and another incoming request
is serviced.

A solution might be to store cookies per-socket.  Needs more feedback
and investigation.
-- 
Colin McCormack <co...@ch...>

Re: [Tclhttpd-users] Problem with form data in threaded tclhttpd

[Jeff S. <hea...@ya...>]

Hi,

I have seen some similar behaviour when I open
multiple forms in a browser and that have the session
ID stored in a cookie. I modified my application not
to use cookies to store the session ID but instead
stored the session ID in the form data and everything
worked as expected.

If you're using self checking forms you need to append
the session ID to Doc_Redirect so the next form in the
chain knows the session ID details.

Kind Regards

Jeff Smith

--- George Harvey <fr...@di...> wrote:

> Hi,
> 
> I'm conducting some stress testing of an
> application, built on threaded
> tclhttpd, and I'm seeing some intermittent failures
> which appear to be
> related to the handling of form and cookie data
> returned to the server.
> I'm still trying to narrow down the problem but I'd
> like to describe the
> symptoms to see if anyone else has seen (or even
> fixed!) anything
> similar.
> 
> The application is written in Tcl, loaded from the
> custom directory and
> accessed as an Application Direct URL. To test it, I
> have written a
> script (using tclwebtest) which will select all the
> menus, fill in
> forms, follow links etc. To stress test it, I run
> multiple copies of the
> test script on different machines.
> 
> What I'm seeing is that a single test script runs OK
> but multiple copies
> will eventually fail. The failure occurs when a
> script does not get back
> the page it expects and can happen anytime from a
> few minutes to a
> couple of hours into the test. I'm using a session
> ID number, stored in
> a cookie, to keep track of session state and when a
> failure occurs the
> server log shows a mis-match between the session ID
> and the expected
> query data. What appears to be happening is that the
> query data for one
> session is somehow getting passed to a different
> session.
> 
> The intermittent nature of the failures suggest that
> they are timing
> dependent, most likely needing a second request to
> arrive within some
> critical time window during processing of the first
> request. I've been
> looking at the tclhttpd code and can see that all
> the header and query
> data is held in a per-socket array in the master
> thread before being
> copied to the worker thread so it is difficult to
> see how it could get
> mixed up. I'm still trying to narrow this down.
> 
> Anyone seen anything like this?
> 
> For reference, I'm running Tcl 8.4.7, tclhttpd
> 3.5.1, tcllib 1.6.1 and
> thread 2.5.2. Tclhttpd is configured to run 5
> threads and I have checked
> that worker threads are being created as expected.
> The server platform
> is Slackware 9.1.
> 
> Regards,
> George
> 
> 
>
-------------------------------------------------------
> This SF.net email is sponsored by: IT Product Guide
> on ITManagersJournal
> Use IT products in your business? Tell us what you
> think of them. Give us
> Your Opinions, Get Free ThinkGeek Gift Certificates!
> Click to find out more
>
http://productguide.itmanagersjournal.com/guidepromo.tmpl
> _______________________________________________
> TclHttpd-users mailing list
> Tcl...@li...
>
https://lists.sourceforge.net/lists/listinfo/tclhttpd-users
> 



		
_______________________________
Do you Yahoo!?
Express yourself with Y! Messenger! Free. Download now. 
http://messenger.yahoo.com

[Tclhttpd-users] Problem with form data in threaded tclhttpd

[George H. <fr...@di...>]

Hi,

I'm conducting some stress testing of an application, built on threaded
tclhttpd, and I'm seeing some intermittent failures which appear to be
related to the handling of form and cookie data returned to the server.
I'm still trying to narrow down the problem but I'd like to describe the
symptoms to see if anyone else has seen (or even fixed!) anything
similar.

The application is written in Tcl, loaded from the custom directory and
accessed as an Application Direct URL. To test it, I have written a
script (using tclwebtest) which will select all the menus, fill in
forms, follow links etc. To stress test it, I run multiple copies of the
test script on different machines.

What I'm seeing is that a single test script runs OK but multiple copies
will eventually fail. The failure occurs when a script does not get back
the page it expects and can happen anytime from a few minutes to a
couple of hours into the test. I'm using a session ID number, stored in
a cookie, to keep track of session state and when a failure occurs the
server log shows a mis-match between the session ID and the expected
query data. What appears to be happening is that the query data for one
session is somehow getting passed to a different session.

The intermittent nature of the failures suggest that they are timing
dependent, most likely needing a second request to arrive within some
critical time window during processing of the first request. I've been
looking at the tclhttpd code and can see that all the header and query
data is held in a per-socket array in the master thread before being
copied to the worker thread so it is difficult to see how it could get
mixed up. I'm still trying to narrow this down.

Anyone seen anything like this?

For reference, I'm running Tcl 8.4.7, tclhttpd 3.5.1, tcllib 1.6.1 and
thread 2.5.2. Tclhttpd is configured to run 5 threads and I have checked
that worker threads are being created as expected. The server platform
is Slackware 9.1.

Regards,
George

Re: [Tclhttpd-users] Re: Checking Page flow redirects etc..????

[Colin M. <co...@ch...>]

On Fri, 2004-10-08 at 01:16, Michael Schlenker wrote:
> Colin McCormack schrieb:
> > On Fri, 2004-10-08 at 00:09, Nicolas Boretos wrote:
> >
> >>Is there any simple or built-in way in tclhttpd for checking your page 
> >>flow, meaning verify that you got to a page from another???
> >  
> > Not that I know of, but it would be a cool project.
> > 
> > What you want (I think) is to drive each user's interaction from a state
> > machine.  Their session starts in state 0, then depending on the inputs
> > they submit, it moves to a different state.
> > 
> > You define a FSM (aka DFSA) which determines which transitions are
> > legal.
> > 
> > There's a tcllib FSM package.  Could be useful for this kind of thing.
> 
> Something like joining the FSM package with the session interpreters and 
> adding some helpers to drive it?

Yes.  I just looked at http://tcllib.sourceforge.net/doc/fa.html and
http://tcllib.sourceforge.net/doc/dexec.html as a result of Nick's
query, and it seems that it would be suitable.

Something like this:

You define a set of states (page names) and a set of transitions and
symbols from state to state.  One of the input variables from a form is
named 'symbol' and the directory .tml checks that symbol represents a
valid transition (by calling [$fa put $symbol] on an fa generated by
[::grammar::fa::dexec] over your DFA.)  One could also calculate
something from form inputs and synthesise a symbol to put into $fa.

If the transition isn't legal, $fa will call an error routine, so you
can generate a redirect and reset the state to some known state.

If the transition is terminal, you close the session, or the
transaction.

The only problem I can see with this model is knowing what state $fa is
currently in, in case that matters ... one could always use a cookie or
a session variable to track the state ... by for example recording the
current page requested (assuming you'd named states for pages, which I
think is a good idea.)

If you felt energetic, you could even write a Tk GUI to specify the DFA.

Anyway, (a) it would ensure that you couldn't end up in a state by
following an illegal path, or in an uncontrolled manner; (b) programming
by DFAs seems seductive, but it can very quickly become unwieldy; (c)
often you need a single transition to some known state (for example, on
input error you might want to restart the transaction) ... if you have a
lot of these, you can nest DFAs - going into a state which means another
DFA is handling input.

I implemented ISO protocols (like X25) in the 80s with this stuff.
-- 
Colin McCormack <co...@ch...>

Re: [Tclhttpd-users] Re: Checking Page flow redirects etc..????

[Michael S. <sc...@un...>]

Colin McCormack schrieb:
> On Fri, 2004-10-08 at 00:09, Nicolas Boretos wrote:
> 
> 
>>Is there any simple or built-in way in tclhttpd for checking your page 
>>flow, meaning verify that you got to a page from another???
> 
> 
> Not that I know of, but it would be a cool project.
> 
> What you want (I think) is to drive each user's interaction from a state
> machine.  Their session starts in state 0, then depending on the inputs
> they submit, it moves to a different state.
> 
> You define a FSM (aka DFSA) which determines which transitions are
> legal.
> 
> There's a tcllib FSM package.  Could be useful for this kind of thing.

Something like joining the FSM package with the session interpreters and 
adding some helpers to drive it?

Michael

Re: [Tclhttpd-users] Sanity check..

[Nicolas B. <ni...@ma...>]

Colin McCormack wrote:

>
>
>Good idea for a project sometime.
>
>DFSA = Deterministic Finite State Automaton.  State machine.
>
>  
>
Wow...

>>>One facility you should become familiar with is [Httpd_Suspend] and
>>>[Httpd_Resume], which allows you to leave the connection open (for some
>>>period) while processing other incoming connections.  It might be useful
>>>in this application.
>>>      
>>>
>
>  
>
>>I think in this case I would probably want the opposite; meaning I want 
>>to complete my current mk transaction before moving on
>>to the next...
>>    
>>
>
>Yes, you want to serialise mk transactions, but you can't serialise web
>transactions ... they come in whenever someone initiates a connection.
>
>So you need to be able to put a web transaction on hold, pending the
>release of the mk lock.  One way to do this, I think, is to place
>pending requests which are waiting on a lock into a queue, suspend the
>connection, and then restart one of them when the lock is released.  A
>kind of poor-man's multitasking.
>  
>
Well, right now what I do is grab a record identifier (oid) from a 
counter when someone wants to add a record.
The next person would grab the next oid...
User fills in form and submits..at this point, I append to the mk db. 
Assuming  someone else tries to appends at the same time,
I would think tclhttpd would first complete the one of the appends and 
then move on to the next..Correct??

 From then on, I just use the oid pointing to the table row to 
read/write from it. I dont delete rows live (which shifts rows and might 
confuse mk)

>Otherwise, when a new connection comes in, what're you going to do?
>
If the server is busy (talking ms here, the users browser responds a 
little slower before it writes and re-directs...)

> 
>Return an error page?
>  
>
got to go,
let's resume this tommorrow..

nicolas

[Tclhttpd-users] Re: Checking Page flow redirects etc..????

[Colin M. <co...@ch...>]

On Fri, 2004-10-08 at 00:09, Nicolas Boretos wrote:

> Is there any simple or built-in way in tclhttpd for checking your page 
> flow, meaning verify that you got to a page from another???

Not that I know of, but it would be a cool project.

What you want (I think) is to drive each user's interaction from a state
machine.  Their session starts in state 0, then depending on the inputs
they submit, it moves to a different state.

You define a FSM (aka DFSA) which determines which transitions are
legal.

There's a tcllib FSM package.  Could be useful for this kind of thing.

> currently doing cludges like the following..
> 
> if {[string match "$::env(HTTP_REFERER)" 
> "http://$::env(HTTP_HOST)/rubia/parts.tml?record_id=[ncgi::value 
> record_id]&rec_id=[ncgi::value rec_id]&session_id=[ncgi::value 
> session_id]&use_name=[ncgi::value use_name]"]} {
> rubia::populate_checkboxes month_collected  [ncgi::value record_id] 
> month_collected session_id [ncgi::value session_id] use_name 
> [ncgi::value use_name]
> rubia::populate_checkboxes month_traded  [ncgi::value record_id] .blahhhhhh
> 
> Or how to check whether form is referred or self posted...??
> 
> proc rubia::is_self_posted {} {
>         if {$::env(REQUEST_URI) != "[eval join 
> {http://$::env(HTTP_HOST)$::page(url)}]"} {
>                         ####Form is refered, so lock record####
>                            return 0
>         } else {
>                 ####Form is self-posted, write data etc
>         return 1
>         }
>  }
> 
> or the inverse...
> 
>  proc rubia::is_referred {} {
>         if {$::env(HTTP_REFERER) != "[eval join 
> {http://$::env(HTTP_HOST)$::page(url)}]"} {
>                         ########
>                            return 1
>         } else {
>                 ####Form is self-posted, write data etc
>         return 0
>         }
>  }
> 
> This used in the .tml pages to decide what action to take ...
> if {[rubia::is_self_posted]} { do someting...
> 
> This gets painfull after a while...

Word.
-- 
Colin McCormack <co...@ch...>

Re: [Tclhttpd-users] Sanity check..

[Colin M. <co...@ch...>]

On Fri, 2004-10-08 at 00:01, Nicolas Boretos wrote:
> Colin McCormack wrote:
> 
> >On Thu, 2004-10-07 at 19:14, Nicolas Boretos wrote:
> >
> >  
> >
> >>As much as I 
> >>understand, "tclhttpd" is my "single user", and that
> >>read/writes are sequential/serialized, the equivalent of opening a 
> >>console and playing with the mk db...
> >>    
> >>
> >
> >That's a good way to look at it.
> >  
> >
> So, does this mean you feel that I should not have concurrency issues w/ 
> th mk?

If you think of it as 'tclhttpd is my single user', then no you won't.

> >>A few key points about the app.
> >>1. A .tclaccess file permits access to the db

> >Wrong place.  Should be a .tml file (just dir/.tml is sourced in.)

> I'm not sure I get it;-(...The .tclaccess file  calls the browsers login 
> stuff...I'm not sure that a .tml file is designed for this
> Or, am I mis-understanding you...

Oh, I get it, you mean .tclaccess grants permission to access.  I was
confused between that and the other access.

> >Handling state through Redirects is a pain though.  Perhaps we should
> >come up with a nicer abstraction to cover the messy details of building
> >systems of DFSAs to represent interactions via templates.

> Great idea!...Errr, WTF is a DFSA?
> I have really gotten burned redirects and form variables with it this 
> time...

Good idea for a project sometime.

DFSA = Deterministic Finite State Automaton.  State machine.

> >One facility you should become familiar with is [Httpd_Suspend] and
> >[Httpd_Resume], which allows you to leave the connection open (for some
> >period) while processing other incoming connections.  It might be useful
> >in this application.

> I think in this case I would probably want the opposite; meaning I want 
> to complete my current mk transaction before moving on
> to the next...

Yes, you want to serialise mk transactions, but you can't serialise web
transactions ... they come in whenever someone initiates a connection.

So you need to be able to put a web transaction on hold, pending the
release of the mk lock.  One way to do this, I think, is to place
pending requests which are waiting on a lock into a queue, suspend the
connection, and then restart one of them when the lock is released.  A
kind of poor-man's multitasking.

Otherwise, when a new connection comes in, what're you going to do? 
Return an error page?
-- 
Colin McCormack <co...@ch...>

[Tclhttpd-users] Checking Page flow redirects etc..????

[Nicolas B. <ni...@ma...>]

Hi Again

Is there any simple or built-in way in tclhttpd for checking your page 
flow, meaning verify that you got to a page from another???

currently doing cludges like the following..

if {[string match "$::env(HTTP_REFERER)" 
"http://$::env(HTTP_HOST)/rubia/parts.tml?record_id=[ncgi::value 
record_id]&rec_id=[ncgi::value rec_id]&session_id=[ncgi::value 
session_id]&use_name=[ncgi::value use_name]"]} {
rubia::populate_checkboxes month_collected  [ncgi::value record_id] 
month_collected session_id [ncgi::value session_id] use_name 
[ncgi::value use_name]
rubia::populate_checkboxes month_traded  [ncgi::value record_id] .blahhhhhh

Or how to check whether form is referred or self posted...??

proc rubia::is_self_posted {} {
        if {$::env(REQUEST_URI) != "[eval join 
{http://$::env(HTTP_HOST)$::page(url)}]"} {
                        ####Form is refered, so lock record####
                           return 0
        } else {
                ####Form is self-posted, write data etc
        return 1
        }
 }

or the inverse...

 proc rubia::is_referred {} {
        if {$::env(HTTP_REFERER) != "[eval join 
{http://$::env(HTTP_HOST)$::page(url)}]"} {
                        ########
                           return 1
        } else {
                ####Form is self-posted, write data etc
        return 0
        }
 }

This used in the .tml pages to decide what action to take ...
if {[rubia::is_self_posted]} { do someting...

This gets painfull after a while...

regards,

nicolas