Webware for Python

Am 28.11.2012 12:10, schrieb Sophana K:
> Look like another thread could be changing the _memoryStore during
> this iteration. Should I change to
 > for key in list(self._memoryStore):

Yes, this or "for key in self._memoryStore.keys()" should fix it. I have 
already committed this as a fix to the trunk.

-- Chris

It seems the problem has now moved to another place.
Should I try the trunk?

 File "/home/.../Webware-1.1/WebKit/Tasks/SessionTask.py", line 13, in run
    self._sessionstore.cleanStaleSessions(self)
  File "/home/.../Webware-1.1/WebKit/SessionDynamicStore.py", line
256, in cleanStaleSessions
    self._memoryStore.cleanStaleSessions(task)
  File "/home/.../Webware-1.1/WebKit/SessionStore.py", line 202, in
cleanStaleSessions
    for key in self:
RuntimeError: dictionary changed size during iteration



On Wed, Nov 28, 2012 at 12:56 PM, Christoph Zwerschke <cito@...:

> Am 28.11.2012 12:10, schrieb Sophana K:
> > Look like another thread could be changing the _memoryStore during
> > this iteration. Should I change to
>  > for key in list(self._memoryStore):
>
> Yes, this or "for key in self._memoryStore.keys()" should fix it. I have
> already committed this as a fix to the trunk.
>
> -- Chris
>
>
> ------------------------------------------------------------------------------
> Keep yourself connected to Go Parallel:
> INSIGHTS What's next for parallel hardware, programming and related areas?
> Interviews and blogs by thought leaders keep you ahead of the curve.
> http://goparallel.sourceforge.net
> _______________________________________________
> Webware-discuss mailing list
> Webware-discuss@...
> https://lists.sourceforge.net/lists/listinfo/webware-discuss
>

Am 30.11.2012 13:48, schrieb Sophana K:
> It seems the problem has now moved to another place.
> Should I try the trunk?
>
>   File "/home/.../Webware-1.1/WebKit/Tasks/SessionTask.py", line 13, in run
>      self._sessionstore.cleanStaleSessions(self)
>    File "/home/.../Webware-1.1/WebKit/SessionDynamicStore.py", line 256, in cleanStaleSessions
>      self._memoryStore.cleanStaleSessions(task)
>    File "/home/.../Webware-1.1/WebKit/SessionStore.py", line 202, in cleanStaleSessions
>      for key in self:
> RuntimeError: dictionary changed size during iteration

Same problem. It's now also fixed in the trunk.

-- Christoph

Thanks

Isn't it strange nobody had this problem before?
Seems that there isn't a lot of webware 1.1 applications in production.
Don't you think?

Sophana


On Fri, Nov 30, 2012 at 5:36 PM, Christoph Zwerschke <cito@...> wrote:

> Am 30.11.2012 13:48, schrieb Sophana K:
> > It seems the problem has now moved to another place.
> > Should I try the trunk?
> >
> >   File "/home/.../Webware-1.1/WebKit/Tasks/SessionTask.py", line 13, in
> run
> >      self._sessionstore.cleanStaleSessions(self)
> >    File "/home/.../Webware-1.1/WebKit/SessionDynamicStore.py", line 256,
> in cleanStaleSessions
> >      self._memoryStore.cleanStaleSessions(task)
> >    File "/home/.../Webware-1.1/WebKit/SessionStore.py", line 202, in
> cleanStaleSessions
> >      for key in self:
> > RuntimeError: dictionary changed size during iteration
>
> Same problem. It's now also fixed in the trunk.
>
> -- Christoph
>
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Keep yourself connected to Go Parallel:
> TUNE You got it built. Now make it sing. Tune shows you how.
> http://goparallel.sourceforge.net
> _______________________________________________
> Webware-discuss mailing list
> Webware-discuss@...
> https://lists.sourceforge.net/lists/listinfo/webware-discuss
>

Hi Sophana,
hi Christoph,

the webware for python is used in two companies, on 4 Servers, I work 
for. It is running very stable.
But however we use a self-implemented session. Thats why it seems, I 
didn't mentioned this problem to Christoph.
But our claims, we have to the session is a little bit bigger one that 
the one from webware has and it was easier to build an own, derived one.

However, Webkit is pretty stable, there is no need not to take it, 
because Christoph very cares about all.

Thank you Christoph btw. for all your help and support everytime so far!

Best Regards,
Tobias
------------------------------------------------------------------------
Am 30.11.2012 18:01, schrieb Sophana K:
> Thanks
>
> Isn't it strange nobody had this problem before?
> Seems that there isn't a lot of webware 1.1 applications in production.
> Don't you think?
>
> Sophana
>
>
> On Fri, Nov 30, 2012 at 5:36 PM, Christoph Zwerschke <cito@... 
> <mailto:cito@...>> wrote:
>
>     Am 30.11.2012 13:48, schrieb Sophana K:
>     > It seems the problem has now moved to another place.
>     > Should I try the trunk?
>     >
>     >   File "/home/.../Webware-1.1/WebKit/Tasks/SessionTask.py", line
>     13, in run
>     >      self._sessionstore.cleanStaleSessions(self)
>     >    File "/home/.../Webware-1.1/WebKit/SessionDynamicStore.py",
>     line 256, in cleanStaleSessions
>     >      self._memoryStore.cleanStaleSessions(task)
>     >    File "/home/.../Webware-1.1/WebKit/SessionStore.py", line
>     202, in cleanStaleSessions
>     >      for key in self:
>     > RuntimeError: dictionary changed size during iteration
>
>     Same problem. It's now also fixed in the trunk.
>
>     -- Christoph
>
>
>
>
>
>
>     ------------------------------------------------------------------------------
>     Keep yourself connected to Go Parallel:
>     TUNE You got it built. Now make it sing. Tune shows you how.
>     http://goparallel.sourceforge.net
>     _______________________________________________
>     Webware-discuss mailing list
>     Webware-discuss@...
>     <mailto:Webware-discuss@...>
>     https://lists.sourceforge.net/lists/listinfo/webware-discuss
>
>
>
>
> ------------------------------------------------------------------------------
> Keep yourself connected to Go Parallel:
> TUNE You got it built. Now make it sing. Tune shows you how.
> http://goparallel.sourceforge.net
>
>
> _______________________________________________
> Webware-discuss mailing list
> Webware-discuss@...
> https://lists.sourceforge.net/lists/listinfo/webware-discuss

Am 30.11.2012 18:01, schrieb Sophana K:
 > Isn't it strange nobody had this problem before?
 > Seems that there isn't a lot of webware 1.1 applications in production.
 > Don't you think?
T

Am 30.11.2012 18:01, schrieb Sophana K:
> Isn't it strange nobody had this problem before? Seems that there
> isn't a lot of webware 1.1 applications in production. Don't you
> think?

That problem should not appear in practice unless you have somehow two 
session sweeper tasks running at the same time. Are you doing anything 
special in this regard? I'm running Webware 1.1 apps with many users 
which use sessions heavily, and I've never seen this issue.

-- Christoph

(Sorry, my mailer just sent an empty reply.)

I have some tasks scheduled every 30m and 24h.
I did nothing special about the session sweeper. Are they related to
taskKit?


On Fri, Nov 30, 2012 at 6:28 PM, Christoph Zwerschke <cito@...> wrote:

> Am 30.11.2012 18:01, schrieb Sophana K:
> > Isn't it strange nobody had this problem before? Seems that there
> > isn't a lot of webware 1.1 applications in production. Don't you
> > think?
>
> That problem should not appear in practice unless you have somehow two
> session sweeper tasks running at the same time. Are you doing anything
> special in this regard? I'm running Webware 1.1 apps with many users
> which use sessions heavily, and I've never seen this issue.
>
> -- Christoph
>
> (Sorry, my mailer just sent an empty reply.)
>
>
> ------------------------------------------------------------------------------
> Keep yourself connected to Go Parallel:
> TUNE You got it built. Now make it sing. Tune shows you how.
> http://goparallel.sourceforge.net
> _______________________________________________
> Webware-discuss mailing list
> Webware-discuss@...
> https://lists.sourceforge.net/lists/listinfo/webware-discuss
>

Am 06.12.2012 10:59, schrieb Sophana K:
> I have some tasks scheduled every 30m and 24h.
> I did nothing special about the session sweeper. Are they related to
> taskKit?

The session sweeper thread is started by the Application instance using 
its own instance of the TaskKit scheduler, available as 
Application.taskManager(). Do you use that scheduler instance or do you 
use your own? To me it looks as if you're either running the session 
sweeper twice or one of your tasks is doing something with the session 
store of the application. Otherwise you shouldn't see that error.

-- Christoph

Yes, I'm creating a Scheduler instance.
I didn't know I was supposed to use the one from the application.
Is it normal that a scheduler instance automatically does session sweeping?
Shouldn't the application create the task explicitely?

I have now fixed my code to use the existing scheduler.
Shouldn't you remove the previous fixes then?

Thanks for the hint.



On Thu, Dec 6, 2012 at 12:23 PM, Christoph Zwerschke <cito@...> wrote:

> Am 06.12.2012 10:59, schrieb Sophana K:
> > I have some tasks scheduled every 30m and 24h.
> > I did nothing special about the session sweeper. Are they related to
> > taskKit?
>
> The session sweeper thread is started by the Application instance using
> its own instance of the TaskKit scheduler, available as
> Application.taskManager(). Do you use that scheduler instance or do you
> use your own? To me it looks as if you're either running the session
> sweeper twice or one of your tasks is doing something with the session
> store of the application. Otherwise you shouldn't see that error.
>
> -- Christoph
>
>
> ------------------------------------------------------------------------------
> LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
> Remotely access PCs and mobile devices and provide instant support
> Improve your efficiency, and focus on delivering more value-add services
> Discover what IT Professionals Know. Rescue delivers
> http://p.sf.net/sfu/logmein_12329d2d
> _______________________________________________
> Webware-discuss mailing list
> Webware-discuss@...
> https://lists.sourceforge.net/lists/listinfo/webware-discuss
>

Am 07.12.2012 12:16, schrieb Sophana K:
> Yes, I'm creating a Scheduler instance.
> I didn't know I was supposed to use the one from the application.

It's ok to create your own instance. But then the two schedulers will 
run as two parallel threads which needs more ressources and you can get 
these race conditions if both schedulers have tasks which access the 
session store. It's also convenient to use the application.taskManager() 
because it's started and stopped automatically together with the 
application.

> Is it normal that a scheduler instance automatically does session sweeping?
> Shouldn't the application create the task explicitely?

But that's excactly how it works. The application creates its own 
scheduler, and then adds just one task, the one which is used for 
session sweeping. You can replace that task with your own or add more 
tasks if you like.

> I have now fixed my code to use the existing scheduler.
> Shouldn't you remove the previous fixes then?

The fixes are still helpful in case anybody changes the session store 
for whatever maybe valid reasons.

It's still unclear to me what caused your problems: Do your own tasks 
modify the session store? Then that's the explanation, because you ran 
them in a parallel scheduler. If your own tasks don't change the 
session, then you must continue searching for the error cause. It looks 
like the session sweeper thread somehow was duplicated, but how this 
happens is unclear to me without seeing your code. If you just create 
your own Scheduler instance, this should not happen.

-- Christoph

My tasks do not access the sessions. (Why would they?)
As it is a new behaviour, (I was using 0.9) a look at the changes could
help...
Could it be the scheduler instances sharing some global task lists?


On Fri, Dec 7, 2012 at 2:38 PM, Christoph Zwerschke <cito@...> wrote:

> Am 07.12.2012 12:16, schrieb Sophana K:
> > Yes, I'm creating a Scheduler instance.
> > I didn't know I was supposed to use the one from the application.
>
> It's ok to create your own instance. But then the two schedulers will
> run as two parallel threads which needs more ressources and you can get
> these race conditions if both schedulers have tasks which access the
> session store. It's also convenient to use the application.taskManager()
> because it's started and stopped automatically together with the
> application.
>
> > Is it normal that a scheduler instance automatically does session
> sweeping?
> > Shouldn't the application create the task explicitely?
>
> But that's excactly how it works. The application creates its own
> scheduler, and then adds just one task, the one which is used for
> session sweeping. You can replace that task with your own or add more
> tasks if you like.
>
> > I have now fixed my code to use the existing scheduler.
> > Shouldn't you remove the previous fixes then?
>
> The fixes are still helpful in case anybody changes the session store
> for whatever maybe valid reasons.
>
> It's still unclear to me what caused your problems: Do your own tasks
> modify the session store? Then that's the explanation, because you ran
> them in a parallel scheduler. If your own tasks don't change the
> session, then you must continue searching for the error cause. It looks
> like the session sweeper thread somehow was duplicated, but how this
> happens is unclear to me without seeing your code. If you just create
> your own Scheduler instance, this should not happen.
>
> -- Christoph
>
>
> ------------------------------------------------------------------------------
> LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
> Remotely access PCs and mobile devices and provide instant support
> Improve your efficiency, and focus on delivering more value-add services
> Discover what IT Professionals Know. Rescue delivers
> http://p.sf.net/sfu/logmein_12329d2d
> _______________________________________________
> Webware-discuss mailing list
> Webware-discuss@...
> https://lists.sourceforge.net/lists/listinfo/webware-discuss
>

Am 08.12.2012 09:51, schrieb Sophana K:
> My tasks do not access the sessions. (Why would they?)
> As it is a new behaviour, (I was using 0.9) a look at the changes could
> help...

In 0.9 and 1.0 the session sweeper loops run over the keys list, not 
over the dictionaries, simply because older Python versions did not 
support that. So the problem with duplicate session sweepers might well 
have existed, you just didn't notice it.

> Could it be the scheduler instances sharing some global task lists?

I thought about that too, because it would be an explanation, but no, 
they don't.

One explanation for the duplicate session sweeping could be that your 
tasks inherit from WebKit.Tasks.SessionTask instead of TaskKit.Task. Or 
that you inherit from Application and run __init__ twice.

To debug the problem, you can put some print statement in the __init__() 
and run() methods of SessionTask and see when it is instantiated (should 
be instantiated once at the start) and how often it is run (should run 
once every 10 minutes, depending on your SessionTimeout).

-- Christoph