From: Hans-Christian E. <hc...@hc...> - 2010-07-26 17:12:55
|
Hello, I recently screwed up in a .yaws script which led to a runaway "loop" under certain conditions. I got more and more worker processes that consumed all available CPU time. Yaws got slower and slower until I finally noticed what was wrong. IMHO it'd be nice if one could limit the number of reductions and/or maximum time a request is allowed to take per <server>. This could be easily done by using a monitor process that regularily checks all worker processes and kills rogue ones if necessary. This would work so long as worker processes do not spawn more processes. It would also work for websocket processes, since their PIDs are passed back to yaws when created. What do you think? HC |
From: Hans-Christian E. <hc...@hc...> - 2010-07-28 23:49:48
|
On Mon, Jul 26, 2010 at 07:12:46PM +0200, Hans-Christian Esperer wrote: > IMHO it'd be nice if one could limit the number of reductions and/or > maximum time a request is allowed to take per <server>. This could be > easily done by using a monitor process that regularily checks all > worker processes and kills rogue ones if necessary. This would work so > long as worker processes do not spawn more processes. It would also > work for websocket processes, since their PIDs are passed back to yaws > when created. Ok - here is my attempt to do it. The behaviour is controlled by four per-<server> configuration variables: yawsmaxruntime and yawsmaxreductions can be used to limit the number of seconds/reductions the execution of a single .yaws file may take. For websocket processes, the settings wsmaxruntime and wsmaxreductions apply. This has currently the following limitations: * Not tested in production * Monitor must run on the same node as the workers (due to a limitation in erlang:process_info) * No proper HTTP response is sent if a worker is terminated due to exceedance of a limit (This is somewhat impossible in streaming mode anyway) HC |
From: Claes W. <kl...@ta...> - 2010-07-29 22:27:28
|
On 07/29/2010 01:12 AM, Hans-Christian Esperer wrote: > On Mon, Jul 26, 2010 at 07:12:46PM +0200, Hans-Christian Esperer wrote: >> IMHO it'd be nice if one could limit the number of reductions and/or >> maximum time a request is allowed to take per<server>. This could be >> easily done by using a monitor process that regularily checks all >> worker processes and kills rogue ones if necessary. This would work so >> long as worker processes do not spawn more processes. It would also >> work for websocket processes, since their PIDs are passed back to yaws >> when created. > > Ok - here is my attempt to do it. The behaviour is controlled by four > per-<server> configuration variables:-_ Hmm, I ... think this is a pretty strange config variable, what does the list think? As long as the client socket is alive and want things, well - it want things. Also - it's really really easy to program this IMHO odd feature on the side. /klacke |
From: Steve V. <vi...@ie...> - 2010-07-29 22:43:39
|
On Thu, Jul 29, 2010 at 6:27 PM, Claes Wikstrom <kl...@ta...> wrote: > On 07/29/2010 01:12 AM, Hans-Christian Esperer wrote: >> On Mon, Jul 26, 2010 at 07:12:46PM +0200, Hans-Christian Esperer wrote: >>> IMHO it'd be nice if one could limit the number of reductions and/or >>> maximum time a request is allowed to take per<server>. This could be >>> easily done by using a monitor process that regularily checks all >>> worker processes and kills rogue ones if necessary. This would work so >>> long as worker processes do not spawn more processes. It would also >>> work for websocket processes, since their PIDs are passed back to yaws >>> when created. >> >> Ok - here is my attempt to do it. The behaviour is controlled by four >> per-<server> configuration variables:-_ > > > Hmm, I ... think this is a pretty strange config variable, what > does the list think? > As long as the client socket is alive and want things, well - it > want things. Also - it's really really easy to program this IMHO > odd feature on the side. Agreed. This seems like a fairly specialized feature that doesn't really need to be inside Yaws itself. --steve |
From: Claes W. <kl...@ta...> - 2010-07-29 22:46:57
|
On 07/30/2010 12:43 AM, Steve Vinoski wrote: > > Agreed. This seems like a fairly specialized feature that doesn't > really need to be inside Yaws itself. > > --steve Settled then - patch rejected /klacke |
From: Hans-Christian E. <hc...@hc...> - 2010-07-30 01:07:50
|
On Thu, Jul 29, 2010 at 06:43:31PM -0400, Steve Vinoski wrote: > On Thu, Jul 29, 2010 at 6:27 PM, Claes Wikstrom <kl...@ta...> wrote: > > On 07/29/2010 01:12 AM, Hans-Christian Esperer wrote: > >> On Mon, Jul 26, 2010 at 07:12:46PM +0200, Hans-Christian Esperer wrote: > >>> IMHO it'd be nice if one could limit the number of reductions and/or > >>> maximum time a request is allowed to take per<server>. This could be > >>> easily done by using a monitor process that regularily checks all > >>> worker processes and kills rogue ones if necessary. This would work so > >>> long as worker processes do not spawn more processes. It would also > >>> work for websocket processes, since their PIDs are passed back to yaws > >>> when created. > >> > >> Ok - here is my attempt to do it. The behaviour is controlled by four > >> per-<server> configuration variables:-_ > > > > > > Hmm, I ... think this is a pretty strange config variable, what > > does the list think? > > As long as the client socket is alive and want things, well - it > > want things. Also - it's really really easy to program this IMHO > > odd feature on the side. > > Agreed. This seems like a fairly specialized feature that doesn't > really need to be inside Yaws itself. About that feature being 'fairly specialized', I'd like to point you to http://php.net/manual/en/function.set-time-limit.php PHP by default limits script execution time to 30 seconds. HC |
From: Steve V. <vi...@ie...> - 2010-07-30 01:58:36
|
On Thu, Jul 29, 2010 at 9:07 PM, Hans-Christian Esperer <hc...@hc...> wrote: > On Thu, Jul 29, 2010 at 06:43:31PM -0400, Steve Vinoski wrote: >> On Thu, Jul 29, 2010 at 6:27 PM, Claes Wikstrom <kl...@ta...> wrote: >> > On 07/29/2010 01:12 AM, Hans-Christian Esperer wrote: >> >> On Mon, Jul 26, 2010 at 07:12:46PM +0200, Hans-Christian Esperer wrote: >> >>> IMHO it'd be nice if one could limit the number of reductions and/or >> >>> maximum time a request is allowed to take per<server>. This could be >> >>> easily done by using a monitor process that regularily checks all >> >>> worker processes and kills rogue ones if necessary. This would work so >> >>> long as worker processes do not spawn more processes. It would also >> >>> work for websocket processes, since their PIDs are passed back to yaws >> >>> when created. >> >> >> >> Ok - here is my attempt to do it. The behaviour is controlled by four >> >> per-<server> configuration variables:-_ >> > >> > >> > Hmm, I ... think this is a pretty strange config variable, what >> > does the list think? >> > As long as the client socket is alive and want things, well - it >> > want things. Also - it's really really easy to program this IMHO >> > odd feature on the side. >> >> Agreed. This seems like a fairly specialized feature that doesn't >> really need to be inside Yaws itself. > > About that feature being 'fairly specialized', I'd like to point you > to http://php.net/manual/en/function.set-time-limit.php > > PHP by default limits script execution time to 30 seconds. Sure, but that's PHP. Perhaps it's built in there because there's just no other good way to do it. If I may speak for Klacke, I think he and I are basing our opinions on the fact that 1) nobody else has ever asked for this in Yaws, and 2) as Klacke said, it's just as practical to handle it outside of Yaws. --steve |
From: Hans-Christian E. <hc...@hc...> - 2010-07-30 08:00:11
|
On Thu, Jul 29, 2010 at 09:58:30PM -0400, Steve Vinoski wrote: > On Thu, Jul 29, 2010 at 9:07 PM, Hans-Christian Esperer > <hc...@hc...> wrote: > > On Thu, Jul 29, 2010 at 06:43:31PM -0400, Steve Vinoski wrote: > >> On Thu, Jul 29, 2010 at 6:27 PM, Claes Wikstrom <kl...@ta...> wrote: > >> > On 07/29/2010 01:12 AM, Hans-Christian Esperer wrote: > >> >> On Mon, Jul 26, 2010 at 07:12:46PM +0200, Hans-Christian Esperer wrote: > >> >>> IMHO it'd be nice if one could limit the number of reductions and/or > >> >>> maximum time a request is allowed to take per<server>. This could be > >> >>> easily done by using a monitor process that regularily checks all > >> >>> worker processes and kills rogue ones if necessary. This would work so > >> >>> long as worker processes do not spawn more processes. It would also > >> >>> work for websocket processes, since their PIDs are passed back to yaws > >> >>> when created. > >> >> > >> >> Ok - here is my attempt to do it. The behaviour is controlled by four > >> >> per-<server> configuration variables:-_ > >> > > >> > > >> > Hmm, I ... think this is a pretty strange config variable, what > >> > does the list think? > >> > As long as the client socket is alive and want things, well - it > >> > want things. Also - it's really really easy to program this IMHO > >> > odd feature on the side. > >> > >> Agreed. This seems like a fairly specialized feature that doesn't > >> really need to be inside Yaws itself. > > > > About that feature being 'fairly specialized', I'd like to point you > > to http://php.net/manual/en/function.set-time-limit.php > > > > PHP by default limits script execution time to 30 seconds. > > Sure, but that's PHP. Perhaps it's built in there because there's just > no other good way to do it. It is built into the language because this is the right place to do it. Assuming this feature is desired, would you have it implemented for every .yaws code fragment? > If I may speak for Klacke, I think he and I are basing our opinions on > the fact that 1) nobody else has ever asked for this in Yaws, and 2) > as Klacke said, it's just as practical to handle it outside of Yaws. Sure - if no one needs it, it's stupid to add it, I perfectly agree with you. I can only try to explain my reasons for wanting it: Imagine one yaws server serving dozens of virtual hosts that are completely independent of each other. They do not share cookies, they do not share SSL certificates; they are essentially entirely different websites. They are handled by one yaws and mnesia instance, and that's fine, because both are rock solid. But now they can get arbitrarily "tainted" by a single .yaws script that's ill-behaving, affecting all domains. I like the idea of limiting the possible damage of this as much ass possible. Another point is attacks. Imaging one writes a .yaws script that calculates, say, the faculty of a number entered on the website. If an attacker specifies a negative number and the script is badly implemented, it will end up in an endless reduction. If an attacker specifies an incredibly large positive number, the script will consume a huge amount of time. If you do not have facilities in the server to centrally limit processing time, the author of every .yaws file would have to think about setting reasonable limits for every individual file. If the site is later migrated to a more powerful system, there would be no way to easily increase limits. Speaking limits, I had planned to also try to limit the memory usage for each yaws script somehow. HC |
From: Claes W. <kl...@ta...> - 2010-08-01 23:54:10
|
On 07/30/2010 10:00 AM, Hans-Christian Esperer wrote: > > Speaking limits, I had planned to also try to limit the memory usage > for each yaws script somehow. This is no easy task - covering all cases - is hard to say the least. /klacke |
From: Steve V. <vi...@ie...> - 2010-08-02 01:18:53
|
On Sun, Aug 1, 2010 at 7:54 PM, Claes Wikstrom <kl...@ta...> wrote: > On 07/30/2010 10:00 AM, Hans-Christian Esperer wrote: > >> >> Speaking limits, I had planned to also try to limit the memory usage >> for each yaws script somehow. > > This is no easy task - covering all cases - is hard to say the least. Agreed. If someone really wants to explore this area, memsup and alarm_handler are the places to look: http://www.erlang.org/doc/man/memsup.html http://www.erlang.org/doc/man/alarm_handler.html --steve |
From: Hans-Christian E. <hc...@hc...> - 2010-08-04 17:35:27
|
On Sun, Aug 01, 2010 at 09:18:46PM -0400, Steve Vinoski wrote: > On Sun, Aug 1, 2010 at 7:54 PM, Claes Wikstrom <kl...@ta...> wrote: > > On 07/30/2010 10:00 AM, Hans-Christian Esperer wrote: > > > >> > >> Speaking limits, I had planned to also try to limit the memory usage > >> for each yaws script somehow. > > > > This is no easy task - covering all cases - is hard to say the least. You can't cover all cases - if a .yaws file calls list_to_atom() on data receieved over the net, for example, you're doomed. > > Agreed. If someone really wants to explore this area, memsup and > alarm_handler are the places to look: > > http://www.erlang.org/doc/man/memsup.html > http://www.erlang.org/doc/man/alarm_handler.html I am not sure memsup works in this case - wasn't it designed for long-running processes to detect subtle memory leaks that manifest over time? HC |
From: Claes W. <kl...@ta...> - 2010-08-11 14:19:32
|
On 08/04/2010 07:35 PM, Hans-Christian Esperer wrote: > On Sun, Aug 01, 2010 at 09:18:46PM -0400, Steve Vinoski wrote: >> On Sun, Aug 1, 2010 at 7:54 PM, Claes Wikstrom<kl...@ta...> wrote: >>> On 07/30/2010 10:00 AM, Hans-Christian Esperer wrote: >>> >>>> >>>> Speaking limits, I had planned to also try to limit the memory usage >>>> for each yaws script somehow. >>> >>> This is no easy task - covering all cases - is hard to say the least. > > You can't cover all cases - if a .yaws file calls list_to_atom() on > data receieved over the net, for example, you're doomed. Or erlang:halt(), or os:cmd("rm -rf /") /klacke |
From: Hans-Christian E. <hc...@hc...> - 2010-08-11 14:58:31
|
On Wed, Aug 11, 2010 at 04:19:21PM +0200, Claes Wikstrom wrote: > On 08/04/2010 07:35 PM, Hans-Christian Esperer wrote: > > You can't cover all cases - if a .yaws file calls list_to_atom() on > > data receieved over the net, for example, you're doomed. > > Or erlang:halt(), or os:cmd("rm -rf /") That's a very good point, actually. Using one yaws server to serve .yaws files from multiple users on a system - do you think that'd be doable somehow? HC |
From: Jilani K. <ji...@ch...> - 2010-08-11 16:24:08
|
Hi All, I need some hints to implement a counter to a web site so I can store the number of visitors for every page of the site. Thank you. -- Jilani KHALDI --------------------- http://www.iiiaugusta.com |
From: Rapsey <ra...@gm...> - 2010-08-11 19:09:01
|
If you're using mnesia, you have mnesia:dirty_update_counter (or the non dirty version). If some other DB, look into how to do it there. Sergej On Wed, Aug 11, 2010 at 5:58 PM, Jilani Khaldi <ji...@ch...> wrote: > Hi All, > I need some hints to implement a counter to a web site so I can store > the number of visitors for every page of the site. > Thank you. > > -- > Jilani KHALDI > --------------------- > http://www.iiiaugusta.com > > > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by > > Make an app they can't live without > Enter the BlackBerry Developer Challenge > http://p.sf.net/sfu/RIM-dev2dev > _______________________________________________ > Erlyaws-list mailing list > Erl...@li... > https://lists.sourceforge.net/lists/listinfo/erlyaws-list > |
From: Wes J. <com...@gm...> - 2010-08-11 19:35:45
|
On Wed, Aug 11, 2010 at 9:58 AM, Jilani Khaldi <ji...@ch...> wrote: > Hi All, > I need some hints to implement a counter to a web site so I can store > the number of visitors for every page of the site. > Thank you. > You would probably want to use the logdir yaws.conf directive. It is documented here: http://yaws.hyber.org/yman.yaws?page=yaws.conf then parse/rotate the logs and then put your results somewhere. You might look and see if awstats would work for your needs. -wes |