#31 mod_qos concurrent requests indication completely different from server-status handler concurrent requests

3.x
open
nobody
concurrency (1)
2
2014-01-21
2014-01-15
Mark Blackman
No

We're large scale users of mod_qos in our production environments using Apache 2.2 (prefork) and in at least one environment (across multiple servers) we're seeing indications that the concurrent requests indicator in mod_qos bears no relationship to the number of busy Apache servers. I've cut and pasted the output from server-status?auto below.

Total Accesses: 1543832
Total kBytes: 29131620
CPULoad: .0185832
Uptime: 53274
ReqPerSec: 28.9791
BytesPerSec: 559950
BytesPerReq: 19322.6
BusyWorkers: 43
IdleWorkers: 2
Scoreboard: KKKKKCKKKKCK.K..C.KC.C..K...C.K..CCKKC.KKK..K...KCKKK..W.K.CKK...K....WC....C...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
v;dwdef00;801;QS_LocRequestLimit;292[/]: 0
v;dw09022;801;QS_LocRequestLimit;100[/]: 0
v;dw06290;801;QS_LocRequestLimit;292[/]: 221

Apache is saying 43 concurrent requests (or connections) but the QOS output at the bottom believes there's 221 concurrent requests for all the virtualhosts and all 221 concurrent requests are in the dw06290 virtual host.

This is Apache 2.2.25 with mod_qos 10.13. I'm aware of the warning about pre-fork Apache and mod_qos but we've not detected any other issues and the notice seems more of a "use at your own risk" point rather than a "it simply will not ever work" and our initial testing didn't reveal any issues.

Is this discrepancy expected? I can find no evidence that it is expected and cannot find any Change log notes addressing it either.

Regards,
Mark Blackman

Discussion

  • No, this discrepancy is not expected. Invalid request counter may be caused by unexpected child exists (process crash). Any yes, mod_qos is not tested to be used within prefork binaries, e.g. I don't know what happens when a child reaches the MaxRequestsPerChild limit.

     
  • Mark Blackman
    Mark Blackman
    2014-01-17

    Thanks for the speedy response.

    I also left out another odd aspect of the concurrency indicator. We note that the QOS concurrency indication there very nearly rises monontonically during the day until it hits the concurrency limit, raises an error message in the log, returns 503 to the unlucky client and then falls to one and starts incrementing again.

    I've cut and pasted the sampled concurrency indicator (for dw06290) below over the course of the day at 1 minute intervals. The underlying connection concurrency indicated by apache server-status is a standard diurnal pattern, where QOS indicates a sawtooth. First column is timestamp, second is QOS concurrency, and finally the configured limit.

    2014-01-14 02:00:13 0 292
    2014-01-14 02:01:16 0 292
    2014-01-14 02:02:13 0 292
    2014-01-14 02:03:26 0 292
    2014-01-14 02:04:13 0 292
    2014-01-14 02:05:12 0 292
    2014-01-14 02:06:23 0 292
    2014-01-14 02:07:15 0 292
    2014-01-14 02:08:27 0 292
    2014-01-14 02:09:16 0 292
    2014-01-14 02:10:15 0 292
    2014-01-14 02:11:13 0 292
    2014-01-14 02:12:12 1 292
    2014-01-14 02:13:32 1 292
    2014-01-14 02:14:13 1 292
    2014-01-14 02:15:15 1 292
    2014-01-14 02:16:13 1 292
    2014-01-14 02:17:23 1 292
    2014-01-14 02:18:16 1 292
    2014-01-14 02:19:14 1 292
    2014-01-14 02:20:14 1 292
    2014-01-14 02:21:15 1 292
    2014-01-14 02:22:15 1 292
    2014-01-14 02:23:21 1 292
    2014-01-14 02:24:14 1 292
    2014-01-14 02:25:13 1 292
    2014-01-14 02:26:13 1 292
    2014-01-14 02:27:12 1 292
    2014-01-14 02:28:12 1 292
    2014-01-14 02:29:13 1 292
    2014-01-14 02:30:13 1 292
    2014-01-14 02:31:15 1 292
    2014-01-14 02:32:13 1 292
    2014-01-14 02:33:24 1 292
    2014-01-14 02:34:14 1 292
    2014-01-14 02:35:12 1 292
    2014-01-14 02:36:12 1 292
    2014-01-14 02:37:12 1 292
    2014-01-14 02:38:19 1 292
    2014-01-14 02:39:12 1 292
    2014-01-14 02:40:17 1 292
    2014-01-14 02:41:13 1 292
    2014-01-14 02:42:12 1 292
    2014-01-14 02:43:19 1 292
    2014-01-14 02:44:20 1 292
    2014-01-14 02:45:13 1 292
    2014-01-14 02:46:13 1 292
    2014-01-14 02:47:13 1 292
    2014-01-14 02:48:13 1 292
    2014-01-14 02:49:18 1 292
    2014-01-14 02:50:12 1 292
    2014-01-14 02:51:12 1 292
    2014-01-14 02:52:14 1 292
    2014-01-14 02:53:28 1 292
    2014-01-14 02:54:13 1 292
    2014-01-14 02:55:17 1 292
    2014-01-14 02:56:13 1 292
    2014-01-14 02:57:12 1 292
    2014-01-14 02:58:20 1 292
    2014-01-14 02:59:17 1 292
    2014-01-14 03:00:15 1 292
    2014-01-14 03:01:14 1 292
    2014-01-14 03:02:12 1 292
    2014-01-14 03:03:37 1 292
    2014-01-14 03:04:20 1 292
    2014-01-14 03:05:12 1 292
    2014-01-14 03:06:15 1 292
    2014-01-14 03:07:14 1 292
    2014-01-14 03:08:23 1 292
    2014-01-14 03:09:13 1 292
    2014-01-14 03:10:15 1 292
    2014-01-14 03:11:12 1 292
    2014-01-14 03:12:14 1 292
    2014-01-14 03:13:41 1 292
    2014-01-14 03:14:14 1 292
    2014-01-14 03:15:13 1 292
    2014-01-14 03:16:24 1 292
    2014-01-14 03:17:18 1 292
    2014-01-14 03:18:13 1 292
    2014-01-14 03:19:13 1 292
    2014-01-14 03:20:13 1 292
    2014-01-14 03:21:17 1 292
    2014-01-14 03:22:13 1 292
    2014-01-14 03:23:29 1 292
    2014-01-14 03:24:14 1 292
    2014-01-14 03:25:21 1 292
    2014-01-14 03:26:13 1 292
    2014-01-14 03:27:18 1 292
    2014-01-14 03:28:24 1 292
    2014-01-14 03:29:12 1 292
    2014-01-14 03:30:12 1 292
    2014-01-14 03:31:12 1 292
    2014-01-14 03:32:12 1 292
    2014-01-14 03:33:12 1 292
    2014-01-14 03:34:13 1 292
    2014-01-14 03:35:12 1 292
    2014-01-14 03:36:12 1 292
    2014-01-14 03:37:12 1 292
    2014-01-14 03:38:12 1 292
    2014-01-14 03:39:12 1 292
    2014-01-14 03:40:12 1 292
    2014-01-14 03:41:12 1 292
    2014-01-14 03:42:12 1 292
    2014-01-14 03:43:12 1 292
    2014-01-14 03:44:12 1 292
    2014-01-14 03:45:13 1 292
    2014-01-14 03:46:12 1 292
    2014-01-14 03:47:12 1 292
    2014-01-14 03:48:12 1 292
    2014-01-14 03:49:12 1 292
    2014-01-14 03:50:12 1 292
    2014-01-14 03:51:12 1 292
    2014-01-14 03:52:12 1 292
    2014-01-14 03:53:12 1 292
    2014-01-14 03:54:12 1 292
    2014-01-14 03:55:12 1 292
    2014-01-14 03:56:17 1 292
    2014-01-14 03:57:13 1 292
    2014-01-14 03:58:13 1 292
    2014-01-14 03:59:12 1 292
    2014-01-14 04:00:12 1 292
    2014-01-14 04:01:13 1 292
    2014-01-14 04:02:12 1 292
    2014-01-14 04:03:35 1 292
    2014-01-14 04:04:12 1 292
    2014-01-14 04:05:12 1 292
    2014-01-14 04:06:12 1 292
    2014-01-14 04:07:13 1 292
    2014-01-14 04:08:12 1 292
    2014-01-14 04:09:12 1 292
    2014-01-14 04:10:12 1 292
    2014-01-14 04:11:12 1 292
    2014-01-14 04:12:12 1 292
    2014-01-14 04:13:13 1 292
    2014-01-14 04:14:12 1 292
    2014-01-14 04:15:12 1 292
    2014-01-14 04:16:12 1 292
    2014-01-14 04:17:12 1 292
    2014-01-14 04:18:12 1 292
    2014-01-14 04:19:12 1 292
    2014-01-14 04:20:12 1 292
    2014-01-14 04:21:12 1 292
    2014-01-14 04:22:12 1 292
    2014-01-14 04:23:12 1 292
    2014-01-14 04:24:12 1 292
    2014-01-14 04:25:13 1 292
    2014-01-14 04:26:14 1 292
    2014-01-14 04:27:12 1 292
    2014-01-14 04:28:20 1 292
    2014-01-14 04:29:12 1 292
    2014-01-14 04:30:12 1 292
    2014-01-14 04:31:13 1 292
    2014-01-14 04:32:12 1 292
    2014-01-14 04:33:12 1 292
    2014-01-14 04:34:12 1 292
    2014-01-14 04:35:12 1 292
    2014-01-14 04:36:13 1 292
    2014-01-14 04:37:12 1 292
    2014-01-14 04:38:12 1 292
    2014-01-14 04:39:12 1 292
    2014-01-14 04:40:12 1 292
    2014-01-14 04:41:12 1 292
    2014-01-14 04:42:13 1 292
    2014-01-14 04:43:12 1 292
    2014-01-14 04:44:12 1 292
    2014-01-14 04:45:11 1 292
    2014-01-14 04:46:11 1 292
    2014-01-14 04:47:11 1 292
    2014-01-14 04:48:11 1 292
    2014-01-14 04:49:11 1 292
    2014-01-14 04:50:11 1 292
    2014-01-14 04:51:11 1 292
    2014-01-14 04:52:11 1 292
    2014-01-14 04:53:11 1 292
    2014-01-14 04:54:11 1 292
    2014-01-14 04:55:11 1 292
    2014-01-14 04:56:11 1 292
    2014-01-14 04:57:11 1 292
    2014-01-14 04:58:13 1 292
    2014-01-14 04:59:11 1 292
    2014-01-14 05:00:11 1 292
    2014-01-14 05:01:11 1 292
    2014-01-14 05:02:14 1 292
    2014-01-14 05:03:11 1 292
    2014-01-14 05:04:12 3 292
    2014-01-14 05:05:11 1 292
    2014-01-14 05:06:11 1 292
    2014-01-14 05:07:11 1 292
    2014-01-14 05:08:11 1 292
    2014-01-14 05:09:11 1 292
    2014-01-14 05:10:12 1 292
    2014-01-14 05:11:11 1 292
    2014-01-14 05:12:11 1 292
    2014-01-14 05:13:11 1 292
    2014-01-14 05:14:11 1 292
    2014-01-14 05:15:11 1 292
    2014-01-14 05:16:12 1 292
    2014-01-14 05:17:11 1 292
    2014-01-14 05:18:11 1 292
    2014-01-14 05:19:11 1 292
    2014-01-14 05:20:11 1 292
    2014-01-14 05:21:11 1 292
    2014-01-14 05:22:12 1 292
    2014-01-14 05:23:11 1 292
    2014-01-14 05:24:11 1 292
    2014-01-14 05:25:11 1 292
    2014-01-14 05:26:11 1 292
    2014-01-14 05:27:11 1 292
    2014-01-14 05:28:11 1 292
    2014-01-14 05:29:11 1 292
    2014-01-14 05:30:11 1 292
    2014-01-14 05:31:11 1 292
    2014-01-14 05:32:11 1 292
    2014-01-14 05:33:12 1 292
    2014-01-14 05:34:11 1 292
    2014-01-14 05:35:11 1 292
    2014-01-14 05:36:11 1 292
    2014-01-14 05:37:12 1 292
    2014-01-14 05:38:12 1 292
    2014-01-14 05:39:11 1 292
    2014-01-14 05:40:11 1 292
    2014-01-14 05:41:11 1 292
    2014-01-14 05:42:11 3 292
    2014-01-14 05:43:12 3 292
    2014-01-14 05:44:12 3 292
    2014-01-14 05:45:11 3 292
    2014-01-14 05:46:11 3 292
    2014-01-14 05:47:11 3 292
    2014-01-14 05:48:11 3 292
    2014-01-14 05:49:11 3 292
    2014-01-14 05:50:12 3 292
    2014-01-14 05:51:11 3 292
    2014-01-14 05:52:11 3 292
    2014-01-14 05:53:11 3 292
    2014-01-14 05:54:11 4 292
    2014-01-14 05:55:11 4 292
    2014-01-14 05:56:12 4 292
    2014-01-14 05:57:11 4 292
    2014-01-14 05:58:11 4 292
    2014-01-14 05:59:11 4 292
    2014-01-14 06:00:12 4 292
    2014-01-14 06:01:12 4 292
    2014-01-14 06:02:12 4 292
    2014-01-14 06:03:35 4 292
    2014-01-14 06:04:11 4 292
    2014-01-14 06:05:12 4 292
    2014-01-14 06:06:11 4 292
    2014-01-14 06:07:11 5 292
    2014-01-14 06:08:11 5 292
    2014-01-14 06:09:11 5 292
    2014-01-14 06:10:11 6 292
    2014-01-14 06:11:11 6 292
    2014-01-14 06:12:11 6 292
    2014-01-14 06:13:12 6 292
    2014-01-14 06:14:11 6 292
    2014-01-14 06:15:11 6 292
    2014-01-14 06:16:11 6 292
    2014-01-14 06:17:11 6 292
    2014-01-14 06:18:12 6 292
    2014-01-14 06:19:12 7 292
    2014-01-14 06:20:11 6 292
    2014-01-14 06:21:11 6 292
    2014-01-14 06:22:13 6 292
    2014-01-14 06:23:11 6 292
    2014-01-14 06:24:12 6 292
    2014-01-14 06:25:13 6 292
    2014-01-14 06:26:11 6 292
    2014-01-14 06:27:13 6 292
    2014-01-14 06:28:11 6 292
    2014-01-14 06:29:11 6 292
    2014-01-14 06:30:11 7 292
    2014-01-14 06:31:11 7 292
    2014-01-14 06:32:11 7 292
    2014-01-14 06:33:15 12 292
    2014-01-14 06:34:12 12 292
    2014-01-14 06:35:12 14 292
    2014-01-14 06:36:12 14 292
    2014-01-14 06:37:11 16 292
    2014-01-14 06:38:11 14 292
    2014-01-14 06:39:12 14 292
    2014-01-14 06:40:14 14 292
    2014-01-14 06:41:11 14 292
    2014-01-14 06:42:13 14 292
    2014-01-14 06:43:12 14 292
    2014-01-14 06:44:12 14 292
    2014-01-14 06:45:12 15 292
    2014-01-14 06:46:11 15 292
    2014-01-14 06:47:12 15 292
    2014-01-14 06:48:13 15 292
    2014-01-14 06:49:11 15 292
    2014-01-14 06:50:12 15 292
    2014-01-14 06:51:11 15 292
    2014-01-14 06:52:13 17 292
    2014-01-14 06:53:11 15 292
    2014-01-14 06:54:13 15 292
    2014-01-14 06:55:12 15 292
    2014-01-14 06:56:12 15 292
    2014-01-14 06:57:11 15 292
    2014-01-14 06:58:11 16 292
    2014-01-14 06:59:11 16 292
    2014-01-14 07:00:11 18 292
    2014-01-14 07:01:13 18 292
    2014-01-14 07:02:18 17 292
    2014-01-14 07:03:21 18 292
    2014-01-14 07:04:14 18 292
    2014-01-14 07:05:12 18 292
    2014-01-14 07:06:12 19 292
    2014-01-14 07:07:11 19 292
    2014-01-14 07:08:12 19 292
    2014-01-14 07:09:12 20 292
    2014-01-14 07:10:12 20 292
    2014-01-14 07:11:12 22 292
    2014-01-14 07:12:28 21 292
    2014-01-14 07:13:12 22 292
    2014-01-14 07:14:12 22 292
    2014-01-14 07:15:12 23 292
    2014-01-14 07:16:12 23 292
    2014-01-14 07:17:12 24 292
    2014-01-14 07:18:12 25 292
    2014-01-14 07:19:12 26 292
    2014-01-14 07:20:12 26 292
    2014-01-14 07:21:11 27 292
    2014-01-14 07:22:12 27 292
    2014-01-14 07:23:13 27 292
    2014-01-14 07:24:11 28 292
    2014-01-14 07:25:11 27 292
    2014-01-14 07:26:12 28 292
    2014-01-14 07:27:12 27 292
    2014-01-14 07:28:11 27 292
    2014-01-14 07:29:11 32 292
    2014-01-14 07:30:12 32 292
    2014-01-14 07:31:12 32 292
    2014-01-14 07:32:12 32 292
    2014-01-14 07:33:16 32 292
    2014-01-14 07:34:11 35 292
    2014-01-14 07:35:12 35 292
    2014-01-14 07:36:14 37 292
    2014-01-14 07:37:12 37 292
    2014-01-14 07:38:12 37 292
    2014-01-14 07:39:11 37 292
    2014-01-14 07:40:12 39 292
    2014-01-14 07:41:11 40 292
    2014-01-14 07:42:22 41 292
    2014-01-14 07:43:11 40 292
    2014-01-14 07:44:13 40 292
    2014-01-14 07:45:12 40 292
    2014-01-14 07:46:12 40 292
    2014-01-14 07:47:12 40 292
    2014-01-14 07:48:12 42 292
    2014-01-14 07:49:20 45 292
    2014-01-14 07:50:12 45 292
    2014-01-14 07:51:11 45 292
    2014-01-14 07:52:12 47 292
    2014-01-14 07:53:11 47 292
    2014-01-14 07:54:11 47 292
    2014-01-14 07:55:12 48 292
    2014-01-14 07:56:12 48 292
    2014-01-14 07:57:12 49 292
    2014-01-14 07:58:12 49 292
    2014-01-14 07:59:12 50 292
    2014-01-14 08:00:12 53 292
    2014-01-14 08:01:12 52 292
    2014-01-14 08:02:12 52 292
    2014-01-14 08:03:34 54 292
    2014-01-14 08:04:12 56 292
    2014-01-14 08:05:12 57 292
    2014-01-14 08:06:13 59 292
    2014-01-14 08:07:11 58 292
    2014-01-14 08:08:12 59 292
    2014-01-14 08:09:12 61 292
    2014-01-14 08:10:12 61 292
    2014-01-14 08:11:12 60 292
    2014-01-14 08:12:12 63 292
    2014-01-14 08:13:12 66 292
    2014-01-14 08:14:12 67 292
    2014-01-14 08:15:12 66 292
    2014-01-14 08:16:12 66 292
    2014-01-14 08:17:12 67 292
    2014-01-14 08:18:14 68 292
    2014-01-14 08:19:12 71 292
    2014-01-14 08:20:12 71 292
    2014-01-14 08:21:12 71 292
    2014-01-14 08:22:14 71 292
    2014-01-14 08:23:12 71 292
    2014-01-14 08:24:18 72 292
    2014-01-14 08:25:12 74 292
    2014-01-14 08:26:12 77 292
    2014-01-14 08:27:12 75 292
    2014-01-14 08:28:12 76 292
    2014-01-14 08:29:12 77 292
    2014-01-14 08:30:19 76 292
    2014-01-14 08:31:12 76 292
    2014-01-14 08:32:12 77 292
    2014-01-14 08:33:33 78 292
    2014-01-14 08:34:14 78 292
    2014-01-14 08:35:19 79 292
    2014-01-14 08:36:12 83 292
    2014-01-14 08:37:12 82 292
    2014-01-14 08:38:14 82 292
    2014-01-14 08:39:13 83 292
    2014-01-14 08:40:12 83 292
    2014-01-14 08:41:12 83 292
    2014-01-14 08:42:12 88 292
    2014-01-14 08:43:12 85 292
    2014-01-14 08:44:12 85 292
    2014-01-14 08:45:12 87 292
    2014-01-14 08:46:14 87 292
    2014-01-14 08:47:12 90 292
    2014-01-14 08:48:12 91 292
    2014-01-14 08:49:13 90 292
    2014-01-14 08:50:12 95 292
    2014-01-14 08:51:19 94 292
    2014-01-14 08:52:12 94 292
    2014-01-14 08:53:27 94 292
    2014-01-14 08:54:13 95 292
    2014-01-14 08:55:12 96 292
    2014-01-14 08:56:13 97 292
    2014-01-14 08:57:21 99 292
    2014-01-14 08:58:17 100 292
    2014-01-14 08:59:12 100 292
    2014-01-14 09:00:12 101 292
    2014-01-14 09:01:26 101 292
    2014-01-14 09:02:26 104 292
    2014-01-14 09:03:15 101 292
    2014-01-14 09:04:12 102 292
    2014-01-14 09:05:12 104 292
    2014-01-14 09:06:15 105 292
    2014-01-14 09:07:12 105 292
    2014-01-14 09:08:13 107 292
    2014-01-14 09:09:12 105 292
    2014-01-14 09:10:15 105 292
    2014-01-14 09:11:12 106 292
    2014-01-14 09:12:12 106 292
    2014-01-14 09:13:12 106 292
    2014-01-14 09:14:12 106 292
    2014-01-14 09:15:12 106 292
    2014-01-14 09:16:12 106 292
    2014-01-14 09:17:12 107 292
    2014-01-14 09:18:12 108 292
    2014-01-14 09:19:12 107 292
    2014-01-14 09:20:12 107 292
    2014-01-14 09:21:16 107 292
    2014-01-14 09:22:12 107 292
    2014-01-14 09:23:29 108 292
    2014-01-14 09:24:12 108 292
    2014-01-14 09:25:14 108 292
    2014-01-14 09:26:12 108 292
    2014-01-14 09:27:12 110 292
    2014-01-14 09:28:12 113 292
    2014-01-14 09:29:13 113 292
    2014-01-14 09:30:12 114 292
    2014-01-14 09:31:18 116 292
    2014-01-14 09:32:12 115 292
    2014-01-14 09:33:12 116 292
    2014-01-14 09:34:13 119 292
    2014-01-14 09:35:12 118 292
    2014-01-14 09:36:12 122 292
    2014-01-14 09:37:13 119 292
    2014-01-14 09:38:12 120 292
    2014-01-14 09:39:12 121 292
    2014-01-14 09:40:12 120 292
    2014-01-14 09:41:12 120 292
    2014-01-14 09:42:20 124 292
    2014-01-14 09:43:15 122 292
    2014-01-14 09:44:12 122 292
    2014-01-14 09:45:12 122 292
    2014-01-14 09:46:14 124 292
    2014-01-14 09:47:12 124 292
    2014-01-14 09:48:12 127 292
    2014-01-14 09:49:15 126 292
    2014-01-14 09:50:14 127 292
    2014-01-14 09:51:12 127 292
    2014-01-14 09:52:13 127 292
    2014-01-14 09:53:13 128 292
    2014-01-14 09:54:14 127 292
    2014-01-14 09:55:12 131 292
    2014-01-14 09:56:17 128 292
    2014-01-14 09:57:13 128 292
    2014-01-14 09:58:12 128 292
    2014-01-14 09:59:12 128 292
    2014-01-14 10:00:14 130 292
    2014-01-14 10:01:12 130 292
    2014-01-14 10:02:12 132 292
    2014-01-14 10:03:12 132 292
    2014-01-14 10:04:12 133 292
    2014-01-14 10:05:12 136 292
    2014-01-14 10:06:12 138 292
    2014-01-14 10:07:12 137 292
    2014-01-14 10:08:12 137 292
    2014-01-14 10:09:12 137 292
    2014-01-14 10:10:12 137 292
    2014-01-14 10:11:12 137 292
    2014-01-14 10:12:12 142 292
    2014-01-14 10:13:12 140 292
    2014-01-14 10:14:12 140 292
    2014-01-14 10:15:12 141 292
    2014-01-14 10:16:12 140 292
    2014-01-14 10:17:12 140 292
    2014-01-14 10:18:13 141 292
    2014-01-14 10:19:12 141 292
    2014-01-14 10:20:12 141 292
    2014-01-14 10:21:12 141 292
    2014-01-14 10:22:12 142 292
    2014-01-14 10:23:12 143 292
    2014-01-14 10:24:13 145 292
    2014-01-14 10:25:12 148 292
    2014-01-14 10:26:12 148 292
    2014-01-14 10:27:12 149 292
    2014-01-14 10:28:12 149 292
    2014-01-14 10:29:14 150 292
    2014-01-14 10:30:12 150 292
    2014-01-14 10:31:12 150 292
    2014-01-14 10:32:13 150 292
    2014-01-14 10:33:12 151 292
    2014-01-14 10:34:13 151 292
    2014-01-14 10:35:12 155 292
    2014-01-14 10:36:14 155 292
    2014-01-14 10:37:12 157 292
    2014-01-14 10:38:12 158 292
    2014-01-14 10:39:12 159 292
    2014-01-14 10:40:12 159 292
    2014-01-14 10:41:12 159 292
    2014-01-14 10:42:12 159 292
    2014-01-14 10:43:12 159 292
    2014-01-14 10:44:12 161 292
    2014-01-14 10:45:13 163 292
    2014-01-14 10:46:12 163 292
    2014-01-14 10:47:12 163 292
    2014-01-14 10:48:12 164 292
    2014-01-14 10:49:12 164 292
    2014-01-14 10:50:12 164 292
    2014-01-14 10:51:12 164 292
    2014-01-14 10:52:12 164 292
    2014-01-14 10:53:12 164 292
    2014-01-14 10:54:12 168 292
    2014-01-14 10:55:18 166 292
    2014-01-14 10:56:12 167 292
    2014-01-14 10:57:12 167 292
    2014-01-14 10:58:12 167 292
    2014-01-14 10:59:12 167 292
    2014-01-14 11:00:12 167 292
    2014-01-14 11:01:12 169 292
    2014-01-14 11:02:12 169 292
    2014-01-14 11:03:12 169 292
    2014-01-14 11:04:18 173 292
    2014-01-14 11:05:12 172 292
    2014-01-14 11:06:12 172 292
    2014-01-14 11:07:12 172 292
    2014-01-14 11:08:12 172 292
    2014-01-14 11:09:12 172 292
    2014-01-14 11:10:12 172 292
    2014-01-14 11:11:12 172 292
    2014-01-14 11:12:12 173 292
    2014-01-14 11:13:12 173 292
    2014-01-14 11:14:12 172 292
    2014-01-14 11:15:12 174 292
    2014-01-14 11:16:13 173 292
    2014-01-14 11:17:12 174 292
    2014-01-14 11:18:12 176 292
    2014-01-14 11:19:12 176 292
    2014-01-14 11:20:12 177 292
    2014-01-14 11:21:12 178 292
    2014-01-14 11:22:12 179 292
    2014-01-14 11:23:12 180 292
    2014-01-14 11:24:12 180 292
    2014-01-14 11:25:12 180 292
    2014-01-14 11:26:12 181 292
    2014-01-14 11:27:12 181 292
    2014-01-14 11:28:12 181 292
    2014-01-14 11:29:12 182 292
    2014-01-14 11:30:12 182 292
    2014-01-14 11:31:12 182 292
    2014-01-14 11:32:12 182 292
    2014-01-14 11:33:12 182 292
    2014-01-14 11:34:14 184 292
    2014-01-14 11:35:16 184 292
    2014-01-14 11:36:12 184 292
    2014-01-14 11:37:12 186 292
    2014-01-14 11:38:12 186 292
    2014-01-14 11:39:12 187 292
    2014-01-14 11:40:12 187 292
    2014-01-14 11:41:12 187 292
    2014-01-14 11:42:12 187 292
    2014-01-14 11:43:14 188 292
    2014-01-14 11:44:12 192 292
    2014-01-14 11:45:12 190 292
    2014-01-14 11:46:12 191 292
    2014-01-14 11:47:13 190 292
    2014-01-14 11:48:12 190 292
    2014-01-14 11:49:12 190 292
    2014-01-14 11:50:12 192 292
    2014-01-14 11:51:12 192 292
    2014-01-14 11:52:12 194 292
    2014-01-14 11:53:13 194 292
    2014-01-14 11:54:12 195 292
    2014-01-14 11:55:12 195 292
    2014-01-14 11:56:12 197 292
    2014-01-14 11:57:12 197 292
    2014-01-14 11:58:12 197 292
    2014-01-14 11:59:13 198 292
    2014-01-14 12:00:13 199 292
    2014-01-14 12:01:12 198 292
    2014-01-14 12:02:12 198 292
    2014-01-14 12:03:12 198 292
    2014-01-14 12:04:13 200 292
    2014-01-14 12:05:17 200 292
    2014-01-14 12:06:12 200 292
    2014-01-14 12:07:12 201 292
    2014-01-14 12:08:12 201 292
    2014-01-14 12:09:12 205 292
    2014-01-14 12:10:13 201 292
    2014-01-14 12:11:12 203 292
    2014-01-14 12:12:12 204 292
    2014-01-14 12:13:20 204 292
    2014-01-14 12:14:12 205 292
    2014-01-14 12:15:17 204 292
    2014-01-14 12:16:12 205 292
    2014-01-14 12:17:14 205 292
    2014-01-14 12:18:12 211 292
    2014-01-14 12:19:14 213 292
    2014-01-14 12:20:12 212 292
    2014-01-14 12:21:12 212 292
    2014-01-14 12:22:12 211 292
    2014-01-14 12:23:12 212 292
    2014-01-14 12:24:13 213 292
    2014-01-14 12:25:12 214 292
    2014-01-14 12:26:13 214 292
    2014-01-14 12:27:12 215 292
    2014-01-14 12:28:12 216 292
    2014-01-14 12:29:14 215 292
    2014-01-14 12:30:13 215 292
    2014-01-14 12:31:12 215 292
    2014-01-14 12:32:12 215 292
    2014-01-14 12:33:12 216 292
    2014-01-14 12:34:12 215 292
    2014-01-14 12:35:18 220 292
    2014-01-14 12:36:13 218 292
    2014-01-14 12:37:12 217 292
    2014-01-14 12:38:12 217 292
    2014-01-14 12:39:12 217 292
    2014-01-14 12:40:12 217 292
    2014-01-14 12:41:12 219 292
    2014-01-14 12:42:12 217 292
    2014-01-14 12:43:12 217 292
    2014-01-14 12:44:12 217 292
    2014-01-14 12:45:12 217 292
    2014-01-14 12:46:12 218 292
    2014-01-14 12:47:13 218 292
    2014-01-14 12:48:12 218 292
    2014-01-14 12:49:12 220 292
    2014-01-14 12:50:12 218 292
    2014-01-14 12:51:12 218 292
    2014-01-14 12:52:12 218 292
    2014-01-14 12:53:15 218 292
    2014-01-14 12:54:12 218 292
    2014-01-14 12:55:12 219 292
    2014-01-14 12:56:12 222 292
    2014-01-14 12:57:11 221 292
    2014-01-14 12:58:11 223 292
    2014-01-14 12:59:11 223 292
    2014-01-14 13:00:11 223 292
    2014-01-14 13:01:13 223 292
    2014-01-14 13:02:11 223 292
    2014-01-14 13:03:17 224 292
    2014-01-14 13:04:11 226 292
    2014-01-14 13:05:14 227 292
    2014-01-14 13:06:11 230 292
    2014-01-14 13:07:11 227 292
    2014-01-14 13:08:11 227 292
    2014-01-14 13:09:11 227 292
    2014-01-14 13:10:11 227 292
    2014-01-14 13:11:11 228 292
    2014-01-14 13:12:11 227 292
    2014-01-14 13:13:11 228 292
    2014-01-14 13:14:12 229 292
    2014-01-14 13:15:11 228 292
    2014-01-14 13:16:11 227 292
    2014-01-14 13:17:11 228 292
    2014-01-14 13:18:11 229 292
    2014-01-14 13:19:12 228 292
    2014-01-14 13:20:11 228 292
    2014-01-14 13:21:11 229 292
    2014-01-14 13:22:12 228 292
    2014-01-14 13:23:11 231 292
    2014-01-14 13:24:12 231 292
    2014-01-14 13:25:11 230 292
    2014-01-14 13:26:11 231 292
    2014-01-14 13:27:11 231 292
    2014-01-14 13:28:11 231 292
    2014-01-14 13:29:12 232 292
    2014-01-14 13:30:13 231 292
    2014-01-14 13:31:11 234 292
    2014-01-14 13:32:12 232 292
    2014-01-14 13:33:11 233 292
    2014-01-14 13:34:11 233 292
    2014-01-14 13:35:11 235 292
    2014-01-14 13:36:13 236 292
    2014-01-14 13:37:12 235 292
    2014-01-14 13:38:11 235 292
    2014-01-14 13:39:11 236 292
    2014-01-14 13:40:11 236 292
    2014-01-14 13:41:12 237 292
    2014-01-14 13:42:12 238 292
    2014-01-14 13:43:11 239 292
    2014-01-14 13:44:12 240 292
    2014-01-14 13:45:12 242 292
    2014-01-14 13:46:11 242 292
    2014-01-14 13:47:12 243 292
    2014-01-14 13:48:11 243 292
    2014-01-14 13:49:13 243 292
    2014-01-14 13:50:22 243 292
    2014-01-14 13:51:11 243 292
    2014-01-14 13:52:11 244 292
    2014-01-14 13:53:12 245 292
    2014-01-14 13:54:11 246 292
    2014-01-14 13:55:12 246 292
    2014-01-14 13:56:11 249 292
    2014-01-14 13:57:11 247 292
    2014-01-14 13:58:11 249 292
    2014-01-14 13:59:11 249 292
    2014-01-14 14:00:11 249 292
    2014-01-14 14:01:12 249 292
    2014-01-14 14:02:11 249 292
    2014-01-14 14:03:25 250 292
    2014-01-14 14:04:11 252 292
    2014-01-14 14:05:11 252 292
    2014-01-14 14:06:11 253 292
    2014-01-14 14:07:11 253 292
    2014-01-14 14:08:12 257 292
    2014-01-14 14:09:11 255 292
    2014-01-14 14:10:11 255 292
    2014-01-14 14:11:12 256 292
    2014-01-14 14:12:11 255 292
    2014-01-14 14:13:11 255 292
    2014-01-14 14:14:12 256 292
    2014-01-14 14:15:11 258 292
    2014-01-14 14:16:11 257 292
    2014-01-14 14:17:11 257 292
    2014-01-14 14:18:11 257 292
    2014-01-14 14:19:12 257 292
    2014-01-14 14:20:12 258 292
    2014-01-14 14:21:12 258 292
    2014-01-14 14:22:12 258 292
    2014-01-14 14:23:11 257 292
    2014-01-14 14:24:13 257 292
    2014-01-14 14:25:11 258 292
    2014-01-14 14:26:11 259 292
    2014-01-14 14:27:11 259 292
    2014-01-14 14:28:13 263 292
    2014-01-14 14:29:11 263 292
    2014-01-14 14:30:12 264 292
    2014-01-14 14:31:12 263 292
    2014-01-14 14:32:11 265 292
    2014-01-14 14:33:12 264 292
    2014-01-14 14:34:11 264 292
    2014-01-14 14:35:12 264 292
    2014-01-14 14:36:11 265 292
    2014-01-14 14:37:12 264 292
    2014-01-14 14:38:12 267 292
    2014-01-14 14:39:12 266 292
    2014-01-14 14:40:13 265 292
    2014-01-14 14:41:12 266 292
    2014-01-14 14:42:12 265 292
    2014-01-14 14:43:11 265 292
    2014-01-14 14:44:15 269 292
    2014-01-14 14:45:12 272 292
    2014-01-14 14:46:11 270 292
    2014-01-14 14:47:12 270 292
    2014-01-14 14:48:12 273 292
    2014-01-14 14:49:12 273 292
    2014-01-14 14:50:12 274 292
    2014-01-14 14:51:12 275 292
    2014-01-14 14:52:11 275 292
    2014-01-14 14:53:21 277 292
    2014-01-14 14:54:11 278 292
    2014-01-14 14:55:12 282 292
    2014-01-14 14:56:12 282 292
    2014-01-14 14:57:11 282 292
    2014-01-14 14:58:12 282 292
    2014-01-14 14:59:12 282 292
    2014-01-14 15:00:12 282 292
    2014-01-14 15:01:11 282 292
    2014-01-14 15:02:13 282 292
    2014-01-14 15:03:11 282 292
    2014-01-14 15:04:15 283 292
    2014-01-14 15:05:13 282 292
    2014-01-14 15:06:12 283 292
    2014-01-14 15:07:14 283 292
    2014-01-14 15:08:12 283 292
    2014-01-14 15:09:12 283 292
    2014-01-14 15:10:13 283 292
    2014-01-14 15:11:12 283 292
    2014-01-14 15:12:12 286 292
    2014-01-14 15:13:24 284 292
    2014-01-14 15:14:12 284 292
    2014-01-14 15:15:12 286 292
    2014-01-14 15:16:12 286 292
    2014-01-14 15:17:28 292 292
    2014-01-14 15:18:12 1 292
    2014-01-14 15:19:12 1 292
    2014-01-14 15:20:12 1 292
    2014-01-14 15:21:12 1 292
    2014-01-14 15:22:12 1 292
    2014-01-14 15:23:12 3 292
    2014-01-14 15:24:12 2 292
    2014-01-14 15:25:12 2 292
    2014-01-14 15:26:12 3 292
    2014-01-14 15:27:11 3 292
    2014-01-14 15:28:12 3 292
    2014-01-14 15:29:11 4 292
    2014-01-14 15:30:14 4 292
    2014-01-14 15:31:12 4 292
    2014-01-14 15:32:12 6 292
    2014-01-14 15:33:12 5 292
    2014-01-14 15:34:12 5 292
    2014-01-14 15:35:12 5 292
    2014-01-14 15:36:12 7 292
    2014-01-14 15:37:14 10 292
    2014-01-14 15:38:12 8 292
    2014-01-14 15:39:12 8 292
    2014-01-14 15:40:12 8 292
    2014-01-14 15:41:13 9 292
    2014-01-14 15:42:12 9 292
    2014-01-14 15:43:13 9 292
    2014-01-14 15:44:12 9 292
    2014-01-14 15:45:12 10 292
    2014-01-14 15:46:12 9 292
    2014-01-14 15:47:12 10 292
    2014-01-14 15:48:12 10 292
    2014-01-14 15:49:12 10 292
    2014-01-14 15:50:12 10 292
    2014-01-14 15:51:12 11 292
    2014-01-14 15:52:12 11 292
    2014-01-14 15:53:12 14 292
    2014-01-14 15:54:12 13 292
    2014-01-14 15:55:12 13 292
    2014-01-14 15:56:12 15 292
    2014-01-14 15:57:12 13 292
    2014-01-14 15:58:12 13 292
    2014-01-14 15:59:12 13 292
    2014-01-14 16:00:12 13 292
    2014-01-14 16:01:12 16 292
    2014-01-14 16:02:12 17 292
    2014-01-14 16:03:34 17 292
    2014-01-14 16:04:13 17 292
    2014-01-14 16:05:12 17 292
    2014-01-14 16:06:12 20 292
    2014-01-14 16:07:14 20 292
    2014-01-14 16:08:12 21 292
    2014-01-14 16:09:15 21 292
    2014-01-14 16:10:12 21 292
    2014-01-14 16:11:19 22 292
    2014-01-14 16:12:13 21 292
    2014-01-14 16:13:12 21 292
    2014-01-14 16:14:16 22 292
    2014-01-14 16:15:12 22 292
    2014-01-14 16:16:15 23 292
    2014-01-14 16:17:12 23 292
    2014-01-14 16:18:12 23 292
    2014-01-14 16:19:12 24 292
    2014-01-14 16:20:13 25 292
    2014-01-14 16:21:12 25 292
    2014-01-14 16:22:12 25 292
    2014-01-14 16:23:12 25 292
    2014-01-14 16:24:12 25 292
    2014-01-14 16:25:13 29 292
    2014-01-14 16:26:12 29 292
    2014-01-14 16:27:12 28 292
    2014-01-14 16:28:12 28 292
    2014-01-14 16:29:12 28 292
    2014-01-14 16:30:12 28 292
    2014-01-14 16:31:12 28 292
    2014-01-14 16:32:12 29 292
    2014-01-14 16:33:12 28 292
    2014-01-14 16:34:12 28 292
    2014-01-14 16:35:13 30 292
    2014-01-14 16:36:12 28 292
    2014-01-14 16:37:12 28 292
    2014-01-14 16:38:15 28 292
    2014-01-14 16:39:12 28 292
    2014-01-14 16:40:12 30 292
    2014-01-14 16:41:14 29 292
    2014-01-14 16:42:12 28 292
    2014-01-14 16:43:15 28 292
    2014-01-14 16:44:12 28 292
    2014-01-14 16:45:12 29 292
    2014-01-14 16:46:12 29 292
    2014-01-14 16:47:12 30 292
    2014-01-14 16:48:12 32 292
    2014-01-14 16:49:12 31 292
    2014-01-14 16:50:12 31 292
    2014-01-14 16:51:12 32 292
    2014-01-14 16:52:12 32 292
    2014-01-14 16:53:12 31 292
    2014-01-14 16:54:13 33 292
    2014-01-14 16:55:12 34 292
    2014-01-14 16:56:12 33 292
    2014-01-14 16:57:12 34 292
    2014-01-14 16:58:12 37 292
    2014-01-14 16:59:14 37 292
    2014-01-14 17:00:13 38 292
    2014-01-14 17:01:12 38 292
    2014-01-14 17:02:12 38 292
    2014-01-14 17:03:12 38 292
    2014-01-14 17:04:12 38 292
    2014-01-14 17:05:15 38 292
    2014-01-14 17:06:12 38 292
    2014-01-14 17:07:13 38 292
    2014-01-14 17:08:12 38 292
    2014-01-14 17:09:12 39 292
    2014-01-14 17:10:12 39 292
    2014-01-14 17:11:20 39 292
    2014-01-14 17:12:12 39 292
    2014-01-14 17:13:12 39 292
    2014-01-14 17:14:12 39 292
    2014-01-14 17:15:12 39 292
    2014-01-14 17:16:12 39 292
    2014-01-14 17:17:12 39 292
    2014-01-14 17:18:12 39 292
    2014-01-14 17:19:12 39 292
    2014-01-14 17:20:12 39 292
    2014-01-14 17:21:12 39 292
    2014-01-14 17:22:12 39 292
    2014-01-14 17:23:12 39 292
    2014-01-14 17:24:12 39 292
    2014-01-14 17:25:12 39 292
    2014-01-14 17:26:12 39 292
    2014-01-14 17:27:12 40 292
    2014-01-14 17:28:12 39 292
    2014-01-14 17:29:12 39 292
    2014-01-14 17:30:12 39 292
    2014-01-14 17:31:12 39 292
    2014-01-14 17:32:12 39 292
    2014-01-14 17:33:12 40 292
    2014-01-14 17:34:12 39 292
    2014-01-14 17:35:17 39 292
    2014-01-14 17:36:12 41 292
    2014-01-14 17:37:12 40 292
    2014-01-14 17:38:23 40 292
    2014-01-14 17:39:12 40 292
    2014-01-14 17:40:12 40 292
    2014-01-14 17:41:12 40 292
    2014-01-14 17:42:12 40 292
    2014-01-14 17:43:12 40 292
    2014-01-14 17:44:12 40 292
    2014-01-14 17:45:12 41 292
    2014-01-14 17:46:12 41 292
    2014-01-14 17:47:12 41 292
    2014-01-14 17:48:12 41 292
    2014-01-14 17:49:12 42 292
    2014-01-14 17:50:12 41 292
    2014-01-14 17:51:12 41 292
    2014-01-14 17:52:12 42 292
    2014-01-14 17:53:12 42 292
    2014-01-14 17:54:13 42 292
    2014-01-14 17:55:12 42 292
    2014-01-14 17:56:12 42 292
    2014-01-14 17:57:19 42 292
    2014-01-14 17:58:12 42 292
    2014-01-14 17:59:12 42 292
    2014-01-14 18:00:12 42 292
    2014-01-14 18:01:12 42 292
    2014-01-14 18:02:16 42 292
    2014-01-14 18:03:12 43 292
    2014-01-14 18:04:14 43 292
    2014-01-14 18:05:12 44 292
    2014-01-14 18:06:12 43 292
    2014-01-14 18:07:13 43 292
    2014-01-14 18:08:12 43 292
    2014-01-14 18:09:12 43 292
    2014-01-14 18:10:12 43 292
    2014-01-14 18:11:12 45 292
    2014-01-14 18:12:12 43 292
    2014-01-14 18:13:12 43 292
    2014-01-14 18:14:12 43 292
    2014-01-14 18:15:12 43 292
    2014-01-14 18:16:12 43 292
    2014-01-14 18:17:12 43 292
    2014-01-14 18:18:12 44 292
    2014-01-14 18:19:13 43 292
    2014-01-14 18:20:12 43 292
    2014-01-14 18:21:12 44 292
    2014-01-14 18:22:12 43 292
    2014-01-14 18:23:12 43 292
    2014-01-14 18:24:12 43 292
    2014-01-14 18:25:12 43 292
    2014-01-14 18:26:12 43 292
    2014-01-14 18:27:12 43 292
    2014-01-14 18:28:12 43 292
    2014-01-14 18:29:12 43 292
    2014-01-14 18:30:13 43 292
    2014-01-14 18:31:12 43 292
    2014-01-14 18:32:12 43 292
    2014-01-14 18:33:12 43 292
    2014-01-14 18:34:12 43 292
    2014-01-14 18:35:14 43 292
    2014-01-14 18:36:12 43 292
    2014-01-14 18:37:12 43 292
    2014-01-14 18:38:12 43 292
    2014-01-14 18:39:12 43 292
    2014-01-14 18:40:12 43 292
    2014-01-14 18:41:13 43 292
    2014-01-14 18:42:12 43 292
    2014-01-14 18:43:12 43 292
    2014-01-14 18:44:12 43 292
    2014-01-14 18:45:12 43 292
    2014-01-14 18:46:15 43 292
    2014-01-14 18:47:12 43 292
    2014-01-14 18:48:12 43 292
    2014-01-14 18:49:12 43 292
    2014-01-14 18:50:12 43 292
    2014-01-14 18:51:12 43 292
    2014-01-14 18:52:13 43 292
    2014-01-14 18:53:12 43 292
    2014-01-14 18:54:12 43 292
    2014-01-14 18:55:12 43 292
    2014-01-14 18:56:12 43 292
    2014-01-14 18:57:12 43 292
    2014-01-14 18:58:12 43 292
    2014-01-14 18:59:12 43 292
    2014-01-14 19:00:12 43 292
    2014-01-14 19:01:12 43 292
    2014-01-14 19:02:12 43 292
    2014-01-14 19:03:15 43 292
    2014-01-14 19:04:12 43 292
    2014-01-14 19:05:14 43 292
    2014-01-14 19:06:12 43 292
    2014-01-14 19:07:12 43 292
    2014-01-14 19:08:16 43 292
    2014-01-14 19:09:12 43 292
    2014-01-14 19:10:12 43 292
    2014-01-14 19:11:12 43 292
    2014-01-14 19:12:12 43 292
    2014-01-14 19:13:31 43 292
    2014-01-14 19:14:12 43 292
    2014-01-14 19:15:12 43 292
    2014-01-14 19:16:12 43 292
    2014-01-14 19:17:12 43 292
    2014-01-14 19:18:12 43 292
    2014-01-14 19:19:13 43 292
    2014-01-14 19:20:13 43 292
    2014-01-14 19:21:16 43 292
    2014-01-14 19:22:13 43 292
    2014-01-14 19:23:12 43 292
    2014-01-14 19:24:17 43 292
    2014-01-14 19:25:13 43 292
    2014-01-14 19:26:12 43 292
    2014-01-14 19:27:12 43 292
    2014-01-14 19:28:12 43 292
    2014-01-14 19:29:12 43 292
    2014-01-14 19:30:12 43 292
    2014-01-14 19:31:12 43 292
    2014-01-14 19:32:12 43 292
    2014-01-14 19:33:12 43 292
    2014-01-14 19:34:12 43 292
    2014-01-14 19:35:12 43 292
    2014-01-14 19:36:15 43 292
    2014-01-14 19:37:12 43 292
    2014-01-14 19:38:12 43 292
    2014-01-14 19:39:12 43 292
    2014-01-14 19:40:12 43 292
    2014-01-14 19:41:13 43 292
    2014-01-14 19:42:12 43 292
    2014-01-14 19:43:12 43 292
    2014-01-14 19:44:12 43 292
    2014-01-14 19:45:13 43 292
    2014-01-14 19:46:12 43 292
    2014-01-14 19:47:13 43 292
    2014-01-14 19:48:12 43 292
    2014-01-14 19:49:12 43 292
    2014-01-14 19:50:12 43 292
    2014-01-14 19:51:12 43 292
    2014-01-14 19:52:12 43 292
    2014-01-14 19:53:12 43 292
    2014-01-14 19:54:11 43 292
    2014-01-14 19:55:11 43 292
    2014-01-14 19:56:11 43 292
    2014-01-14 19:57:11 43 292
    2014-01-14 19:58:11 43 292
    2014-01-14 19:59:11 43 292
    2014-01-14 20:00:12 43 292
    2014-01-14 20:01:12 43 292
    2014-01-14 20:02:11 43 292
    2014-01-14 20:03:11 43 292
    2014-01-14 20:04:12 43 292
    2014-01-14 20:05:11 43 292
    2014-01-14 20:06:11 43 292
    2014-01-14 20:07:11 43 292
    2014-01-14 20:08:13 44 292
    2014-01-14 20:09:11 44 292
    2014-01-14 20:10:12 44 292
    2014-01-14 20:11:11 44 292
    2014-01-14 20:12:11 44 292
    2014-01-14 20:13:11 44 292
    2014-01-14 20:14:11 44 292
    2014-01-14 20:15:11 44 292
    2014-01-14 20:16:11 44 292
    2014-01-14 20:17:11 44 292
    2014-01-14 20:18:11 44 292
    2014-01-14 20:19:11 44 292
    2014-01-14 20:20:11 44 292
    2014-01-14 20:21:12 44 292
    2014-01-14 20:22:11 44 292
    2014-01-14 20:23:11 44 292
    2014-01-14 20:24:11 45 292
    2014-01-14 20:25:11 44 292
    2014-01-14 20:26:12 44 292
    2014-01-14 20:27:11 44 292
    2014-01-14 20:28:11 44 292
    2014-01-14 20:29:11 44 292
    2014-01-14 20:30:11 44 292
    2014-01-14 20:31:11 44 292
    2014-01-14 20:32:11 44 292
    2014-01-14 20:33:11 44 292
    2014-01-14 20:34:11 44 292
    2014-01-14 20:35:11 44 292
    2014-01-14 20:36:15 44 292
    2014-01-14 20:37:12 44 292
    2014-01-14 20:38:11 44 292
    2014-01-14 20:39:11 44 292
    2014-01-14 20:40:11 44 292
    2014-01-14 20:41:12 44 292
    2014-01-14 20:42:12 44 292
    2014-01-14 20:43:11 44 292
    2014-01-14 20:44:11 44 292
    2014-01-14 20:45:11 44 292
    2014-01-14 20:46:11 44 292
    2014-01-14 20:47:12 44 292
    2014-01-14 20:48:12 44 292
    2014-01-14 20:49:11 44 292
    2014-01-14 20:50:11 44 292
    2014-01-14 20:51:11 44 292
    2014-01-14 20:52:11 44 292
    2014-01-14 20:53:11 44 292
    2014-01-14 20:54:11 44 292
    2014-01-14 20:55:12 44 292
    2014-01-14 20:56:11 44 292
    2014-01-14 20:57:11 44 292
    2014-01-14 20:58:11 44 292
    2014-01-14 20:59:11 44 292
    2014-01-14 21:00:11 44 292
    2014-01-14 21:01:11 44 292
    2014-01-14 21:02:11 44 292
    2014-01-14 21:03:11 44 292
    2014-01-14 21:04:11 44 292
    2014-01-14 21:05:11 44 292
    2014-01-14 21:06:18 44 292
    2014-01-14 21:07:11 44 292
    2014-01-14 21:08:13 44 292
    2014-01-14 21:09:11 44 292
    2014-01-14 21:10:11 44 292
    2014-01-14 21:11:11 44 292
    2014-01-14 21:12:11 44 292
    2014-01-14 21:13:11 44 292
    2014-01-14 21:15:11 0 292
    2014-01-14 21:16:11 0 292
    2014-01-14 21:17:11 0 292
    2014-01-14 21:18:11 0 292
    2014-01-14 21:19:11 0 292
    2014-01-14 21:20:12 0 292
    2014-01-14 21:21:11 0 292
    2014-01-14 21:22:12 0 292
    2014-01-14 21:23:11 0 292
    2014-01-14 21:24:11 0 292
    2014-01-14 21:25:11 0 292
    2014-01-14 21:26:13 0 292
    2014-01-14 21:27:11 0 292
    2014-01-14 21:28:11 0 292
    2014-01-14 21:29:11 0 292
    2014-01-14 21:30:12 0 292
    2014-01-14 21:31:12 0 292
    2014-01-14 21:32:11 0 292
    2014-01-14 21:33:11 0 292
    2014-01-14 21:34:11 0 292
    2014-01-14 21:35:11 0 292
    2014-01-14 21:36:14 0 292
    2014-01-14 21:37:11 0 292
    2014-01-14 21:38:11 0 292
    2014-01-14 21:39:11 0 292
    2014-01-14 21:40:12 0 292
    2014-01-14 21:41:11 0 292
    2014-01-14 21:42:11 0 292
    2014-01-14 21:43:12 0 292
    2014-01-14 21:44:11 0 292
    2014-01-14 21:45:11 0 292
    2014-01-14 21:46:11 0 292
    2014-01-14 21:47:11 0 292
    2014-01-14 21:48:12 0 292
    2014-01-14 21:49:11 0 292
    2014-01-14 21:50:12 0 292
    2014-01-14 21:51:11 0 292
    2014-01-14 21:52:12 0 292
    2014-01-14 21:53:11 0 292
    2014-01-14 21:54:11 0 292
    2014-01-14 21:55:12 0 292
    2014-01-14 21:56:11 0 292
    2014-01-14 21:57:11 0 292
    2014-01-14 21:58:11 0 292
    2014-01-14 21:59:11 0 292
    2014-01-14 22:00:13 0 292
    2014-01-14 22:01:11 0 292
    2014-01-14 22:02:12 0 292
    2014-01-14 22:03:32 0 292
    2014-01-14 22:04:12 0 292
    2014-01-14 22:05:12 0 292
    2014-01-14 22:06:12 0 292
    2014-01-14 22:07:12 0 292
    2014-01-14 22:08:12 0 292
    2014-01-14 22:09:11 0 292
    2014-01-14 22:10:11 0 292
    2014-01-14 22:11:12 0 292
    2014-01-14 22:12:12 0 292
    2014-01-14 22:13:12 0 292
    2014-01-14 22:14:12 0 292
    2014-01-14 22:15:11 0 292
    2014-01-14 22:16:11 0 292
    2014-01-14 22:17:11 0 292
    2014-01-14 22:18:12 0 292
    2014-01-14 22:19:11 0 292
    2014-01-14 22:20:12 0 292
    2014-01-14 22:21:11 2 292
    2014-01-14 22:22:13 0 292
    2014-01-14 22:23:12 0 292
    2014-01-14 22:24:12 0 292
    2014-01-14 22:25:12 0 292
    2014-01-14 22:26:12 0 292
    2014-01-14 22:27:12 0 292
    2014-01-14 22:28:24 0 292
    2014-01-14 22:29:11 0 292
    2014-01-14 22:30:12 0 292
    2014-01-14 22:31:11 0 292
    2014-01-14 22:32:12 0 292
    2014-01-14 22:33:12 0 292
    2014-01-14 22:34:12 0 292
    2014-01-14 22:35:12 0 292
    2014-01-14 22:36:12 0 292
    2014-01-14 22:37:12 0 292
    2014-01-14 22:38:11 0 292
    2014-01-14 22:39:12 0 292
    2014-01-14 22:40:12 0 292
    2014-01-14 22:41:12 0 292
    2014-01-14 22:42:12 0 292
    2014-01-14 22:43:12 0 292
    2014-01-14 22:44:12 0 292
    2014-01-14 22:45:12 0 292
    2014-01-14 22:46:12 0 292
    2014-01-14 22:47:12 0 292
    2014-01-14 22:48:12 0 292
    2014-01-14 22:49:12 0 292
    2014-01-14 22:50:12 0 292
    2014-01-14 22:51:12 0 292
    2014-01-14 22:52:12 0 292
    2014-01-14 22:53:20 0 292
    2014-01-14 22:54:11 0 292
    2014-01-14 22:55:12 0 292
    2014-01-14 22:56:12 0 292
    2014-01-14 22:57:12 0 292
    2014-01-14 22:58:12 0 292
    2014-01-14 22:59:12 0 292
    2014-01-14 23:00:12 0 292
    2014-01-14 23:01:12 0 292
    2014-01-14 23:02:12 0 292
    2014-01-14 23:03:26 0 292
    2014-01-14 23:04:12 0 292
    2014-01-14 23:05:12 0 292
    2014-01-14 23:06:12 0 292
    2014-01-14 23:07:12 0 292
    2014-01-14 23:08:12 0 292
    2014-01-14 23:09:12 0 292
    2014-01-14 23:10:12 0 292
    2014-01-14 23:11:12 0 292
    2014-01-14 23:12:12 0 292
    2014-01-14 23:13:12 0 292
    2014-01-14 23:14:12 0 292
    2014-01-14 23:15:12 0 292
    2014-01-14 23:16:12 0 292
    2014-01-14 23:17:12 0 292
    2014-01-14 23:18:12 0 292
    2014-01-14 23:19:12 0 292
    2014-01-14 23:20:12 0 292
    2014-01-14 23:21:14 0 292
    2014-01-14 23:22:12 0 292
    2014-01-14 23:23:12 0 292
    2014-01-14 23:24:12 0 292
    2014-01-14 23:25:12 0 292
    2014-01-14 23:26:12 0 292
    2014-01-14 23:27:12 0 292
    2014-01-14 23:28:23 0 292
    2014-01-14 23:29:12 0 292
    2014-01-14 23:30:12 0 292
    2014-01-14 23:31:12 0 292
    2014-01-14 23:32:12 0 292
    2014-01-14 23:33:12 0 292
    2014-01-14 23:34:12 0 292
    2014-01-14 23:35:12 0 292
    2014-01-14 23:36:12 0 292
    2014-01-14 23:37:12 0 292
    2014-01-14 23:38:12 0 292
    2014-01-14 23:39:12 0 292
    2014-01-14 23:40:12 0 292
    2014-01-14 23:41:12 0 292
    2014-01-14 23:42:12 0 292
    2014-01-14 23:43:12 0 292
    2014-01-14 23:44:12 0 292
    2014-01-14 23:45:12 0 292
    2014-01-14 23:46:12 0 292
    2014-01-14 23:47:12 0 292
    2014-01-14 23:48:12 0 292
    2014-01-14 23:49:12 0 292
    2014-01-14 23:50:12 0 292
    2014-01-14 23:51:12 0 292
    2014-01-14 23:52:12 0 292
    2014-01-14 23:53:12 0 292
    2014-01-14 23:54:12 0 292
    2014-01-14 23:55:12 0 292
    2014-01-14 23:56:12 0 292
    2014-01-14 23:57:12 0 292
    2014-01-14 23:58:12 0 292
    2014-01-14 23:59:12 0 292
    2014-01-15 00:00:12 0 292
    2014-01-15 00:01:12 0 292
    2014-01-15 00:02:12 0 292
    2014-01-15 00:03:12 0 292
    2014-01-15 00:04:12 0 292
    2014-01-15 00:05:12 0 292
    2014-01-15 00:06:12 0 292
    2014-01-15 00:07:16 0 292
    2014-01-15 00:08:12 0 292
    2014-01-15 00:09:12 0 292
    2014-01-15 00:10:12 0 292
    2014-01-15 00:11:12 0 292
    2014-01-15 00:12:12 0 292
    2014-01-15 00:13:16 0 292
    2014-01-15 00:14:12 0 292
    2014-01-15 00:15:12 0 292
    2014-01-15 00:16:12 0 292
    2014-01-15 00:17:12 0 292
    2014-01-15 00:18:12 0 292
    2014-01-15 00:19:12 0 292
    2014-01-15 00:20:12 0 292
    2014-01-15 00:21:12 0 292
    2014-01-15 00:22:12 0 292
    2014-01-15 00:23:12 0 292
    2014-01-15 00:24:12 0 292
    2014-01-15 00:25:12 0 292
    2014-01-15 00:26:12 0 292
    2014-01-15 00:27:12 0 292
    2014-01-15 00:28:38 0 292
    2014-01-15 00:29:12 0 292
    2014-01-15 00:30:12 0 292
    2014-01-15 00:31:12 0 292
    2014-01-15 00:32:12 0 292
    2014-01-15 00:33:12 0 292
    2014-01-15 00:34:12 0 292
    2014-01-15 00:35:12 0 292
    2014-01-15 00:36:12 0 292
    2014-01-15 00:37:12 0 292
    2014-01-15 00:38:12 0 292
    2014-01-15 00:39:12 0 292
    2014-01-15 00:40:12 0 292
    2014-01-15 00:41:12 0 292
    2014-01-15 00:42:12 0 292
    2014-01-15 00:43:22 0 292
    2014-01-15 00:44:12 0 292
    2014-01-15 00:45:12 0 292
    2014-01-15 00:46:12 0 292
    2014-01-15 00:47:12 0 292
    2014-01-15 00:48:15 0 292
    2014-01-15 00:49:12 0 292
    2014-01-15 00:50:12 0 292
    2014-01-15 00:51:12 0 292
    2014-01-15 00:52:12 0 292
    2014-01-15 00:53:41 0 292
    2014-01-15 00:54:12 0 292
    2014-01-15 00:55:12 0 292
    2014-01-15 00:56:12 0 292
    2014-01-15 00:57:12 0 292
    2014-01-15 00:58:15 0 292
    2014-01-15 00:59:12 0 292
    2014-01-15 01:00:12 0 292
    2014-01-15 01:01:12 0 292
    2014-01-15 01:02:12 0 292
    2014-01-15 01:03:12 0 292
    2014-01-15 01:04:12 0 292
    2014-01-15 01:05:12 0 292
    2014-01-15 01:06:12 0 292
    2014-01-15 01:07:14 0 292
    2014-01-15 01:08:13 0 292
    2014-01-15 01:09:12 0 292
    2014-01-15 01:10:16 0 292
    2014-01-15 01:11:12 0 292
    2014-01-15 01:12:12 0 292
    2014-01-15 01:13:13 0 292
    2014-01-15 01:14:12 0 292
    2014-01-15 01:15:16 0 292
    2014-01-15 01:16:12 0 292
    2014-01-15 01:17:12 0 292
    2014-01-15 01:18:13 0 292
    2014-01-15 01:19:12 0 292
    2014-01-15 01:20:12 0 292
    2014-01-15 01:21:12 0 292
    2014-01-15 01:22:12 0 292
    2014-01-15 01:23:12 0 292
    2014-01-15 01:24:12 0 292
    2014-01-15 01:25:12 0 292
    2014-01-15 01:26:12 0 292
    2014-01-15 01:27:12 0 292
    2014-01-15 01:28:14 0 292
    2014-01-15 01:29:12 0 292
    2014-01-15 01:30:13 0 292
    2014-01-15 01:31:12 0 292
    2014-01-15 01:32:12 0 292
    2014-01-15 01:33:18 0 292
    2014-01-15 01:34:12 0 292
    2014-01-15 01:35:12 0 292
    2014-01-15 01:36:13 0 292
    2014-01-15 01:37:12 0 292
    2014-01-15 01:38:12 0 292
    2014-01-15 01:39:12 0 292
    2014-01-15 01:40:12 0 292
    2014-01-15 01:41:12 0 292
    2014-01-15 01:42:13 0 292
    2014-01-15 01:43:12 0 292
    2014-01-15 01:44:12 0 292
    2014-01-15 01:45:12 0 292
    2014-01-15 01:46:12 0 292
    2014-01-15 01:47:12 0 292
    2014-01-15 01:48:13 0 292
    2014-01-15 01:49:12 0 292
    2014-01-15 01:50:12 0 292
    2014-01-15 01:51:12 0 292
    2014-01-15 01:52:12 0 292
    2014-01-15 01:53:12 0 292
    2014-01-15 01:54:13 0 292
    2014-01-15 01:55:12 0 292
    2014-01-15 01:56:12 0 292
    2014-01-15 01:57:12 0 292
    2014-01-15 01:58:12 0 292
    2014-01-15 01:59:12 0 292

    Is the latest version of mod_qos likely address this and are there actual architecutral reasons why mod_qos shouldn't work with prefork?

     
    • Mark Blackman
      Mark Blackman
      2014-01-17

      Sorry I missed the add attachment link for that.

       
  • Have you seen anything in the ErrorLog (which correlates with this behaviour)? Or is there any other "action" (special request, special module)?

    About the "architecture": mod_qos uses a counter for each rule. The counter is incremented at the beginning of the request processing and decremented at the end. There must be something hat prevents the correct termination of the request processing if these counters leak (e.g. child exit).

     
  • Mark Blackman
    Mark Blackman
    2014-01-17

    1701 segmentation faults that day, that server. No trace of what triggered the segmentation fault though.

     
  • Mark Blackman
    Mark Blackman
    2014-01-21

    We've replicated this behaviour by firing SIGSEGV at children in a test environment and presumably this just demonstrates the counters don't have much defense against child processes existing abnormally. Could mod_qos trap the SIGSEGV to decrement counters?

     
  • Apache httpd does not offer any hook to modules to handle child exits.