[Rabbit-proxy-users] RabbIT - frequent freezes

[Costa T. <kt...@ho...>]

Hi,

I am using RabbIT 2.0.35 with blackdown-jdk 1.4.2.01 on Gentoo Linux.
RabbIT is installed on 2x 3GHz Xeon with 4GB RAM, and it has other proxies
in front (clients' requests) and behind it (server replies).

RabbIT is run with this command:

/opt/blackdown-jdk-1.4.2.01/bin/java -Xms2m -Xmx64m -Xss256k -classpath
/opt/RabbIT2/current rabbit.proxy.Proxy

This setup is serving about 2.5 mil requests per day.

I have the following issue:

There must be some recurring job inside java that freezes RabbIT every a
few minutes, especially during peak hours.

Bellow, is the output of vmstat that shows the problem. Follow the 'us'
(CPU/user) column at the rows where it shows 100. At that point the 'r'
(PROCS/ready) column shows that there are dozens of processes ready but
there is no CPU available to serve them.

The machine is running only RabbIT.

I have tried using -Xincgc (incremental garbage collector) but then RabbIT
exits with something like:

Exception in thread "RabbIT2: 193" java.lang.OutOfMemoryError: requested
24 bytes for CHeapObj-new. Out of swap space?

I have tried increasing -Xms (512MB) and -Xmx (1,7GB) but the problem gets
worse.

The problem appears only during peak times.

Any help is appreciated.

procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy
id wa
 2  0      0 1852248 429796 765508    0    0     0   984 1980   656  8  5
79  8
 0  0      0 1854488 429796 765644    0    0     0     0 2028   685  8  6
86  0
10  0      0 1838872 429796 765780    0    0     0     0 2527  1466 18 14
68  0
23  0      0 1811480 429796 765916    0    0     0     0 2889  3319 60 40 
0  0
 1  0      0 1851816 429796 765848    0    0     0     0 2711  2911 24 12
64  0
 0  0      0 1853176 429796 765848    0    0     0   960 2174   799  9  5
76 10
 0  0      0 1852712 429796 765916    0    0     0     0 2199   969 17 11
72  0
 0  0      0 1853088 429796 765984    0    0     0     0 2178   807  8  6
87  0
21  0      0 1844416 429796 766188    0    0     0     0 2510  1394 32 17
52  0
 7  0      0 1835736 429796 766188    0    0     0     0 1905  2449 89 11 
0  0
 2  0      0 1838736 429796 766324    0    0     0   868 2570  3340 46 29
20  4
 0  0      0 1851824 429796 766324    0    0     0     0 2377   979 14 10
76  0
 1  0      0 1851952 429796 766324    0    0     0     0 2020   747  5  4
91  0
 1  0      0 1849176 429796 766324    0    0     0     0 2134   860 14 10
76  0
25  0      0 1842528 429804 766452    0    0     0     0 2464   848 55  6
40  0
 1  0      0 1849184 429804 766860    0    0     0  1316 2766 21895 57 34 
8  0
28  0      0 1844112 429804 766928    0    0     0     0 2233   980 88  7 
6  0
80  0      0 1844224 429804 766928    0    0     0     0 1766   565 100  0
 0  0
61  0      0 1844224 429804 766928    0    0     0     0 1510   315 100  0
 0  0
61  0      0 1844160 429804 766928    0    0     0     0 1556   235 100  0
 0  0
12  0      0 1847888 429804 767132    0    0     0   580 2004 15191 93  6 
0  0
12  0      0 1847888 429804 767132    0    0     0     0 1430   167 100  0
 0  0
11  0      0 1848536 429804 767132    0    0     0     0 1356   181 100  0
 0  0
15  0      0 1844312 429804 767200    0    0     0     0 1441  2540 93  5 
2  0
65  1      0 1848568 429804 767200    0    0     0   884 2539  1669 96  4 
0  0
63  0      0 1848760 429804 767268    0    0     0   292 1602   294 100  0
 0  0
97  0      0 1848824 429804 767268    0    0     0     0 1504   288 100  0
 0  0
10  0      0 1838664 429804 766928    0    0     0     0 3150 12033 62 37 
1  0
 9  0      0 1845464 429804 767064    0    0     0     0 3824 28732 66 31 
3  0
56  1      0 1836752 429804 766928    0    0     0  1020 2147  3557 91  8 
0  0
14  0      0 1837936 429804 766860    0    0     0   308 1955  4486 91  9 
0  0
11  0      0 1821648 429804 766860    0    0     0     0 2411  3643 63 37 
0  0
 1  0      0 1848680 429804 766792    0    0     0     0 1971   787 18 10
72  0
 1  0      0 1848680 429804 766860    0    0     0     0 2245  1164 29 20
51  0
 1  1      0 1850536 429804 766928    0    0     0   560 2152   876  8  6
83  3
 2  0      0 1849448 429804 766860    0    0     0   296 2249   920 10  7
80  2
 1  0      0 1850408 429804 766928    0    0     0     0 2117   901 10  8
81  0
 0  0      0 1850728 429804 766928    0    0     0     0 2230   829  9  6
85  0
 1  0      0 1848552 429804 766996    0    0     0     0 1914   551  4  3
92  0
 1  1      0 1843576 429804 766996    0    0     0   532 1840   680  7  5
86  2
 1  0      0 1850424 429808 767060    0    0     0   284 2190   875  7  6
85  2
 0  0      0 1850424 429808 767196    0    0     0     0 1881   711 12  8
81  0
 9  0      0 1837592 429808 767196    0    0     0     0 1979   944 12 10
78  0
 0  0      0 1849976 429808 767332    0    0     0     0 1966  1056 36 22
42  0
 0  1      0 1850232 429808 767332    0    0     0   544 2097   833  5  4
89  2
 0  0      0 1849976 429808 767400    0    0     0   272 2035   779  9  5
83  3
 0  0      0 1850680 429808 767400    0    0     0     0 1915   661  7  5
89  0

-- 
Costa