Re: [Rabbit-proxy-users] RabbIT - frequent freezes

[Costa T. <kt...@ho...>]

> Do anyone of the proxies in front of rabbit handle the broken chunking
> from apache?
> If not you should probably try the 2.0.36-pre3 release (will be full 
> 2.0.36 in short time
> if nothing more shows up in my testing).

Hm! I don't know. I'll try 2.0.36-pre3 (where can I get it from?)

> Could be the cache cleaner?
> What value do you have for cleanloop?

I have tried setting size to 0 and loop to 0, to 3600, 86400, etc.
Nothing helped. Currently, I have size and time set to 0 and loop set to
3600.

> How big cache do you have? (I guess 0, since you have other proxies).

Right. I don't need a cache for RabbIT.
Actually, I restart rabbit once per day. At that time I completely clean
any cache it has.

> Can you check the cache statistics and see if rabbit builds a cache?
> (even if you set it to 0 it will use a small cache for image 
> recompression).

Actually, I have disabled the front ends. Is it possible to get this info
by other means?
 
> Could also be GC. Can you try to to add " -verbose:gc " to your
> argument 
> list
> and see if it is GC-related?.

Well, check the log bellow. At the point the memory usage went above
100MB, RabbIT became completely unusable. 100% CPU utilization and just
very few requests served. I also noticed that when RabbIT freezes, the
garbage collector does not run (normally it logs one row every 1-2
seconds), but when it is frozen, the GC stops logging for several
seconds.

This log has been created with -Xmx1536m (64m was giving OutOfMemory
errors).

[GC 512K->234K(1984K), 0.0039340 secs]
[GC 745K->291K(1984K), 0.0023450 secs]
[GC 803K->328K(1984K), 0.0034220 secs]
[GC 840K->419K(1984K), 0.0036420 secs]
[GC 931K->497K(1984K), 0.0028920 secs]
[GC 1009K->547K(1984K), 0.0020810 secs]
[GC 1059K->613K(1984K), 0.0014280 secs]
[GC 1125K->655K(1984K), 0.0011000 secs]
[GC 1167K->681K(1984K), 0.0015260 secs]
[GC 1193K->736K(1984K), 0.0016470 secs]
[GC 1241K->763K(1984K), 0.0011300 secs]
[GC 1275K->836K(1984K), 0.0011320 secs]
[GC 1348K->949K(1984K), 0.0019430 secs]
[GC 1461K->1039K(1984K), 0.0013120 secs]
[GC 1550K->1125K(1984K), 0.0018400 secs]
[GC 1637K->1156K(1984K), 0.0010890 secs]
[GC 1668K->1208K(1984K), 0.0013630 secs]
[GC 1720K->1289K(1984K), 0.0013090 secs]
[GC 1794K->1347K(1984K), 0.0012780 secs]
[GC 1859K->1345K(1984K), 0.0008460 secs]
[GC 1857K->1365K(1984K), 0.0009890 secs]
[GC 1548K->1344K(1984K), 0.0007000 secs]
[Full GC[Unloading class sun.reflect.GeneratedConstructorAccessor3]
[Unloading class sun.reflect.GeneratedConstructorAccessor1]
[Unloading class sun.reflect.GeneratedConstructorAccessor2]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor1]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor3]
[Unloading class sun.reflect.GeneratedMethodAccessor5]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor4]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor2]
 1344K->800K(1984K), 0.0196280 secs]
[GC 7809K->7374K(8484K), 0.0017450 secs]
[GC 7886K->7416K(8484K), 0.0011560 secs]
[GC 7928K->7479K(8484K), 0.0016540 secs]
[GC 7991K->7582K(8484K), 0.0011850 secs]
[GC 8094K->7615K(8484K), 0.0010430 secs]
[GC 8127K->7676K(8484K), 0.0013240 secs]
[GC 8188K->7730K(8484K), 0.0010260 secs]
[GC 8242K->7811K(8484K), 0.0015890 secs]
[GC 8323K->7880K(8484K), 0.0015550 secs]
[GC 8391K->7916K(8484K), 0.0012330 secs]
[GC 8428K->8002K(8612K), 0.0016510 secs]
[Full GC[Unloading class
sun.reflect.GeneratedSerializationConstructorAccessor5]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor6]
 8002K->7775K(8612K), 0.0199320 secs]
[GC 8799K->7900K(14112K), 0.0019680 secs]
[GC 8924K->7945K(14112K), 0.0016690 secs]
[GC 8969K->7982K(14112K), 0.0017250 secs]
[GC 9005K->8115K(14112K), 0.0027720 secs]
[GC 9131K->8205K(14112K), 0.0018150 secs]
[GC 9229K->8253K(14112K), 0.0017360 secs]
[GC 9277K->8386K(14112K), 0.0025100 secs]
[GC 9410K->8479K(14112K), 0.0022580 secs]
[GC 9503K->8527K(14112K), 0.0017070 secs]
[GC 9550K->8667K(14112K), 0.0024520 secs]
[GC 9691K->8717K(14112K), 0.0017000 secs]
[GC 9740K->8776K(14112K), 0.0018140 secs]
[GC 9800K->8837K(14112K), 0.0019040 secs]
[GC 9859K->8808K(14112K), 0.0016000 secs]
[GC 9832K->8874K(14112K), 0.0016230 secs]
[GC 9898K->8938K(14112K), 0.0022560 secs]
[GC 9962K->8995K(14112K), 0.0019040 secs]
[GC 10019K->9001K(14112K), 0.0021020 secs]
[GC 10024K->9004K(14112K), 0.0019970 secs]
[GC 10028K->9058K(14112K), 0.0016320 secs]
[GC 10073K->9094K(14112K), 0.0015690 secs]
[GC 10118K->9233K(14112K), 0.0020420 secs]
[GC 10257K->9297K(14112K), 0.0025300 secs]
[GC 10321K->9367K(14112K), 0.0019540 secs]
[GC 10391K->9431K(14112K), 0.0025270 secs]
[GC 10455K->9555K(14112K), 0.0027590 secs]
[GC 10579K->9611K(14112K), 0.0018400 secs]
[GC 10635K->9684K(14112K), 0.0023350 secs]
[GC 10707K->9723K(14112K), 0.0021800 secs]
[GC 10747K->9770K(14112K), 0.0018520 secs]
[GC 10794K->9801K(14112K), 0.0017750 secs]
[GC 10825K->9835K(14112K), 0.0017920 secs]
[GC 10859K->9823K(14112K), 0.0016860 secs]
[GC 10846K->9827K(14112K), 0.0013060 secs]
[GC 10851K->9829K(14112K), 0.0011300 secs]
[GC 10853K->9837K(14112K), 0.0011300 secs]
[GC 10861K->9837K(14112K), 0.0012570 secs]
[GC 10861K->9859K(14112K), 0.0015970 secs]
[GC 10883K->9870K(14112K), 0.0017110 secs]
[GC 10893K->9877K(14112K), 0.0019500 secs]
[GC 10900K->9945K(14112K), 0.0021570 secs]
[GC 10967K->9967K(14112K), 0.0019390 secs]
[GC 10991K->10002K(14112K), 0.0022250 secs]
[GC 11023K->10027K(14112K), 0.0018740 secs]
[GC 11051K->10135K(14112K), 0.0017790 secs]
[GC 11159K->10182K(14112K), 0.0016170 secs]
[GC 11206K->10259K(14112K), 0.0018220 secs]
[GC 11283K->10254K(14112K), 0.0018570 secs]
[GC 11273K->10309K(14112K), 0.0017820 secs]
[GC 11332K->10395K(14112K), 0.0017540 secs]
[GC 11419K->10472K(14112K), 0.0021140 secs]
[GC 11495K->10524K(14112K), 0.0020100 secs]
[GC 11548K->10611K(14112K), 0.0017850 secs]
[GC 11635K->10620K(14112K), 0.0018190 secs]
[GC 11640K->10658K(14112K), 0.0019240 secs]
[GC 11681K->10752K(14112K), 0.0017810 secs]
[GC 11776K->10828K(14112K), 0.0020860 secs]
[GC 11852K->10892K(14112K), 0.0018880 secs]
[GC 11906K->10973K(14112K), 0.0019260 secs]
[GC 11997K->10993K(14112K), 0.0019050 secs]
[GC 12017K->11079K(14112K), 0.0020250 secs]
[GC 12096K->11168K(14112K), 0.0031120 secs]
[GC 12191K->11247K(14112K), 0.0021610 secs]
[GC 12271K->11273K(14112K), 0.0027600 secs]
[GC 12297K->11351K(14112K), 0.0033470 secs]
[GC 12375K->11371K(14112K), 0.0019710 secs]
[GC 12391K->11456K(14112K), 0.0023760 secs]
[GC 12479K->11482K(14112K), 0.0020480 secs]
[GC 12505K->11592K(14112K), 0.0020170 secs]
[GC 12616K->11665K(14112K), 0.0021420 secs]
[GC 12688K->11693K(14112K), 0.0029010 secs]
[GC 12716K->11784K(14112K), 0.0019910 secs]
[GC 12808K->12017K(14112K), 0.0022830 secs]
[GC 13041K->12072K(14112K), 0.0023400 secs]
[GC 13096K->12168K(14112K), 0.0021880 secs]
[GC 13186K->12204K(14112K), 0.0031370 secs]
[GC 13228K->12206K(14112K), 0.0018540 secs]
[GC 13229K->12249K(14112K), 0.0024660 secs]
[GC 13265K->12252K(14112K), 0.0016400 secs]
[GC 13275K->12290K(14112K), 0.0018060 secs]
[GC 13314K->12332K(14112K), 0.0019640 secs]
[GC 13356K->12405K(14112K), 0.0027590 secs]
[GC 13428K->12450K(14112K), 0.0019810 secs]
[GC 13474K->12533K(14112K), 0.0022940 secs]
[GC 13553K->12589K(14112K), 0.0017760 secs]
[GC 13612K->12646K(14112K), 0.0026020 secs]
[GC 13665K->12677K(14112K), 0.0019050 secs]
[GC 13701K->12743K(14112K), 0.0023470 secs]
[GC 13767K->12795K(14112K), 0.0023770 secs]
[GC 13819K->12882K(14112K), 0.0024370 secs]
[GC 13905K->12942K(14112K), 0.0020070 secs]
[GC 13965K->12979K(14112K), 0.0021600 secs]
[GC 14003K->12968K(14112K), 0.0018170 secs]
[GC 13990K->13027K(14112K), 0.0018560 secs]
[GC 14051K->13046K(14112K), 0.0022420 secs]
[GC 14070K->13134K(14240K), 0.0021590 secs]
[Full GC[Unloading class sun.reflect.GeneratedConstructorAccessor5]
[Unloading class sun.reflect.GeneratedMethodAccessor7]
[Unloading class sun.reflect.GeneratedConstructorAccessor8]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor8]
[Unloading class sun.reflect.GeneratedConstructorAccessor6]
[Unloading class sun.reflect.GeneratedConstructorAccessor7]
[Unloading class sun.reflect.GeneratedConstructorAccessor4]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor9]
[Unloading class sun.reflect.GeneratedMethodAccessor6]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor7]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor10]
 13134K->9755K(14240K), 0.0348110 secs]
[GC 11226K->9956K(17860K), 0.0035540 secs]
[GC 11428K->10074K(17860K), 0.0026080 secs]
[GC 11546K->10180K(17860K), 0.0022610 secs]
[GC 11645K->10312K(17860K), 0.0023860 secs]
[GC 11784K->10505K(17860K), 0.0027650 secs]
[GC 11976K->10691K(17860K), 0.0026710 secs]
[GC 12163K->10777K(17860K), 0.0021580 secs]
[GC 12249K->10919K(17860K), 0.0021360 secs]
[GC 12391K->11043K(17860K), 0.0024700 secs]
[GC 12515K->11215K(17860K), 0.0026310 secs]
[GC 12687K->11399K(17860K), 0.0023470 secs]
[GC 12870K->11543K(17860K), 0.0027870 secs]
[GC 13005K->11620K(17860K), 0.0021770 secs]
[GC 13084K->11726K(17860K), 0.0025340 secs]
[GC 13198K->11869K(17860K), 0.0024920 secs]
[GC 13341K->12018K(17860K), 0.0024830 secs]
[GC 13490K->12095K(17860K), 0.0024170 secs]
[GC 13557K->12244K(17860K), 0.0025470 secs]
[GC 13716K->12366K(17860K), 0.0026920 secs]
[GC 13838K->12588K(17860K), 0.0028810 secs]
[GC 14060K->12648K(17860K), 0.0025780 secs]
[GC 14120K->12704K(17860K), 0.0023200 secs]
[GC 14167K->12892K(17860K), 0.0030100 secs]
[GC 14364K->12994K(17860K), 0.0024820 secs]
[GC 14466K->13059K(17860K), 0.0041980 secs]
[GC 14531K->13175K(17860K), 0.0027810 secs]
[GC 14642K->13243K(17860K), 0.0022220 secs]
[GC 14715K->13257K(17860K), 0.0023560 secs]
[GC 14722K->13372K(17860K), 0.0041560 secs]
[GC 14843K->13478K(17860K), 0.0045770 secs]
[GC 14950K->13575K(17860K), 0.0027440 secs]
[GC 15047K->13692K(17860K), 0.0029840 secs]
[GC 15164K->13822K(17860K), 0.0025630 secs]
[GC 15294K->14028K(17860K), 0.0027180 secs]
[GC 15498K->14165K(17860K), 0.0033270 secs]
[GC 15637K->14269K(17860K), 0.0023710 secs]
[GC 15741K->14373K(17860K), 0.0040380 secs]
[GC 15845K->14331K(17860K), 0.0019270 secs]
[GC 15803K->14395K(17860K), 0.0024440 secs]
[GC 15867K->14518K(17860K), 0.0027550 secs]
[GC 15986K->14683K(17860K), 0.0029100 secs]
[GC 16154K->14855K(17860K), 0.0029000 secs]
[GC 16318K->14887K(17860K), 0.0027730 secs]
[GC 16357K->14971K(17860K), 0.0024210 secs]
[GC 16443K->15102K(17860K), 0.0028780 secs]
[GC 16574K->15200K(17860K), 0.0024850 secs]
[GC 16670K->15311K(17860K), 0.0026940 secs]
[GC 16783K->15414K(17860K), 0.0028400 secs]
[GC 16886K->15530K(17860K), 0.0045520 secs]
[GC 17000K->15665K(17860K), 0.0032010 secs]
[GC 17137K->15789K(17860K), 0.0032460 secs]
[GC 17261K->15851K(17860K), 0.0028360 secs]
[GC 17323K->15931K(17860K), 0.0026640 secs]
[GC 17403K->16051K(17860K), 0.0032500 secs]
[GC 17523K->16154K(17860K), 0.0048960 secs]
[GC 17626K->16255K(17860K), 0.0028620 secs]
[GC 17728K->16319K(17860K), 0.0026340 secs]
[GC 17791K->16440K(17988K), 0.0025530 secs]
[Full GC 16440K->10447K(17988K), 0.0597210 secs]
[GC 12047K->10683K(19204K), 0.0040940 secs]
[GC 12280K->10734K(19204K), 0.0030960 secs]
[GC 12330K->10791K(19204K), 0.0029980 secs]
[GC 12391K->10888K(19204K), 0.0028260 secs]
[GC 12488K->11025K(19204K), 0.0032710 secs]
[GC 12625K->11064K(19204K), 0.0029460 secs]
[GC 12662K->11092K(19204K), 0.0026510 secs]
[GC 12692K->11186K(19204K), 0.0034690 secs]
[GC 12786K->11261K(19204K), 0.0033740 secs]
[GC 12861K->11324K(19204K), 0.0036640 secs]
[GC 12924K->11329K(19204K), 0.0027210 secs]
[GC 12928K->11386K(19204K), 0.0031880 secs]
[GC 12985K->11464K(19204K), 0.0030650 secs]
[GC 13062K->11512K(19204K), 0.0029720 secs]
[GC 13112K->11545K(19204K), 0.0029240 secs]
[GC 13145K->11597K(19204K), 0.0033030 secs]
[GC 13197K->11661K(19204K), 0.0029980 secs]
[GC 13261K->11669K(19204K), 0.0030330 secs]
[GC 13268K->11757K(19204K), 0.0033080 secs]
[GC 13355K->11782K(19204K), 0.0028600 secs]
[GC 13382K->11863K(19204K), 0.0032340 secs]
[GC 13463K->11952K(19204K), 0.0033340 secs]
[GC 13551K->12016K(19204K), 0.0031490 secs]
[GC 13609K->12088K(19204K), 0.0032420 secs]
[GC 13685K->12177K(19204K), 0.0032320 secs]
[GC 13777K->12241K(19204K), 0.0031310 secs]
[GC 13841K->12296K(19204K), 0.0031240 secs]
[GC 13895K->12367K(19204K), 0.0033750 secs]
[GC 13962K->12463K(19204K), 0.0034730 secs]
[GC 14063K->12561K(19204K), 0.0034150 secs]
[GC 14161K->12652K(19204K), 0.0033480 secs]
[GC 14252K->12675K(19204K), 0.0033670 secs]
[GC 14275K->12811K(19204K), 0.0037510 secs]
[GC 14411K->12947K(19204K), 0.0036320 secs]
[GC 14547K->12974K(19204K), 0.0032380 secs]
[GC 14574K->13025K(19204K), 0.0035040 secs]
[GC 14625K->13123K(19204K), 0.0034620 secs]
[GC 14723K->13217K(19204K), 0.0035630 secs]
[GC 14817K->13258K(19204K), 0.0033890 secs]
[GC 14859K->13294K(19204K), 0.0056130 secs]
[GC 14894K->13350K(19204K), 0.0032090 secs]
[GC 14948K->13489K(19204K), 0.0035530 secs]
[GC 15089K->13549K(19204K), 0.0035310 secs]
[GC 15148K->13599K(19204K), 0.0032650 secs]
[GC 15199K->13700K(19204K), 0.0033100 secs]
[GC 15298K->13802K(19204K), 0.0036500 secs]
[GC 15402K->13950K(19204K), 0.0032740 secs]
[GC 15560K->14015K(19204K), 0.0034990 secs]
[GC 15615K->13979K(19204K), 0.0031580 secs]
[GC 15576K->14052K(19204K), 0.0035900 secs]
[GC 15652K->14034K(19204K), 0.0034410 secs]
[GC 15634K->14051K(19204K), 0.0030440 secs]
[GC 15644K->14129K(19204K), 0.0034930 secs]
[GC 15729K->14169K(19204K), 0.0036320 secs]
[GC 15769K->14220K(19204K), 0.0029380 secs]
[GC 15817K->14305K(19204K), 0.0036340 secs]
[GC 15900K->14433K(19204K), 0.0038610 secs]
[GC 16032K->14524K(19204K), 0.0036200 secs]
[GC 16124K->14601K(19204K), 0.0035810 secs]
[GC 16199K->14652K(19204K), 0.0037040 secs]
[GC 16252K->14754K(19204K), 0.0036340 secs]
[GC 16354K->14884K(19204K), 0.0038260 secs]
[GC 16484K->14908K(19204K), 0.0037320 secs]
[GC 16499K->14945K(19204K), 0.0036060 secs]
[GC 16537K->15016K(19204K), 0.0036160 secs]
[GC 16616K->15106K(19204K), 0.0042390 secs]
[GC 16706K->15237K(19204K), 0.0041370 secs]
[GC 16836K->15325K(19204K), 0.0034910 secs]
[GC 16927K->15403K(19204K), 0.0036210 secs]
[GC 17003K->15428K(19204K), 0.0081670 secs]
[GC 17028K->15467K(19204K), 0.0036350 secs]
[GC 17067K->15507K(19204K), 0.0036630 secs]
[GC 17107K->15513K(19204K), 0.0035120 secs]
[GC 17113K->15485K(19204K), 0.0036300 secs]
[GC 17085K->15491K(19204K), 0.0034370 secs]
[GC 17083K->15526K(19204K), 0.0034540 secs]
[GC 17125K->15557K(19204K), 0.0050430 secs]
[GC 17153K->15589K(19204K), 0.0035980 secs]
[GC 17187K->15601K(19204K), 0.0034010 secs]
[GC 17201K->15622K(19204K), 0.0035320 secs]
[GC 17222K->15700K(19204K), 0.0037730 secs]
[GC 17291K->15737K(19204K), 0.0040130 secs]
[GC 17337K->15751K(19204K), 0.0066420 secs]
[GC 17348K->15749K(19204K), 0.0034200 secs]
[GC 17339K->15743K(19204K), 0.0034060 secs]
[GC 17343K->15806K(19204K), 0.0035900 secs]
[GC 17400K->15810K(19204K), 0.0034230 secs]
[GC 17410K->15788K(19204K), 0.0033930 secs]
[GC 17388K->15828K(19204K), 0.0036850 secs]
[GC 17427K->15858K(19204K), 0.0037180 secs]
[GC 17452K->15883K(19204K), 0.0039180 secs]
[GC 17483K->15931K(19204K), 0.0035940 secs]
[GC 17531K->15981K(19204K), 0.0035560 secs]
[GC 17581K->16027K(19204K), 0.0036940 secs]
[GC 17618K->16151K(19204K), 0.0044930 secs]
[GC 17751K->16226K(19204K), 0.0043750 secs]
[GC 17825K->16301K(19204K), 0.0038530 secs]
[GC 17896K->16340K(19204K), 0.0034640 secs]
[GC 17938K->16363K(19204K), 0.0063920 secs]
[GC 17960K->16406K(19204K), 0.0037640 secs]
[GC 18005K->16446K(19204K), 0.0035670 secs]
[GC 18040K->16540K(19204K), 0.0035410 secs]
[GC 18140K->16529K(19204K), 0.0035480 secs]
[GC 18124K->16575K(19204K), 0.0059970 secs]
[GC 18175K->16579K(19204K), 0.0067210 secs]
[GC 18179K->16669K(19204K), 0.0048300 secs]
[GC 18269K->16737K(19204K), 0.0076220 secs]
[GC 18337K->16793K(19204K), 0.0039120 secs]
[GC 18393K->16861K(19204K), 0.0072320 secs]
[GC 18453K->16916K(19204K), 0.0041300 secs]
[GC 18516K->16981K(19204K), 0.0038480 secs]
[GC 18581K->17044K(19204K), 0.0039720 secs]
[GC 18644K->17147K(19204K), 0.0080300 secs]
[GC 18747K->17230K(19204K), 0.0104950 secs]
[GC 18827K->17365K(19204K), 0.0093330 secs]
[GC 18965K->17576K(19204K), 0.0099360 secs]
[GC 19176K->17724K(19332K), 0.2783080 secs]
[Full GC 17724K->12334K(19332K), 0.0969750 secs]
[GC 14382K->12776K(22864K), 0.0114740 secs]
[GC 14824K->12939K(22864K), 0.0150260 secs]
[GC 14987K->13125K(22864K), 0.0104470 secs]
[GC 15173K->13111K(22864K), 0.0043060 secs]
[GC 15159K->13156K(22864K), 0.0040750 secs]
[GC 15199K->13253K(22864K), 0.0039230 secs]
[GC 15301K->13294K(22864K), 0.0061790 secs]
[GC 15339K->13280K(22864K), 0.0039790 secs]
[GC 15328K->13362K(22864K), 0.0039190 secs]
[GC 15410K->13434K(22864K), 0.0053220 secs]
[GC 15480K->13459K(22864K), 0.0060960 secs]
[GC 15507K->13524K(22864K), 0.0077810 secs]
[GC 15569K->13467K(22864K), 0.0041180 secs]
[GC 15515K->13543K(22864K), 0.0040020 secs]
[GC 15582K->13621K(22864K), 0.0041150 secs]
[GC 15664K->13674K(22864K), 0.0043610 secs]
[GC 15722K->13753K(22864K), 0.0044410 secs]
[GC 15801K->13766K(22864K), 0.0093460 secs]
[GC 15814K->13841K(22864K), 0.0042710 secs]
[GC 15889K->13858K(22864K), 0.0040020 secs]
[GC 15905K->13916K(22864K), 0.0040660 secs]
[GC 15964K->13928K(22864K), 0.0043220 secs]
[GC 15976K->13946K(22864K), 0.0041480 secs]
[GC 15994K->14103K(22864K), 0.0045940 secs]
[GC 16151K->14242K(22864K), 0.0048350 secs]
[GC 16287K->14324K(22864K), 0.0044560 secs]
[GC 16372K->14487K(22864K), 0.0044490 secs]
[GC 16535K->14742K(22864K), 0.0087380 secs]
[GC 16790K->14785K(22864K), 0.0047370 secs]
[GC 16826K->14842K(22864K), 0.0046690 secs]
[GC 16889K->14977K(22864K), 0.0044480 secs]
[GC 17025K->15033K(22864K), 0.0041880 secs]
[GC 17079K->15095K(22864K), 0.0042140 secs]
[GC 17143K->15235K(22864K), 0.0041030 secs]
[GC 17283K->15265K(22864K), 0.0047150 secs]
[GC 17313K->15401K(22864K), 0.0053060 secs]
[GC 17449K->15518K(22864K), 0.0047610 secs]
[GC 17566K->15644K(22864K), 0.0053130 secs]
[GC 17692K->15696K(22864K), 0.0045630 secs]
[GC 17744K->15813K(22864K), 0.0047480 secs]
[GC 17859K->15796K(22864K), 0.0042540 secs]
[GC 17842K->15927K(22864K), 0.0047670 secs]
[GC 17970K->16013K(22864K), 0.0049730 secs]
[GC 18061K->16099K(22864K), 0.0050140 secs]
[GC 18147K->16324K(22864K), 0.0057410 secs]
[GC 18372K->16520K(22864K), 0.0051360 secs]
[GC 18568K->16577K(22864K), 0.0045540 secs]
[GC 18624K->16754K(22864K), 0.0049260 secs]
[GC 18802K->16881K(22864K), 0.0049020 secs]
[GC 18929K->17030K(22864K), 0.0048870 secs]
[GC 19078K->17163K(22864K), 0.0050820 secs]
[GC 19203K->17232K(22864K), 0.0046480 secs]
[GC 19279K->17280K(22864K), 0.0043290 secs]
[GC 19328K->17358K(22864K), 0.0099010 secs]
[GC 19406K->17400K(22864K), 0.0047810 secs]
[GC 19448K->17489K(22864K), 0.0045130 secs]
[GC 19537K->17619K(22864K), 0.0050730 secs]
[GC 19667K->17618K(22864K), 0.0042940 secs]
[GC 19666K->17707K(22864K), 0.0043890 secs]
[GC 19755K->17750K(22864K), 0.0049030 secs]
[GC 19798K->17848K(22864K), 0.0045990 secs]
[GC 19896K->17901K(22864K), 0.0044070 secs]
[GC 19948K->17939K(22864K), 0.0053280 secs]
[GC 19987K->18030K(22864K), 0.0049060 secs]
[GC 20074K->18237K(22864K), 0.0051640 secs]
[GC 20283K->18330K(22864K), 0.0046130 secs]
[GC 20378K->18472K(22864K), 0.0052890 secs]
[GC 20520K->18568K(22864K), 0.0045550 secs]
[GC 20616K->18570K(22864K), 0.0045910 secs]
[GC 20617K->18614K(22864K), 0.0048650 secs]
[GC 20662K->18705K(22864K), 0.0044690 secs]
[GC 20753K->18834K(22864K), 0.0047370 secs]
[GC 20882K->18962K(22864K), 0.0054050 secs]
[GC 21010K->19128K(22864K), 0.0046420 secs]
[GC 21174K->19285K(22864K), 0.0052510 secs]
[GC 21333K->19334K(22864K), 0.0050150 secs]
[GC 21382K->19463K(22864K), 0.0051450 secs]
[GC 21509K->19478K(22864K), 0.0050760 secs]
[GC 21526K->19493K(22864K), 0.0054140 secs]
[GC 21541K->19600K(22864K), 0.0047440 secs]
[GC 21648K->19700K(22864K), 0.0048700 secs]
[GC 21748K->19718K(22864K), 0.0046540 secs]
[GC 21766K->19713K(22864K), 0.0046040 secs]
[GC 21761K->19829K(22864K), 0.0058690 secs]
[GC 21877K->20046K(22864K), 0.0105550 secs]
[GC 22093K->20233K(22864K), 0.0064990 secs]
[GC 22281K->20323K(22864K), 0.0052270 secs]
[GC 21401K->20334K(22864K), 0.0043700 secs]
[Full GC 20334K->13844K(22864K), 0.0717630 secs]
[GC 22898K->20665K(25828K), 0.0069570 secs]
[GC 23156K->20830K(25828K), 0.0055670 secs]
[GC 23326K->20926K(25828K), 0.0056010 secs]
[GC 23422K->21134K(25828K), 0.0057690 secs]
[GC 23351K->21224K(25828K), 0.0057390 secs]
[Full GC 21224K->18933K(25828K), 0.0738370 secs]
[GC 28608K->25985K(34948K), 0.0062890 secs]
[GC 29056K->26082K(34948K), 0.0075950 secs]
[GC 29144K->26161K(34948K), 0.0071200 secs]
[GC 29230K->26217K(34948K), 0.0062510 secs]
[GC 29288K->26318K(34948K), 0.0060720 secs]
[GC 29390K->26398K(34948K), 0.0057490 secs]
[GC 29470K->26442K(34948K), 0.0100110 secs]
[GC 29513K->26521K(34948K), 0.0055260 secs]
[GC 28762K->26592K(34948K), 0.0055340 secs]
[Full GC 26592K->24584K(34948K), 0.0943330 secs]
[GC 46276K->43433K(63564K), 0.0061710 secs]
[Full GC 43433K->43429K(63564K), 0.0561070 secs]
[GC 67440K->62218K(78848K), 0.0073480 secs]
[GC 67978K->62349K(78848K), 0.0083420 secs]
[GC 68109K->62489K(78848K), 0.0088360 secs]
[GC 68248K->62647K(78848K), 0.0093530 secs]
[GC 63122K->62121K(78848K), 0.0044700 secs]
[Full GC 62121K->62121K(78848K), 0.0572980 secs]
[GC 82917K->80927K(112304K), 0.0057840 secs]
[Full GC 80927K->73235K(112304K), 0.1588800 secs]
[GC 100581K->92469K(132236K), 0.0101350 secs]
[GC 101557K->92542K(132236K), 0.0109990 secs]
[GC 101628K->92761K(132236K), 0.0099430 secs]
[GC 97487K->92554K(132236K), 0.0133420 secs]
[Full GC 92554K->91768K(132236K), 0.2328910 secs]
[GC 114541K->110630K(165492K), 0.0073170 secs]
[Full GC 110630K->110626K(165492K), 0.0608510 secs]
[GC 142133K->130202K(199228K), 0.0146400 secs]
[GC 143442K->130343K(199228K), 0.0204090 secs]
[GC 133386K->129651K(199228K), 0.0091440 secs]
[Full GC 129651K->129651K(199228K), 0.0690250 secs]
[GC 150117K->148360K(233368K), 0.0069660 secs]
[Full GC 148360K->148360K(233368K), 0.0672830 secs]
[GC 184027K->168339K(266852K), 0.0170140 secs]
[GC 175384K->167759K(266852K), 0.0110320 secs]
[Full GC 167759K->159480K(266852K), 0.3219960 secs]
[GC 178232K->178030K(286860K), 0.0355170 secs]
[GC 179157K->178114K(286860K), 0.0049690 secs]
[GC 196866K->196333K(286860K), 0.0362860 secs]
[GC 215079K->198156K(286860K), 0.0168510 secs]
[GC 199231K->196734K(286860K), 0.0070810 secs]
[GC 214998K->214711K(286860K), 0.0370000 secs]
[GC 233463K->233343K(286860K), 0.0369330 secs]
[GC 252095K->234899K(286860K), 0.0179200 secs]
[GC 247850K->234560K(286860K), 0.0256440 secs]
[GC 253312K->251750K(286860K), 0.0372250 secs]
[GC 270500K->270345K(305484K), 0.0510770 secs]
[Full GC 270345K->270345K(305484K), 0.0942260 secs]
[GC 298303K->272901K(485520K), 0.0247080 secs]
[GC 292040K->289100K(485520K), 0.0395150 secs]
[GC 320204K->308801K(485520K), 0.0541180 secs]
[GC 334175K->327025K(485520K), 0.0530490 secs]
[GC 358129K->347239K(485520K), 0.0716530 secs]
[GC 361156K->347768K(485520K), 0.0292580 secs]
[GC 398598K->386819K(485520K), 0.0705530 secs]
[GC 417923K->406672K(485520K), 0.0512470 secs]