SV: [Simpleweb-Support] Scaling Problems with SimpleWeb

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Hi Niall,

I am using both Java 1.5.0_03 and 1.4.2_09 - both SDK and JRE. Both
versions have the problems.

> Hi Kristian,
>=20
> Firstly, are you using Java 1.5 or 1.4?
>=20
> Also, what is your memory profile like under high
> loads? An OutOfMemoryError  can cause services to end
> without closing the connection.

No Out of memory errors. We use the standard memory limitation.=20

>=20
> The OutOfMemoryError can also cause, on occasion,
> confusion within many data structures both within the
> Simple and Java class libraries.
>=20
> Have you used Jconsole (using the
> -Dcom.sun.management.jmxremote VM argument) to inspect
> the state of the VM? What is the memory usage like,
> also are there any dead threads (there should be
> none).

I have sent a SIGQUIT to the process. This gives a dump of the Threads
being in WAIT_STATE. I have attached the Thread Dump to this mail.=20

>=20
> If you are sure it is not an OutOfMemoryError, then I
> will certainly investigate it further.
>=20
> Thanks for the feedback, I really appreciate it.
>=20
> Niall

Hope we will get it to work. You are welcome to ask me other questions.=20

Kristian

>=20
> --- Kristian Reesen Skouboe <kri...@ja...> wrote:
>=20
> > Hello,
> >
> >
> >
> > I have done some scaling tests of SimpleWeb 2.7.4.
> > using a Microsoft Web
> > Application Stress Tool.
> >
> >
> >
> > The OS is Linux 2.4.21-32.ELsmp #1 SMP Fri Apr 15
> > 21:17:59 EDT 2005 i686
> > i686 i386 GNU/Linux. 2 CPUs - 3 Ghz and 2 GB of RAM.
> > A Normal HP server.
> > The tests have also been run on two other hardware
> > architectures.
> >
> >
> >
> > The number of connections to SimpleWeb have been set
> > for 4000
> > simultaneous connections during a test running
> > period of 20 minutes.
> >
> >
> >
> > The connections are being done onto different images
> > that are being
> > served using SimpleWeb and an implementation of the
> > Process method. The
> > Process method reads the images from HashMap using
> > the URL given.
> >
> >
> >
> > Furthermore, I have put in some monitoring of the
> > number of sockets that
> > are being opened and closed when SimpleWeb is
> > running.
> >
> > Both the GranularPoller and the DefaultPoller have
> > been used using both
> > standard and lower timeout rate settings.
> >
> > Furthermore we use the LinuxConfigurator and we have
> > also been using the
> > default configuration. Here is a snapshot of the
> > LinuxConfigurator:
> >
> > xxx
> >
> >
> >
> >
> >
> > The tests have been run multiple times. The results
> > of the tests are
> > that SimpleWeb halts its execution and stops serving
> > images to the Web
> > clients that connect (the Stress Tool). This happens
> > when there are
> > about 500 simultaneous connections have been done to
> > SimpleWeb.
> > Furthermore the number of connections then raises to
> > about 2000
> > connections, where it stops increasing (max is 4000
> > connections in
> > Stress Tool). Then when the stress tool is done, the
> > number of
> > connections sometimes decrease to about 500
> > connections, other times to
> > about 10-20 connections. But it never decreases to 0
> > connections.
> >
> >
> >
> > When the test is run again, the number of
> > connections arises again, but
> > no requests are being served by SimpleWeb. It only
> > enqueues the
> > requests, it never dequeues them and gives a
> > response back.
> >
> >
> >
> > If the tests are done with lower connections, e.g.
> > 1000 connections for
> > about 3-4 minutes, the SimpleWeb functions fine and
> > serves requests as
> > designed. It never halts. This only happens when the
> > connections are
> > high (4000) connections.
> >
> >
> >
> > After doing some analysis on the java classes, I
> > think I have found four
> > java classes that may be having the problems:
> >
> > - Processor
> >
> > - Scheduler
> >
> > - SchedulerQueue
> >
> > - PriorityQueue
> >
> >
> >
> > The Processor class is opening the sockets and
> > letting the pipeline
> > method execute on the sockets. However the sockets
> > are never really
> > closed (except when GranularPoller closes a pipeline
> > - a socket closes
> > also), and a lot of the sockets convert to "Can't
> > Identify protocols" in
> > Linux, and when the number of sockets are as high as
> > the max number of
> > open file handles in Linux, SimpleWeb cannot serve
> > any more web content
> > (images in this case). The GranularPoller looks like
> > it is working when
> > the number of simultaneous connections are lower
> > than 500 connections.
> > However when the number of connections increases
> > above this to about
> > 1000-2000 connections, the GranularPoller stops
> > closing down
> > connections.
> >
> >
> >
> > The Scheduler class is used for enqueuing and
> > dequeuing objects into a
> > queue. However when SimpleWeb is running under high
> > load, the enqueuing
> > continues to work, but the dequeuing of objects
> > stops working. I have
> > put logging onto the class and dequeueing never
> > happens again when
> > SimpleWeb has been overloaded with requests.
> >
> >
> >
> > The SchedulerQueue class works together with
> > Scheduler class, the
> > dequeuing of objects never happens here either.
> >
> >
> >
> > The PriorityQueue continues to grow with new
> > requests from the Stress
> > Tool, however it is not decreased when SimpleWeb is
> > overloaded.
> >
> >
> >
> > Two Solutions that work currently:
> >
> > I have tried out two solutions that tend to solve
> > the problem, however
> > the solutions are not good, because they kick away
> > users from the
> > webserver.
> >
> >
> >
> > Solution 1: Implementing a monitoring thread that
> > monitors the number of
> > connections open. This is the same as implementing a
> > "Max connections
> > allowed" in SimpleWeb, as many other web servers
> > have. When the number
> > of connections are too high, then we stop serving.
> > This tends to work.
> >
> >
> >
> > Solution 2: Try to notify the Daemons, PriorityQueue
> > and other Threads
> > running, that they need to work. This is done using
> > a "notifyAll" call
> > to the different threads. This tends to kick in the
> > dequeuing of the
> > PriorityQueue, however this it does not work
> > completety.
> >
> >
> >
> > Please help me with deep design details of
> > SimpleWebs different queue
> > structures and how it handles many connections
> > (unlimited?) and sockets
> > and thread locking. There seems to be some problems
> > when running
> > SimpleWeb with high loads.
> >
> > Med venlig hilsen / Best Regards,
> > Kristian Reesen Skouboe
> > Denmark, Europe
> >
> =3D=3D=3D message truncated =3D=3D=3D
>=20
>=20
> Niall Gallagher
>=20
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>=20
>=20
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log
> files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD
SPLUNK!
> http://ads.osdn.com/?ad_id=3D7637&alloc_id=3D16865&op=3Dclick
> _______________________________________________
> Simpleweb-Support mailing list
> Sim...@li...
> https://lists.sourceforge.net/lists/listinfo/simpleweb-support