Menu
▾
▴
Re: [AVIS-NOTICE] Re: [Springframework-developer] Cleanup of context resources on webapp reload
|
From: Guillaume P. <gpo...@gl...> - 2004-05-31 23:22:59
|
=3E=3E Of course=2C it would be ideal if the JavaBeans Introspector used = a WeakHashMap =3E=3E with WeakReference values - none of this would be necessary then=2E= After all=2C it=27s a =3E=3E pretty obvious bug in the current JDK=3A What=27s the point in usi= ng a WeakHashMap =3E=3E with values that reference the key but are not held in WeakReferen= ces=3F If they wouldn=27t =3E=3E care about garbage collection in the first place=2C they could use= a plain HashMap=2E=2E=2E =3E=3E Currently=2C they seem to go just half the way=2E =3E=3E With the current JDK implementation=2C it would also be nice if th= e servlet container did =3E=3E an Introspector=2EflushCaches call on web app shutdown=2C or prefe= rably a flushFromCaches =3E=3E call for each class loader by the corresponding class loader=2E Ho= wever=2C it=27s unrealistic that =3E=3E all containers will do this any time soon - and this wouldn=27t be= necessary either if that d*** =3E=3E JavaBeans Introspector implementation properly used WeakReferences= =2E From what I understand=2C Sun simply prefered to have caching of BeanInfo= s=2C at the risk of causing memory leaks=2C for performance reasons=2E I= don=27t think that caching with WeakReference has much performance impro= vement=2E From the javadoc =3A -------------------------------------------------------------------------= ------------- http=3A//java=2Esun=2Ecom/j2se/1=2E4=2E1/docs/api/java/beans/Introspector= =2Ehtml Because the Introspector caches BeanInfo classes for better performance=2C= = take care if you use it in an application that uses multiple class loader= s=2E = In general=2C when you destroy a ClassLoader that has been used to intros= pect classes=2C you should use the Introspector=2EflushCaches or Introspe= ctor=2EflushFromCaches method to flush all of the introspected classes ou= t of the cache=2E = -------------------------------------------------------------------------= ------------- I tried to find the quote where they say what I just paraphared above=2C = but I can=27t find it anymore=2E But one of the bug URL I = sent on this thread earlier has just been updated this weekend=2C it seem= s like they have a fix for this problem in Tiger =3A http=3A//bugs=2Esun=2Ecom/bugdatabase/view=5Fbug=2Edo=3Fbug=5Fid=3D480900= 8 Quote =3A =22Strong references to Method=2C Method=5B=5D=2C Class and Cla= ss=5B=5D have been changed to a combination of Soft and Weak = refererences=2E=22 So the Instrospector will still hold a strong reference on the BeanInfo=2C= but the BeanInfo won=27t have a strong reference on = the Class anymore=2E =3E=3E So I guess I=27ll add IntrospectorCleanupListener and Introspector= CleanupServlet classes=2E=2E=2E thoughts=3F Sound good to me=2C but just to be sure=2C Servlets that never throw Unav= alaibleException from the service method cannot be destroyed unless the c= ontext is being destroyed=2C right=3F Guillaume ---- Messages d=B4origine ---- De=3A j=FCrgen h=F6ller =5Bwerk3AT=5D =3Cjuergen=2Ehoeller=40werk3at=2Eco= m=3E Date=3A lundi=2C mai 31=2C 2004 8=3A22 am Objet=3A =5BAVIS-NOTICE=5D Re=3A =5BSpringframework-developer=5D Cleanup = of context resources on webapp reload =3E I=27ve just found out that our JPetStore=27s remaining resource leak = =3E is not caused by iBATIS or JDOM but by Struts=3A Struts uses Commons = =3E BeanUtils which in turn uses the JavaBeans Introspector without = =3E flushing=2E=2E=2E So on web app shutdown=2C the Introspector cache ho= lds = =3E references to Struts Actions from the web app classloder=2E This = =3E prevents garbage collection of JDOM=27s singletons because the web = =3E app classloader (which holds its loaded classes and thus their = =3E static references) is still around=2E So it=27s not JDOM=27s fault=3B= it=27s = =3E rather like with Spring=27s SQLErrorCodesFactory and co=2E =3E = =3E Which gives 2 candidates that would benefit from a general = =3E Introspector=2EflushCaches call on shutdown so far=3A Struts and = =3E Quartz=2E As per my mail from yesterday=2C Spring does not need that = =3E anymore as it flushes the JavaBeans Introspector for each class = =3E that it caches introspection results for itself=2E If you still see = =3E Spring classes hanging around after web app shutdown=2C that=27s = =3E caused by leaks of *other* tools that prevent the class loader = =3E (which is referencing Spring classes) from being disposed=2E =3E = =3E So the remaining question is=3A Should we provide a way to call = =3E Introspector=2EflushCaches on web app shutdown=3F Doing this in = =3E ContextLoaderListener will just affect apps that actually use = =3E ContextLoaderListener=3A What about DispatcherServlet-only apps=3F = =3E What about Struts apps that load a Spring context via a plugin=3F = =3E What about Servlet 2=2E2 apps that can=27t register listeners=3F I gu= ess = =3E it would be better to provide separate IntrospectorCleanupListener = =3E and IntrospectorCleanupServlet classes (analogous to = =3E Log4jConfigListener and Log4jConfigServlet)=2C just doing a = =3E Introspector=2EflushCaches call on shutdown=2E =3E = =3E Of course=2C it would be ideal if the JavaBeans Introspector used a = =3E WeakHashMap with WeakReference values - none of this would be = =3E necessary then=2E After all=2C it=27s a pretty obvious bug in the = =3E current JDK=3A What=27s the point in using a WeakHashMap with values = =3E that reference the key but are not held in WeakReferences=3F If they = =3E wouldn=27t care about garbage collection in the first place=2C they = =3E could use a plain HashMap=2E=2E=2E Currently=2C they seem to go just = half = =3E the way=2E =3E = =3E With the current JDK implementation=2C it would also be nice if the = =3E servlet container did an Introspector=2EflushCaches call on web app = =3E shutdown=2C or preferably a flushFromCaches call for each class = =3E loader by the corresponding class loader=2E However=2C it=27s = =3E unrealistic that all containers will do this any time soon - and = =3E this wouldn=27t be necessary either if that d*** JavaBeans = =3E Introspector implementation properly used WeakReferences=2E =3E = =3E So I guess I=27ll add IntrospectorCleanupListener and = =3E IntrospectorCleanupServlet classes=2E=2E=2E thoughts=3F =3E = =3E Juergen =3E = =3E = =3E =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F =3E = =3E Von=3A springframework-developer-admin=40lists=2Esourceforge=2Enet im= = =3E Auftrag von j=FCrgen h=F6ller =5Bwerk3AT=5D =3E Gesendet=3A Mo 31=2E05=2E2004 13=3A08 =3E An=3A springframework-developer=40lists=2Esourceforge=2Enet =3E Betreff=3A Re=3A =5BSpringframework-developer=5D Cleanup of context = =3E resources on webapp reload =3E = =3E = =3E = =3E Guillaume=2C =3E = =3E As far as I see=2C the only benefit that we would gain with such a = =3E classloader check is that we avoid recaching introspection results = =3E during application runtime=2C for the case where the garbage = =3E collector has eagerly removed them=2E However=2C the latter can only = =3E happen when there are no other references to the given Class=3A This = =3E won=27t be the case with typical Spring usage=2C as both bean = =3E factories and web command controllers (i=2Ee=2E the main usages of = =3E BeanWrapper) hold a reference to the bean class respectively = =3E command class=2E =3E = =3E And if there are no other references to a specific Class=2C I = =3E suppose the garbage collector is free to remove the cached = =3E introspection results for it=3B this can be considered a desirable = =3E thing=2E So I guess it=27s better to leave CachedIntrospectionResults= = =3E as it is=2C as there=27s no noteworthy difference in all other = =3E respects=2E Do you see any negative impact on performance=2C given th= e = =3E scenario from above=3F Your earlier point comes to my mind here=3A = =3E It=27s also about a simple implementation that doesn=27t obscure the = =3E purpose of the code=2E =3E = =3E Juergen =3E = =3E = =3E =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F =3E = =3E Von=3A springframework-developer-admin=40lists=2Esourceforge=2Enet im= = =3E Auftrag von Guillaume Poirier =3E Gesendet=3A Mo 31=2E05=2E2004 00=3A26 =3E An=3A springframework-developer=40lists=2Esourceforge=2Enet =3E Betreff=3A Re=3A =5BSpringframework-developer=5D Cleanup of context = =3E resources on webapp reload =3E = =3E = =3E = =3E Juergen=2C =3E = =3E I think that Introspector=2EflushCaches() on a = =3E ServletContextListener it still =3E the lesser of evils for that memory leak=2C the effect on other = =3E webapps would =3E not be that bad anyway=2C just slow a little bit the next few calls t= o =3E Introspector=2EgetBeanInfo()=2E However=2C I agree with you that thi= s = =3E shouldprobably be dealt with by the application rather than = =3E Spring=2E But may be it =3E could be an option with Spring=27s ContextLoaderListener (probably = =3E off by =3E default)=3F =3E = =3E Concerning your solution with CachedIntrospectionResults=2C I would = =3E have a =3E sugestion for performance=2E For the majority of the use cases=2C =3E CachedIntrospectionResults would be in the same ClassLoader as the = =3E targetclass=2E So I would suggest to first check the ClassLoader to = =3E know if it=27s =3E safe to cache the class without WeakReference=2E I=27ve attached a = =3E patch to the =3E E-Mail if you would like to include my suggestion=2E =3E = =3E Thanks=2C =3E Guillaume =3E = =3E ----- Original Message ----- =3E From=3A =22j=FCrgen h=F6ller =5Bwerk3AT=5D=22 =3Cjuergen=2Ehoeller=40= werk3at=2Ecom=3E =3E To=3A =3Cspringframework-developer=40lists=2Esourceforge=2Enet=3E =3E Sent=3A Sunday=2C May 30=2C 2004 11=3A09 AM =3E Subject=3A Re=3A =5BSpringframework-developer=5D Cleanup of context = =3E resources on =3E webapp reload =3E = =3E = =3E Guillaume=2C =3E = =3E I=27m using a profiler=2C explicitly running garbage collection after= =3E application shutdown and then seeing what remains on the heap=2E = =3E It=27s actually =3E my first really instance profiler work for about two years=3B = =3E haven=27t had any =3E need for it in the meantime=2E =3E = =3E Anyway=2C I did some further tests today=2C changing = =3E SQLErrorCodesFactory and =3E GlobalAdvisorAdapterRegistry to classic singletons again=2C with an =3E Introspector=2EflushCaches call on shutdown=2E While this didn=27t ha= ve = =3E any effect =3E with Petclinic=2C it did result in proper cleanup with Image Database= =2E =3E = =3E So the Introspector=2EflushCaches call *does* have some effect=2C = =3E contrary to my =3E assumption from yesterday=2C but not in all scenarios=2E After some = =3E furthertests=2C I found out that Hibernate (-=3E Petclinic) has its = =3E own leaks=2C thus =3E keeps the web app class loader around=2C thus no cleanup of Spring=27= s =3E singletons in that class loader=2E If Hibernate isn=27t used=2C = =3E everything gets =3E cleaned up properly=2E =3E = =3E This means that you=27re perfectly right=3A Coding singletons with =3E WeakReferences only cleans up that particular singleton object but = =3E of course =3E doesn=27t do anything about the class loader itself with its = =3E remaining leaks=2E =3E As the code *is* uglier with WeakReferences=2C I=27ve permanently cha= nged =3E SQLErrorCodesFactory and GlobalAdvisorAdapterRegistry back to classic= =3E singletons (like they were until a week ago)=2E =3E = =3E I=27d like to leave CachedIntrospectionResults as-is with a = =3E WeakHashMap and =3E WeakReferences=2C as this holds references to classes and might = =3E reside in a =3E parent classloader of the referenced classes=2E That=27s how the JDK=27= s =3E Introspector class should be coded too=2E=2E=2E =3E = =3E The remaining issue is=3A Where to invoke Introspector=2EflushCaches=3F= = =3E EveryApplicationContext shutdown=3F How to avoid cleaning all = =3E BeanInfos *of the =3E entire VM*=2C when all we want to do is clean up all BeanInfos of = =3E the current =3E app=3F =3E Sure=2C a general flushCaches call cleans up all BeanInfo leaks of = =3E the app =3E too=2C be it from Spring or Quartz or whatever=2C but all we want to = =3E provideout-of-the-box is proper cleanup of Spring itself=2E =3E = =3E So my solution for Spring is simple=3A CachedIntrospectionResults = =3E flushes the =3E Introspector cache for the given class (and its superclasses) = =3E right after it =3E fetched the BeanInfo for it (i=2Ee=2E right after it=27s been cached)= =2E = =3E As we cache =3E the introspection results ourselves anyway=2C we wouldn=27t benefit = =3E from the =3E Introspector cache in the first place=2E That shouldn=27t have any = =3E negativeimpact on performance=2E =3E = =3E I did quite a lot of tests with this new strategy=2C and everything g= ets =3E cleaned up nicely within Spring=2E Just if you add Hibernate or = =3E Quartz to the =3E mix=2C you=27ll get leaks from them that prevent the class loader fro= m = =3E gettinggarbage collected=2E With Quartz=2C a general flushCaches call= = =3E helps=3B with =3E Hibernate=2C even that doesn=27t=2E =3E = =3E So in certain scenarios=2C it might help to add a custom =3E ServletContextListener that does an Introspector=2EflushCaches call o= n =3E shutdown - but for Spring itself=2C this isn=27t necessary=2C as the = beans =3E infrastructure does not cause any leaks anymore=2E I=27m happy with = =3E the current =3E solution=3A classic singletons as before=2C just a flushFromCaches = =3E call for each =3E introspected class right when building the CachedIntrospectionResults= =3E object=2E =3E = =3E We should probably tell the Quartz guys to apply specific = =3E flushFromCachescalls after their Introspector usage=2E I=27ve also = =3E noticed that JDOM=2C as used =3E within iBATIS SQL Maps 1=2E3=2C has a resource leak too=2E I haven=27= t = =3E researchedwhere the Hibernate leaks come from in detail=2C but the = =3E root of the problem =3E seems to be CGLIB there=2E =3E = =3E Many thanks for pointing this out! =3A-) =3E = =3E Juergen =3E = =3E = =3E =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F =3E = =3E Von=3A springframework-developer-admin=40lists=2Esourceforge=2Enet im= = =3E Auftrag von =3E Guillaume Poirier =3E Gesendet=3A So 30=2E05=2E2004 07=3A01 =3E An=3A springframework-developer=40lists=2Esourceforge=2Enet =3E Betreff=3A Re=3A =5BSpringframework-developer=5D Cleanup of context = =3E resources on =3E webapp reload =3E = =3E = =3E = =3E Juergen=2C I=27m curious about how exactly you assert what is leaking= =2C = =3E what is =3E not=2C and exactly what is it better with WeakReference=3F Are you = =3E using a =3E profiler that tells you the non-garbage collected objects in the = =3E JVM or do =3E you use other means=3F =3E = =3E Concerning SQLErrorCodesFactory=2C while the class=27 constructor doe= s = =3E cause a =3E leak (indirectly by the use of XmlBeanFactory)=2C it is not itself = =3E the leak=2C =3E it=27s the class instance of SQLErrorCodes that prevent the = =3E ClassLoader=27scollection=2E I did a test calling =3E SQLErrorCodesFactory=2EgetInstance()=2EgetErrorCodes(=22DB2=22) in a = child =3E ClassLoader=2C and it caused a leak when the ClassLoader is thrown = =3E away=2E Then =3E I tried to create my own SQLErrorCodesFactory implementation=2C and = =3E it kept =3E leaking until I stopped using the XmlBeanFactory to create the = =3E instances of =3E SQLErrorCodes=2E Then I tried to just use an Introspector directly = =3E to set the =3E properties on the SQLErrorCodes instances by reflection=2C and it = =3E leaked just =3E as the SQLErrorCodesFactory=2E You have to keep in mind that if e=2Eg= =2E =3E Introspector has an hard reference on SQLErrorCodes=2Eclass=2C then = =3E none of the =3E class loaded by its ClassLoader can be collected=2E So it would be = =3E the cause =3E of SQLErrorCodesFactory singleton to be kept alive=2C not because =3E SQLFactoryErrorCodes has an hard reference on it=2C but because the =3E ClassLoader does=2C and SQLErrorCodes=2Eclass has an hard reference o= n the =3E ClassLoader=2C and the Introspector has an hard reference on =3E SQLErrorCodes=2Eclass=2E So the singleton of SQLFactoryCodesFactory = =3E is really =3E kept alive because Introspector has an hard reference on the class = =3E instanceof SQLErrorCodes=2E =3E = =3E I might misunderstand the situation=2C but as I see it=2C you=27re = =3E working on the =3E symptom rather than the cause=2E If you allow the singleton to be = =3E garbagecollected when the class itself is retained=2C then you might = =3E save some =3E memory=2C but it=27s just like adding memory to the JVM to solve a = =3E leak=2C it will =3E only delay the OutOfMemoryError=2C it won=27t prevent it=2E However=2C= if = =3E you allow =3E the ClassLoader to be collected by removing any hard reference to its= =3E classes=2C that would prevent the leak to even take place at all=2E =3E = =3E That=27s why I=27m curious as to why you say it=27s better with = =3E WeakReference on =3E singleton and Introspector=2EflushCaches() does nothing=2C how do you= = =3E make that =3E assertion=3F I realize that if there=27s something else than = =3E Introspector that =3E has an hard reference on a class of the ClassLoader being thown away=2C= =3E flushing the Introspector=27s cache will have no effect on the leak=2C= = =3E it would =3E still leak as fast=2E But while the WeakReference on the singleton = =3E will delay =3E the OutOfMemoryError=2C does it really help that much=2C since anyway= = =3E all the =3E Class definitions and static members cannot be collected=3F You = =3E have to =3E consider that coding defensively on this might reduce performance = =3E because of =3E more object creation and use of synchronization=2C while also = =3E complicating the =3E code=2E Is it really worth it=2C did your tests really showed a = =3E significanteffect on a typical application=3F =3E = =3E Guillaume =3E = =3E = =3E ----- Original Message ----- =3E From=3A =22j=FCrgen h=F6ller =5Bwerk3AT=5D=22 =3Cjuergen=2Ehoeller=40= werk3at=2Ecom=3E =3E To=3A =3Cspringframework-developer=40lists=2Esourceforge=2Enet=3E =3E Sent=3A Saturday=2C May 29=2C 2004 2=3A28 PM =3E Subject=3A Re=3A =5BSpringframework-developer=5D Cleanup of context = =3E resources on =3E webapp reload =3E = =3E = =3E Guillaume=2C =3E = =3E I=27ve prototypically added an Introspector=2EflushCaches call to con= text =3E shutdown=3A I don=27t see any difference in the profiler=2E That=27s = not too =3E surprising=2C as the originally leaking classes are not managed by be= ans =3E facilities in the first place=3A for example=2C SQLErrorCodesFactory=2C= = =3E which is =3E just used internally by SQLErrorCodeSQLExceptionTranslator=2E =3E = =3E Of course=2C since my changes from a week ago=2C those classes don=27= t leak =3E anymore=2C as they hold their singleton instance in a WeakReference = =3E now=2E=2E=2E I =3E still don=27t understand why this is necessary=2C but I=27m 100=25 su= re = =3E that it does =3E make a difference on both Sun JDK 1=2E4=2E2 and Sun JDK 1=2E3=2E1=2E = I=27ve = =3E also tried =3E various GC configuration options - always the same effect=2E =3E = =3E Juergen =3E = =3E = =3E =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F =3E = =3E Von=3A springframework-developer-admin=40lists=2Esourceforge=2Enet im= = =3E Auftrag von =3E Guillaume Poirier =3E Gesendet=3A Sa 29=2E05=2E2004 06=3A12 =3E An=3A springframework-developer=40lists=2Esourceforge=2Enet =3E Betreff=3A Re=3A =5BSpringframework-developer=5D Cleanup of context = =3E resources on =3E webapp reload =3E = =3E = =3E = =3E I experimented some more about this cleanup issue=2C and I was able = =3E to narrow =3E down the problem to the caching done by the = =3E java=2Ebeans=2EIntrospector=2E It =3E stores the BeanInfo instances in a WeakHashMap=2C but in that Map =3E implementation=2C only the keys uses WeakReference=2C the values are = =3E stored with =3E hard references since BeanInfo has an hard reference on the class = =3E it gives =3E info about (indirectly through BeanDescriptor and others)=2C any time= =3E Introspector=2EgetBeanInfo(Class) is used=2C that class and it=27s = =3E static members =3E will not ever be able to be garbage collected=2E I kind of remember = =3E someonementioning something related to this in the mailling list=2C = =3E but I cannot find =3E the mail=2E I wonder if there=27s other case where the java=5Bx=5D = =3E classes might =3E have an hard reference on a class or its instances=2E A fix for this= =3E particular problem is to have a ServletContextListener call =3E Introspector=2EflushCaches() when the context is destroyed=2E =3E = =3E It seems like a known issue at Sun =3A =3E = =3E http=3A//bugs=2Esun=2Ecom/bugdatabase/view=5Fbug=2Edo=3Fbug=5Fid=3D42= 91376 =3E http=3A//bugs=2Esun=2Ecom/bugdatabase/view=5Fbug=2Edo=3Fbug=5Fid=3D47= 30581 =3E http=3A//bugs=2Esun=2Ecom/bugdatabase/view=5Fbug=2Edo=3Fbug=5Fid=3D48= 09008 =3E = =3E So=2C unless I=27m missing something here=2C that means fixes like us= ing =3E synchronization and WeakReference on singleton probably won=27t help = =3E much if =3E at all=2E The only way that I can see for a class not to be gargage = =3E collectedwhen no more active thread use it=2C is if another = =3E ClassLoader has an hard =3E reference to the class instance=2C or an instance of that class=2E = =3E Having the =3E class itself have an hard reference on a its own singleton has no = =3E effect=2Cit=27s a circular reference that will not prevent the class = =3E or the instance to =3E be gargabe collected when neither is being refered to by something = =3E else=2E =3E Guillaume =3E = =3E = =3E = =3E = =3E ------------------------------------------------------- =3E This SF=2ENet email is sponsored by=3A Oracle 10g =3E Get certified on the hottest thing ever to hit the market=2E=2E=2E = =3E Oracle 10g=2E =3E Take an Oracle 10g class now=2C and we=27ll give you the exam FREE=2E= =3E http=3A//ads=2Eosdn=2Ecom/=3Fad=5Fid=3D3149=26alloc=5Fid=3D8166=26op=3D= click =3E =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F =3E Springframework-developer mailing list =3E Springframework-developer=40lists=2Esourceforge=2Enet =3E https=3A//lists=2Esourceforge=2Enet/lists/listinfo/springframework-de= veloper =3E = =3E = =3E = =3E = =3E ------------------------------------------------------- =3E This SF=2ENet email is sponsored by=3A Oracle 10g =3E Get certified on the hottest thing ever to hit the market=2E=2E=2E = =3E Oracle 10g=2E =3E Take an Oracle 10g class now=2C and we=27ll give you the exam FREE=2E= =3E http=3A//ads=2Eosdn=2Ecom/=3Fad=5Fid149=26alloc=5Fid=3F66=26op=3Dick =3E =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F =3E Springframework-developer mailing list =3E Springframework-developer=40lists=2Esourceforge=2Enet =3E https=3A//lists=2Esourceforge=2Enet/lists/listinfo/springframework-de= veloper =3E = =3E = =3E = =3E = =3E ------------------------------------------------------- =3E This SF=2ENet email is sponsored by=3A Oracle 10g =3E Get certified on the hottest thing ever to hit the market=2E=2E=2E = =3E Oracle 10g=2E =3E Take an Oracle 10g class now=2C and we=27ll give you the exam FREE=2E= =3E http=3A//ads=2Eosdn=2Ecom/=3Fad=5Fid=3D3149=26alloc=5Fid=3D8166=26op=3D= click =3E =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F =3E Springframework-developer mailing list =3E Springframework-developer=40lists=2Esourceforge=2Enet =3E https=3A//lists=2Esourceforge=2Enet/lists/listinfo/springframework-de= veloper =3E = =3E = =3E = =3E = =3E ------------------------------------------------------- =3E This SF=2ENet email is sponsored by=3A Oracle 10g =3E Get certified on the hottest thing ever to hit the market=2E=2E=2E = =3E Oracle 10g=2E =3E Take an Oracle 10g class now=2C and we=27ll give you the exam FREE=2E= =3E http=3A//ads=2Eosdn=2Ecom/=3Fad=5Fid149=26alloc=5Fid=3F66=26op=3Dick =3E =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F =3E Springframework-developer mailing list =3E Springframework-developer=40lists=2Esourceforge=2Enet =3E https=3A//lists=2Esourceforge=2Enet/lists/listinfo/springframework-de= veloper =3E = =3E = =3E = =3E = =3E ------------------------------------------------------- =3E This SF=2ENet email is sponsored by=3A Oracle 10g =3E Get certified on the hottest thing ever to hit the market=2E=2E=2E = =3E Oracle 10g=2E =3E Take an Oracle 10g class now=2C and we=27ll give you the exam FREE=2E= =3E http=3A//ads=2Eosdn=2Ecom/=3Fad=5Fid149=26alloc=5Fid=3F66=26op=3Dick =3E =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F =3E Springframework-developer mailing list =3E Springframework-developer=40lists=2Esourceforge=2Enet =3E https=3A//lists=2Esourceforge=2Enet/lists/listinfo/springframework-de= veloper =3E = =3E = =3E = =3E = =3E ------------------------------------------------------- =3E This SF=2ENet email is sponsored by=3A Oracle 10g =3E Get certified on the hottest thing ever to hit the market=2E=2E=2E = =3E Oracle 10g=2E = =3E Take an Oracle 10g class now=2C and we=27ll give you the exam FREE=2E= =3E http=3A//ads=2Eosdn=2Ecom/=3Fad=5Fid149=26alloc=5Fid=3F66=26op=D5ick =3E =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F =3E Springframework-developer mailing list =3E Springframework-developer=40lists=2Esourceforge=2Enet =3E https=3A//lists=2Esourceforge=2Enet/lists/listinfo/springframework-de= veloper =3E |