Re: [Wala-wala] GC overhead limit exceeded when frequently invoking getCallsiteSignature

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

It’s possible something is wrong in WALA, but I would want to see more
evidence from a tool like MAT before digging more.

--Manu

On Fri, Sep 2, 2022 at 12:03 Parjanya Vyas <par...@uw...>
wrote:

> Hi Manu,
>
> Thank you very much for your response.
>
> Sure, I will try using that particular tool to check if I can find a
> memory leak within my code.
>
> However, during multiple runs of the code the stack trace always points to
> the getSignature method whenever this exception is thrown.
>
> Moreover, the below StackOverflow answer states that this exception with
> 'GC overhead limit exceeded' message is thrown when GC is running
> excessively without being able to claim enough memory:
>
> https://stackoverflow.com/questions/1393486/error-java-lang-outofmemoryerror-gc-overhead-limit-exceeded
>
> Do you think there might be a chance the issue is indeed within WALA code?
>
> Thanks,
> Parjanya
> ------------------------------
> *From:* Manu Sridharan <ma...@sr...>
> *Sent:* Friday, September 2, 2022 2:33:36 PM
>
> *To:* WALA discussion and Q&A <wal...@li...>
> *Subject:* Re: [Wala-wala] GC overhead limit exceeded when frequently
> invoking getCallsiteSignature
>
> Hi Parjanya,
>
> An OutOfMemoryError occurs when the GC is unable to collect enough memory
> to allow for future allocations.  This does not occur due to frequent
> allocation; it happens because too much memory is being retained by the
> program.  Also, the stack trace associated with an OutOfMemoryError usually
> does not give good guidance as to what is causing the problem.
>
> Based on what you wrote below, I suspect the issue here is within your own
> code.  Do you have a large List or Map for caching analysis results that
> keeps growing?  That’s often the root cause of a leak.  When I want to
> debug this sort of issue I use the Eclipse MAT tool to analyze a heap
> snapshot:
>
> https://www.eclipse.org/mat/
>
> To collect a heap snapshot when the program runs out of memory, pass the
> -XX:+HeapDumpOnOutOfMemoryError parameter to the JVM.  I would test this
> with a much smaller heap than 12000m, since the heap dump will be roughly
> the size of the heap.  Once you open the heap snapshot in MAT, you should
> be able to see what data structure is consuming a lot of memory.
>
> Thanks,
> Manu
>
> On Sep 2, 2022 at 11:07:00, Parjanya Vyas <par...@uw...>
> wrote:
>
> Hi Manu,
>
> Thanks a lot for your response.
> It is the JVM that throws the 'GC overhead limit exceeded' exception that
> stems from a WALA method.
> Here is the stack trace:
> Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit
> exceeded
> at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:68)
> at java.lang.StringBuilder.<init>(StringBuilder.java:89)
> at
> com.ibm.wala.types.TypeName$TypeNameKey.toUnicodeString(TypeName.java:319)
> at com.ibm.wala.types.TypeName.toUnicodeString(TypeName.java:123)
> at
> com.ibm.wala.types.Descriptor$Key.appendSemicolonIfNeeded(Descriptor.java:227)
> at com.ibm.wala.types.Descriptor$Key.toString(Descriptor.java:200)
> at com.ibm.wala.types.Descriptor.toString(Descriptor.java:109)
> at java.lang.String.valueOf(String.java:2994)
> at java.lang.StringBuilder.append(StringBuilder.java:131)
> at
> com.ibm.wala.types.MethodReference.getSignature(MethodReference.java:223)
>
> My guess is that the issue is due to the method 'UTF8Convert::fromUTF8'
> allocating a new Array every time it needs to construct the string. Rapid
> repeated invocation of this method can probably result in a frequent
> allocation/deallocation of the array resulting in the GC being invoked very
> frequently. However, I might be wrong.
>
> For your queries:
>
>    1. The total size of all the dex files included is 87.6 MB
>    2. No, I am not constructing a callgraph, only the class hierarchy.
>
> Note that creating the class hierarchy is not an issue. The exception
> occurs only when I am analyzing the constructed class hierarchy during
> which I invoke the 'CallSiteReference.getDeclaredTarget.getSignature'
> repeatedly for different methods.
>
> Thanks,
> Parjanya
>
> ------------------------------
> *From:* Manu Sridharan <ma...@sr...>
> *Sent:* Friday, September 2, 2022 11:25 AM
> *To:* WALA discussion and Q&A <wal...@li...>
> *Subject:* Re: [Wala-wala] GC overhead limit exceeded when frequently
> invoking getCallsiteSignature
>
> After Googling I see the error message is likely due to an
> OutOfMemoryError.  WALA class hierarchy construction has been battle-tested
> with large classpaths of jars, but I haven’t done much with large dex
> files.  It’s possible there is a memory leak somewhere.  I’d say the
> simplest thing to try for you is to increase the max heap size, running on
> a machine with more RAM if possible; do you still get an OOM with 24G of
> heap?
>
> If you do still get an OOM with more heap, we can try to dig deeper.  Out
> of curiosity, what is the total size of the dex files in your class
> hierarchy?  And are you just trying to build a class hierarchy, or also a
> call graph?
>
> --Manu
>
> On Sep 2, 2022 at 07:48:13, Manu Sridharan <ma...@sr...> wrote:
>
> Hi Parjanya, are you saying that WALA is throwing an exception, or that
> there is an error from the JVM? If it’s the former can you link to the line
> of code throwing the exception and paste a stack trace excerpt?
>
> Thanks,
> Manu
>
> On Fri, Sep 2, 2022 at 07:46 Parjanya Vyas <par...@uw...>
> wrote:
>
> Hi Raffi,
>
> Thanks very much for your response.
> I use '12000m' with -Xmx right now.
>
> Thanks,
> Parjanya
> ------------------------------
> *From:* Raffi T Khatchadourian <raf...@hu...>
> *Sent:* Thursday, September 1, 2022 10:44 AM
> *To:* wal...@li... <wal...@li...>
> *Subject:* Re: [Wala-wala] GC overhead limit exceeded when frequently
> invoking getCallsiteSignature
>
> Parjanya,
>
> What is the value that you are giving to -Xmx when you run java?
>
> On Thu, 2022-09-01 at 03:28 +0000, Parjanya Vyas wrote:
>
> Hi all,
>
> I apologize in advance if this question has been already addressed or is
> too naive. When using class hierarchy analysis with a large code base, I
> encounter the 'GC overhead limit exceeded' exception that stems from
> MethodReference::getCallsiteSignature method when it is invoked frequently.
>
> The exception only occurs when analyzing a large analysis scope that
> consists of multiple large dex files. There is no fixed point that I could
> observe when this exception is triggered.
>
> Can someone please help me out with this? Any guidance will be highly
> appreciated! I can provide more details on the analysis data/processing
> that I am attempting, if needed.
>
> Thanks,
> Parjanya Vyas
> _______________________________________________
> Wala-wala mailing list
> Wal...@li...
> https://lists.sourceforge.net/lists/listinfo/wala-wala
>
> _______________________________________________
> Wala-wala mailing list
> Wal...@li...
> https://lists.sourceforge.net/lists/listinfo/wala-wala
>
> --
> Sent from my phone. Please pardon brevity
>
> _______________________________________________
> Wala-wala mailing list
> Wal...@li...
> https://lists.sourceforge.net/lists/listinfo/wala-wala
>
> _______________________________________________
> Wala-wala mailing list
> Wal...@li...
> https://lists.sourceforge.net/lists/listinfo/wala-wala
>