#401 UNEXPECTED TOP-LEVEL EXCEPTION SimException with Android

[pjv]

Eric, first of all, thanks for this excellent and extensive product. It works really fast, and there is really a lot you can do with it. Though the learning curve is a bit steep, and unfortunately there's some bugs for Android specifically. Together, that makes it exceptionally difficult to get a working final product and to debug the proguard setup.

Many an Android developer might have seen dex fail to process a jar from proguard, when they use "-dontobfuscate", because dex considers the LocalVariableTable incorrect. I understand so far this has been somewhat of a lower priority to get fixed as it deals with debug information, which is very understandable, but let me explain why I'm still running into problems. This bug report spun out of http://stackoverflow.com/questions/5701126/compile-with-proguard-gives-unexpected-top-level-exception/5721965#comment-7569258.

What I get:
[apply] UNEXPECTED TOP-LEVEL EXCEPTION:
[apply] com.android.dx.cf.code.SimException: local variable type mismatch: attempt to set or access a value of type java.lang.Object using a local variable of type int. This is symptomatic of .class transformation tools that ignore local variable information.
[apply] at com.android.dx.cf.code.BaseMachine.throwLocalMismatch(BaseMachine.java:550)
[apply] at com.android.dx.cf.code.Simulator$SimVisitor.visitLocal(Simulator.java:570)
[apply] at com.android.dx.cf.code.BytecodeArray.parseInstruction(BytecodeArray.java:411)
[apply] at com.android.dx.cf.code.Simulator.simulate(Simulator.java:99)
[apply] at com.android.dx.cf.code.Ropper.processBlock(Ropper.java:684)
[apply] at com.android.dx.cf.code.Ropper.doit(Ropper.java:639)
[apply] at com.android.dx.cf.code.Ropper.convert(Ropper.java:252)
[apply] at com.android.dx.dex.cf.CfTranslator.processMethods(CfTranslator.java:252)
[apply] at com.android.dx.dex.cf.CfTranslator.translate0(CfTranslator.java:131)
[apply] at com.android.dx.dex.cf.CfTranslator.translate(CfTranslator.java:85)
[apply] at com.android.dx.command.dexer.Main.processClass(Main.java:369)
[apply] at com.android.dx.command.dexer.Main.processFileBytes(Main.java:346)
[apply] at com.android.dx.command.dexer.Main.access$400(Main.java:59)
[apply] at com.android.dx.command.dexer.Main$1.processFileBytes(Main.java:294)
[apply] at com.android.dx.cf.direct.ClassPathOpener.processArchive(ClassPathOpener.java:244)
[apply] at com.android.dx.cf.direct.ClassPathOpener.processOne(ClassPathOpener.java:130)
[apply] at com.android.dx.cf.direct.ClassPathOpener.process(ClassPathOpener.java:108)
[apply] at com.android.dx.command.dexer.Main.processOne(Main.java:313)
[apply] at com.android.dx.command.dexer.Main.processAllFiles(Main.java:233)
[apply] at com.android.dx.command.dexer.Main.run(Main.java:185)
[apply] at com.android.dx.command.dexer.Main.main(Main.java:166)
[apply] at com.android.dx.command.Main.main(Main.java:90)
[apply] ...at bytecode offset 00000071
[apply] locals[0000]: Lnet/lp/collectionista/providers/ProductContentProvider;
[apply] locals[0001]: Landroid/net/Uri;
[apply] locals[0002]: Ljava/lang/String;
[apply] locals[0003]: [Ljava/lang/String;
[apply] locals[0004]: Landroid/database/sqlite/SQLiteDatabase;
[apply] locals[0005]: Ljava/lang/String;
[apply] locals[0006]: [Ljava/lang/Object;
[apply] stack[0002]: Landroid/database/sqlite/SQLiteDatabase;
[apply] stack[0001]: Ljava/lang/String;
[apply] stack[top0]: Ljava/lang/StringBuilder;
[apply] ...while working on block 0071
[apply] ...while working on method delete:(Landroid/net/Uri;Ljava/lang/String;[Ljava/lang/String;)I
[apply] ...while processing delete (Landroid/net/Uri;Ljava/lang/String;[Ljava/lang/String;)I
[apply] ...while processing net/lp/collectionista/providers/ProductContentProvider.class
[apply]
[apply] 1 error; aborting

I will add the jar, the proguard.cfg and the dump.txt below.

A similar bug is #3161222, but the Scala part does not apply in general, and the advice was to exclude the offending classes, which I'm unable to do here given that it involves basic classes of my own such as net/lp/collectionista/providers/ProductContentProvider.class.

Second, there is bug #3002656, which fixed some issues but evidently not all. Finally there's Sean from ZXing/Google who reported bug #1998750.

The problem goes away if you only do the shrinking (no optimization), but the apk will be bigger. If you want optimization, you can turn it on, but then you need to drop the debug flags in javac AND in dex. There seems to be no easy way to let proguard drop them for you (see #1998750). The big problem you then get if you release your app this way, is your stacktraces miss line number information, making them pretty useless. I use mobile analytics to send me uncaught exceptions from the field, so even my released app needs to have this.

Here, it's a bit vague to me. Are there any other downsides? Is there a way to get this debug information back? If not, then I do think this is a high priority issue for Android.

I've picked up that if you also obfuscate, your problem might be solved, because it'll garble anything anyway and give you the mapping.txt. Not sure if you can reenable the debug flags then, and if the problem also goes away if you don't. What I do know, is that I have other problems when I enable obfuscation (XStream fails to work properly: http://stackoverflow.com/questions/6418299/proguard-and-xstream-with-omitfield-on-android\). Also, I'm not sure if I'd really want to use obfuscation. ReTrace-ing using the mapping file really works nice, but even then it is an extra step. I'd have to copy the many stacktraces from the field out of the mobile analytics, run retrace, and classify the bug. For me it reduces recognizability, and thus an early opportunity to prioritize. So I really would like a fully working optimization step as the end goal, if not also as a stepping stone to setup the whole proguard configuration and debug it.

And I need to compress my apk somewhat because of the unsuited size of some libraries, which have always been targeted towards Java and not Android. Some early phones really have limited storage memory, so Android very much needs small apk sizes. I feel like I'm stuck between many enclosing walls with proguard atm.

If it's not worth it solving this issue, can somebody shed some light on what options there are to work around it in different ways, and maybe also update the documentation?

Thanks,
pjv

[pjv]

proguard.cfg from -printconfiguration

[pjv]

The files I couldn't attach here:

dump.txt.zip: proguard -dump http://dl.dropbox.com/u/13504115/dump.txt.zip
obfuscated.jar: output jar from proguard http://dl.dropbox.com/u/13504115/obfuscated.jar

My project is open-source if you want to have a look.

[Eric Lafortune]

Thanks for the details. In this case, you can avoid the problem by not specifying the option -keepparameternames in your configuration. This option is actually intended (but entirely optional) for processing APIs, so it's unlikely that you really need it. It preserves a subset of the local variable names (-keepattributes LocalVariableTable, LocalVariableTypeTable). Unfortunately, it can suffer from the same problem; the debug information sometimes conflicts with the optimized code, in more or less subtle ways. It can be difficult to keep debug information intact while optimizing code beyond recognition.

The options only affect local variable names, so you can still keep line number information (-keepattributes SourceFile, LineNumberTable), for useful stack traces. If you'd rather not obfuscate class names and method names, you can specify keep them (-keep,allowshrinking,allowoptimization class * { <methods>; }).

I'm not sure what the problem is with XStream serialization. If it's not because some attribute that it expects (maybe Signature?) is removed, it may be a class or field that is removed or renamed. You could see if broad -keep options work (-keep class * { *; }), and then gradually narrow down the problem.

For some reason, I can't find bugs 3002656 and 1998750 in the tracker search.

[pjv]

Hi Eric,

Thanks for your quick and constructive response. The two other bugs are here, just for future reference: http://sourceforge.net/tracker/index.php?func=detail&aid=3002656&group_id=54750&atid=474704 and http://sourceforge.net/tracker/index.php?func=detail&aid=1998750&group_id=54750&atid=474704 .

Let me get back to you on the rest after I've looked into it. My idea was to file another bug report for the XStream issue, if I can't get it diagnosed, but I'll try your advice first. I also thought of the Signature in the mean time because I saw it somewhere. I was not aware that -keepparameternames implied the other -keepattributes, because it just is not transparent to the end user in that way.

[pjv]

Hi,

I've gotten XStream to work with obfuscation, by adding a few more -keepattributes (which carried me over the blocker issue and then the rest of the minor issues with XStream were easy to solve). Thanks for the successful advice. I now have:

-keepattributes SourceFile,LineNumberTable
-keepattributes *Annotation*
-keepattributes Signature
-keepattributes EnclosingMethod
-keepattributes InnerClasses

-keepattributes Exceptions # can be removed?
-keepattributes Deprecated # can be removed?
-keepattributes Synthetic # can be removed?

As for the topic of this bug, I still haven't been able to get just shrinking + optimization working without getting the SimException. I removed the -keepparameternames to no avail. The exception stacktrace is still exactly the same. Does anything else have a relationship with these "-keepattributes LocalVariableTable, LocalVariableTypeTable"?

As a third option, how useful is your suggestion to do obfuscation but with keeping class and method names? Would that equate just doing optimization AND having a mapping.txt file?

Thanks,
pjv

[Eric Lafortune]

The options -keepattributes or -keepparameternames are only relevant in the obfuscation step, which removes the debug attributes and renames classes and class members. So you probably want to have an obfuscation step (to work around the SimException), but just disable the renaming (to simplify debugging and maybe avoid problems with introspection).

[pjv]

Hi Eric,

Ok, just for clarity, let me sort my thoughts: I'm not interested in -keepparameternames at all. What I understand now is that with enabling also obfuscation I can accomplish pretty much the same effect as with just shrinking and optimizing, if I disable the renaming. I will need to keep the mapping.txt, but that is manageable, and on the upper hand the apk will be a bit smaller, and in this way I circumvent the SimException.

When you say 'disable the renaming', do you just mean your earlier suggestion of:
-keep,allowshrinking,allowoptimization class * { <methods>; } ?

Obfuscation (i.e. also renaming) is no longer a problem for the correctness of the program for me. It's just that I would like to have decent readable stacktraces like after the optimization step.

I will need to try out the above line. I'm not sure why it wouldn't bring back the SimException, seeing that maybe the affected parts of the code are again not obfuscated.

Thanks a lot, I'll get back to you in a bit.

pjv

[pjv]

I noticed I still hadn't answered back here. Sorry about the long delay.

Meanwhile I was able to have it working as you proposed. I'm doing obfuscation, but am overriding it to keep the information I need for the stacktraces. And XStream has been working as well.

The delay means I've at least seen this working for this long while, so I would advise others reading this to make the same choice.

I think we can close this issue, though it would be nice if it worked more out of the box on Android.

I have answered the StackOverflow questions. I also created a public Google+ post about this.

Thanks a lot

[codeslicer]

Following the solution on:

http://stackoverflow.com/a/7587680/832776

resolved my problem. I just needed to add !code/allocation/variable to the optimizations.