#385 Possible Proguard (or scala, or Android) bug causes dx error

open
None
6
2014-09-10
2011-01-18
joe
No

Hi,

First, I'd like to thank you, the Proguard developers, very much for
your work. Proguard has been a wonderful tool for my forays into android.

I think I've found what might be a bug in Proguard. It might also be a
bug in the Scala compiler--I'm not quite sure. The actual error
message is printed by Android's dx bytecode compiler and blames
".class transformation tools" (e.g. Proguard), so I thought it would
ask here first.

The bug is that when I try to dx a jar created with Proguard, I get the following error:

com.android.dx.cf.code.SimException: local variable type mismatch: attempt to set or access a value of type int using a local variable of type java.lang.Object. This is symptomatic of .class transformation tools that ignore local variable information.

I've isolated the code that produces the error. To try it, you'll need
android.jar, dx.jar and the dx script from an Android development
environment (I tried android-6 and android-8), scala-library.jar from
Scala 2.8.1, and proguard.jar (I tried it with 4.4 and with 4.6b2 with
the same results). Since all of these are open-source, to save you the
trouble of finding them, I've packaged up the jars into a .tgz with a
run.sh script that will trigger the bug immediately. (I'd attach it,
but it's 11 megs and SF doesn't seem to accept files this big.)

http://squarechan.com/static/temp/proguard-test.tgz
(Note: I'm not sure how long this link will be up.)

If you have the necessary jars already,
here's how you can trigger the bug by hand:

$ cd /tmp
$ mkdir test
$ cd test
$ cat>Ohai.scala
package com.test.hello

import scala.collection.JavaConversions._
import _root_.java.util.HashSet

class Ohai {
def foo() {
new HashSet("12,34,12".split(",").toList)
}
}
^D
$ mkdir classes
$ scalac -d classes Ohai.scala
$ java -jar ./proguard-4.4.jar -injars ./classes -injars './scala-library.jar(!META-INF/MANIFEST.MF,!library.properties)' -outjars ./myclasses.min.jar -libraryjars ./android.jar -forceprocessing -dontobfuscate -dontpreverify -dontwarn -keep 'class com.test.hello.** { <fields>; <methods>;}'
$ ./dx -JXmx512m --dex --output=./classes.dex ./myclasses.min.jar

{If you're not using the .tgz, you will probably need to add the
appropriate paths to the jars and the dx program above.}

The result:

$ ./run.sh
#!/bin/sh -v
mkdir -p classes
scalac -d classes Ohai.scala
java -jar ./proguard-4.4.jar -injars ./classes -injars './scala-library.jar(!META-INF/MANIFEST.MF,!library.properties)' -outjars ./myclasses.min.jar -libraryjars ./android.jar -forceprocessing -dontobfuscate -dontpreverify -dontwarn -keep 'class com.test.hello.** { <fields>; <methods>;}'
ProGuard, version 4.4
Reading program directory [/tmp/test/classes]
Reading program jar [/tmp/test/scala-library.jar] (filtered)

Reading library jar [/tmp/test/android.jar]
Note: scala.xml.include.sax.Main$$anonfun$3 calls '(org.xml.sax.EntityResolver)Class.forName(variable).newInstance()'
Note: scala.Enumeration accesses a field 'MODULE$' dynamically
Maybe this is program field 'scala.Array$ { scala.Array$ MODULE$; }'
Maybe this is program field 'scala.Cell$ { scala.Cell$ MODULE$; }'
Maybe this is program field 'scala.Console$ { scala.Console$ MODULE$; }'
Maybe this is program field 'scala.Either$ { scala.Either$ MODULE$; }'
{several pages of 'Maybe this is program field' snipped}
Note: there were 1 class casts of dynamically created class instances.
You might consider explicitly keeping the mentioned classes and/or
their implementations (using '-keep').
Note: there were 1 accesses to class members by means of introspection.
You should consider explicitly keeping the mentioned class members
(using '-keep' or '-keepclassmembers').
Note: You're ignoring all warnings!
Preparing output jar [/tmp/test/myclasses.min.jar]
Copying resources from program directory [/tmp/test/classes]
Copying resources from program jar [/tmp/test/scala-library.jar] (filtered)
./dx -JXmx512m --dex --output=./classes.dex ./myclasses.min.jar

UNEXPECTED TOP-LEVEL EXCEPTION:
com.android.dx.cf.code.SimException: local variable type mismatch: attempt to set or access a value of type int using a local variable of type java.lang.Object. This is symptomatic of .class transformation tools that ignore local variable information.
at com.android.dx.cf.code.BaseMachine.throwLocalMismatch(BaseMachine.java:537)
at com.android.dx.cf.code.Simulator$SimVisitor.visitLocal(Simulator.java:523)
at com.android.dx.cf.code.BytecodeArray.parseInstruction(BytecodeArray.java:481)
at com.android.dx.cf.code.Simulator.simulate(Simulator.java:99)
at com.android.dx.cf.code.Ropper.processBlock(Ropper.java:678)
at com.android.dx.cf.code.Ropper.doit(Ropper.java:633)
at com.android.dx.cf.code.Ropper.convert(Ropper.java:250)
at com.android.dx.dex.cf.CfTranslator.processMethods(CfTranslator.java:252)
at com.android.dx.dex.cf.CfTranslator.translate0(CfTranslator.java:131)
at com.android.dx.dex.cf.CfTranslator.translate(CfTranslator.java:85)
at com.android.dx.command.dexer.Main.processClass(Main.java:299)
at com.android.dx.command.dexer.Main.processFileBytes(Main.java:278)
at com.android.dx.command.dexer.Main.access$100(Main.java:56)
at com.android.dx.command.dexer.Main$1.processFileBytes(Main.java:229)
at com.android.dx.cf.direct.ClassPathOpener.processArchive(ClassPathOpener.java:244)
at com.android.dx.cf.direct.ClassPathOpener.processOne(ClassPathOpener.java:130)
at com.android.dx.cf.direct.ClassPathOpener.process(ClassPathOpener.java:108)
at com.android.dx.command.dexer.Main.processOne(Main.java:247)
at com.android.dx.command.dexer.Main.processAllFiles(Main.java:183)
at com.android.dx.command.dexer.Main.run(Main.java:139)
at com.android.dx.command.dexer.Main.main(Main.java:120)
at com.android.dx.command.Main.main(Main.java:89)
...at bytecode offset 00000023
locals[0000]: Lscala/collection/mutable/ArrayBuilder$ofRef;
locals[0001]: Ljava/lang/Object;
locals[0002]: Lscala/collection/mutable/ArrayBuilder$ofRef;
stack[0001]: Lscala/collection/mutable/ArrayBuilder$ofRef;
stack[top0]: I
...while working on block 0021
...while working on method $plus$eq:(Ljava/lang/Object;)Lscala/collection/mutable/ArrayBuilder$ofRef;
...while processing $plus$eq (Ljava/lang/Object;)Lscala/collection/mutable/ArrayBuilder$ofRef;
...while processing scala/collection/mutable/ArrayBuilder$ofRef.class

1 error; aborting

If I use proguard 4.6b2, the final error is a little different:

UNEXPECTED TOP-LEVEL EXCEPTION:
com.android.dx.cf.code.SimException: local variable type mismatch: attempt to set or access a value of type java.lang.Object using a local variable of type int. This is symptomatic of .class transformation tools that ignore local variable information.
at com.android.dx.cf.code.BaseMachine.throwLocalMismatch(BaseMachine.java:537)
at com.android.dx.cf.code.Simulator$SimVisitor.visitLocal(Simulator.java:523)
at com.android.dx.cf.code.BytecodeArray.parseInstruction(BytecodeArray.java:517)
at com.android.dx.cf.code.Simulator.simulate(Simulator.java:99)
at com.android.dx.cf.code.Ropper.processBlock(Ropper.java:678)
at com.android.dx.cf.code.Ropper.doit(Ropper.java:633)
at com.android.dx.cf.code.Ropper.convert(Ropper.java:250)
at com.android.dx.dex.cf.CfTranslator.processMethods(CfTranslator.java:252)
at com.android.dx.dex.cf.CfTranslator.translate0(CfTranslator.java:131)
at com.android.dx.dex.cf.CfTranslator.translate(CfTranslator.java:85)
at com.android.dx.command.dexer.Main.processClass(Main.java:299)
at com.android.dx.command.dexer.Main.processFileBytes(Main.java:278)
at com.android.dx.command.dexer.Main.access$100(Main.java:56)
at com.android.dx.command.dexer.Main$1.processFileBytes(Main.java:229)
at com.android.dx.cf.direct.ClassPathOpener.processArchive(ClassPathOpener.java:244)
at com.android.dx.cf.direct.ClassPathOpener.processOne(ClassPathOpener.java:130)
at com.android.dx.cf.direct.ClassPathOpener.process(ClassPathOpener.java:108)
at com.android.dx.command.dexer.Main.processOne(Main.java:247)
at com.android.dx.command.dexer.Main.processAllFiles(Main.java:183)
at com.android.dx.command.dexer.Main.run(Main.java:139)
at com.android.dx.command.dexer.Main.main(Main.java:120)
at com.android.dx.command.Main.main(Main.java:89)
...at bytecode offset 00000077
locals[0000]: Lscala/collection/immutable/HashSet$HashTrieSet;
locals[0001]: Lscala/collection/immutable/HashSet;
locals[0002]: I
locals[0003]: I
locals[0004]: [Lscala/collection/immutable/HashSet;
locals[0005]: Lscala/collection/immutable/HashSet;
locals[0006]: I
locals[0007]: Lscala/collection/immutable/HashSet$HashTrieSet;
stack[0005]: N0065Lscala/collection/immutable/HashSet$HashTrieSet;
stack[0004]: N0065Lscala/collection/immutable/HashSet$HashTrieSet;
stack[0003]: I
stack[0002]: [Lscala/collection/immutable/HashSet;
stack[0001]: Lscala/collection/immutable/HashSet$HashTrieSet;
stack[top0]: Lscala/collection/immutable/HashSet$HashTrieSet;
...while working on block 0070
...while working on method updated0:(Ljava/lang/Object;II)Lscala/collection/immutable/HashSet;
...while processing updated0 (Ljava/lang/Object;II)Lscala/collection/immutable/HashSet;
...while processing scala/collection/immutable/HashSet$HashTrieSet.class

1 error; aborting

Any suggestions about where this bug might actually reside (in Scala,
in Proguard, or in dx) would be very helpful.

Thanks very much in advance.

Joe

Discussion

  • Eric Lafortune
    Eric Lafortune
    2011-01-19

    Thanks for your detailed report. As a general note, you should really remove -dontwarn and solve all warnings, for instance by at least specifying Java's rt.jar as a library jar. You can then probably work around the problem by removing the (optional) LocalVariableTable attributes, either by not letting scalac generate them or by letting ProGuard remove them in the obfuscation step. It seems ProGuard's optimization step is not updating them correctly at some point. I've fixed some issues in this area before, but I'll have to investigate which case is still causing trouble.

    Note that ProGuard's Examples page contains Scala and Android configurations that might be useful.

     
  • Eric Lafortune
    Eric Lafortune
    2011-01-19

    • assigned_to: nobody --> lafortune
    • priority: 5 --> 6
     
  • joe
    joe
    2011-01-19

    Thanks so much for the reply! Removing -dontobfuscate was a great temporary fix: dx now runs without a problem.

    I agree that -dontwarn is a bad idea; I'd rather not use it either. I'm using it because I haven't yet figured out how to simultaneously build against scala-library.jar (which depends on a number of symbols in rt.jar) and android.jar (which implements much, but not all, of the symbols in rt.jar). Fortunately, none of the pieces of Scala that I use appear to rely on those missing symbols. If i turn off -dontwarn, Proguard will rightly complain about those missing dependencies If I add rt.jar, android.jar's symbols will clash with rt.jar's symbols. I hope to figure out the correct solution (rebuild scala-library.jar from source without those dependencies?) at some point. (In my defense, the android plugin for scala-build-tool, from which I cribbed the first draft of my proguard settings, also uses -dontwarn.)

    Thanks very much again. (The scala and android examples were very helpful, by the way.)

    Joe

     
  • Eric Lafortune
    Eric Lafortune
    2011-01-20

    I understand. You may be able to get rid of all warnings about references to missing classes by filtering out the offending classes from scala-library.jar (with a filter on the -injars option). The end result would be the same, but the build process would be cleaner.