Problems with incremental obfuscation.

[Allen Unueco]

When running ProGuard on incremental versions of our program it seems that it will randomly change previously mapped names.

These are the options Im using

------------

libraryjars  <java.home>/lib/rt.jar
-injars nonobfuscated.jar
-outjars obfuscated.jar

-printmapping proguard.map
-applymapping proguard.map

-keepclasseswithmembers public class * {
    public static void main(java.lang.String[]);
}

------------

I ran ProGuard on v1.0 (without the -applymapping option) and then took that proguard.map and used that on v1.1

When I do a diff of the old and new map file I see some strange things. Here are a few bits from the diff between the input and output map files.

-    void onDocumentEvent() -> e
+    void onDocumentEvent() -> aH_

-    void initialiseLookAndFeel() -> q
+    void initialiseLookAndFeel() -> aS_

-    java.lang.String[] TABLEMODEL_COLUMN_NAMES -> h
+    java.lang.String[] TABLEMODEL_COLUMN_NAMES -> ay_

-    boolean _alarmTreeCopied -> o
-    boolean _isFirstAlarmStatusReading -> p
+    boolean _alarmTreeCopied -> aC_
+    boolean _isFirstAlarmStatusReading -> aD_

When I take the output map file from v1.1 and use it as input for v1.2 I again get some random problems. For example

-    java.util.jar.JarFile[] _inputJars -> e
+    java.util.jar.JarFile[] _inputJars -> a_

-    void copyContent(java.lang.String) -> d
+    void copyContent(java.lang.String) -> b_

Also it is not deterministic, I will get different output when I run it a number of times.

For example it between version v1.0 and v1.1 it seems to always change onDocumentEvent() but the first time it was aH_ and the next time it was aW_

We first found this error while using ProGuard 2.1. That led me to test out 3.2 but I get the same errors.

Im using a Windows 2000 system running Java 1.4

>java -version
java version "1.4.2"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2-b28)
Java HotSpot(TM) Client VM (build 1.4.2-b28, mixed mode)

[Gili Tzabari]

-applymapping and -printmapping should not point to the same file otherwise you're reading and writing from and to the same file.

Change them and try again.

Gili

[Allen Unueco]

You bring up a good point but it seems as if ProGuard buffers the output map file and writes it at the end.

Anyway, during my testing for the past day I have changed it to use separate files for input and output and I have the same behavior.

[Eric Lafortune]

Names with underscores result from the obfuscation algorithm finding a conflict between method names in class hierarchies. Suppose method X has been assigned a name A in a given class hierarchy, and at a later point method Y has been assigned the same name A in a different class hierarchy. If the algorithm later finds out both methods are actually occurring in yet a third class hierarchy, it resolves the naming conflict by reassigning names with underscores. Sometimes, this is simply unavoidable, when applying existing maps to class hierarchies that have changed. Sometimes, this happens during the obfuscation process itself.

This piece of theory may not help you much, though. I'll think about ways to avoid the renaming when possible, especially when using maps.

Eric.

[Allen Unueco]

Eric,

Thank you for the background information.

I am still having an issue with incremental obfuscation though. Here is the what Im seeing.

There was already this mapping in Version "A" for the name d_

217:222:wmt.common.framework.Model buildProblemStatusModel(java.lang.String) -> d_

A new method was added in Version "B" which got mapped to the same name "d_"

181:184:wmt.common.framework.Model buildAlarmViewModel(java.lang.String) -> d_

So I get a ClassFormatError while trying to run version "B" due to two methods with the same name.

What am I doing wrong and how can I get around this problem?

Here are a few snippets from the mapfiles and the stack trace.

--- proguard.map output from Version "A"

wmt.radiodist.models.RadioDacSmxModelBuilder -> mA:
    47:126:wmt.common.framework.Model buildModel(wmt.common.models.ModelType,java.lang.String) -> a
    132:140:wmt.common.framework.Model buildAlarmDefinitionsModel(java.lang.String) -> a
    146:150:wmt.common.framework.Model buildAlarmSeveritySensor(java.lang.String) -> g
    156:159:wmt.common.framework.Model buildBusTributaryDiagnosticsModel(java.lang.String) -> q
    165:174:wmt.common.framework.Model buildBusTributaryDiagnosticsViewModel(java.lang.String) -> s
    180:183:wmt.common.framework.Model buildBusTributaryModel(java.lang.String) -> a_
    189:192:wmt.common.framework.Model buildBusTributaryViewModel(java.lang.String) -> c
    198:202:wmt.common.framework.Model buildBusTributaryStatusModel(java.lang.String) -> r
    208:211:wmt.common.framework.Model buildConnectableDetailsModel(java.lang.String) -> e
    217:222:wmt.common.framework.Model buildProblemStatusModel(java.lang.String) -> d_
    228:232:wmt.common.framework.Model buildSensorsModel(java.lang.String) -> f
    238:241:wmt.common.framework.Model buildStmMuxModel(java.lang.String) -> h
    247:252:wmt.common.framework.Model buildTributaryBasicModel(java.lang.String) -> t
    258:264:wmt.common.framework.Model buildTributaryModel(java.lang.String) -> k
    271:276:wmt.common.framework.Model buildTributaryCapabilityModel(java.lang.String) -> d
    282:289:wmt.common.framework.Model buildTributaryDiagnosticsModel(java.lang.String) -> p
    295:305:wmt.common.framework.Model buildTributaryDiagnosticsViewModel(java.lang.String) -> o
    311:315:wmt.common.framework.Model buildTributaryStatusModel(java.lang.String) -> l
    321:329:wmt.common.framework.Model buildTributaryViewModel(java.lang.String) -> n

--- proguard.map output from Version "B"

wmt.radiodist.models.RadioDacSmxModelBuilder -> mA:
    48:143:wmt.common.framework.Model buildModel(wmt.common.models.ModelType,java.lang.String) -> a
    149:157:wmt.common.framework.Model buildAlarmDefinitionsModel(java.lang.String) -> a
    163:165:wmt.common.framework.Model buildAlarmIsUsedModel(java.lang.String) -> m
    171:175:wmt.common.framework.Model buildAlarmSeveritySensor(java.lang.String) -> g
    181:184:wmt.common.framework.Model buildAlarmViewModel(java.lang.String) -> d_
    190:193:wmt.common.framework.Model buildBusTributaryDiagnosticsModel(java.lang.String) -> q
    199:211:wmt.common.framework.Model buildBusTributaryDiagnosticsViewModel(java.lang.String) -> s
    217:220:wmt.common.framework.Model buildBusTributaryModel(java.lang.String) -> a_
    226:229:wmt.common.framework.Model buildBusTributaryViewModel(java.lang.String) -> c
    235:239:wmt.common.framework.Model buildBusTributaryStatusModel(java.lang.String) -> r
    245:248:wmt.common.framework.Model buildConnectableDetailsModel(java.lang.String) -> e
    254:257:wmt.common.framework.Model buildPrbsModel(java.lang.String) -> b
    263:263:wmt.common.framework.Model buildPrbsStatusSensor(java.lang.String) -> u
    269:276:wmt.common.framework.Model buildProblemStatusModel(java.lang.String) -> d_
    282:286:wmt.common.framework.Model buildSensorsModel(java.lang.String) -> f
    292:295:wmt.common.framework.Model buildStmMuxModel(java.lang.String) -> h
    301:306:wmt.common.framework.Model buildTributaryBasicModel(java.lang.String) -> t
    312:318:wmt.common.framework.Model buildTributaryModel(java.lang.String) -> k
    325:330:wmt.common.framework.Model buildTributaryCapabilityModel(java.lang.String) -> d
    336:343:wmt.common.framework.Model buildTributaryDiagnosticsModel(java.lang.String) -> p
    349:362:wmt.common.framework.Model buildTributaryDiagnosticsViewModel(java.lang.String) -> o
    368:372:wmt.common.framework.Model buildTributaryStatusModel(java.lang.String) -> l
    378:386:wmt.common.framework.Model buildTributaryViewModel(java.lang.String) -> n

-- ClassFormatError while running Version "B"

java.lang.ClassFormatError: mA (Repetitive method name/signature)
        at java.lang.ClassLoader.defineClass0(Native Method)
        at java.lang.ClassLoader.defineClass(Unknown Source)
        at java.security.SecureClassLoader.defineClass(Unknown Source)
        at java.net.URLClassLoader.defineClass(Unknown Source)
        at java.net.URLClassLoader.access$100(Unknown Source)
        at java.net.URLClassLoader$1.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClassInternal(Unknown Source)
        at oQ.b(Unknown Source)
        at gB.a(Unknown Source)

[Eric Lafortune]

This turns out to be a bug in ProGuard. When resolving a naming conflict between class hierarchies, by picking a name ending in a '_', ProGuard assumes this name will be unique. It doesn't check whether it is already in use due to the -applymapping option. Once more, performing incremental obfuscation on classes whose fundamental structure has changed can be problematic (and sometimes impossible). An optimal solution may be difficult to find, with related methods having to receive the same name across class hierarchies, most methods having to receive different names across class hierarchies, and certain methods having to receive mapped names -- all this trying to keep the names short. I'll think about improvements in this area.

Eric.

[Allen Unueco]

Eric,

Do you have any ETA for a fix to this bug?

I have a hack that Im currently using where I changed the call to setNewMemberName() in the case where a '_' would be used to also include 3 random chars.

This makes it very unlikely that a duplicate name will be selected. Our builds are now not failing at runtime.

To detect this runtime problem I was in the process of creating a post-build step which would verify that each of the class files in the resulting jar file could load. This seems like a good thing for proguard to check before ending successfully.

-allen

[Eric Lafortune]

I haven't fixed the problem yet, but it is fairly high on my list and I have some ideas. No ETA, sorry.

Eric.

[Allen Unueco]

Originally I thought the problem we were seeing was due to names changing during subsequent obfuscations. But it looks the real problem is the duplicate name issue.

As long as the mapped names are unique, thus valid, Im less concerned about names changing from what they were in the -applymapping.

-allen

[Tim]

I just started using ProGuard and ran into a similar problem, but what I'm seeing happens when I run ProGuard twice on the same set of jar files.  When I run the first time, I do not use an input map.  I take the output map from the first run and make it the input map for the second run.  I then compare the input and output maps from the second run, assuming that since there are no additional classes or members that the maps should be identical.  Turns out there are some differences.  In all cases where there are differences, the first run produced names without any underscores, but the second used names with underscores.  Subsequent runs using the second output map as the input map produce the same output map every time.

To keep download times short, I was hoping to be able to ship fixes for our product as small jar files containing only those classes that have changed since the release of the product.  We would use the output map from subsequent builds as the input map for each fix build.  If unchanged classes or members can have their obfuscated names changed from an initial build to a subsequent build, it then seems like our scheme may not work.

Has anyone else tried putting the same code through the rinse cycle a second time to see if the obfuscated names change?  Can you comment on why this might be happening?  I was following the argument about naming conflicts between class hierarchies, but since I didn't see any underscores in the initial list, it seems that ProGuard did not detect any conflicts the first time through.

- Tim