xmlvm-users

[xmlvm-users] Java shrinkers

[Panayotis K. <pan...@pa...>]

After the talk about Scala under iPhone, and since the Scala library is too big (~8MB), I thought to have a look at java shrinkers.

Since most people were talking about ProGuard & Android projects, I give it a shot.
If this is successful enough, this will be a gain not only for Scala or whatever, but also for OpenJDK and similar approaches that depend on other JAR files :)

I have to say that I was quite successful with ProGuard. It works even for strangely crafted classes, like the Scala's ones.
The only problem with ProGuard is that it performs a quick class-based optimization only. For a simple hello-world application (with some Scala-based features), it required ~350kb of classes from the Scala framework (from a total of ~6MB library). Not bad, but not good enough.

I also had a look at yGuard, which is a free (but not completely open source) project, which performs method- and variable- based optimizations. Unfortunately I wasn't lucky, at least with Scala projects, it crashed with a message:
"yShrink encountered an unknown problem!"

In ProGuard site there is a list of other shrinkers. Of course we heading only for open source, or at least free implementations.
Here is the list:
http://proguard.sourceforge.net/alternatives.html
Since ProGuard does a good job with class-based shrinking, we are actually looking for method-based shrinking.
Even if this is too slow, we could have ProGuard make a pre-selection of classes and then let the second library do the mathod-based selection.

What do you think? I believe that this task is really necessary and will help a lot with  every project, especially with the new upcoming C backend, but it needs testers ;) 

Re: [xmlvm-users] Java shrinkers

[Sascha H. <sa...@xm...>]

Panayotis,

just to understand your proposal: Are you talking about trying to reduce the
amount of code we need to translate based on a call graph analysis? Thus
throwing out methods that are never called in an application and classes
that are never referenced?

// Sascha

On Tue, Aug 31, 2010 at 5:54 PM, Panayotis Katsaloulis <
pan...@pa...> wrote:

> After the talk about Scala under iPhone, and since the Scala library is too
> big (~8MB), I thought to have a look at java shrinkers.
>
> Since most people were talking about ProGuard & Android projects, I give it
> a shot.
> If this is successful enough, this will be a gain not only for Scala or
> whatever, but also for OpenJDK and similar approaches that depend on other
> JAR files :)
>
> I have to say that I was quite successful with ProGuard. It works even for
> strangely crafted classes, like the Scala's ones.
> The only problem with ProGuard is that it performs a quick class-based
> optimization only. For a simple hello-world application (with some
> Scala-based features), it required ~350kb of classes from the Scala
> framework (from a total of ~6MB library). Not bad, but not good enough.
>
> I also had a look at yGuard, which is a free (but not completely open
> source) project, which performs method- and variable- based optimizations.
> Unfortunately I wasn't lucky, at least with Scala projects, it crashed with
> a message:
> "yShrink encountered an unknown problem!"
>
> In ProGuard site there is a list of other shrinkers. Of course we heading
> only for open source, or at least free implementations.
> Here is the list:
> http://proguard.sourceforge.net/alternatives.html
> Since ProGuard does a good job with class-based shrinking, we are actually
> looking for method-based shrinking.
> Even if this is too slow, we could have ProGuard make a pre-selection of
> classes and then let the second library do the mathod-based selection.
>
> What do you think? I believe that this task is really necessary and will
> help a lot with  every project, especially with the new upcoming C backend,
> but it needs testers ;)
>
> ------------------------------------------------------------------------------
> This SF.net Dev2Dev email is sponsored by:
>
> Show off your parallel programming skills.
> Enter the Intel(R) Threading Challenge 2010.
> http://p.sf.net/sfu/intel-thread-sfd
> _______________________________________________
> xmlvm-users mailing list
> xml...@li...
> https://lists.sourceforge.net/lists/listinfo/xmlvm-users
>

Re: [xmlvm-users] Java shrinkers

[Panayotis K. <pan...@pa...>]

On 31 Αυγ 2010, at 7:18 μ.μ., Sascha Haeberling wrote:

> Panayotis,
> 
> just to understand your proposal: Are you talking about trying to reduce the amount of code we need to translate based on a call graph analysis? Thus throwing out methods that are never called in an application and classes that are never referenced?
> 
> // Sascha

Yes, exactly.
There are already some projects, which do that. Having only class-based dependency is rather expensive, so unfortunately projects like ProGuard is of minimum help.

I don't know why yGuard which does method-based analysis didn't work, but 1) we can investigate it, 2) bug report it 3) try another library :)
I suspect it had to do with some Scala optimizations; probably with pure Java code we won't have this problem. Still I believe that a more generic solution (i.e. one that also supports other JVM-based languages, like Scala) should be considered. 

Re: [xmlvm-users] Java shrinkers

[Sascha H. <sa...@gm...>]

I am actually working on such an optimized call graph dependency optimizer,
which should shrink down the whole compilation size.

In addition we will remove the need to compile classes that don't need to be
compiled. E.g. all the JDK classes need to be compiled once and we can
provide it to our users. These two measures should speed up compilation time
a lot.

// Sascha

On Aug 31, 2010 7:08 PM, "Panayotis Katsaloulis" <pan...@pa...>
wrote:
>
> On 31 Αυγ 2010, at 7:18 μ.μ., Sascha Haeberling wrote:
>
>> Panayotis,
>>
>> just to understand your proposal: Are you talking about trying to reduce
the amount of code we need to translate based on a call graph analysis? Thus
throwing out methods that are never called in an application and classes
that are never referenced?
>>
>> // Sascha
>
> Yes, exactly.
> There are already some projects, which do that. Having only class-based
dependency is rather expensive, so unfortunately projects like ProGuard is
of minimum help.
>
> I don't know why yGuard which does method-based analysis didn't work, but
1) we can investigate it, 2) bug report it 3) try another library :)
> I suspect it had to do with some Scala optimizations; probably with pure
Java code we won't have this problem. Still I believe that a more generic
solution (i.e. one that also supports other JVM-based languages, like Scala)
should be considered.
>
------------------------------------------------------------------------------
> This SF.net Dev2Dev email is sponsored by:
>
> Show off your parallel programming skills.
> Enter the Intel(R) Threading Challenge 2010.
> http://p.sf.net/sfu/intel-thread-sfd
> _______________________________________________
> xmlvm-users mailing list
> xml...@li...
> https://lists.sourceforge.net/lists/listinfo/xmlvm-users

Re: [xmlvm-users] Java shrinkers

[Panayotis K. <pan...@pa...>]

On 31 Αυγ 2010, at 8:44 μ.μ., Sascha Haeberling wrote:

> I am actually working on such an optimized call graph dependency optimizer, which should shrink down the whole compilation size.
> 
> In addition we will remove the need to compile classes that don't need to be compiled. E.g. all the JDK classes need to be compiled once and we can provide it to our users. These two measures should speed up compilation time a lot.
> 
> // Sascha

The problem has to do more with size, not time.
It is really easy to produce static libraries, and thus reduce compilation time to minimum.
The way it is done right now, though, is again object-level optimization - similar to what ProGuard does (and it does it extremely well - it is even able to deal with simple reflection cases).

The problem is with method-level optimizations. That is why I propose to investigate such methods. 

In which level is the optimization you are working on?

Re: [xmlvm-users] Java shrinkers

[Sascha H. <sa...@xm...>]

As I explained, I am working on two things: Optimizing the amount of code
being pulled in as well as having things pre-compiled.

The first will reduce the size of the binary. The dependency analysis I
implemented a while a go is a very naive approach that pulls in all classes
and all their references, thus pulling in almost the complete JDK. It was
good for getting us started with the C backend. What I am currently working
on does the analysis on a Class::method level, which should reduce code size
significantly.

Stay tuned for updates around this this week, I will let you know when I
have something to try out.

// Sascha

On Tue, Aug 31, 2010 at 11:07 PM, Panayotis Katsaloulis <
pan...@pa...> wrote:

>
> On 31 Αυγ 2010, at 8:44 μ.μ., Sascha Haeberling wrote:
>
> > I am actually working on such an optimized call graph dependency
> optimizer, which should shrink down the whole compilation size.
> >
> > In addition we will remove the need to compile classes that don't need to
> be compiled. E.g. all the JDK classes need to be compiled once and we can
> provide it to our users. These two measures should speed up compilation time
> a lot.
> >
> > // Sascha
>
> The problem has to do more with size, not time.
> It is really easy to produce static libraries, and thus reduce compilation
> time to minimum.
> The way it is done right now, though, is again object-level optimization -
> similar to what ProGuard does (and it does it extremely well - it is even
> able to deal with simple reflection cases).
>
> The problem is with method-level optimizations. That is why I propose to
> investigate such methods.
>
> In which level is the optimization you are working on?
>
> ------------------------------------------------------------------------------
> This SF.net Dev2Dev email is sponsored by:
>
> Show off your parallel programming skills.
> Enter the Intel(R) Threading Challenge 2010.
> http://p.sf.net/sfu/intel-thread-sfd
> _______________________________________________
> xmlvm-users mailing list
> xml...@li...
> https://lists.sourceforge.net/lists/listinfo/xmlvm-users
>

Re: [xmlvm-users] Java shrinkers

[Jeff P. <jcp...@ro...>]

I have used Proguard, and it definitely can delete things below Class level,
so I am not sure what you mean about method-level optimizations.

If you add the '-printusage filename' to your config or Ant build.xml you
can see everything removed.  A class name by itself means the whole class
was delete.  If it is followed by a colon, only the stuff below was removed
along with line numbers, e.g.

myNamespace.com.stuff.morestuff.CurrentPlatform:
    public static final byte SERVER_PLATFORM
    public static final byte DESKTOP_PLATFORM
    public static final byte PHONE_PLATFORM
    32:32:public static boolean isServer()
    34:34:public static boolean isPhone()
    39:39:public static boolean isValidPlatform(byte)
    40:40:public static boolean isValidClientPlatform(byte)

It also does much optimization in the methods themselves.  Probably the one
that should be of most interest to you is method in-lining. Sounds like you
only want to do this when the space penalty is very low.  This can be
adjusted using system properties, only documented within Proguard's source
code.

The package to look at is proguard.optimize.peephole .  The Class is
MethodInliner.  You will see that there are 2 properties consulted for the
maximum resulting code length to decide if it should be done.  One for J2SE
& the other for J2ME.  Specifying a lower value for the J2SE version might
help.  Looking through the source of this entire directory for other system
properties may also be good.

I know of another way to get the size down, but think they are going to be
sort of "empty calories".  Here it is just in case:
use option '-repackageclasses ${single-package}'.  Use something like
'<property name="single-package" value="_"/>

Hope this helps


On 8/31/10 5:07 PM, "Panayotis Katsaloulis" <pan...@pa...> wrote:

> 
On 31 Αυγ 2010, at 8:44 μ.μ., Sascha Haeberling wrote:

> I am actually
> working on such an optimized call graph dependency optimizer, which should
> shrink down the whole compilation size.
> 
> In addition we will remove the
> need to compile classes that don't need to be compiled. E.g. all the JDK
> classes need to be compiled once and we can provide it to our users. These two
> measures should speed up compilation time a lot.
> 
> // Sascha

The problem
> has to do more with size, not time.
It is really easy to produce static
> libraries, and thus reduce compilation time to minimum.
The way it is done
> right now, though, is again object-level optimization - similar to what
> ProGuard does (and it does it extremely well - it is even able to deal with
> simple reflection cases).

The problem is with method-level optimizations.
> That is why I propose to investigate such methods. 

In which level is the
> optimization you are working
> on?
--------------------------------------------------------------------------
> ----
This SF.net Dev2Dev email is sponsored by:

Show off your parallel
> programming skills.
Enter the Intel(R) Threading Challenge
> 2010.
http://p.sf.net/sfu/intel-thread-sfd
___________________________________
> ____________
xmlvm-users mailing
> list
xml...@li...
https://lists.sourceforge.net/lists/lis
> tinfo/xmlvm-users

Re: [xmlvm-users] Java shrinkers

[Panayotis K. <pan...@pa...>]

First of all, sorry for the late answer.
Last time I thought that it couldn't be done - if it really can then
this is a nice surprise!
Do you remember the actual parameter used for this issue?
I believe if we find a solution with ProGuard, this will help a *lot*
the XMLVM progress and the whole acceptance of XMLVM, even with the
relaxed new iOS SDK agreement.

On 9/1/10, Jeff Palmer <jcp...@ro...> wrote:
> I have used Proguard, and it definitely can delete things below Class level,
> so I am not sure what you mean about method-level optimizations.
>
> If you add the '-printusage filename' to your config or Ant build.xml you
> can see everything removed.  A class name by itself means the whole class
> was delete.  If it is followed by a colon, only the stuff below was removed
> along with line numbers, e.g.
>
> myNamespace.com.stuff.morestuff.CurrentPlatform:
>     public static final byte SERVER_PLATFORM
>     public static final byte DESKTOP_PLATFORM
>     public static final byte PHONE_PLATFORM
>     32:32:public static boolean isServer()
>     34:34:public static boolean isPhone()
>     39:39:public static boolean isValidPlatform(byte)
>     40:40:public static boolean isValidClientPlatform(byte)
>
> It also does much optimization in the methods themselves.  Probably the one
> that should be of most interest to you is method in-lining. Sounds like you
> only want to do this when the space penalty is very low.  This can be
> adjusted using system properties, only documented within Proguard's source
> code.
>
> The package to look at is proguard.optimize.peephole .  The Class is
> MethodInliner.  You will see that there are 2 properties consulted for the
> maximum resulting code length to decide if it should be done.  One for J2SE
> & the other for J2ME.  Specifying a lower value for the J2SE version might
> help.  Looking through the source of this entire directory for other system
> properties may also be good.
>
> I know of another way to get the size down, but think they are going to be
> sort of "empty calories".  Here it is just in case:
> use option '-repackageclasses ${single-package}'.  Use something like
> '<property name="single-package" value="_"/>
>
> Hope this helps
>
>
> On 8/31/10 5:07 PM, "Panayotis Katsaloulis" <pan...@pa...> wrote:
>
>>
> On 31 Αυγ 2010, at 8:44 μ.μ., Sascha Haeberling wrote:
>
>> I am actually
>> working on such an optimized call graph dependency optimizer, which should
>> shrink down the whole compilation size.
>>
>> In addition we will remove the
>> need to compile classes that don't need to be compiled. E.g. all the JDK
>> classes need to be compiled once and we can provide it to our users. These
>> two
>> measures should speed up compilation time a lot.
>>
>> // Sascha
>
> The problem
>> has to do more with size, not time.
> It is really easy to produce static
>> libraries, and thus reduce compilation time to minimum.
> The way it is done
>> right now, though, is again object-level optimization - similar to what
>> ProGuard does (and it does it extremely well - it is even able to deal
>> with
>> simple reflection cases).
>
> The problem is with method-level optimizations.
>> That is why I propose to investigate such methods.
>
> In which level is the
>> optimization you are working
>> on?
> --------------------------------------------------------------------------
>> ----
> This SF.net Dev2Dev email is sponsored by:
>
> Show off your parallel
>> programming skills.
> Enter the Intel(R) Threading Challenge
>> 2010.
> http://p.sf.net/sfu/intel-thread-sfd
> ___________________________________
>> ____________
> xmlvm-users mailing
>> list
> xml...@li...
> https://lists.sourceforge.net/lists/lis
>> tinfo/xmlvm-users
>
>
>
>


-- 
Panayotis Katsaloulis

Re: [xmlvm-users] Java shrinkers

[Jeff P. <jcp...@ro...>]

Switched to an HTML format.  The forced line wrap at 80? made this hard to
read.
The lines found in MethodInliner.java are:

private static final int MAXIMUM_INLINED_CODE_LENGTH =
Integer.parseInt(System.getProperty("maximum.inlined.code.length", "8"));
private static final int MAXIMUM_RESULTING_CODE_LENGTH_JSE =
Integer.parseInt(System.getProperty("maximum.resulting.code.length",
"8000"));
private static final int MAXIMUM_RESULTING_CODE_LENGTH_JME =
Integer.parseInt(System.getProperty("maximum.resulting.code.length",
"2000"));

I have not figured out how to specify system properties with ant, but here's
what I would experiment with, added to a call to Proguard directly from the
java command line:
java -DMAXIMUM_RESULTING_CODE_LENGTH_JSE=2000 -jar proguard.jar
@myProfile.pro

You might also be able to use the following option(s) in the Proguard
profile to achieve the same result.
-microedition (& probably also)
-dontpreverify

If this does the same thing without any side effects, then it has the
advantages of both being documented, & easy to do inside of ant.  I have not
tried either for the Android Remote Assistant part of my product.  Just a
standard JSE run through Proguard with the exception of NOT specifying
-overloadaggressively
  
Dex throws a VerifyError Exception, if you do.

Activities to get this part of the system production ready will hopefully be
re-starting this week.  This has been on hold while working on an AI
goalseeker in OpenCL, which took forever.  I'll try this as well.