gamedevlists-brew

[Gamedevlists-brew] porting a database file from BREW to J2ME

[Aaron I. <ais...@ap...>]

Hello,
 
Well I figure I might as well take advantage of our new freedoms...

I'm doing a port of our Spanish/English dictionary and phrase book
application from BREW to J2ME, and I'm a bit annoyed there is no file i/o
functions in J2ME, just a stream reading function.  The compressed database
was designed to be very efficient in BREW, but I think it needs to be
redesigned from the ground up to be efficient in J2ME.  One small advantage
is that I can rely on the built in zip/jar support to do some of the
decompression work.  The database is read-only, so I don't need to mess with
RecordStores.

1) Has anyone noticed a phone that completely decompresses the JAR,
including non-class files, into memory when the app is first loaded?  That
is, do all known J2ME phones keep the resources compressed until requested?

2) Do any phones degrade in performance when large numbers of small files
are stored in the jar file?

Thank you!

-Aaron

RE: [Gamedevlists-brew] porting a database file from BREW to J2ME

[Tom H. <to...@am...>]

> -----Original Message-----
> 1) Has anyone noticed a phone that completely decompresses the JAR,
> including non-class files, into memory when the app is first loaded?  That
> is, do all known J2ME phones keep the resources compressed until
> requested?

Yes. A file is only loaded into memory when you open it. To be even more
specific, if you're loading a 100k file and using the stream mechanism to
get to a record 50k in, then 50k of the file will be loaded into memory.

So far I haven't found a phone that would let you seek to the 50k mark
without having the preceding 50k in your heap. As I figure it, this is due
to the file you're loading being zipped inside the jar, so you need to
decompress from the start to the point where you want ... you can't just
jump to it. In case it isn't obvious, this also means you have significant
performance issues if you're going to be seeking all over the place in a
file.

It might be interesting to just store the file in the jar, and not deflate
it. Maybe it would allow you do seek with good performance and without
hitting the heap if the file isn't compressed. You would need to create the
it with a tool other than jar, or modify the jar with WinZip after or
something.

> 2) Do any phones degrade in performance when large numbers of small files
> are stored in the jar file?

I haven't run into any performance issues, but there's a pretty large
overhead per file (~150 bytes) so if download size is a concern having lots
of files is a very bad thing.

Tom

Re: [Gamedevlists-brew] porting a database file from BREW to J2ME

[Tom P. <bre...@ac...>]

> > 1) Has anyone noticed a phone that completely decompresses the JAR,
> > including non-class files, into memory when the app is first loaded?  That
> > is, do all known J2ME phones keep the resources compressed until
> > requested?
> 
> Yes. A file is only loaded into memory when you open it. To be even more
[...]
> It might be interesting to just store the file in the jar, and not deflate
> it. Maybe it would allow you do seek with good performance and without

No, I'm assuming that Aaron is already asking whether it's
okay to leave the file decompressed in the JAR, because he's
worried that if the phone decompressed the JAR then it'd be
too big.

Nextel iDEN phones definitely decompress the JAR upon install.
Install is a different step from download. The classes go into
a program space, and resources go into a data space. There are
iDEN-specific JAD properties for the decompressed size, e.g.:

   iDEN-Data-Space-Requirement: 112
   iDEN-Program-Space-Requirement: 157

where the values are in KB.

Offhand, I don't know of other phones that decompress the JAR,
but I wouldn't assume Motorola is not or will not be the only
one to use this optimization for J2ME performance.
--t

RE: [Gamedevlists-brew] porting a database file from BREW to J2ME

[Aaron I. <ais...@ap...>]

Thank you Tom P and Tom H for your insights.

I imagine that I can speed up the seek process by breaking up the database
into several smaller chunks, each stored as a separate file in the jar (but
not too small, as Tom H points out).  In addition, I had already planning on
reworking the record search algorithm so it didn't have to do any reverse
seeks, which I could tell would be inefficient from the stream API.

Given that some phones decompress the application at install time, as Tom P
explains, it seems that I can't rely on the zip compression in the jar file
alone.  

Is there sneaky way to use the built-in zip decompressor that is required
for decompressing the jar and png files?  I was thinking of putting the data
in a fake PNG 24-bit image, but I don't see a way to read it.  It would be a
shame to have to bring along a ZIP decompressor in my jar, just because they
didn't expose it in the MIDP API.

-Aaron

> -----Original Message-----
> From: gam...@li... 
> [mailto:gam...@li...] On 
> Behalf Of Tom Park
> Sent: Saturday, December 11, 2004 5:32 PM
> To: gam...@li...
> Subject: Re: [Gamedevlists-brew] porting a database file from 
> BREW to J2ME
> 
> > > 1) Has anyone noticed a phone that completely 
> decompresses the JAR, 
> > > including non-class files, into memory when the app is 
> first loaded?  
> > > That is, do all known J2ME phones keep the resources compressed 
> > > until requested?
> > 
> > Yes. A file is only loaded into memory when you open it. To be even 
> > more
> [...]
> > It might be interesting to just store the file in the jar, and not 
> > deflate it. Maybe it would allow you do seek with good 
> performance and 
> > without
> 
> No, I'm assuming that Aaron is already asking whether it's 
> okay to leave the file decompressed in the JAR, because he's 
> worried that if the phone decompressed the JAR then it'd be too big.
> 
> Nextel iDEN phones definitely decompress the JAR upon install.
> Install is a different step from download. The classes go 
> into a program space, and resources go into a data space. 
> There are iDEN-specific JAD properties for the decompressed 
> size, e.g.:
> 
>    iDEN-Data-Space-Requirement: 112
>    iDEN-Program-Space-Requirement: 157
> 
> where the values are in KB.
> 
> Offhand, I don't know of other phones that decompress the 
> JAR, but I wouldn't assume Motorola is not or will not be the 
> only one to use this optimization for J2ME performance.
> --t
> 
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide Read honest & 
> candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now. 
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Gamedevlists-brew mailing list
> Gam...@li...
> https://lists.sourceforge.net/lists/listinfo/gamedevlists-brew
> 

Re: [Gamedevlists-brew] porting a database file from BREW to J2ME

[Tom P. <bre...@ac...>]

> Is there sneaky way to use the built-in zip decompressor that is required
> for decompressing the jar and png files?  I was thinking of putting the data
> in a fake PNG 24-bit image, but I don't see a way to read it.  It would be a
> shame to have to bring along a ZIP decompressor in my jar, just because they
> didn't expose it in the MIDP API.

Oh, have you not tried this method?

   InputConnection conn = Connector.open("jar:/dictionary.jar", Connector.READ);
   DataInputStream dis = conn.openDataInputStream();

Just kidding. It's annoying that they didn't expose an
inflate in MIDP2. They did add Image.getRGB, but that
still doesn't help because the values can be altered
to phone-safe colors.

You said you already have a decompressor, but you might
be interested in looking at this one:
http://www.java4ever.com/index.php?section=j2me&project=apime&menu=download&lang=en

> Given that some phones decompress the application at install time, as Tom P
> explains, it seems that I can't rely on the zip compression in the jar file
> alone.  

Well, you could make a separate version for iDEN phones.
--t

Re: [Gamedevlists-brew] porting a database file from BREW to J2ME

[David P. <da...@ma...>]

What i usually do in my java games, is to split up the big resources (>80kb) 
into small ones, ~20kb each. Therefore i dont end up having too many files 
(which would increase the jar/zip header), and it also reduces the memory 
overhead when opening the filestream. Never had to use any compression algo, 
appart for rle8/16 for some maps..


----- Original Message ----- 
From: "Tom Park" <bre...@ac...>
To: <gam...@li...>
Sent: Sunday, December 12, 2004 2:09 AM
Subject: Re: [Gamedevlists-brew] porting a database file from BREW to J2ME


>> Is there sneaky way to use the built-in zip decompressor that is required
>> for decompressing the jar and png files?  I was thinking of putting the 
>> data
>> in a fake PNG 24-bit image, but I don't see a way to read it.  It would 
>> be a
>> shame to have to bring along a ZIP decompressor in my jar, just because 
>> they
>> didn't expose it in the MIDP API.
>
> Oh, have you not tried this method?
>
>   InputConnection conn = Connector.open("jar:/dictionary.jar", 
> Connector.READ);
>   DataInputStream dis = conn.openDataInputStream();
>
> Just kidding. It's annoying that they didn't expose an
> inflate in MIDP2. They did add Image.getRGB, but that
> still doesn't help because the values can be altered
> to phone-safe colors.
>
> You said you already have a decompressor, but you might
> be interested in looking at this one:
> http://www.java4ever.com/index.php?section=j2me&project=apime&menu=download&lang=en
>
>> Given that some phones decompress the application at install time, as Tom 
>> P
>> explains, it seems that I can't rely on the zip compression in the jar 
>> file
>> alone.
>
> Well, you could make a separate version for iDEN phones.
> --t
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Gamedevlists-brew mailing list
> Gam...@li...
> https://lists.sourceforge.net/lists/listinfo/gamedevlists-brew
>