j2me zipfile reading patch (no permission nags anymore)
Brought to you by:
james22
Menu
▾
▴
#59 j2me zipfile reading patch (no permission nags anymore)
This patch allows users to have a .jar created by Osm2GpsMid on the StorageCard/MMC. It handles ZipFiles using minimal, but complete code based on the APPNOTE.TXT by pkware. The ZipFile class is written from scratch, ZipEntry/ZipConstants were taken from GNU Classpath, but modified quite a bit.
This patch save us from annoying permission messages and preserves space on the storage card!! The zip file is opened once and the file connection is held open throughout the usage of GpsMid. It was tested _thoroughly_ using a (crapy) 6230i phone, so the class is carefully crafted to be fast and avoid memory hogging. While it is a tiny bit less speedy than from phone memory, it is surprisingly stable (on the 6230i I tested with, I hope you can confirm it).
IMPORTANT NOTE (will save you hours of fiddling): Nokia (at least on the 6230i) prevents loading files that have ".jar" as file extension, so you need to rename the jar (I chose "osmdata", w/o extension). If you do not rename your jar file, an Exception will be thrown: "invalid filename" or "filename not found" or similar..
Have fun,
cmuelle8
(trendypack)
j2me zipfile reading patch (no permission nags anymore)
Thank you very much for this amazing patch!!!
On my new mobile sometimes it seems to render even faster than from the built-in map. So your code seems to be rather efficient :-))
Unfortunately it doesn't work on my old mobile - an SE K750i. It still nags a few times with security requesters for reading but finally does not get to the search or map screen.
So we probalbly still should support both - map directories and zip maps. Changing the "Select directory" command to a "Select map zip/dir" should be possible, I think.
Would you like to implement this as well? Thus we could incorporate your patch earlier to CVS.
I'll add it to the menuMapSrc stuff, so that one can fall back to the uncompressed files. But I really feel we should make it work this method with the SE K750i .. unfortunately I don't have this phone (in fact, I don't have any SE phone)
Can you uncomment the fc2/log stuff in ZipFile and post the log.txt it writes to the storage card (gpsmid needs write permissions obviously, if you activate this).. I know, a better way would be to use your logging functionality you use throughout GpsMid, but I had many reboots of the 6230i, so it needed to be really minimal during testing (this is also the reason for the many log.flush()'s appear)..
I'll post an updated patch this evening..
Greetings,
Christian / cmuelle8
Hi Christian,
here are my results testing on K750i with the log stuff:
The midlet behaves different now (in addition to the still existing security requests) -
on the Splash screen it gives this error:
Failed to load basic configuration! Check your map date source:null
When trying to go to the map it says:
Failed to display map: java.lang.NullPointerException:null
The log.txt tells the following:
beforereopenMETA-INF/MANIFEST.MF
attempting391done391Manifest-Version:1.0
MIDlet-Name:GpsMidTest
MIDlet-Version:0.4.55
MIDlet-Vendor:HaraldMueller
MIDlet-Icon:/GpsMid.png
MIDlet-Info-URL:http://gpsmid.sourceforge.net
MIDlet-1:GpsMidTest,/GpsMid.png,de.ueller.midlet.gps.GpsMid
MIDlet-Delete-Confirm:Doyoureallywanttokillme?
MicroEdition-Configuration:CLDC-1.1
MicroEdition-Profile:MIDP-2.0
Polish-Version:2.0.1(2008-03-03)
afterreopenMETA-INF/MANIFEST.MF
beforereopenMETA-INF/MANIFEST.MF
attempting391done391Manifest-Version:1.0
MIDlet-Name:GpsMidTest
MIDlet-Version:0.4.55
MIDlet-Vendor:HaraldMueller
MIDlet-Icon:/GpsMid.png
MIDlet-Info-URL:http://gpsmid.sourceforge.net
MIDlet-1:GpsMidTest,/GpsMid.png,de.ueller.midlet.gps.GpsMid
MIDlet-Delete-Confirm:Doyoureallywanttokillme?
MicroEdition-Configuration:CLDC-1.1
MicroEdition-Profile:MIDP-2.0
Polish-Version:2.0.1(2008-03-03)
afterreopenMETA-INF/MANIFEST.MF
So this actuallly seems to work?!
Btw, there's high demand for a functionality like this,
so it should be easy to get more testers:
- on Blackberry (they need a non-obfuscated midlet): http://sourceforge.net/forum/message.php?msg_id=7172241
- on Nokia 3120: http://sourceforge.net/forum/message.php?msg_id=5455577
...and many more I don't find again right now.
Yes,
parsing the file works,
reopening the fileconnection more than one time works and
reading out files works too (the MANIFEST.MF in this test case).
So all three steps where something could go wrong are done successfully. If you commented out my test code, it is normal that you get the null pointer stuff, since I do <<contents = new HashMap();>> at the end of the ZipFile constructor, which destroys the parsed entries again (don't ask, it kept my 6230i from dying while testing - this also resulted in the synchronized statement for getInputStream(), though I did not try w/o this now that everything else works, I suppose synchronized is necessary..).
So how come it does not work for the 750i? .. weird. I modified the testing code in the constructor a bit to print out the file listing to log.txt too.. feel free to also try w/o the last statement that kills the contents. If you don't clear the contents HashMap messages should be the same you experienced the first time testing..
>>-----SNIP
if (!(readEntries()>0))
throw new IOException("no entries in file");
/*
InputStream tmp;
for (int v, i=2; i!=0; i--) {
log.writeChars("before reopen META-INF/MANIFEST.MF\n\n\n"); log.flush();
tmp = getInputStream(getEntry("META-INF/MANIFEST.MF"));
while ((v=tmp.read())!=-1)
log.writeChar(v);
tmp.close();
tmp = null;
}
log.writeChars("before reopen for printing index\n\n\n"); log.flush();
for (de.enough.polish.util.Iterator i = contents.keysIterator();i.hasNext();)
log.writeChars(((String) i.next())+((ZipEntry)contents.get(s)).getSize());
log.close();
log = null;
contents = new HashMap(); // destroy the HashMap just built
*/
----->>SNIP
Greetings..
When compiling this I get:
[javac] Internal J2ME Polish class: C:\j2me\eclipseworkspace\GpsMid\build\real\Generic\full\en_US\source\net\sourceforge\util\zip\ZipFile.java:68: cannot find symbol
[javac] symbol : variable s
[javac] location: class net.sourceforge.util.zip.ZipFile
[javac] i.next())+((ZipEntry)contents.get(s)).getSize());
Ok, replaced s with (String) i.next(), now the output is like below.
So this might just be some kind of hard to find
timing / threading issue of GpsMid rather than a problem
of your unzipping code. :/
beforereopenMETA-INF/MANIFEST.MF
attempting391done391Manifest-Version:1.0
MIDlet-Name:GpsMidTest
MIDlet-Version:0.4.55
MIDlet-Vendor:HaraldMueller
MIDlet-Icon:/GpsMid.png
MIDlet-Info-URL:http://gpsmid.sourceforge.net
MIDlet-1:GpsMidTest,/GpsMid.png,de.ueller.midlet.gps.GpsMid
MIDlet-Delete-Confirm:Doyoureallywanttokillme?
MicroEdition-Configuration:CLDC-1.1
MicroEdition-Profile:MIDP-2.0
Polish-Version:2.0.1(2008-03-03)
beforereopenMETA-INF/MANIFEST.MF
attempting391done391Manifest-Version:1.0
MIDlet-Name:GpsMidTest
MIDlet-Version:0.4.55
MIDlet-Vendor:HaraldMueller
MIDlet-Icon:/GpsMid.png
MIDlet-Info-URL:http://gpsmid.sourceforge.net
MIDlet-1:GpsMidTest,/GpsMid.png,de.ueller.midlet.gps.GpsMid
MIDlet-Delete-Confirm:Doyoureallywanttokillme?
MicroEdition-Configuration:CLDC-1.1
MicroEdition-Profile:MIDP-2.0
Polish-Version:2.0.1(2008-03-03)
beforereopenforprintingindex
bear.amr13761d421.d9963d425.d268y.class491d27.d5346t23.d5452street.png72left.png557d41.d259t41.d97t410.d20straighton.amr641t49.d596x.class114t421.d1757t429.d418satelit.png1892aj.class1730crossing_small.png1074t443.d470cafe.png938t454.d929c2.d6154t461.d461t469.d799t472.d56mark_display.png4524t483.d434s12.d830an.class38c23.d119c27.d391t498.d426lighthouse.png896telephone.png1964c30.d3534de/ueller/midlet/gps/importexport/BtObexServer.class2014s34.d145dc.class1414c45.d66c49.d192am.class442c52.d19779c56.d1821bw.class581c63.d1847db.class3114s63.d1334s67.d232c70.d884t4113.d5594s74.d749s78.d520c81.d1532library.png4284s81.d128c89.d2504c96.d16s96.d944c107.d1134c114.d3524d210.d111c121.d894df.class893leave_motorway.amr597ap.class2453into_tunnel.amr535bz.class2141t210.d8991terminal.png6383ao.class11820t225.d11383by.class761de/ueller/midlet/gps/importexport/FileExportSession.class8907t236.d136t243.d10010t250.d267t254.d7089pharmacy.png8301t265.d18135t272.d13472as.class15788names-0.dat1382ar.class188dg.class2743COPYING175again.amr282continue.amr799de/ueller/midlet/gps/importexport/Jsr172GpxParser.class126connect.amr1628laundry.png6754prepare.amr577d417.d160soon.amr6434d420.d14170d424.d225d26.d8849t26.d984cinema.png563beacon.png1047dict-4.dat438d44.d97t44.d29t48.d1766di.class677t424.d1217helipad.png254t435.d686hospital.png272straighton.png1550t453.d1469c1.d8045c5.d1061c9.d38t464.d261t468.d1896t471.d235t475.d1849t479.d15dm.class349c11.d497nightclub.png4274t486.d5129c19.d4064t493.d3904s22.d1125t497.d590dl.class1854c37.d228fuel.png268dict-3.dat4924c40.d794c48.d182s48.d9068c51.d4504s51.d2204dk.class3004t4101.d2244s62.d1811s66.d1271c73.d1082c77.d685t4116.d72c80.d4474t4123.d3024s84.d59beach.png344c95.d3774c102.d217c106.d3660c113.d930c117.d485target.png102c120.d314ay.class517cd.class671post_box.png230dict-2.dat495t213.d6143biergarten.png545t220.d15595t228.d521t231.d6788t239.d3179t242.d9580motorway_exit.png305ch.class7369t257.d8speed_limit.amr3924t260.d6921GpsMid.png3216t271.d19485t279.d5267t282.d391dq.class552cf.class225de/ueller/midlet/gps/importexport/CommGpxImportSession.class5056d.class286fastfood.png266c.class8201cj.class638dt.class3606t03.d151LCD_horiz.png97LCD_vert.png202t14.d577d423.d205ds.class225t21.d299t25.d7373tram.png225exchange.png305d43.d603t43.d1487t47.d254services.png614t427.d3580t430.d1397t438.d227t441.d1622t449.d269e.class1046bb.class1127c4.d44cl.class1019t467.d185t470.d371t478.d568c10.d1829c14.d1541c18.d47s18.d5926castle.png4924t492.d4944s21.d1469c29.d208du.class253c32.d4714s32.d208kiosk.png4684c47.d156s47.d4096th.amr460c50.d1614c58.d134s58.d644t4100.d30960c65.d380c69.d1244be.class1674200.amr4964t4111.d124s72.d1910butcher.png1461s76.d6055th.amr3424t4122.d2114s83.d99g.class12657c90.d364c98.d18atm.png22c101.d4224c109.d831cn.class266c112.d3054d212.d907d216.d297c123.d90post_office.png2604600.amr165aerodrome.png163t212.d6144500.amr6556t227.d331t230.d9083rd.amr11970t238.d5167bg.class7513t245.d487t249.d3624legend.dat15692follow_street.amr9128theater.png258t263.d5414t267.d4916t270.d221t274.d4611t281.d793800.amr3839bj.class307700.amr3243diy_store.png137icecream.png1395bi.class562cs.class319bh.class214t02.d513d411.d19603zoo.png614d415.d1181d419.d10969d422.d151theme_park.png282d24.d73t24.d7999p.class53742d42.d10403d46.d1370t411.d1766ab.class1541t419.d1040bl.class1649t426.d2924t433.d290aa.class1595t444.d1028n.class1052t451.d1721t459.d4095c3.d4374t462.d830t473.d155t480.d74t484.d5394t488.d2234t491.d3164t495.d552c28.d514t499.d1801c31.d308c35.d120c39.d1402c42.d4494s42.d382ad.class439c53.d4234s53.d565q.class21149c60.d1734t4103.d2104s64.d1928s68.d281cx.class4894t4110.d2954t4114.d2294s75.d1451s79.d499disconnect.amr908c82.d1109c86.d321t4125.d149bm.class3164c97.d52to.amr1834cw.class964c108.d2794c115.d1714d211.d266d215.d3894monument.png5733t.class3502out_of_tunnel.amr8802t215.d5t219.d12127af.class15388s.class3253t233.d6409cz.class7048hotel.png11826t248.d312t251.d3839t259.d1235bo.class12391t266.d3313table_tennis.png19871t277.d17597railstation.png1989charMap.txt346city.png156camping.png240v.class5616toilets.png1347u.class2283tower.png353t01.d151t05.d201d414.d255
Ooops :-/
I'd say we found a sourceforge bug regarding display of the comment section ;-) ..
sorry for the untested code post.. (I had it different before.., changed too fast..)
you need to i.next() only once though, in the loop, to get a proper file listing - so
String s = i.next().toString();
in the while loop would do, if you do i.next() twice you get the file intermixed with the size of the following file ;-) .. and maybe put a + "\n" I forgot for prettiness..
apart from that, this should not matter at all here - it delivered what we wanted to see. I would suggest that maybe the Java VM of the 750i handles synchronized stuff worse than my 6230i (which really works awesome - tested for hours, map and search, not a single issue..) - but this is just a very vague guess.
i can't think of anything else though - the tests code you ran should let us derive the following:
- file skipping works
- file reading works
- reopening the InputStream on a single FileConnection object works
_maybe_ the 750i runs the GC at different times, so that sometimes a closed inputstream is still open, you should be able to stress test this by increasing the number in the for loop from i=2 to i=200 .. and see if that triggers any issues .. i would just read out the file 190 times, not writing to log (so there is a higher open/close frequency), and put stuff to log the last 10 times..
actually synchronized should work on the 750i - you have lots of threading in gpsmid and that works too, doesn't it?
i think it's simple in the end, but not easy to spot as of now (as always ;-)
good nite,
Christian
oh, and you can print to log the name too in
getEntry() method,
just to see what getMapResource() gives it - this should not be different (e.g. I get urls of the form "/keyMap.txt" on my mobile), since, I suppose, you build the resource strings yourself in the code,
so that should not be the issue either
ps: did not get to GuiDiscover modifications yet, bear with me .. it should not be to hard, I guess, but don't think it's time critical - I like "when it's done" as a timeframe ;-) .. I'll post back..
> if you do i.next() twice you get the file
> intermixed with the size of the following file
Yes, but that saved us not garbling up the tracker a bit more ;)
I've not done your additional tests yet -
the thing is I get all the time reading permission requesters.
But I've made yet another observation - testing with 3 different zip files:
1) a 500 kb midlet - seems to work completely
2) a 1500 kb midlet - map works at least in the regions I tested (security requesters when panning), GuiSearch works partially. I get reproducibly a NullPointer exception
when I search for "55", but it works when I search for "44".
3) a 20 MB midlet - doesn't work at all. This is the one I've initially tested with.
So we've actually two K750 issues:
- probably a size limitation of the zip file
- the security requesters for every file opened from the zip file
Sweet patch! I haven't yet had a chance of applying the patch and testing it, but I am looking forward to doing so. Never-the-less the chance of getting proper filesystem support for map data just made me want to quickly add a positive comment here! So thanks for posting this patch! I can test it here on a SE K550 (JP-7.3) and a Nokia 6301 (S40 5th FP1) and will report if things work well or what issues there are.
I have had a chance to give it a quick test on my phones and it does seem to work nicely here. Neither the K550 nor the 6301 ask for the permissions dialog more than once, even if the permissions are set to ask always, so the reuse of the connection works fine on these phones. I haven't done any direct speed tests, but the performance seems comparable to that of the built-in maps on both my phones, so hopefully there is a large class of phones on which this works well and in most cases quite a bit better than the unpacked directory. On my 6301 I have also this way for the first time been able to try out larger maps of several MB. Although the jvm at least doesn't put an artificial size limit on the jar size of 1Mb, as do several other models, all java applets get stored on the phone, rather than the memory card and so you run out of space after about 10Mb of midlet.
So all together this looks like a really nice patch and I think we should try and get this in as soon as possible for all those who have been wanting this feature for a long time. Given that sk750 still has some problems with this and that potentially others might be using the map directory, perhaps we could keep the old code and depending on if one selects a directory or a file use one or the other.
I will have a look at the actual patch soon and see if I can do a quick code review. What I have noticed is that the code doesn't compile with the Generic-minimal setting any more, as some preprocessor directives were broken by comments and others are missing around the filesystem access, but those are minor points and should be easily fixable
Altogether many thanks for this great patch
I'm working out the configuration issues at the moment .. and will post back a revised patch then (I'm touching Configuration.java and GuiDiscover.java) .. it will provide uncompressed loading and the zipfile way (depending wheter you select a directory or a file in the browser)
there is one issue I still have with the record store or the code not keeping my selection between restarts, but it could also be a messed up recordstore on the 6230i I'm testing with.. so once that is resolved I'll upload it
be well,
Christian
In the meanwhile I've found out more about when there's the NullPointer Exception
(in the 1500kb midlet when pressing 55 in the Search screen):
No names match the search string 55 in the midle,
thus there is no s55.d file in the zip. So it seems the requested but
not found file leads to the NPE.
Did you try the 1500kb midlet on a different phone from phone memory? Maybe this is a bug unrelated to the patch.. Is the s55.d present if you look into the zip on your desktop computer?
The s55.d file does not exist in the zip. The midlet works fine when used
from phone memory.
I've looked further now and it seems the line
is = zipfile.getInputStream(zipfile.getEntry(name));
in the Configuration class passes the NullPointer returned from getEntry() on to zipfile.getInputStream() which does not handle Nullpointers.
> there is one issue I still have with the record store or the code not
> keeping my selection between restarts
Is the additional information you want to store
if the mapFileUrl is a file or a directory?
If you use setCfgBitState(byte bit, boolean state, boolean setAsDefault)
for this purpose, setAsDefault must be true.
There's also a small update on:
3) a 20 MB midlet - doesn't work at all. This is the one I've initially
tested with.
Actually the running midlet is able to read the legend.dat file from the external midlet.
The legend.dat contains the midlet version and date and it is displayed in the startup screen. However any subsequent zip accesses do not seem to work. When pressing
"Map" the midlet does nothing for a while (no security requester or anything).
I've added some more log output for getEntry() and it actually seems to try
to read only the legend.dat and nothing more:
afterreopenMETA-INF/MANIFEST.MF
getEntry/legend.datfromoffset3637777
attempting6608done6608
At least it shows legend.dat from offset 3,637,777 could be read successfully.
Here's the additional logging I've added, maybe you can
also put it commented out in the midlet so it is available later:
public ZipEntry getEntry(String name) {
ZipEntry z = (ZipEntry) contents.get(
name.charAt(0)=='/'
?name.substring(1)
:name);
try {
if (z != null) {
log.writeChars("getEntry " + name + " from offset " + z.offset + "\n" ); log.flush();
} else {
log.writeChars("getEntry " + name + " failed\n" ); log.flush();
}
} catch (IOException e) {
;
}
return z;
}
public synchronized InputStream getInputStream(ZipEntry e) throws IOException {
InputStream ret;
if (e == null) {
return null;
}
byte [] b = new byte [(int)e.getCompressedSize()+8];
int i = b.length-8;
log.writeChars("attempting "+e.getSize()); log.flush();
ret = fc.openInputStream();
int skipped = (int) ret.skip(e.offset);
if (skipped != e.offset) {
log.writeChars(e.getName() + ": asked to skip " + e.offset + " but skipped " + skipped + "\n" ); log.flush();
}
int bytesRead = ret.read(b, 0, i);
if (bytesRead != i) {
log.writeChars(e.getName() + ": asked to read from offs " + e.offset + " " + i + " bytes but read " + bytesRead + "\n" ); log.flush();
}
ret.close();
ret = null;
log.writeChars("done "+e.getSize()); log.flush();
...
>I've looked further now and it seems the line
>is = zipfile.getInputStream(zipfile.getEntry(name));
>in the Configuration class passes the NullPointer returned from getEntry()
>on to zipfile.getInputStream() which does not handle Nullpointers.
Yes, this is already handled now in getMapResource in my code base (I need a try catch there anyway for the uncompressed access - so this is a good place to handle the possible exception from getEntry too)
Good thing to know though, that GpsMid might request files that do not actually exist - thanks for your investigation on that..
at the moment getMapResource looks like the following, but I still have problems with the record store stuff mentioned earlier (despite your post with the cfgbit):
public static InputStream getMapResource(String name) throws IOException {
InputStream is = null;
if (mapFromJar) {
is = QueueReader.class.getResourceAsStream(name);
} else {
//#if polish.api.fileconnection
try {
if (mapFileUrl.endsWith("/")) {
// directory mode
name = mapFileUrl + name.substring(1);
FileConnection fc = (FileConnection) Connector.open(name, Connector.READ);
is = fc.openInputStream();
}
else {
// zipfile mode
if (mapZipFile == null)
mapZipFile = new ZipFile(mapFileUrl);
is = mapZipFile.getInputStream(mapZipFile.getEntry(name.substring(1)));
}
} catch (Exception e) {
//#debug info
logger.info("Failed to open: " + name);
throw new IOException(e.getMessage());
}
//#else
logger.fatal("missing JSR75, no filesystem support");
//#endif
}
return is;
}
oh, if you already want to use that last code snippet, you need to change getEntry too (just use name there, not name.substring(..) stuff -
this is better anyway, as stripping the leading '/' should not belong to the ZipFile class (think of an entry in the ZipFile that has a leading slash - you couldn't retrieve it then - stripping the leading '/' needs to be done in getMapResource as it is special case behavior needed for GpsMid only
I've added the check for (e != null) in ZipFile.getInputStream(), despite the exception handling done in getMapResource() too..
even it is not necessary for GpsMid with the current implementation of getMapResource(), other j2me developers reusing this class will surely appreciate this check.., so instead of throwing an NPE, getInputStream gracefully returns null, if it's input is null..
again, muchos gracias for pointing this out..
> the record store or the code not keeping my selection between restarts,
> but it could also be a messed up recordstore on the 6230i I'm testing with..
Should I give the current version of your patch a try on my mobile
so we can see if the 6230i recordstore is ok?