Menu
▾
▴
Re: [JPP-Devel] [Fwd: OpenJump, Shapefile and Dbase]
|
From: Larry B. <bec...@gm...> - 2010-07-06 00:46:21
|
Hi Michaël, Thanks for the testing. I agree with your analysis that something more needs to be done to keep users from losing precision. I think perhaps that your idea of computing the maximum needed width for numeric fields may be the best plan. Of course, it will be difficult to determine the needed digits to the right of the decimal point for some floating point values. If you want, go ahead and revert my change and I"ll work some more on it next month. regards, Larry 2010/7/4 Michaël Michaud <mic...@fr...> > Hi, > > Thanks Larry. I tested it and the change works as you describe. > I just want to warn that the change may have side effects and make sure > that these side effects are acceptable for users. > > Now, numeric attribute one can see in OpenJUMP may be truncated while > using Save Selected Datasets (ex. change 9999 to 10000 in a field of > width 4 : you can see 10000 in OpenJUMP, but finally save '0000' in the > dbf file). > IMHO, OpenJUMP default behaviour is better until we have a full support > of attribute length to prevent or to warn users from this kind of side > effect. > > A solution which may be halfway between default behaviour and your patch > is to compute numeric field width as we do for characters (computing the > maximum width useful to write all the values of a layer - with a min > width of 1). Or to mix this solution with your patch so that a field > cannot be narrowed but can be enlarged... > > my 2 cents > > Michaël > > Larry Becker a écrit : > > Hi, > > > > I have committed an experimental change regarding this post. I > > modified ShapefileWriter writeDbf() to read the original dbf file > > header and use the existing numeric field definitions. This occurs > > mostly with Save Selected Datasets, but will also happen when you Save > > Dataset as and overwrite the original file. If you save to a new > > path, your original numeric field widths will be changed to OJ defaults. > > > > The rule for when the original dbf numeric field size is used is > > simply this: if the name has not changed and the type is still numeric. > > > > Please let me know if this causes any problems! I'll be away from my > > development computer for the next three weeks, but will be following > > email. I'm sure if it turns out to be bad, Stefan or someone will > > revert it. > > > > regards, > > Larry > > > > On Wed, Jun 23, 2010 at 9:16 AM, Larry Becker <bec...@gm... > > <mailto:bec...@gm...>> wrote: > > > > Excellent points! Let's see if we can address them. > > > > > > how do you know that the dataset you want to "save" or "save > > as" still > > has the same schema as the original dbf ? > > > > > > I guess the first thing to do is to have the code verify that the > > original column names have not changed., and if they have then > > abandon the attempt and revert to the current technique. Column > > names would be used as keys in a HashMap of DbfFieldDefs . We > > could probably allow some data type changes, and deleting or > > adding columns (but not both deleting and adding). The rule could > > be to continue if either the set of original column names or the > > set of new column names is a superset of the other (I.e. one > > completely contains the other), although this is debatable and > > other rules are possible. > > > > Your example of layer a being saved over file b is confusing, > > true. It would be confusing in the current method. All that is > > going to change in the proposed code is that the column > > specification will not change (assuming layer b has the same > > columns as layer a). > > > > regards, > > Larry > > > > 2010/6/23 Michaël Michaud <mic...@fr... > > <mailto:mic...@fr...>> > > > > Hi, > > > > > > Where do you want to store dbf header information > > exactly ? In the > > > > > > DriverProperties ? > > > > > > I'm not going to store the header info. I'm going to detect > > that the > > > file exists before saving over it and read the header first. > > Ah, OK, I see now why you want to know when the dbf is deleted. > > So, here are a few questions related to your approach : > > how do you know that the dataset you want to "save" or "save > > as" still > > has the same schema as the original dbf ? > > - if you load a.shp, then add a column to the schema, then > > "save dataset" > > - if you load a.shp, then edit a user attribute from Integer > > to String > > then "save dataset" > > > > Also here is a brief sequence to understand how "save dataset" > > and "save > > layer as" work (IMHO works well, but can be a bit confusing if > > user is > > not perfectly aware about what he does) > > - load a.shp, > > - save layer b as a.shp : a.shp is replaced by b data > > - modify a layer > > - save a layer dataset : a.shp is replaced again by a data > > (note that if > > a has not been modified, a.shp still contains b data) > > > > Michaël > > > > > > Larry > > > > > > 2010/6/22 Michaël Michaud <mic...@fr... > > <mailto:mic...@fr...> > > > <mailto:mic...@fr... > > <mailto:mic...@fr...>>> > > > > > > Larry Becker a écrit : > > > > Hi Michaël, > > > > > > > > The approach I'm taking to reach the same goal is > > quite different. > > > > If it works, the modification will only be in one method: > > > > ShapefileWriter.writeDbf(FeatureCollection > > featureCollection, String > > > > fname). What I intend is to read fname's dbf header > using > > > > DbfFile(String file), if the file exists, and use the > > DbfFieldDefs > > > > contained within. At this point I don't know when the > > "Replace > > > > existing file" dialog occurs, and if it deletes it before > > > writeDbf is > > > > called. Do you see any other pitfalls of this approach? > > > Yes, the shapefile only solution will be more simple to > > implement. > > > I did > > > not understand why you worry about when "Replace > > existing file" > > > occurs. > > > Where do you want to store dbf header information > > exactly ? In the > > > DriverProperties ? > > > > > > Michaël > > > > > > > > regards, > > > > Larry > > > > > > > > 2010/6/22 Michaël Michaud <mic...@fr... > > <mailto:mic...@fr...> > > > <mailto:mic...@fr... > > <mailto:mic...@fr...>> > > > > <mailto:mic...@fr... > > <mailto:mic...@fr...> > > <mailto:mic...@fr... > > <mailto:mic...@fr...>>>> > > > > > > > > Hi Larry, > > > > > > > > As you go on working about this feature, I just > > want to let you > > > > know the > > > > work I did in 2004 to add new attribute types in JUMP > > > > You'll find a patch and a complete doc here : > > > > > > http://michael.michaud.free.fr/jump/plugins/plugins.html (hoping > > > > you can > > > > read french) > > > > My concern was exactly "not to change input schema". > I > > > abandonned it > > > > because my concern changed, and because I could > > not update > > > every piece > > > > of JUMP code that should have been updated to > > take new > > > attribute > > > > types > > > > into account (especially database-access, but not > > only). > > > > > > > > Michaël > > > > > > > > Larry Becker a écrit : > > > > > I agree with Michaël. This isn't really a bug, > > but it is > > > annoying to > > > > > have your DBF header modified. I have already > > spent some > > > time > > > > trying > > > > > to implement a "don't modify the DBF header" > > approach to > > > > > SaveSelectedDatasets, but it seems to be a > difficult > > > > modification and > > > > > I haven't gotten very far on it yet. The idea > > is to read the > > > > existing > > > > > header and associate with the attributes being > > saved (by name) > > > > > whenever possible, and convert using custom > > format specifiers. > > > > > > > > > > I'll keep working on it because this is an > important > > > feature since > > > > > shape files are OJ's primary dataset. > > > > > > > > > > regards, > > > > > Larry > > > > > > > > > > 2010/6/22 Michaël Michaud > > <mic...@fr... <mailto:mic...@fr...> > > > <mailto:mic...@fr... > > <mailto:mic...@fr...>> > > > > <mailto:mic...@fr... > > <mailto:mic...@fr...> > > > <mailto:mic...@fr... > > <mailto:mic...@fr...>>> > > > > > <mailto:mic...@fr... > > <mailto:mic...@fr...> > > > <mailto:mic...@fr... > > <mailto:mic...@fr...>> > > <mailto:mic...@fr... <mailto:mic...@fr...> > > > <mailto:mic...@fr... > > <mailto:mic...@fr...>>>>> > > > > > > > > > > Hi, > > > > > > > > > > Would be interesting to have more details. > > > > > Different bytes surrounded in your pdf file > > is related > > > to a > > > > recent > > > > > discussion with Larry on the forum. > > > > > > > > > > OpenJUMP does not support length/decimal > > count in > > > attribute > > > > > definition. > > > > > It exports : > > > > > integers to 11 bytes long dbf field (thanks > > to Larry > > > who reduced > > > > > it from > > > > > 16 to 11) > > > > > doubles to 33 bytes long dbf field including > > 16 decimal > > > > digits (hence > > > > > the 21 10 bytes from OJ dbf instead of 0c 03 > > which > > > means 12 > > > > bytes > > > > > including 3 decimal digit) > > > > > > > > > > It is not easy to workaround this, but it > > can not > > > considered > > > > as a bug. > > > > > OpenOffice.org calc can read these files > > very well and > > > write > > > > the field > > > > > length this way : > > > > > before : > > > > > poiShapeID,N,4,0 poiTypID,N,4,0 > > poiName,C,50 > > > > > > > > > > after > > > > > poiShapeID,N,11,0 poiTypID,N,11,0 > > > poiName,C,32 > > > > > > > > > > > > > > > Notice that the character field has also > > been changed > > > because > > > > > there were > > > > > no field with more than 32 characters in the > > dataset > > > > > > > > > > If OpenJUMP dbf cannot be uploaded to access > > with > > > Borland-BDE, I > > > > > suppose > > > > > there is another problem. > > > > > I do not understand what is the matter with > > the length of > > > > file names. > > > > > > > > > > my 2 cents > > > > > > > > > > Michaël > > > > > > > > > > > > > > > > > > > > Stefan Steiniger a écrit : > > > > > > Hei All/Developers, > > > > > > > > > > > > I got an email from Bernd describing the > > following > > > problem > > > > with > > > > > > OpenJUMP writing shape/dbf files. [my > > translation]: > > > > > > > > > > > > ============= > > > > > > It seem like the dbf headers written by > > OpenJUMP are not > > > > conform to > > > > > > the DBASE III standard. > > > > > > > > > > > > Working with Borland-BDE dbase-tests > > failed and an Error > > > > 8961 is > > > > > > returned. The BDE was/is actually used to > > include > > > the dbf > > > > files > > > > > into a > > > > > > MS Access database - in particular as the > > BDE supports > > > > long file > > > > > names. > > > > > > > > > > > > With short file names the BDE is not used > > and hence > > > there > > > > are no > > > > > > problems, also other programs don't seem > > to test for the > > > > correct dbf > > > > > > header. but... short filenames are not so > > > comfortable, etc. > > > > > > > > > > > > Attached is a file (hexdump_dbf.pdf) that > > shows the > > > > differences > > > > > before > > > > > > and after OJ writes the shape file - and > > the changes in > > > > the header. > > > > > > > > > > > > The saved file contains several spaces and > > quite a > > > bit of > > > > digits > > > > > after > > > > > > the comma [sound very similar to what > > Michael looked > > > into > > > > lately?] - > > > > > > hence the file size is bigger too. > > > > > > > > > > > > ============== > > > > > > > > > > > > Anyone who is interested in fixing that or > > has any > > > > suggestions? > > > > > > > > > > > > the dbase standard description is also > > attached > > > > > > > > > > > > mhm.. I actually would not be surprised > > when the > > > original > > > > shape file > > > > > > writer is from the 90ies > > > > > > > > > > > > stefan > > > > > > > > > > > > -------- Original Message -------- > > > > > > Subject: OpenJump, Shape und Dbase > > > > > > Date: Thu, 17 Jun 2010 13:41:52 +0200 > > > > > > From: <ber...@er... > > <mailto:ber...@er...> > > > <mailto:ber...@er... > > <mailto:ber...@er...>> > > > > <mailto:ber...@er... > > <mailto:ber...@er...> > > > <mailto:ber...@er... > > <mailto:ber...@er...>>> <mailto:ber...@er... > > <mailto:ber...@er...> > > > <mailto:ber...@er... > > <mailto:ber...@er...>> > > > > <mailto:ber...@er... > > <mailto:ber...@er...> <mailto:ber...@er... > > <mailto:ber...@er...>>>>> > > > > > > To: <ss...@ge... > > <mailto:ss...@ge...> > > > <mailto:ss...@ge... <mailto:ss...@ge...>> > > <mailto:ss...@ge... <mailto:ss...@ge...> > > > <mailto:ss...@ge... <mailto:ss...@ge...>>> > > > > <mailto:ss...@ge... > > <mailto:ss...@ge...> <mailto:ss...@ge... > > <mailto:ss...@ge...>> > > > <mailto:ss...@ge... <mailto:ss...@ge...> > > <mailto:ss...@ge... <mailto:ss...@ge...>>>>> > > > > > > > > > > > > > > > > > > > > > > > > Hallo Stefan, > > > > > > > > > > > > Im speziellen Fall geht es um die numerischen > > > Datenfelder (Typ > > > > > double) > > > > > > aus den DBF-Dateien, die nicht ganz den > > > Spezifikationen von > > > > > DBASE III > > > > > > genügen. Mitbekommen habe ich das bei > > Einsatz der > > > Borland-BDE, > > > > > die den > > > > > > Header offensichtlich genau prüft und > > einen Fehler 8961 > > > > meldet. > > > > > > > > > > > > Viele Programme scheint die Veränderung > > des Headers > > > nicht zu > > > > > stören, die > > > > > > BDE schon. Zur Erinnerung: Ich habe die > > BDE ausprobiert, > > > > um die > > > > > DBFs aus > > > > > > den SHAPES in eine MS-ACCESS-Datenbank > > einzubinden, weil > > > > die BDE > > > > > auch > > > > > > lange Dateinamen unterstützt. ACCESS > > deswegen, weil ich > > > > dort mit > > > > > > einfachen Abfragen aus Sachdatenbanken > > (Informix) > > > die Geodaten > > > > > ergänzen > > > > > > und aufwerten kann. > > > > > > > > > > > > Mit kurzen Dateinamen brauche ich die BDE > > nicht, > > > habe keine > > > > > Probleme mit > > > > > > ACCESS, aber GIS mit kurzen Dateinamen > > sieht recht > > > komisch > > > > aus. > > > > > > > > > > > > Hier nun meine Forschungsergebnisse: > > > > > > > > > > > > 1. Daten im Originalzustand in File > > hexdump_ok.pdf > > > > > > hier ist nach den Datentypen N=Numerical zu > > > erkennen, daß die > > > > > Feldgrößen > > > > > > sich in typischen Werten bewegen. > > > > > > 2. Daten im File hexdump_err.pdf > > > > > > Nach "Speichern unter …" im Openjump sind > > diese > > > Eintragungen > > > > > deutlich > > > > > > anders und die numerischen Werte haben > > führende > > > > Leerzeichen und ganz > > > > > > viele Nachkommastellen. > > > > > > > > > > > > Dieses Verhalten erklärt auch, warum > > Fremddaten nach > > > > Bearbeitung > > > > > mit OJ > > > > > > mehr Speicherplatz brauchen als im Original. > > > > > > > > > > > > Die DBASE-Spezifikationen habe ich Dir als > > 3. Datei > > > angehängt. > > > > > > > > > > > > In der Hoffnung, daß Du jemanden findest, > > der das > > > Problem > > > > lösen > > > > > kann, > > > > > > verbleibe ich mit vielen Grüßen aus dem > > sonnigen > > > Thüringen. > > > > > > > > > > > > Bernd Wehle > > > > > > Tiefbau- und Verkehrsamt Erfurt > > > > > > 99085 Erfurt, Steinplatz1 > > > > > > Tel.: 0361 655 3108 > > > > > > Fax: 0361 655 3178 > > > > > > e-Mail: ___...@er..._ > > > > <mailto:ber...@er... > > <mailto:ber...@er...> <mailto:ber...@er... > > <mailto:ber...@er...>> > > > <mailto:ber...@er... > > <mailto:ber...@er...> <mailto:ber...@er... > > <mailto:ber...@er...>>> > > > > > <mailto:ber...@er... > > <mailto:ber...@er...> > > > <mailto:ber...@er... > > <mailto:ber...@er...>> <mailto:ber...@er... > > <mailto:ber...@er...> > > > <mailto:ber...@er... > > <mailto:ber...@er...>>>>> > > > > > > > > > > > > <<hexdump_ok.pdf>> <<hexdump_err.pdf>> > > > > <<dbase_strukturen.pdf>> > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------ > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > > > > ThinkGeek and WIRED's GeekDad team up for > > the Ultimate > > > > > > GeekDad Father's Day Giveaway. ONE MASSIVE > > PRIZE to the > > > > > > lucky parental unit. See the prize list > > and enter > > > to win: > > > > > > http://p.sf.net/sfu/thinkgeek-promo > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------ > > > > > > > > > > > > > > _______________________________________________ > > > > > > Jump-pilot-devel mailing list > > > > > > Jum...@li... > > <mailto:Jum...@li...> > > > <mailto:Jum...@li... > > <mailto:Jum...@li...>> > > > > <mailto:Jum...@li... > > <mailto:Jum...@li...> > > > <mailto:Jum...@li... > > <mailto:Jum...@li...>>> > > > > > > > <mailto:Jum...@li... > > <mailto:Jum...@li...> > > > <mailto:Jum...@li... > > <mailto:Jum...@li...>> > > > > <mailto:Jum...@li... > > <mailto:Jum...@li...> > > > <mailto:Jum...@li... > > <mailto:Jum...@li...>>>> > > > > > > > > > > > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > > > ThinkGeek and WIRED's GeekDad team up for > > the Ultimate > > > > > GeekDad Father's Day Giveaway. ONE MASSIVE > > PRIZE to the > > > > > lucky parental unit. See the prize list and > > enter to win: > > > > > http://p.sf.net/sfu/thinkgeek-promo > > > > > _______________________________________________ > > > > > Jump-pilot-devel mailing list > > > > > Jum...@li... > > <mailto:Jum...@li...> > > > <mailto:Jum...@li... > > <mailto:Jum...@li...>> > > > > <mailto:Jum...@li... > > <mailto:Jum...@li...> > > > <mailto:Jum...@li... > > <mailto:Jum...@li...>>> > > > > > > > <mailto:Jum...@li... > > <mailto:Jum...@li...> > > > <mailto:Jum...@li... > > <mailto:Jum...@li...>> > > > > <mailto:Jum...@li... > > <mailto:Jum...@li...> > > > <mailto:Jum...@li... > > <mailto:Jum...@li...>>>> > > > > > > > > > > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------ > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > > > ThinkGeek and WIRED's GeekDad team up for the > > Ultimate > > > > > GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE > > to the > > > > > lucky parental unit. See the prize list and > > enter to win: > > > > > http://p.sf.net/sfu/thinkgeek-promo > > > > > > > > > > > > > > > ------------------------------------------------------------------------ > > > > > > > > > > _______________________________________________ > > > > > Jump-pilot-devel mailing list > > > > > Jum...@li... > > <mailto:Jum...@li...> > > > <mailto:Jum...@li... > > <mailto:Jum...@li...>> > > > > <mailto:Jum...@li... > > <mailto:Jum...@li...> > > > <mailto:Jum...@li... > > <mailto:Jum...@li...>>> > > > > > > > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > > ThinkGeek and WIRED's GeekDad team up for the > Ultimate > > > > GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE > > to the > > > > lucky parental unit. See the prize list and enter > > to win: > > > > http://p.sf.net/sfu/thinkgeek-promo > > > > _______________________________________________ > > > > Jump-pilot-devel mailing list > > > > Jum...@li... > > <mailto:Jum...@li...> > > > <mailto:Jum...@li... > > <mailto:Jum...@li...>> > > > > <mailto:Jum...@li... > > <mailto:Jum...@li...> > > > <mailto:Jum...@li... > > <mailto:Jum...@li...>>> > > > > > > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------ > > > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > > ThinkGeek and WIRED's GeekDad team up for the Ultimate > > > > GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the > > > > lucky parental unit. See the prize list and enter to > win: > > > > http://p.sf.net/sfu/thinkgeek-promo > > > > > > > > > > ------------------------------------------------------------------------ > > > > > > > > _______________________________________________ > > > > Jump-pilot-devel mailing list > > > > Jum...@li... > > <mailto:Jum...@li...> > > > <mailto:Jum...@li... > > <mailto:Jum...@li...>> > > > > > > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > > > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > ThinkGeek and WIRED's GeekDad team up for the Ultimate > > > GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the > > > lucky parental unit. See the prize list and enter to win: > > > http://p.sf.net/sfu/thinkgeek-promo > > > _______________________________________________ > > > Jump-pilot-devel mailing list > > > Jum...@li... > > <mailto:Jum...@li...> > > > <mailto:Jum...@li... > > <mailto:Jum...@li...>> > > > > > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > > > > > > > > > > ------------------------------------------------------------------------ > > > > > > > > > ------------------------------------------------------------------------------ > > > ThinkGeek and WIRED's GeekDad team up for the Ultimate > > > GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the > > > lucky parental unit. See the prize list and enter to win: > > > http://p.sf.net/sfu/thinkgeek-promo > > > > > > ------------------------------------------------------------------------ > > > > > > _______________________________________________ > > > Jump-pilot-devel mailing list > > > Jum...@li... > > <mailto:Jum...@li...> > > > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > > > > > > > > ------------------------------------------------------------------------------ > > ThinkGeek and WIRED's GeekDad team up for the Ultimate > > GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the > > lucky parental unit. See the prize list and enter to win: > > http://p.sf.net/sfu/thinkgeek-promo > > _______________________________________________ > > Jump-pilot-devel mailing list > > Jum...@li... > > <mailto:Jum...@li...> > > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > > > > > > ------------------------------------------------------------------------ > > > > > ------------------------------------------------------------------------------ > > This SF.net email is sponsored by Sprint > > What will you do first with EVO, the first 4G phone? > > Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > Jump-pilot-devel mailing list > > Jum...@li... > > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > > > > > > ------------------------------------------------------------------------------ > This SF.net email is sponsored by Sprint > What will you do first with EVO, the first 4G phone? > Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first > _______________________________________________ > Jump-pilot-devel mailing list > Jum...@li... > https://lists.sourceforge.net/lists/listinfo/jump-pilot-devel > |