Program crashes with large DBF file
Brought to you by:
sdteffen
Menu
▾
▴
#9 Program crashes with large DBF file
open
nobody
None
5
2012-08-22
2011-02-09
Anonymous
No
Program crashes opening a large DBF file.
Using the task manager I could see it happen when it reach 2GB ram use.
Windowx XP sp3 with 8 GB ram (only 4GB recognized in Windows, its a multiboot mac)
2.21 GB DBF file.
Discussion
Anonymous
I've never used such a large DBF file. And http://en.wikipedia.org/wiki/Shapefile#Data_storage mentions a 2 GB size limit. Which software did you use to create your file? Are you sure that it's not corrupted? I would definitely recommend a system with more RAM if you want to deal with such large files. Which version of GTK DBF editor do you use?
What do you want to do with this large file? How many records are you looking at? I don't think you'll be able to do anything meaningful with such a large file.
I.e. we should rather try to get a Mac version of GTK DBF Editor running, because it'll allow you to get more out of your RAM.
Hi sdteffen;
Thanks for your quick answer.
"try to get a Mac version of GTK DBF Editor running"
Is there one? I will like to.
Which software did you use to create your file?
DBFViewer 2000 v 1.53
Are you sure that it's not corrupted?
I am able to open it with DBFViewer 2000 v 1.53, but not to sort it.
RAM
I got 8GB. More?
GTK DBF version.
Last one. I downloaded it today from this site.
What do you want to do with this large file?
Just curious. Its a list of 23344805 persons of my country.
Honestly, this data is not a big deal for me, I inform the bug just to contribute to an open source project.
Thanks for this program
Regards
Marcelo
Thanks for the feedback. Especially as you're just curious. Your effort is appreciated.
Right now there's no Mac version of GTK DBF Editor. But it should be possible to create it. I'll try.
Regarding the RAM usage: When I created the software, I didn't take memory usage in mind. My files were really small (less than a thousand records) - it could be well that the program is using a multitude of the file size from the memory. I'll have to test that.
I'm glad that you're not trying to use a DBF file for data of that size "for real". - Other technologies should be much better suited to deal with data of that size...
I've made some tests with files in the MB range - they use easily 3 times the file size as RAM - I even had some files with factor 8. So your 8GB might come to a use...
I didn't work on GTK DBF Editor for a while. So today was "wasted" getting up and running on a new dev system + new autotools, etc and migrating from SVN to git. Hope to try the Mac build over the weekend.
Hi Sdteffen;
Thanks for your effort.
I hope you can make a mac version.
I will be the first to try it¡¡¡
Marcelo
Thanks to the excellent work of the MacPorts guys, compilation on the Mac worked beautifully.
There's still some installation issues to sort out and placing the whole thing into a DMG will take me a while, but
I just opened the first DBF file in GTK DBF Editor on the Mac...
You'll need the X Server installation like explained in
http://sdteffen.blogspot.com/2010/04/dia-0971-available-for-mac-os-x.html
If you're using MacPorts yourself, please tell me. Then there would be no need to create the DMG...
Hi Steffen:
Good to know. Congratulations.
No, sorry, I am not using MacPorts.
So, I should install X11 from the Mac Os X install dvd? then?
Thanks.
Status update: Builds and installs now using MacPorts. But there's still 3 scripts to master until there's a DMG...
A Mac OS X version of GTK DBF Editor 0.4.0 is now available from http://sdteffen.de/gtkdbfeditor/index_en.html.
Sorry that it took so long. Looking forward to your feedback.
Hi Sdteffen:
I´ve just read your last comment.
I install X11 from the MAC OS dvd, check for a update (no one), restart, and try.
Sorry, I coudnt make it work.
The crash report said something like:
Library not loaded: /opt/local/lib/libglade-2.0.0.dylib
Referenced from: /Applications/Gtkdbfeditor.app/Contents/Resources/bin/gtkdbfeditor-bin
Reason: image not found
Am I doing something wrong?
Thanks
Marcelo
Marcelo, thanks for the feedback. You're not doing anything wrong. I have to rework the DMG once more...
Step 1: I was able to reproduce your problem...
Library problem solved, now there's something with the fonts...
Font problem is solved. An updated DMG is available from
http://sdteffen.de/gtkdbfeditor/index_en.html
Sorry for the incovenience.
Hi Sdteffen:
Congratulations.
I ve just installed it.
I bit of waiting for the first time to start (2 - 4 minutes), a Unix way of sorting files, but it works without problems.
By the way the "Activity monitaor" saids its Intel (64 bits).
I try some 100 MB file and it takes 800 MB in memory when its open.
Last, thar monster big 2 GB file doesnt open at all. Nothing, no message, no wait, like it wasnt there. (doesnt matter at all)
So, quite a programmer. Thanks.
64bit: Yes, my processor is 64bit, but my Mac OS is 32bit, so I'm not really sure if the current GTK DBF Editor could really use your 8GB RAM - I would expect the limit at 4GB / 8 = 500MB DBF file size.
I've added a feature request "Use less RAM":
https://sourceforge.net/tracker/?func=detail&aid=3194045&group_id=37651&atid=420353
Without major restructuring, I see no way to overcome your problem.
Marcelo, I replied to your mail, but my reply didn't go through. Could you provide an alternative email?