From: Don A. <don...@co...> - 2006-09-14 02:51:08
|
Gerald, The database portability problem does not seem to affect all platforms or all distributions. We have not been able to figure out why it doesn't. For example, on my Ubuntu systems, I have never seen this. Alex, on the other hand, running Debian, sees it frequently. Windows also seems to be highly susceptible to this problem. It may have to do with the particular combination of python and the Berkeley DB. Unfortunately, this problem is beyond our control. We have a workaround for it (Alex's UseTXN change), but the real bug can only be fixed by python.=20 Once your distribution upgrades to python 2.5 (along with providing the corresponding GTK/GNOME libraries), the problem will go away without an upgrade to GRAMPS. Don On Wed, 2006-09-13 at 17:19 -0400, Gerald Britton wrote: > I don't get it! I've bee using rsync to copy my gramps dbs back and > forth between linux boxes for YEARS with no ill effects. >=20 > Is this a new problem with 2.1? Does the database have hard disk > pointers in it? >=20 > I just scp'd my 2.0.11 db from linux to windows, fired up 2.1 and > opened the database with no problems and no lost data. >=20 > Am I just lucky? >=20 >=20 >=20 > On 9/13/06, Alex Roitman <sh...@gr...> wrote: > > Steve, > > > > On Wed, 2006-09-13 at 08:48 -0700, Steve Hall wrote: > > > > > > > > Gramps databases are NOT portable. This means you can not move your > > > > familytree.gramps file from linux to windows and expect it to be > > > > safe. You MUST export to XML from Linux and import the XML on > > > > Windows. > > > > > > > > Not only that, but you can't even move your file from one Linux box > > > > to another. Any time your gramps file leaves you computer, you must > > > > export it. > > > > > > > > Portability may change with future versions of python, but for now, > > > > this is what we are stuck with. > > > > > > This is a horrible situation, it is unreasonable to require a user to > > > manage what the application is supposed to do. Can you imagine the > > > reaction if word processors required this? > > > > We learned about this limitation a few months ago, well into developing > > the 2.1.x branch. It is a horrible problem, I absolutely agree. > > I've been trying to get people's attention to it for a while. > > Now that few people actually tried 2.1.x we may get some meaningful > > discussion on this matter. Before all my calls were just the vapor > > to everybody :-) > > > > > If GRAMPS files can not be exchanged AS IS with other GRAMPS users, > > > then the program has a fundamental problem. > > > > Let me describe the situation in detail. > > > > The 2.0.x versions used grdb files that did not use the transactional > > environment. The new 2.1/2.2 series can work both with and without the > > transactional environment (TXN for short). > > > > The benefits of the transactional environment are: > > 1. Data can never be corrupted on the low level. > > Before flaming me for an outright lie, let me explain > > what this means. The changes to the data are sometimes > > simple (changing a note will only affect one object, > > so exactly one piece of data will be modified). But sometimes > > they are complex (adding a new child to the family (1) adds > > a new person, (2) adds that person to the family). There > > are even more complex changes, involving the modifications > > of several records in several database tables. > > > > With the TXN in places, the whole transaction is either > > committed or abandoned, so the database is always in the > > consistent state. Even if the program crashes in the middle > > of the transaction, either the whole thing will be committed > > or not at all. As many people noticed when testing unstable > > 2.1.x releases, no amount of crashing in whatever places > > (well, except initial upgrade from 2.0.x) can damage the data. > > > > This is no small matter, and is very important to anybody > > working with large data files. > > > > 2. The performance. The TXN uses logging, and we currently utilize > > asynchronous transaction logging, so that we don't have to > > physically write and clear the logs for every transaction. > > This allows us to transactionally write much faster than we did > > in 2.0.x before the transactions. For anybody who tried to > > import huge dataset in 2.0.x and in 2.1.x this performance gain > > is pretty obvious. > > > > BUT, as we learned well into developing 2.1 branch, the TXN-capable > > databases cannot be simply taken away and opened in another environment > > (that is, another machine, or even another user on the same machine). > > Sometimes it works, depending on how the log files are synced and > > released, but we have no control over that. So we cannot count on this. > > > > The workarounds could be either copy the whole environment directory > > (the ~/.gramps/bsddbenv dir), or try to "clear" the connection between > > the environment and the datafile. While the bsddb libraries (libdb4.3) > > have the necessary methods, the python bindings were lacking. Because > > of the bug reports that we filed the necessary method has been added > > only recently. > > > > So the next version of python (2.5) will allow us to disassociate > > the database from the environment dir on cleanly closing the db. > > In fact, the code is already in place, so if the method is available > > in python then it will be automatically used. > > > > Now, what are we to do in the meantime? The following steps are > > the workarounds of different degree of ugliness: > > > > 1. Use XML format natively: open XML files, work with them. > > Downside: slow to open, the whole data will stay in memory. > > OK for small files, but impractical for large files. > > > > 2. Work in GRDB and export into XML for portability/archiving. > > > > 3. Always drag the ~/.gramps/bsddbenv directory along with the file. > > Downside: very cumbersome and inconvenient. > > > > 4. We may disable the TXN functionality, or make it a user-adjustable > > parameter. This way the users would decide whether they want to trade > > portability of grdb files for reliability/performance. The TXN-grdb is > > fast and reliable (until you move it), but not portable. > > > > > > None of this is really good. But we're in a tough situation here: > > we want TXN and portability. Hopefully, python2.5 will be released > > soon and solve the problem. However, realistically it will be a while > > until python2.5 is shipped in every distro. > > > > I sincerely welcome any feedback on what to do here. If people want > > to experiment with portable TXN-less grdbs, right now it can be done > > by setting the "UseTXN =3D False" (instead of the current True) > > in the src/GrampsDb/_GrampsBSDDB.py file in the source tree > > or in /usr/share/gramps/GrampsDb/_GrampsBSDDB.py in your installed > > version (or wherever it is installed -- ymmv). > > > > If there are any bsddb experts or any database experts that can > > provide useful advice, we need you now! > > > > Alex > > > > -- > > Alexander Roitman http://www.gramps-project.org > > > > >=20 > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job ea= sier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronim= o > http://sel.as-us.falkag.net/sel?cmd=3Dlnk&kid=3D120709&bid=3D263057&dat= =3D121642 > _______________________________________________ > Gramps-windows mailing list > Gra...@li... > https://lists.sourceforge.net/lists/listinfo/gramps-windows >=20 >=20 |