From: Jon C. <jo...@go...> - 2009-12-07 16:23:35
|
1) If it ain't broke, don't fix it springs to mind here. I see nothing wrong with keeping the gen.lib as is, but choosing a suitable ORM (such as Django's). It should be perfectly possible to use a Person, Event, Family etc... class based model that behaves quite happily as a Django model. I need to catch up from the archives to see if this approach would suit Doug and Co (does anyone have a time frame to look for re: this, or does someone happen to have a copy of the threads they could forward to me?) 2 + 3) If we took the first step as determine what the models and high-level interface looks like, and what functionality it should offer, then further down the line, and making sure we've got some ORM in place, the underlying engine can be swapped out and replaced with something else (for instance a remote DB server so that multiple users can collaborate for instance -- multiple researchers on the same tree?). I suppose the next step would be to 'forward port' all the reports/other stuff... to use the higher-level API -- but we're getting a little ahead of ourselves now... Although I don't think I'm wrong in saying an abstracted DB API will not only make reports easier, but also Gramps in general... If any of this needs clarification, please let me know: I did promise to take the missus out to eat tonight, so I may not be able to kick off any replies this evening without getting told off! It's encouraging to see a positive interest in this: perhaps if we can determine what would be best for Gramps/projects out of this, then I'm more than happy to spend time progressing it. Cheers, Jon. Gerald Britton wrote: > 3. (sent the email too soon!) What will this mean for conversion and > modules that access the database using the current model? > > On Mon, Dec 7, 2009 at 9:58 AM, Gerald Britton <ger...@gm...> wrote: > >> The current database, built on bsddb, is based on a simple concept of >> a "handle", which is generated when a record is added, and data, which >> is everything else for a given record. we have bsddbs for people, >> families, events, places, sources, media, notes and repositories. In >> addition we have cross-reference databases for easy access by gramps >> id and by surname. These are explicitly supported by bsddb which does >> the heavy lifting for us. >> >> Though not used today, it is currently possible to refer to a person like this: >> >> db.person_map[handle] >> >> where db is an instance of the GrampsDbRead class. Of course, this >> reference returns the entire record associated with the handle in the >> person database, which contains many fields, lists and references to >> other records as well. >> >> I've been following the discussions around Django and now your work, >> Jon and I'm wondering about a few things: >> >> 1. Should we move from the current 7-fold bsddb model to one with >> higher abstraction? >> 2. If the answer to the first question is "yes", what should we be >> looking at for the database engine? >> 3. >> >> On Mon, Dec 7, 2009 at 9:35 AM, Jon Clements <jo...@go...> wrote: >> >>> I was basically approaching it from the leave gen.lib alone and >>> implement a "fully blown" SimpleAccess-esque solution. >>> >>> At the moment I basically have a 'DB' object which represents an open >>> database. This at the moment is populated from a Gramps XML file. This >>> is then basically stored as lxml.objectify objects. Internally a graph >>> structure is built to represent the linking inside the database (so >>> relationships and ref. integrity is made easier). >>> >>> 'DBItem' objects consist of the 'node' data, the basic save/delete >>> etc... Deleting an event automatically removes all other references to >>> it (which has caught me out previously). >>> >>> class Person(DBItem): >>> DBTYPE = 'person' >>> >>> Basically registers an object that 'wraps' a basic DBItem, but >>> containing useful attributes/methods. So for a person, we can write >>> attributes such as .birth, .mother, .families etc... etc... It can also >>> over-ride how it should be saved/retrieved etc... >>> >>> I chose this approach because it keeps the process incremental. We can >>> still access the 'raw' data in a DBItem for the stuff I'm not caring >>> about at the moment, but someone can write a 'Place' class later for >>> instance. >>> >>> The DB itself is an xpath queryable object (adds a bit of flexibility >>> for selections that don't have convenient attributes as of yet). >>> >>> I'll see if I can get the code example out this week. >>> >>> Anyway, does this seem a reasonable approach? >>> >>> Cheers, >>> >>> Jon. >>> >>> >>> >>> >>> Benny Malengier wrote: >>> >>>> This deserves some more discussion, and attention from Gerald and Brian! >>>> >>>> The current gen.lib implementation is quite minimal and tied closely >>>> to the available fields, almost all getters and setters. There has >>>> been talk to moving to attributes and removing the getters and >>>> setters, but no time to implement this invasive change which has no >>>> direct benefit but is much work. The present way is a design decision >>>> from before my time with the project. Business logic was moved to the >>>> utility modules. I am not a big fan of this approach as gen.lib is the >>>> obvious choice to look for some functionality, not some utils.py file. >>>> >>>> So, for me too, a higher level functionality in gen.lib would be nice. >>>> I'm in favour if it can be done cleanly. The current higher level >>>> methods are in src/Simple/_SimpleAccess.py, and implemented as >>>> functions. I'd rather see that approach deprecated, and things done in >>>> gen.lib, and some formatting utilities in something like >>>> gen.lib.utils. The main roadblock is that gen.lib does not store the >>>> db one is working on, so one cannot do higher level logic. >>>> SimpleAccess gets around this because .... it is initialized with the >>>> db, so functions can use it. >>>> One of the things I started trying for GEP13 was delayedaccess (aiming >>>> for speed increase), which among others needs to keep track of an >>>> 'engine' which much be used for extra access not done during >>>> initialization. In short, a hidden __db var for objects was something >>>> I was planning for. I do not like that officially we cannot add db to >>>> the gen.lib functions, but at the same time the serialize function has >>>> only one design: reading from a bsddb database. >>>> I would rather see gen.lib being more "easy" and "inviting" instead of >>>> a two track approach. What Jon mentions is a way to do that. It >>>> certainly would be more pythonic, but it needs broad consensus on how >>>> to achieve this without destroying the simplicity inherent in how >>>> gen.lib works. >>>> >>>> A higher level database access is more tricky, reading >>>> for person in db.person(birth=lambda L: 1920 <= L.date.year <= 1985, >>>> last='CLEMENTS'): >>>> it might be better to do that in a layer over the lowest database >>>> access, to not have any speed impacts. Well, it depends on the >>>> implementation and what you mean with db here. Perhaps something >>>> wrapping the general db api and extending it with higher level >>>> functionality. Without seeing code from Jon or thinking about it more, >>>> can't say how I think this is best done. >>>> >>>> Benny >>>> >>>> 2009/12/7 Jon Clements <jo...@go...>: >>>> >>>> >>>>> Doug Blank wrote: >>>>> >>>>> >>>>>> On Sun, Dec 6, 2009 at 4:29 PM, Jon Clements <jo...@go...> wrote: >>>>>> >>>>>> >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> A colleague of mine has found Gramps extremely useful in recording his >>>>>>> family tree. He initially requested a few changes to the timeline chart >>>>>>> to show marriages and births -- which was relatively easy to do. We then >>>>>>> looked into removing duplicated events (which he'd achieved by importing >>>>>>> a database or some such thing...) >>>>>>> >>>>>>> However, after delving around the API, I found it somewhat >>>>>>> 'un-inviting'... specifically for purposes of reporting. >>>>>>> >>>>>>> I have therefore put in some effort on re-organising the DB API and the >>>>>>> way it functions. Specifically on the ease of querying and integrity. >>>>>>> >>>>>>> Now, to be quite honest, it was an adhoc job, but served our purposes. >>>>>>> The reason I'm posting here is whether or not it would either 1) be >>>>>>> useful to the gramps core, or 2) be a useful third party add-on. If >>>>>>> either of these, then I'll certainly work on the code to tidy it up, and >>>>>>> do some documentation of usage. >>>>>>> >>>>>>> A quick example usage: >>>>>>> >>>>>>> jon = db.person(last='CLEMENTS', first='Jon') # retrieve a person or persons >>>>>>> jon_grandmum = jon.mother.mother # as it says >>>>>>> jon.mother in jon.mother.mother.children # just to check >>>>>>> jon.mother.nchildren # only child, so returns 1 >>>>>>> >>>>>>> It also maintains the referential integrity of the DB, so that deleting >>>>>>> an event, deletes all objects that refer to the event. It's also xpath >>>>>>> queryable. It also mimics a graph, so that shortest paths and >>>>>>> relationships can be worked out. It does have some rough edges though. >>>>>>> >>>>>>> It enables you to write something such as: >>>>>>> >>>>>>> # Simple query for timeline report >>>>>>> for person in db.person(birth=lambda L: 1920 <= L.date.year <= 1985, >>>>>>> last='CLEMENTS'): >>>>>>> events = [ev.date.year for ev in >>>>>>> person.events(include=['birth','death','marriage','child']) >>>>>>> >>>>>>> # Check families >>>>>>> clements = db.family(last='CLEMENTS') >>>>>>> jon in clements # True >>>>>>> jon & clements # Returns families I'm a member of >>>>>>> >>>>>>> Hope this makes sense, and might be of some use, feedback appreciated. >>>>>>> >>>>>>> Cheers, >>>>>>> >>>>>>> Jon. >>>>>>> >>>>>>> >>>>>>> >>>>>> Hi, Jon! I'm just one of the contributors, only speaking for myself, >>>>>> but what you speak of might be interesting to look at. At the least, >>>>>> you (or someone) could develop it as a third-party library plugin >>>>>> (what we now call an "addon") which you could build your own reports >>>>>> on top of. This would allow your colleague to continue to use it, and >>>>>> reports based on it, as gramps continues to evolve. >>>>>> >>>>>> Your interface looks a little bit like Django's API, and we are >>>>>> actively developing under Django for a web-based version of gramps. >>>>>> So, that might make it more interesting. On the other hand, it would >>>>>> need to be able to integrate into what we have in order for us to >>>>>> commit to maintaining more code. >>>>>> >>>>>> If you could make it available someplace, I (for one) would like to >>>>>> see it (if you have a GPL-compatible license). >>>>>> >>>>>> Thanks! >>>>>> >>>>>> -Doug >>>>>> >>>>>> >>>>>> >>>>> Hi Doug, >>>>> >>>>> Thanks for the reply and the encouragement, I'll clean up the code, doc. >>>>> it, and >>>>> release under a GPL compat. licence. Then you (or someone else) can >>>>> sanitise it for me. >>>>> (This may not be before xmas, as obviously I need to do work that pays >>>>> the bills :( ) >>>>> >>>>> >>>>> And yes, I realise that it does look a lot like Django's API (as well as >>>>> BeautifulSoup's) as I'm an active user of the framework Although I play >>>>> around with TurboGears and Zope, Django just seems to make the most >>>>> amount of sense. >>>>> >>>>> If you'd like any help with the development of gramps under Django, >>>>> please let me know -- I'm sure I can put some time aside to help out. >>>>> >>>>> When I'm happy enough with it, I'll put my code in SVN with a trac web >>>>> interface. >>>>> >>>>> Cheers, >>>>> >>>>> Jon. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> ------------------------------------------------------------------------------ >>>>> Join us December 9, 2009 for the Red Hat Virtual Experience, >>>>> a free event focused on virtualization and cloud computing. >>>>> Attend in-depth sessions from your desk. Your couch. Anywhere. >>>>> http://p.sf.net/sfu/redhat-sfdev2dev >>>>> _______________________________________________ >>>>> Gramps-devel mailing list >>>>> Gra...@li... >>>>> https://lists.sourceforge.net/lists/listinfo/gramps-devel >>>>> >>>>> >>>>> >>> ------------------------------------------------------------------------------ >>> Join us December 9, 2009 for the Red Hat Virtual Experience, >>> a free event focused on virtualization and cloud computing. >>> Attend in-depth sessions from your desk. Your couch. Anywhere. >>> http://p.sf.net/sfu/redhat-sfdev2dev >>> _______________________________________________ >>> Gramps-devel mailing list >>> Gra...@li... >>> https://lists.sourceforge.net/lists/listinfo/gramps-devel >>> >>> >> >> -- >> Gerald Britton >> >> > > > > |