[Gramps-users] How do I use gramps to come from evidence to conclusion?

[Ralf M. <Ra...@t-...>]

Hello all,

as the following text is quite long, I'll start with a summary:

I have a lot of documents, and often don't know how the persons 
mentioned link up. I'd like gramps help me find that out.

I know how to enter concluded information into gramps, e.g. that John 
Smith was born 1885 in Boston (birth certificate) and married 1910 in 
New York (marriage certificate).
I even saw hints on how to document the reasoning leading to 
conclusions, and how to enter details from different sources about one 
person (such that the details of that person can be compared easily).

But I do *not* know how to enter information of different documents 
independently (e.g. birth and marriage certificate) and *afterwards* 
have gramps help me draw a conclusion, e.g. whether the John Smiths of 
the birth and the marriage certificate are the same or different persons.
And that's what I would like to do:
1. Enter data of documents into gramps.
2. Use the entered data, within gramps, on screen, to come to 
conclusions about who is who.
3. Consolidate details (gramps doing the routine work, I solving conflicts).

I examined two ideas how to do that in gramps (using person records for 
evidence persons, using forms), but both didn't work out. One problem 
was that it's very hard in gramps to display details of several 
(unrelated) persons simultaneously.


In the rest of this email I'll
- describe shortly my set of documents and what I'd like to achieve,
- outline in more detail the three steps I mentioned above,
- ask my questions using more words than above,
- explain the two ideas I examined and why both didn't work for me, 
comparing them in the end.


Currently I've got roughly 900 source documents: About 540 birth / 
marriage / death records or certificates, about 290 population register 
entries, and the rest a mixture of obituary notices, letters, photos of 
tombstones and war memorials, notes of what relatives told me, excerpts 
from books, a will, court decisions, etc.
The official records are basically structured documents, but often 
contain additional remarks of widely varying content.
Of some of the people mentioned in the documents I know how they are 
related to me. I am confident that many others are related somehow to 
me. Some of the people mentioned in the documents won't be my relatives 
at all.

I would like to turn the content of these documents into a gramps 
database which I can generate various reports from.
For this traditionally I would have to study the documents, draw my 
conclusions, and enter those into the database. To find out how 
documents / the people link up I used to write the key data on slips of 
paper and move those around on a big table, or create little boxes in a 
graphics software, type the data into these boxes and move them around 
the screen, or the like. And once I have drawn my conclusion I have to 
type the data in once more, this time into gramps.
I don't like the idea of having to retype data over and over again. I'd 
rather like to enter them once into gramps, then have gramps help me to 
draw my conclusions and record those conclusions.
In other words: I'd like to use gramps not as a "dumb" database, but as 
a tool to interactively work on the evidence data entered. The main 
purpose is to save work and time compared to classical offline 
conclusion drawing. (A secondary purpose is better traceability from 
conclusion to source.)
I think at least some of these aspects have been discussed on this list 
occasionally as keeping evidence and conclusion separated within gramps.


My idea is to use a three step procedure, which can be gone through 
repeatedly:

Step 1 Data entry
Enter all relevant data of the documents as evidence data into gramps, 
i.e. without having to draw conclusions at this point.
The data entry method should be efficient and error-preventing. An 
example might be to enter all usual data of a common document type into 
a form that somewhat resembles the document. Thus data entry is 
straightforward and it's easy to compare the entered data with the 
document for correctness and completeness.
It's OK if entering unusual remarks in common documents is more 
complicated. The same with entering data from unstructured documents 
like letters. But entering both kinds of unusual evidence data must be 
possible.
On data entry the proper source citation should be applied to all data 
entered from one document.

Step 2 Assign evidence persons to conclusion persons
I.e. come to a conclusion which persons mentioned in the documents refer 
to the same real person and record the connection, creating the 
conclusion person if it doesn't exist already.
It's easy to come to a conclusion about close relatives one knows a lot 
about, but for persons beyond that it's more complicated:
I would like to filter the persons (evidence and conclusion together) by 
name, dates, places etc. The filter result should be a list of persons 
with basic data like name, time and place of birth and death etc.
After selecting the most likely persons from the filter result list it 
should be possible to obtain in-depth data about the selected persons. 
By in-depth data I mean something like what the relationship view shows, 
amended by occupation and residence information (and possibly religion). 
The in-depth data should be presented such that it can be compared 
easily, i.e. the data of all selected persons are visible in parallel 
and can be rearranged ("moving boxes around").
After selecting persons to be connected in the detail data, the 
connection ought to be recorded e.g. by clicking a context menu entry.
In addition to the stepwise refinement outlined above a tool to find 
similar evidence persons might be helpful, similar to the find duplicate 
persons tool.
The whole step 2 is similar to solving a jigsaw puzzle:
At first you look for easily identifiable puzzle pieces like the edge 
pieces and combine them to form the edge. This is similar to combining 
close relatives one knows a lot about to a small tree.
Afterwards you sort the remaining pieces by superficial similarity (what 
looks like sky, trees, ...). That is similar to filtering persons on 
surname, place of birth or the like.
Finally you put similar pieces side by side to compare them in detail 
and combine what fits, either connecting pieces to the edges or 
combining them to a cluster that later can be placed in the picture. 
This is similar to looking at in-depth data of multiple persons and 
deciding how to either connect them to the growing tree or combine them 
to a cluster of persons that later can be connected to the tree.

Step 3 Combine details of evidence persons and copy to conclusion person
During step 2 only persons were combined. Birth and death events, 
occupations etc. were not consolidated yet. That has to happen in step 3.
As this is a lot of routine work I would like the software to help with 
this. The idea is that the software makes suggestions and the user 
either accepts them or overrides them based on his/her judgment.
Example birth event:
Lets assume three evidence birth events: "Between 1885-03-14 and 
1886-03-13 (no place)"; "1885-11-06 in Bavaria"; "about 1885 in Munich". 
The software should suggest "1885-11-06 in Munich" (provided there is a 
place hierarchy in which Bavaria encloses Munich). If the last evidence 
were "1875 in Munich" the software should provide no suggestion but 
leave it to the user to resolve the conflict.
Example occupation events:
Lets assume 6 occupation events: "1860 student", "1870 civil engineer", 
"1880 civil engineer", "1890 engineer", "1900 civil engineer", "1910 
pensioner". The software might suggest to either copy all 6 events 
separately or to combine 1870 and 1880 to "from 1870 to 1880 civil 
engineer". The user might decide to only have the first and last 
evidence event copied as is and combine the rest to "from 1870 to 1900 
civil engineer".
During this step 3 the software should take care of proper source 
reference(s).
There are various alternatives regarding the user interface for this 
step 3. I don't care too much about that, though; I'm mainly interested 
in the functionality as described above. (If I had a choice regarding 
the user interface I would prefer one that is flexible and prevents errors.)

In steps 2 and 3 it would be nice if there were a way to document why a 
conclusion was drawn. However, that is not my highest priority.

I would greatly appreciate if the software would allow for simple 
handling of changed data.
Example: After going through steps 1 to 3 I notice that I misread a name 
and mistyped a date. Now I would like to simply correct the evidence 
data and have the software point me to the conclusions I have to check 
because they depend on the corrected data.


Questions

How do I do steps 1 to 3 with gramps?
I'm not asking for a click-here-then-click-there answer, but am 
interested in what features / tools / ... of gramps to use for the 
steps. Also comments like "I do it this way and it works great for ... 
but I noticed these drawbacks" are very welcome. As are remarks like 
"Forget those 3 steps and do it like ...".

If what I would like to do can't be done with gramps today:
How about the near future (e.g. this year)? If I will be able to do 
steps 1 to 3 in the future, is there work I can start with now (e.g. 
data entry)?

If what I would like to do can't be done with gramps neither today nor 
in the foreseeable future:
Is there other software that would let me do it?
(I'm not interested in public web services, I want to keep my data on a 
local drive.)


Two ideas that didn't work out

Idea 1 was inspired by a remark Enno Borgsteede made sometime mid of 
last year. I thought to remember he wrote that evidence about persons 
could be stored in source persons etc. and that gramps had all the 
database structures to do so. Searching my mailbox I only found a mail 
dated 21. May 2015 which doesn't go quite that far, but at least my 
somewhat erroneous memory inspired the following:

The general idea is to use the gramps persons database structure for 
conclusion persons as well as for evidence persons. They are 
distinguished only by an attribute, an id convention (IC1234 for 
conclusion individual, IE1234 for evidence individual) or something 
similar. Same with families, events and so on.
A conclusion person is linked to the evidence persons it's derived from 
by an association of type "concluded from".

With that idea in mind I look at the three steps described earlier:

Step 1 Data entry
All data has to be entered using the usual gramps editors, i.e. many 
editor windows need to be opened for every document. This is neither 
efficient nor error-preventing (see GEPS 034).
A possible workaround might be to enter the data into a spreadsheet and 
use CSV import. Unfortunately CSV import only supports a very limited 
number of fields, and I'm not sure how much work it is to add further 
fields (could I do that myself?).
An advantage of using the standard gramps editors is that arbitrarily 
complex and exotic data can be entered, as well es data from 
unstructured sources like letters. Even if there were a data entry 
method for structured document data, unusual remarks or the like could 
afterwards be added using the usual gramps editors.

Step 2 Assign evidence persons to conclusion persons
For filtering persons the standard filter gramplet can be used, which is 
not exactly easy to use, but very powerful.
But I couldn't find a way to afterwards show details of multiple 
persons: Neither the Relationship View nor the Charts View allow to show 
multiple unrelated persons. Also gramplets seem to display data of one 
selected person only.
Multiple Quick Views can be visible simultaneously, but I can open them 
only one by one: If I select multiple persons in Person View and use the 
context menu to display a Quick View, the Quick View opens only for the 
first selected person, not for all.
A suitable Quick View with all details of a person is not available yet, 
but according to the docs even I should be able to create one. However I 
don't know whether Quick Views can be made such that one can select 
(entries in) multiple Quick Views and create associations between the 
persons.

Step 3 Combine details of evidence persons and copy to conclusion person
I didn't find any tool that does this or even comes close. Everything 
has to be copied manually.
(I just came across GEPS 009. It discusses merging in context of import 
/ export, but anything done there might be useful for merging details of 
evidence persons to a single conclusion person.)

Pro:
* Uses standard gramps database structures
* Thus all gramps tools can be used
* Strict separation evidence / conclusion
* Data entry flexible

Con:
* Data entry inefficient, complex and error-prone
* No (easy) way to display detail data of several persons
* No solution for step 3
* Needs filter for reports etc. to suppress evidence people
* Evidence and conclusion not easily distinguishable in GUI
The last two items won't hinder initial use, but will be annoying in the 
long run.

Summary:
No solution for step 1 and 3, thus not usable for me. I guess most 
shortcomings could be remedied by addons or tools, but creating those is 
beyond my capabilities.


Idea 2 is the Form Gramplet Nick Hall announced end of October last year.
On 1. October he wrote: "Ideally we should be able to store source 
information separately from our conclusions." This is not completely the 
case yet, but I take the quote as indication that it will be eventually.

The Form Gramplet allows data to be entered in forms. The form is 
connected to a gramps event (thus a conclusion event). To enter a person 
into a form one has to select (or create) a gramps person, thus forms 
can only handle conclusion persons. All other data entered, the evidence 
data, is stored as attributes of event refs (and the event).

With that in mind I look at the three steps described earlier:

Step 1 Data entry
Data of documents are entered into predefined forms.
Users can define forms. A form basically consists of sections, each 
containing a person and fields for data about the person. Sections for a 
family or a list of persons instead of a single person are possible. 
There seem to be no restrictions for number of sections, number of 
fields per section or field names.
All form fields are text only. Relationships between persons in the form 
are defined by the form; there seems to be no mechanism for input fields 
for relationships.
Entering data is easy for textual standard data of structured documents. 
As the form and all entered data are visible it's easy to compare the 
entered data with the document for correctness and completeness and 
directly correct any errors. It's even possible to reopen a form and 
edit the data previously entered.
But it's difficult to enter source remarks not anticipated by the author 
of the form. As a counter measure forms might contain extra fields for 
such data, like a "Remark" field per person. Structural information can 
only be entered as textual description.
It's even worse for unstructured documents like letters. The only form I 
can think of for unstructured documents contains one section for 
multiple persons and a number of fields like "Remark1", "Remark2" and so 
on for a fixed number of fields. That's not exactly comfortable.
As entered data are not stored as the usual gramps structures, the 
gramps standard editors can not be used to enter additional information. 
The only way would be to open an event ref and add attributes (using 
arbitrary attribute names).
Data entered into a form are tied to a "real" gramps event. For census 
data special census events are used, but the example birth certificate 
form uses a birth event. Thus if there are two birth certificates for 
one person, the person will get two birth events. (This can be avoided 
by tying forms to documentation events only, not to personal events, 
e.g. to a birth-certificate-issued event instead of a birth event.)

Step 2 Assign evidence persons to conclusion persons
As forms use "real" persons, i.e. conclusion persons, this has to be 
done as part of step 1. That's not what I wanted, but maybe it's not 
that bad: in some (many?) cases the conclusion may be obvious on data 
entry, so I can use the correct conclusion person immediately. Otherwise 
if I'm not sure whether John Smith mentioned in a document is John Smith 
born 1895 or John Smith born 1888 (both already in my database) or 
whether it's a third John Smith altogether, I can create a new person 
"Smith ***, John", treat him as evidence person and wait till further 
information comes up. Later I can change the person in the form to the 
correct John Smith and delete Smith ***, John. The *** in the name 
reminds me not to attach further information (events, ...) to the person 
as the person is likely to be deleted later.
There is a problem though when looking for the right conclusion person 
in the database: many data entered previously is not visible in the 
person selection window. Examples: The birth date of a groom, given in 
the marriage record, won't be shown; the birth date of a deceased 
person, given in the death record, will not be shown; father and mother 
of a child, given in a birth record, will not be shown as partners of 
each other. All this is because the person selection window gathers the 
data shown from the standard gramps structures (events etc.), but form 
data are stored in a different place.
The same applies to the person view, thus filtering is difficult. And 
like for idea 1, I couldn't find a way to show details of multiple persons.
To use the jigsaw puzzle analogy from above: When using forms, sorting 
and combining the pieces takes place in a dim environment where you can 
only discern a few details of the pieces.

Step 3 Combine details of evidence persons and copy to conclusion person
There doesn't seem to be any tool that does this or even comes close. 
Everything has to be copied manually. However there is a gramplet 
(Overview Gramplet) that shows a subset of evidence details of a person, 
so you can draw your conclusions regarding these details without having 
to collect all pieces of data from different windows. The gramplet can 
probably be extended to show more details. Insofar forms are a bit 
better than idea 1, but you still have to enter all conclusions manually.
Because form fields can be arbitrarily named it might be difficult to 
impossible to implement a generic algorithm that semi-automatically 
combines evidence details.

Pro:
* User-definable forms for entry of structured data
* Data entered in a form can be edited in that form any time later
* Actively developed

Con:
* Complicated to impossible to enter unstructured / extra data, 
especially relations between persons
* Most entered data are not visible in person selection list, Person 
View, ...
* No way to display detail data of several persons
* Separation evidence / conclusion only for some data (not persons)
* A little help, but no real solution for step 3

Summary:
Nice concept for step 1, i.e. entering data; unfortunately it only works 
for completely structured data. The limited visibility of entered data 
makes step 2 difficult, and no real solution for step 3. Thus not usable 
for me.
I guess it might be difficult to create a generic semi-automatic tool 
for step 3 because input fields can have arbitrary names.


Comparison:
If I had to choose between the two ideas solely based on the principle 
of operation, I would prefer idea 1. Looking at other aspects, idea 2 
has the huge benefit of someone already working on it with the prospect 
of implementing a complete separation of evidence and conclusions. If in 
the course of further development the support for unstructured sources 
and the visibility of entered data were improved, I would gladly use 
idea 2.
In general I'd rather use tools and methods that are mainstream gramps 
and commonly used by the gramps community if it works good enough for 
me, and not use exotic hand-made tools and methods that might break with 
every gramps update even if the exotic tools are perfect for my tasks.


Regards,
Ralf M.