From: Gerald B. <ger...@gm...> - 2010-05-12 15:17:25
|
On Wed, May 12, 2010 at 11:01 AM, Doug Blank <dou...@gm...> wrote: > On Wed, May 12, 2010 at 10:53 AM, Gerald Britton > <ger...@gm...> wrote: >> On Wed, May 12, 2010 at 10:47 AM, Doug Blank <dou...@gm...> wrote: >>> On Wed, May 12, 2010 at 10:04 AM, Gerald Britton >>> <ger...@gm...> wrote: >>>> I think that there may be some misunderstanding in how to use the >>>> proxies or how they work. The basic idea is that, when you use a >>>> proxy, you use it for every database access. That way, if you are >>>> using the "private" proxy, for example, every access will be checked >>>> to see if the record is flagged as private. If a record is flagged as >>>> private, then None is returned, effectively making the record >>>> invisible to the caller. Because of this, it does not matter if a >>>> proxy is the first thing or the last thing done. In fact, the idea of >>>> "first" or "last" should not apply at all, since the proxy is your >>>> access method to the data. >>>> >>>> If, however, you combine proxy access with unproxied-access, you are >>>> bypassing the proxy and results may be unpredictable. >>> >>> Gerald, >>> >>> If there is confusion, I think it is in how the user can best use >>> these proxies to get the selection they want. >>> >>> If you look at the examples that Benny and Jérôme both had, there was >>> a confusion on how to accomplish their goals. Benny's example was >>> using the Private Proxy to select only the people that could be >>> accessed through non-private people, and Jérôme's example needed to >>> use the private people first in the selection, before having the >>> private people removed. >>> >>> One possible solution would be to change the order of how the proxies >>> are applied. But that would require the user to have a sophisticated >>> understanding of what these do. More importantly, one wouldn't know >>> who was selected. >>> >>> The agreed upon solution is to put most of the power of the proxies >>> back in the hands of the users. That way, they can run their hand-made >>> filters, and see the results. >>> >>> You may be onto an issue though: can a user hand-craft a filter that >>> gives the same result as applying the Private proxy followed by a >>> filter? If not, we should change the filtering system to allow exactly >>> the same results. >> >> Just the way you say this shows me that the operation is not well >> understood. You don't apply the Private proxy; you access the >> database through it. That means that it hides private data. > > Gerald, > > I guess I should be more precise: I should say that one wraps the > database, and then accesses the data through it. > >> There is >> no logical difference between: >> >> if not private and passes my filter / if passes my filter and not private. > > That would be true if filters were only based on single object > properties, but filters can be very complex. > > There has been confusion over this issue. In fact I have just found a > bug in that the living proxy only used the data available in the > living proxy. It should be the case that we use the full knowledge of > the database, and not limit that to just what is available via the > proxy. No that is not correct. Each proxy access the database through the method given to it (which may be another proxy). Also, each proxy acts as if it is the entire world. So it makes sense that the living proxy uses its internal logic to access the data. > > -Doug > >>> >>> -Doug >>> >>>> On Tue, May 11, 2010 at 8:43 AM, jerome <rom...@ya...> wrote: >>>>>> It is a change that moves 5 lines in two files >>>>>> (ExportGedcom, ExportXml). >>>>> >>>>> Does it mean there is no more crash on gen/proxy/filter.py ? >>>>> >>>>> http://www.gramps-project.org/bugs/view.php?id=3878#c14220 >>>>> >>>>>> 1) is this a bug? >>>>> >>>>> Someone using Gramps for the first time, who generates a filter rule and check the result, then uses the Exporter (privacy option by default on primary formats) might say : "Gramps is not consistent according output format" (gedcom, XML versus geneweb, csv) !!! >>>>> >>>>> Note, I didn't understand "descendant of" arguments (Benny) because descendants of private persons are often matching "living". They are properly filtered (like private). Anyway, I suppose we can also reproduce this issue with "descendant of" filter rule and someone private on the chain of descendants ? >>>>> >>>>> I get a mistake of more than 2000 persons with my primary database + around 900 persons matching the filter ignored on output ! True, as user, I did not export any private records and did not get any crash, but I have the strange feeling that Exporter does not follow my instructions ... That's a little bit annoying. :( >>>>> >>>>>> 2) if yes, do you want to wait enough time after the change >>>>>> to make >>>>>> sure it is well-tested? >>>>> >>>>> The first clue was highlighted on 3.1.x ... >>>>> http://www.gramps-project.org/bugs/view.php?id=3877#c14204 >>>>> >>>>> And manual says : "Remember, GRAMPS is making its best guess, and it may not always be able to guess correctly all the time. Please double check your data." (was for alive, but could be more general tip...) >>>>> http://gramps-project.org/wiki/index.php?title=Gramps_3.2_Wiki_Manual_-_Manage_Family_Trees#Filters_and_privacy >>>>> >>>>> Maybe it can be more tested on trunk or after release (no privacy issue, no crash, just a wrong filter output). >>>>> >>>>> >>>>> Jérôme >>>>> >>>>> >>>>> --- En date de : Mar 11.5.10, Doug Blank <dou...@gm...> a écrit : >>>>> >>>>>> De: Doug Blank <dou...@gm...> >>>>>> Objet: Re: [Gramps-devel] Bug in Filters/Proxies? >>>>>> À: "Benny Malengier" <ben...@gm...> >>>>>> Cc: "Gramps Development List" <gra...@li...>, "Gerald Britton" <ger...@gm...> >>>>>> Date: Mardi 11 mai 2010, 14h01 >>>>>> On Tue, May 11, 2010 at 7:42 AM, >>>>>> Benny Malengier >>>>>> <ben...@gm...> >>>>>> wrote: >>>>>> > >>>>>> > >>>>>> > 2010/5/11 Doug Blank <dou...@gm...> >>>>>> >> >>>>>> >> On Tue, May 11, 2010 at 7:12 AM, Benny Malengier >>>>>> >> <ben...@gm...> >>>>>> wrote: >>>>>> >> > >>>>>> >> > >>>>>> >> > 2010/5/11 Doug Blank <dou...@gm...> >>>>>> >> >> >>>>>> >> >> On Tue, May 11, 2010 at 6:03 AM, Benny >>>>>> Malengier >>>>>> >> >> <ben...@gm...> >>>>>> wrote: >>>>>> >> >> > >>>>>> >> >> > >>>>>> >> >> > 2010/5/11 Gerald Britton <ger...@gm...> >>>>>> >> >> >> >>>>>> >> >> >> I actually think that things are >>>>>> working correctly. If there is a >>>>>> >> >> >> problem it is that the complex >>>>>> interactions possible with proxies >>>>>> >> >> >> and >>>>>> >> >> >> filters using nested >>>>>> combinations of "match" and "do not match" can >>>>>> >> >> >> produce results that are >>>>>> perfectly logical but not always easy to >>>>>> >> >> >> explain. >>>>>> >> >> >> >>>>>> >> >> > >>>>>> >> >> > Yes, the new logic might lead to >>>>>> other unwanted behavior that is >>>>>> >> >> > perhaps >>>>>> >> >> > worse than the original behavior. >>>>>> That is, perhaps it is better not >>>>>> >> >> > to >>>>>> >> >> > do >>>>>> >> >> > this thing. Consider: >>>>>> >> >> > >>>>>> >> >> > A person is a private descendant of >>>>>> a family. But it is not private >>>>>> >> >> > who >>>>>> >> >> > his >>>>>> >> >> > children are. >>>>>> >> >> > >>>>>> >> >> > When you do a descendant report, >>>>>> you obtain all children, then >>>>>> >> >> > remove >>>>>> >> >> > private. In the new setup, you still >>>>>> end up with the children of the >>>>>> >> >> > private >>>>>> >> >> > child, which you would want to >>>>>> remain hidden. >>>>>> >> >> > >>>>>> >> >> > Same with the original ancestor >>>>>> question: from a report you can >>>>>> >> >> > deduce >>>>>> >> >> > somebody is a private ancestor of >>>>>> somebody else, which you might not >>>>>> >> >> > want to >>>>>> >> >> > share with other people. >>>>>> >> >> > >>>>>> >> >> > So, for now, it might be best _not_ >>>>>> to change the way the proxy >>>>>> >> >> > works, >>>>>> >> >> > and >>>>>> >> >> > do as Gerald says: offer a checkbox: >>>>>> apply private on final result, >>>>>> >> >> > as >>>>>> >> >> > opposed to, apply private at the >>>>>> beginning. >>>>>> >> >> > >>>>>> >> >> > One gives results you as a >>>>>> researcher want to see, the other gives >>>>>> >> >> > results >>>>>> >> >> > you allow to share with people. >>>>>> >> >> >>>>>> >> >> I have now looked at all of the >>>>>> possibilities of how filters/proxies >>>>>> >> >> can interact, and I agree with Brian that >>>>>> we should make this change. >>>>>> >> >> The most important reason is that we want >>>>>> the export output to match >>>>>> >> >> expectations. Currently, applying the >>>>>> private and living filters first >>>>>> >> >> can dramatically change the meaning of a >>>>>> filter that someone has >>>>>> >> >> hand-crafted to give the right results. >>>>>> >> >> >>>>>> >> >> The private and living filters should be >>>>>> seen as a final cleansing of >>>>>> >> >> selected data, not a part of the >>>>>> selection process itself. >>>>>> >> >> >>>>>> >> >> If one wants private (or living) to be >>>>>> part of the selection process, >>>>>> >> >> you can add that to your filter. In >>>>>> Benny's example above, you want to >>>>>> >> >> remove the private people *and their >>>>>> descendants*, so you should make >>>>>> >> >> that part of the filter (eq, use a filter >>>>>> that uses a private >>>>>> >> >> criteria). >>>>>> >> >> >>>>>> >> >> The litmus test for the correct solution, >>>>>> I think, is how easy is it >>>>>> >> >> to tell what people will be selected. >>>>>> Right now, the only way to tell >>>>>> >> >> who will be selected is to export the >>>>>> people, create a new tree, and >>>>>> >> >> import them back in and examine. In the >>>>>> new way, you can construct >>>>>> >> >> your filter, and imagine the single >>>>>> people that are alive or private >>>>>> >> >> will be removed. Much easier. >>>>>> >> >> >>>>>> >> >> (There is one complication: living people >>>>>> should be determined by >>>>>> >> >> examining the whole database, not the >>>>>> filtered version. But that can >>>>>> >> >> be fixed.) >>>>>> >> >> >>>>>> >> >> Even the notes filter can be seen as just >>>>>> removing from the selection, >>>>>> >> >> and should be applied after the selection >>>>>> process. Otherwise, you can >>>>>> >> >> end up with the note filter affecting >>>>>> which people are selected, which >>>>>> >> >> then changes the whole group. >>>>>> >> >> >>>>>> >> >> Combining the private and living filters >>>>>> in the selection process >>>>>> >> >> makes the results too different from what >>>>>> one gets interactively, and >>>>>> >> >> makes it too hard to see who was actually >>>>>> selected. >>>>>> >> > >>>>>> >> > Nicely put. I fully agree with this >>>>>> analysis. >>>>>> >> >>>>>> >> Cool. Should we keep 3.2.x working the old way, >>>>>> and only make this >>>>>> >> change in trunk? Or apply it to the >>>>>> soon-to-be-released 3.2.3? >>>>>> > >>>>>> > bug fixes can go in 3.2.3 if they do not bring API >>>>>> changes/large code >>>>>> > commits and are well tested. >>>>>> >>>>>> It is a change that moves 5 lines in two files >>>>>> (ExportGedcom, >>>>>> ExportXml). The questions are: >>>>>> >>>>>> 1) is this a bug? >>>>>> 2) if yes, do you want to wait enough time after the change >>>>>> to make >>>>>> sure it is well-tested? >>>>>> >>>>>> This is the last issue on the roadmap for 3.2.3: >>>>>> >>>>>> http://www.gramps-project.org/bugs/roadmap_page.php >>>>>> >>>>>> -Doug >>>>>> >>>>>> > >>>>>> > Benny >>>>>> >> >>>>>> >> -Doug >>>>>> >> >>>>>> >> > Benny >>>>>> >> >> >>>>>> >> >> -Doug >>>>>> >> >> >>>>>> >> >> > Benny >>>>>> >> >> >> >>>>>> >> >> >> On Tue, May 11, 2010 at 10:03 >>>>>> AM, jerome <rom...@ya...> >>>>>> wrote: >>>>>> >> >> >> > As "standalone" everything >>>>>> works fine (filter or private proxy >>>>>> >> >> >> > results). >>>>>> >> >> >> > I did not look at reports >>>>>> or tools, but I only get problem on >>>>>> >> >> >> > Exporter (XML >>>>>> >> >> >> > or gedcom formats). >>>>>> >> >> >> > >>>>>> >> >> >> > Otherwise, I did not test a >>>>>> filter on notes and private notes or >>>>>> >> >> >> > persons. >>>>>> >> >> >> > >>>>>> >> >> >> > By changing the >>>>>> central/root person on testing gramplet (added on >>>>>> >> >> >> > bug >>>>>> >> >> >> > report) from I0001 to I37, >>>>>> I learned that adding sort() function >>>>>> >> >> >> > on a >>>>>> >> >> >> > filtered list returns >>>>>> ordered objects (sorted by the filter rule) >>>>>> >> >> >> > or >>>>>> >> >> >> > by >>>>>> >> >> >> > entry/key (surname for >>>>>> person). Unfortunately, I was not able to >>>>>> >> >> >> > understand >>>>>> >> >> >> > if to use proxy.filter and >>>>>> proxy.private together will try to >>>>>> >> >> >> > match >>>>>> >> >> >> > common >>>>>> >> >> >> > entries on lists or always >>>>>> to pass one into the second one. >>>>>> >> >> >> > >>>>>> >> >> >> > Maybe only a coding issue >>>>>> on Exporter. But why no more invert mark >>>>>> >> >> >> > on >>>>>> >> >> >> > filter after matching a >>>>>> private record ? So, there is a temp list >>>>>> >> >> >> > somewhere >>>>>> >> >> >> > (invert of inverted filter >>>>>> result !) added on output :( >>>>>> >> >> >> > >>>>>> >> >> >> > There is multiple variables >>>>>> on this issue : >>>>>> >> >> >> > * invert mark on filter >>>>>> >> >> >> > * proxy.private >>>>>> >> >> >> > * proxy.filter >>>>>> >> >> >> > >>>>>> >> >> >> > Patching export formats >>>>>> (XML, gedcom) will generate the same >>>>>> >> >> >> > number >>>>>> >> >> >> > of >>>>>> >> >> >> > people as GeneWeb, csv, etc >>>>>> ... formats >>>>>> >> >> >> > But (in my case) why need >>>>>> to also patch proxy (sourcebase, >>>>>> >> >> >> > notebase) >>>>>> >> >> >> > ? >>>>>> >> >> >> > And why proxy stop >>>>>> iteration on filtered list if someone is >>>>>> >> >> >> > private ? >>>>>> >> >> >> > This >>>>>> >> >> >> > one sounds like a related >>>>>> issue ! >>>>>> >> >> >> > >>>>>> >> >> >> > >>>>>> >> >> >> > >>>>>> >> >> >> > Regards, >>>>>> >> >> >> > Jérôme >>>>>> >> >> >> > >>>>>> >> >> >> > >>>>>> >> >> >> > --- En date de : Mar >>>>>> 11.5.10, Gerald Britton >>>>>> >> >> >> > <ger...@gm...> >>>>>> >> >> >> > a écrit : >>>>>> >> >> >> > >>>>>> >> >> >> >> De: Gerald Britton >>>>>> <ger...@gm...> >>>>>> >> >> >> >> Objet: Re: >>>>>> [Gramps-devel] Bug in Filters/Proxies? >>>>>> >> >> >> >> À: "Doug Blank" <dou...@gm...> >>>>>> >> >> >> >> Cc: "Gramps Development >>>>>> List" >>>>>> >> >> >> >> <gra...@li...> >>>>>> >> >> >> >> Date: Mardi 11 mai >>>>>> 2010, 8h59 >>>>>> >> >> >> >> this is a unique use >>>>>> case. i >>>>>> >> >> >> >> cannot see that there >>>>>> is one correct way >>>>>> >> >> >> >> to do it. probably >>>>>> some would ask for a way to >>>>>> >> >> >> >> specify the order in >>>>>> >> >> >> >> which the proxies are >>>>>> evaluated. however that would >>>>>> >> >> >> >> mean changes to a >>>>>> >> >> >> >> few dialogues I think >>>>>> >> >> >> >> >>>>>> >> >> >> >> On Tue, May 11, 2010 at >>>>>> 2:00 AM, Doug Blank >>>>>> >> >> >> >> <dou...@gm...> >>>>>> >> >> >> >> wrote: >>>>>> >> >> >> >> > Devs, >>>>>> >> >> >> >> > >>>>>> >> >> >> >> > In working with >>>>>> Jérôme on this bug: >>>>>> >> >> >> >> > >>>>>> >> >> >> >> > http://www.gramps-project.org/bugs/view.php?id=3878 >>>>>> >> >> >> >> > >>>>>> >> >> >> >> > We've encountered >>>>>> a filter interaction that doesn't >>>>>> >> >> >> >> seem right, and we >>>>>> >> >> >> >> > have a proposed >>>>>> solution, but we want to bounce it off >>>>>> >> >> >> >> you. Here is >>>>>> >> >> >> >> > the situation: >>>>>> >> >> >> >> > >>>>>> >> >> >> >> > 1) You have a >>>>>> filter that is: everyone that is *not* >>>>>> >> >> >> >> an ancestor of a >>>>>> >> >> >> >> > specific person. >>>>>> In the chain of ancestors are a >>>>>> >> >> >> >> couple of people that >>>>>> >> >> >> >> > are private. There >>>>>> are 29 people that match when >>>>>> >> >> >> >> exported (eg, they >>>>>> >> >> >> >> > are not ancestors >>>>>> of the person). >>>>>> >> >> >> >> > >>>>>> >> >> >> >> > 2) If you add the >>>>>> additional constraint of "Do not >>>>>> >> >> >> >> include records >>>>>> >> >> >> >> > marked private" >>>>>> with this filter, then the number of >>>>>> >> >> >> >> people exported >>>>>> >> >> >> >> > increases to 35, >>>>>> which includes some ancestors. >>>>>> >> >> >> >> > >>>>>> >> >> >> >> > The reason is that >>>>>> the Private filter is applied >>>>>> >> >> >> >> first, so that the >>>>>> >> >> >> >> > chain of ancestors >>>>>> no longer contains everyone that it >>>>>> >> >> >> >> did before, and >>>>>> >> >> >> >> > so less people are >>>>>> found to be ancestors, and thus >>>>>> >> >> >> >> more people are not >>>>>> >> >> >> >> > ancestors. The >>>>>> exported people includes some >>>>>> >> >> >> >> ancestors, but not the >>>>>> >> >> >> >> > private ones. >>>>>> >> >> >> >> > >>>>>> >> >> >> >> > It seems that the >>>>>> right thing to do is to apply the >>>>>> >> >> >> >> Private (and >>>>>> >> >> >> >> > Living) filters >>>>>> last, so that the constructed filter >>>>>> >> >> >> >> behaves as it >>>>>> >> >> >> >> > does interactively >>>>>> in Gramps. We have swapped the >>>>>> >> >> >> >> order of the filter >>>>>> >> >> >> >> > applications >>>>>> having Private go after Filter, and it >>>>>> >> >> >> >> then does the >>>>>> >> >> >> >> > correct thing. It >>>>>> seems from SVN that it has always >>>>>> >> >> >> >> been in the wrong >>>>>> >> >> >> >> > application order >>>>>> since it was introduced in Gramps >>>>>> >> >> >> >> since before >>>>>> >> >> >> >> > September 2007. >>>>>> >> >> >> >> > >>>>>> >> >> >> >> > Does anyone see an >>>>>> error in our logic, or see that in >>>>>> >> >> >> >> some use cases >>>>>> >> >> >> >> > it wouldn't do the >>>>>> right thing? Otherwise, we think >>>>>> >> >> >> >> the order needs to >>>>>> >> >> >> >> > be swapped. >>>>>> >> >> >> >> > >>>>>> >> >> >> >> > -Doug >>>>>> >> >> >> >> > >>>>>> >> >> >> >> > [1] >>>>>> >> >> >> >> > >>>>>> >> >> >> >> > >>>>>> >> >> >> >> > http://gramps.svn.sourceforge.net/viewvc/gramps/trunk/src/plugins/export/ExportGedcom.py?view=diff&r1=8930&r2=8931 >>>>>> >> >> >> >> > >>>>>> >> >> >> >> > >>>>>> >> >> >> >> >>>>>> >> >> >> >> >>>>>> >> >> >> >> >>>>>> >> >> >> >> >>>>>> ------------------------------------------------------------------------------ >>>>>> >> >> >> >> > >>>>>> >> >> >> >> > >>>>>> _______________________________________________ >>>>>> >> >> >> >> > Gramps-devel >>>>>> mailing list >>>>>> >> >> >> >> > Gra...@li... >>>>>> >> >> >> >> > https://lists.sourceforge.net/lists/listinfo/gramps-devel >>>>>> >> >> >> >> > >>>>>> >> >> >> >> >>>>>> >> >> >> >> >>>>>> >> >> >> >> >>>>>> >> >> >> >> -- >>>>>> >> >> >> >> Gerald Britton >>>>>> >> >> >> >> >>>>>> >> >> >> >> >>>>>> >> >> >> >> >>>>>> >> >> >> >> >>>>>> >> >> >> >> >>>>>> ------------------------------------------------------------------------------ >>>>>> >> >> >> >> >>>>>> >> >> >> >> >>>>>> _______________________________________________ >>>>>> >> >> >> >> Gramps-devel mailing >>>>>> list >>>>>> >> >> >> >> Gra...@li... >>>>>> >> >> >> >> https://lists.sourceforge.net/lists/listinfo/gramps-devel >>>>>> >> >> >> >> >>>>>> >> >> >> > >>>>>> >> >> >> > >>>>>> >> >> >> > >>>>>> >> >> >> > >>>>>> >> >> >> >>>>>> >> >> >> >>>>>> >> >> >> >>>>>> >> >> >> -- >>>>>> >> >> >> Gerald Britton >>>>>> >> >> >> >>>>>> >> >> >> >>>>>> >> >> >> >>>>>> >> >> >> >>>>>> >> >> >> >>>>>> ------------------------------------------------------------------------------ >>>>>> >> >> >> >>>>>> >> >> >> >>>>>> _______________________________________________ >>>>>> >> >> >> Gramps-devel mailing list >>>>>> >> >> >> Gra...@li... >>>>>> >> >> >> https://lists.sourceforge.net/lists/listinfo/gramps-devel >>>>>> >> >> > >>>>>> >> >> > >>>>>> >> >> > >>>>>> >> >> > >>>>>> >> >> > >>>>>> ------------------------------------------------------------------------------ >>>>>> >> >> > >>>>>> >> >> > >>>>>> >> >> > >>>>>> _______________________________________________ >>>>>> >> >> > Gramps-devel mailing list >>>>>> >> >> > Gra...@li... >>>>>> >> >> > https://lists.sourceforge.net/lists/listinfo/gramps-devel >>>>>> >> >> > >>>>>> >> >> > >>>>>> >> > >>>>>> >> > >>>>>> > >>>>>> > >>>>>> >>>>>> ------------------------------------------------------------------------------ >>>>>> >>>>>> _______________________________________________ >>>>>> Gramps-devel mailing list >>>>>> Gra...@li... >>>>>> https://lists.sourceforge.net/lists/listinfo/gramps-devel >>>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Gerald Britton >>>> >>> >> >> >> >> -- >> Gerald Britton >> > -- Gerald Britton |