Re: [Gramps-devel] Bug in Filters/Proxies?

[Gerald B. <ger...@gm...>]

This is more or less how I felt about this from the get-go.  However,
I do understand that in some cases, you may or may not get what you
want.  e.g.

Say you want to filter out private records and living people.  If you
stack the proxies:

living
  private
     real database

the living proxy may not have as much data to work with to determine
if someone is living and should be excluded.  OTOH if you stack them:

private
  living
    real database

the living proxy has free reign to exclude living folks.  The private
proxy is at the top and so may exclude others as well.

Whether either approach is incorrect is up for discussion.  I can see
it both ways.  However, if you add a filter proxy to the mix, then all
bets are off.  I don't think that anyone can say that one combination
is correct and another combination is incorrect.  It's all in the eye
of the beholder.

Doug is wondering if we should let the user specify proxy order.  I'm
wondering if it is worth the effort.

On Tue, May 18, 2010 at 4:11 PM, Benny Malengier
<ben...@gm...> wrote:
>
>
> 2010/5/18 Nick Hall <nic...@ho...>
>>
>>
>> Doug Blank wrote:
>>>
>>> On Tue, May 18, 2010 at 10:40 AM, Nick Hall <nic...@ho...>
>>> wrote:
>>>
>>>>
>>>> In reading this thread, I am concerned that this may be getting too
>>>> complicated for the average user.  Do we really want to expose the
>>>> concept
>>>> of both a proxy (or view) and a filter to the end user?
>>>>
>>>
>>> We already do, implicitly. I think the Private proxy is the only one
>>> that may need to be handled in this way, and only when you use a
>>> filter, so it is:
>>>
>>> ( ) Include Private Objects
>>> ( ) Remove Private Objects before filter
>>> ( ) Remove Private Objects after filter
>>>
>>>
>>
>> There is no problem doing it implicitly like this.  The user is not
>> exposed to the implementation.  The options you have listed here are
>> unambiguous.
>>
>>>> Using the A, B, C descendants example, in selecting an option to
>>>> "exclude
>>>> private data" I would expect to see person A and C.  If the user doesn't
>>>> want person C then they should mark them as private.  This could be
>>>> achieved
>>>> by applying the proxy last.  I would consider the current results as a
>>>> bug.
>>>>
>>>> I can see why applying the proxy first produces different results, and
>>>> in
>>>> some circumstances why this may be useful, but surely this approach
>>>> would be
>>>> confusing for "Aunt Martha" and may lead to unexpected results.  Does
>>>> "proxy
>>>> first" really have to be an option?
>>>>
>>>
>>> One compelling argument is that this is the way that Gramps has always
>>> operated, and people (such as Gerald) have constructed filters that
>>> depend on this operation. If we remove the private-proxy-first option,
>>> users would need to do what you suggest: mark everyone past the
>>> private person as private, too.
>>>
>>>
>>
>> In general, I would recommend fixing a bug even if some code relies on a
>> side effect of the bug.  I don't think that the proxy-first option would be
>> the functionality that most people would expect.
>
> If you want to keep it private that C is a descendant of B (child reference
> can be made private), then you _don't_ want to expose this in eg a
> descendant report you want to give to somebody.
>
> So I do not agree with your interpretation of this as a bug. The proxy sees
> another database, where all private info is removed. It works as created.
>
> Changing private flag on objects should never be needed, so suggesting users
> to change private flags is not the way to go.
>
> As far as I see it, these are the use-cases:
>
> 1/ this is private, I really don't want anybody to know about this info. So
> eg ex-wifes, estranged brothers, illegal children. In all these cases,
> nothing should seep out to the outside world by accident. The consequences
> of such accidents lead to huge problems. I am very happy no users complained
> anymore the last two years about this on the user list. This is why the
> private proxie was created, and it seems to work excellent in doing this.
>
> 2/I don't want to see living people. Then all should be removed of those.
>
> The 'bug' of Jerome is hence no bug. If one is allowed to know about a
> private person, then it should not be set as private. Furthermore, for his
> own use, a researcher should never use the private flag, so if a researcher
> wants to know if somebody is family, ..., he should not activate the private
> exclusion.
>
> Benny
>
>>
>>
>>> Another argument is that what one considers private can change
>>> depending on the use of the output. For example, in one export, you
>>> might want to consider C and all of their family as off-limits.
>>> However, you might want to then turn around and create an export that
>>> has D (say, C's spouse) as private, and all of their family as
>>> off-limits. This would require a lot of work from the user to be able
>>> to do this without private-proxy-first.
>>>
>>>
>>
>> This is a better argument.  I can see why people would find this useful.
>>
>> If I understand you correctly, the export is descendants of A.  In one
>> case you want to also exclude descendants of C, and in the other descendants
>> of D.  An easy way to do this is to change the private flag from C to D.
>>
>> The only way I use the private flag is more static - I don't change
>> private flags for a particular export.  Do we want to encourage users to
>> change private flags for a particular export?
>>
>> The alternative would be to create more complicated filters.  Would this
>> be too difficult for the average user?
>>
>>>> I should add that I don't use this functionality much myself, but an
>>>> "apply
>>>> proxies before or after" option doesn't seem like a neat solution to me.
>>>>
>>>
>>> Yes, looking for the best solution, consider ease-of-use, power, and
>>> flexibility.
>>>
>>>
>>
>> I'm sure that you'll come up with a good solution.  I just wanted to
>> remind people of new and Aunt Martha type users and to avoid adding
>> complexity if it is not needed.
>>
>>
>> Nick.
>>
>>
>>> -Doug
>>>
>>>
>>>>
>>>> Regards,
>>>>
>>>>
>>>> Nick.
>>>>
>>>>
>>>> Doug Blank wrote:
>>>>
>>>>>
>>>>> On Tue, May 18, 2010 at 3:10 AM, Benny Malengier
>>>>> <ben...@gm...> wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> 2010/5/17 Doug Blank <dou...@gm...>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Mon, May 17, 2010 at 10:05 AM, Gerald Britton
>>>>>>> <ger...@gm...> wrote:
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, May 17, 2010 at 8:22 AM, Doug Blank <dou...@gm...>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, May 17, 2010 at 4:32 AM, Benny Malengier
>>>>>>>>> <ben...@gm...> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2010/5/12 Gerald Britton <ger...@gm...>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [snip]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> The same thing is done for filters when you use the rule "people
>>>>>>>>>>> matching filter x."  If that is the first rule in a new filter
>>>>>>>>>>> "y",
>>>>>>>>>>> then the other rules in "y" will only see records that pass
>>>>>>>>>>> filter
>>>>>>>>>>> "x".  The beauty here is that you can reverse the rules so the
>>>>>>>>>>> "people
>>>>>>>>>>> matching filter" rule comes last.  You have the flexibility to
>>>>>>>>>>> put
>>>>>>>>>>> it
>>>>>>>>>>> together to get the results you want.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I agree here. There is no bug in the proxie databases, they work
>>>>>>>>>> as
>>>>>>>>>> they
>>>>>>>>>> should.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Yes, I agree that there is no bug in the proxies, nor in the
>>>>>>>>> filters.
>>>>>>>>>
>>>>>>>>> I've been continuing to play with proxies and filters, and I'm not
>>>>>>>>> sure that just changing the order is the right thing to do. But we
>>>>>>>>> all
>>>>>>>>> have to at least agree on what the issues are.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The proxies are not commutative however according to Doug, first
>>>>>>>>>> filter,
>>>>>>>>>> then private is different from first private then filter proxie
>>>>>>>>>> (see
>>>>>>>>>> the
>>>>>>>>>> descendant of and then private versus private then descendant).
>>>>>>>>>> Thinking
>>>>>>>>>> about this, this seems indeed strange, as Gerald indicates, they
>>>>>>>>>> should
>>>>>>>>>> return the same result...
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This is easy to see that Filters/Proxies are not commutative:
>>>>>>>>> Consider
>>>>>>>>> a database with three people: Person A who has a son, Person B.
>>>>>>>>> Person
>>>>>>>>> B also has a son, Person C. Now mark Person B as private.
>>>>>>>>>
>>>>>>>>> If you apply the Private proxy first, then you effectively remove
>>>>>>>>> Person B from the database, and you will not find any connection
>>>>>>>>> between Person A and Person C. Thus, any filter that looks for a
>>>>>>>>> relationship between A and C will not find one. Descendants of A
>>>>>>>>> will
>>>>>>>>> not include C. This is how Gramps has worked for the last few
>>>>>>>>> years,
>>>>>>>>> when you export with a Private Proxy.
>>>>>>>>>
>>>>>>>>> On the other hand, if you do not apply the Private proxy first,
>>>>>>>>> there
>>>>>>>>> is no easy way to hide that relationship from the filters. You can
>>>>>>>>> try
>>>>>>>>> to make selections based on one's Private status, but that isn't
>>>>>>>>> considered when, for example, looking for descendants or ancestors.
>>>>>>>>>
>>>>>>>>> So, the first thing to realize is that filter-then-proxy is not
>>>>>>>>> guaranteed to be the same as proxy-then-filter. They are not
>>>>>>>>> commutative.
>>>>>>>>>
>>>>>>>>> The next thing to realize is that sometimes one may want to use
>>>>>>>>> private people in a filter, before having them removed. Thus,
>>>>>>>>> always
>>>>>>>>> applying the private proxy first may have effects on the filter
>>>>>>>>> that
>>>>>>>>> you may not realize, until you run an export.
>>>>>>>>>
>>>>>>>>> The final thing to realize is that a private filter does not take
>>>>>>>>> the
>>>>>>>>> place of the private proxy. Using a private filter only looks at a
>>>>>>>>> single person. Combining a private filter with, say, a descendant
>>>>>>>>> filter does not have any effect on limiting who is a descendant. As
>>>>>>>>> far as I can see, filters are commutative, and thus cannot be used
>>>>>>>>> to
>>>>>>>>> replace a proxy.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The change of Doug means private is applied last. There is no real
>>>>>>>>>> reasoning
>>>>>>>>>> on why one or the other should be first, ..., well, perhaps there
>>>>>>>>>> is?
>>>>>>>>>> If
>>>>>>>>>> they really are not commutative, I think the order should be in
>>>>>>>>>> the
>>>>>>>>>> way that
>>>>>>>>>> is most difficult (or impossible) to achieve via a normal filter
>>>>>>>>>> setup.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> If Benny's criteria is used, then the way that Gramps is now is the
>>>>>>>>> correct way: there is no easy way (that I see) to remove Person B
>>>>>>>>> (private) and Person C (public descendant) using just filters.
>>>>>>>>>
>>>>>>>>> On the other hand, selecting the proxy first can remove people that
>>>>>>>>> were important in constructing a filter, and a user might not
>>>>>>>>> notice
>>>>>>>>> until the export has been made, and then verified by examining what
>>>>>>>>> was exported.
>>>>>>>>>
>>>>>>>>> So what to do? I see four possibilities:
>>>>>>>>>
>>>>>>>>> 1) Leave as it is, Proxies first. To avoid having a Private proxy
>>>>>>>>> have
>>>>>>>>> adverse affects, a user can opt to not use the proxy, and use a
>>>>>>>>> private filter. I believe that a private filter is equivalent to a
>>>>>>>>> private proxy last. But this method causes confusion, even among
>>>>>>>>> long-time users (and developers) for why an exported group is what
>>>>>>>>> it
>>>>>>>>> is.
>>>>>>>>>
>>>>>>>>> 2) Change to proxies last. This is too limiting---users don't have
>>>>>>>>> the
>>>>>>>>> power that they need with just filters.
>>>>>>>>>
>>>>>>>>> 3) Allow each proxy to be applied before or after. Users would need
>>>>>>>>> to
>>>>>>>>> know the implications. However, this is still confusing and hard to
>>>>>>>>> verify.
>>>>>>>>>
>>>>>>>>> 4) Allow proxies to be used as a filter. I have no idea how this
>>>>>>>>> might
>>>>>>>>> work, but it would allow users to verify the selection and final
>>>>>>>>> output.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> Do you mean, "allow the logic of the proxies to be used in
>>>>>>>> pre-defined
>>>>>>>> filters"?  If so, I can see the value in that, though the "private"
>>>>>>>> proxy goes well beyond filtering.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> To understand the proposed option 4, "allow proxies to be used as a
>>>>>>> filter" one needs to appreciate that the three non-filter based
>>>>>>> proxies (Private, Living, and Note) are more powerful than
>>>>>>> user-constructed filters. That is, they alter what is accessible. For
>>>>>>> example, no combination of filters will prevent you from being able
>>>>>>> to
>>>>>>> select the children of a private person in a descendant filter. Only
>>>>>>> a
>>>>>>> proxy can do that.
>>>>>>>
>>>>>>> What I am suggesting is some manner to put the power of the proxy
>>>>>>> into
>>>>>>> the hands of the user, so that they can mix proxies
>>>>>>> between/before/after filters. This does two things: 1) it allows
>>>>>>> users
>>>>>>> to interactively and easily inspect their application of proxies and
>>>>>>> filters to check for correctness (without having to export and then
>>>>>>> import), and 2) it gives users the most control over when and how the
>>>>>>> proxies get applied.
>>>>>>>
>>>>>>> Having thoroughly examined the two use-cases in this thread (Jerome's
>>>>>>> and Benny's) we can get by with our current sequence (proxies before
>>>>>>> filters). Benny's example is the default behavior when applying the
>>>>>>> Private proxy (eg, block access to children of private people when
>>>>>>> combined with a descendant filter). Jerome can get exactly what he
>>>>>>> wants (I think) by writing a more complicated filter. The easiest way
>>>>>>> to construct Jerome's filter is with three filters, and no proxies:
>>>>>>>
>>>>>>> 1) not an ancestor of X filter
>>>>>>> 2) not private filter
>>>>>>> 3) mark "all rules must apply"
>>>>>>>
>>>>>>> This can also be accomplished with one filter, 2 rules:
>>>>>>>
>>>>>>> 1) an ancestor of X rule
>>>>>>> 2) private rule
>>>>>>> 3) mark "at least one rule must apply"
>>>>>>> 4) mark "those that do not match"
>>>>>>>
>>>>>>> BTW, Jerome's selection could also be done perhaps easier if we had
>>>>>>> the ability to add the "not" to each rule rather than to the entire
>>>>>>> filter selection. For example, one could select:
>>>>>>>
>>>>>>> 1) not ancestors of X rule
>>>>>>> 2) not private rule
>>>>>>> 3) mark "all rules must apply"
>>>>>>>
>>>>>>> What does not work in this case is to select "not ancestor of X"
>>>>>>> filter with the Private proxy (as it is applied first), as that
>>>>>>> messes
>>>>>>> up the ancestors (because it happens to have a private person in the
>>>>>>> ancestor chain).
>>>>>>>
>>>>>>> This suggests Option 3, where Jerome could have gotten the output he
>>>>>>> wanted with a "not ancestor of X" with the Private proxy applied
>>>>>>> last.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> I would not like to see the actual
>>>>>>>> modules available for use as filters, since that would likely
>>>>>>>> require
>>>>>>>> logic to be added to them of the sort:
>>>>>>>>
>>>>>>>> if I am a proxy:
>>>>>>>>  do this
>>>>>>>> else:
>>>>>>>>  do that
>>>>>>>>
>>>>>>>> or would require the caller to be knowledgeable of the
>>>>>>>> implementation
>>>>>>>> proxy, which can lead to maintenance headaches.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> Yeah, I can't see how we could mix proxies and filters together,
>>>>>>> other
>>>>>>> than having an interface like we do in export---allowing one to
>>>>>>> select
>>>>>>> the proxies and their ordering. One could do that in the filter
>>>>>>> sidebar... seeing as that would be complicated (and I have no
>>>>>>> interest
>>>>>>> in doing it) perhaps we can slightly tweak the export to do two
>>>>>>> fairly
>>>>>>> simple things:
>>>>>>>
>>>>>>> 1) allow proxies to be applied before/after the filter
>>>>>>> 2) right before the actual export, the display could show who would
>>>>>>> be
>>>>>>> selected
>>>>>>>
>>>>>>> These two things would go some distance to making the export more
>>>>>>> user
>>>>>>> friendly. It doesn't allow all use-cases (I can think of some, but
>>>>>>> they are not uses that people are asking for).
>>>>>>>
>>>>>>> -Doug
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> This tread is becoming difficult to follow :-)
>>>>>>
>>>>>>
>>>>>
>>>>> This thread was difficult to write! This is complicated. I'm glad
>>>>> we're taking this slow and thoughtful.
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> From what I read, am I correct to assume you would undo the change in
>>>>>> svn?
>>>>>> So first private again, because the other behaviour can be obtained
>>>>>> with
>>>>>> filters?
>>>>>>
>>>>>>
>>>>>
>>>>> Yes, even with the problems it raises, private proxy first has to be
>>>>> at least an option.
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> I agree with 1) above, but for users better to use another word
>>>>>> (restricted
>>>>>> view or so). For 2), more like a preview button, people with large
>>>>>> databases
>>>>>> would not like to wait to see the result.
>>>>>>
>>>>>>
>>>>>
>>>>> Perhaps for 2) we can give more details (eg, display who is selected)
>>>>> as we are exporting, and a way to cancel.
>>>>>
>>>>> I think "view" is the right word, in description, and also related to
>>>>> what Gerald said. We have effectively created a view:
>>>>>
>>>>> self.dbase =  \
>>>>>  gen.proxy.ReferencedProxyDb(
>>>>>  gen.proxy.LivingProxyDb(
>>>>>    gen.proxy.PrivateProxyDb(
>>>>>      gen.proxy.FilterProxyDb(
>>>>>          gen.proxy.FilterProxyDb(
>>>>>             self.dbase,
>>>>>             option_box.cfilter),
>>>>>          note_filter=option_box.nfilter
>>>>>      )
>>>>>    ),
>>>>>   gen.proxy.LivingProxyDb.MODE_INCLUDE_LAST_NAME_ONLY)
>>>>>  )
>>>>>
>>>>> A "View" (what Gerald called a meta-proxy) might be a useful object to
>>>>> have users construct in a future Gramps to get the full power of what
>>>>> is possible. We have all of the code to do it---we're just missing a
>>>>> UI.
>>>>>
>>>>> For now, I will undo the changes in trunk, and try an ordering choice.
>>>>>
>>>>> -Doug
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Benny
>>>>>>
>>>>>>
>>>>>>
>>>>>>>>
>>>>>>>> Also, the filter proxy already uses filters, so that would be
>>>>>>>> redundant I suppose.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> The ideal would be option 4, but this might be hard or impossible,
>>>>>>>>> but
>>>>>>>>> should be explored. Option 3 is easier than 1 for the user, covers
>>>>>>>>> all
>>>>>>>>> known use-cases, but is still hard to verify. Option 2, I think, is
>>>>>>>>> out because it makes Gramps less powerful. Option 1 prevents some
>>>>>>>>> known use-cases.
>>>>>>>>>
>>>>>>>>> -Doug
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Benny
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ------------------------------------------------------------------------------
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Gramps-devel mailing list
>>>>>>>>> Gra...@li...
>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Gerald Britton
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>>
>>>>> _______________________________________________
>>>>> Gramps-devel mailing list
>>>>> Gra...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/gramps-devel
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>>>
>
>



-- 
Gerald Britton