Menu
▾
▴
[Vufind-admins] [VuFind Issues] (VUFIND-661) Undocumented side-effect of sharding across cores w/ different schema (aka field stripping)
|
From: David M. (JIRA) <dem...@vi...> - 2012-09-17 11:36:28
|
<style>
/* Changing the layout to use less space for mobiles */
@media screen and (max-device-width: 480px), screen and (-webkit-min-device-pixel-ratio: 2) {
#email-body { min-width: 30em !important; }
#email-page { padding: 8px !important; }
#email-banner { padding: 8px 8px 0 8px !important; }
#email-avatar { margin: 1px 8px 8px 0 !important; padding: 0 !important; }
#email-fields { padding: 0 8px 8px 8px !important; }
#email-gutter { width: 0 !important; }
}
</style>
<div id="email-body">
<table id="email-wrap" align="center" border="0" cellpadding="0" cellspacing="0" style="background-color:#f0f0f0;color:#000000;width:100%;">
<tr valign="top">
<td id="email-page" style="padding:16px !important;">
<table align="center" border="0" cellpadding="0" cellspacing="0" style="background-color:#ffffff;border:1px solid #bbbbbb;color:#000000;width:100%;">
<tr valign="top">
<td bgcolor="#FFFFFF" style="background-color:#FFFFFF;color:#395fdb;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;line-height:1;"><img src="http://vufind.org/demo/images/vufind.jpg" alt="" style="vertical-align:top;" /></td>
</tr><tr valign="top">
<td id="email-banner" style="padding:32px 32px 0 32px;">
<table align="left" border="0" cellpadding="0" cellspacing="0" width="100%" style="width:100%;">
<tr valign="top">
<td style="color:#505050;font-family:Arial,FreeSans,Helvetica,sans-serif;padding:0;">
<img id="email-avatar" src="http://vufind.org/jira/secure/useravatar?avatarId=10072" alt="" height="48" width="48" border="0" align="left" style="padding:0;margin: 0 16px 16px 0;" />
<div id="email-action" style="padding: 0 0 8px 0;font-size:12px;line-height:18px;">
<a class="user-hover" rel="dmaus" id="email_dmaus" href="http://vufind.org/jira/secure/ViewProfile.jspa?name=dmaus" style="color:#326ca6;">David Maus</a>
created <img src="http://vufind.org/jira/images/icons/bug.gif" height="16" width="16" border="0" align="absmiddle" alt="Bug"> <a style='color:#326ca6;text-decoration:none;' href='http://vufind.org/jira/browse/VUFIND-661'>VUFIND-661</a>
</div>
<div id="email-summary" style="font-size:16px;line-height:20px;padding:2px 0 16px 0;">
<a style='color:#326ca6;text-decoration:none;' href='http://vufind.org/jira/browse/VUFIND-661'><strong>Undocumented side-effect of sharding across cores w/ different schema (aka field stripping)</strong></a>
</div>
</td>
</tr>
</table>
</td>
</tr>
<tr valign="top">
<td id="email-fields" style="padding:0 32px 32px 32px;">
<table border="0" cellpadding="0" cellspacing="0" style="padding:0;text-align:left;width:100%;" width="100%">
<tr valign="top">
<td id="email-gutter" style="width:64px;white-space:nowrap;"></td>
<td>
<table border="0" cellpadding="0" cellspacing="0" width="100%">
<tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Issue Type:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
<img src="http://vufind.org/jira/images/icons/bug.gif" height="16" width="16" border="0" align="absmiddle" alt="Bug"> Bug
</td>
</tr> <tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Affects Versions:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
1.3, 2.0beta </td>
</tr>
<tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Assignee:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
Unassigned </td>
</tr> <tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Components:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
Documentation, Search </td>
</tr>
<tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Created:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
17/Sep/12 7:35 AM
</td>
</tr> <tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Description:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
Subissue of <a href="http://vufind.org/jira/browse/VUFIND-316" title="Use multiple indices for search"><strike>VUFIND-316</strike></a>
<br/>
<br/>
Quote:
<br/>
---
<br/>
We definitely need to get the terminology straight: SHARDING is splitting a single index across multiple cores. Searching across cores with different schema is a ... hacky implementation of sharding and called MULTIINDEX (or MINDEX) for short. Sharding is easy, mindex is not.
<br/>
<br/>
Stripping fields, sys\Solr::getStrippedFields(): VF creates an array of the stripped fields based on the selected shards. This array is used in sys\Solr::isStripped() in order to decide whether a particular field is stripped or not. The magic phrase is "stripped fields based on the selected shards". If a shard is not selected, we don't need to strip its special fields, right?
<br/>
<br/>
Demian mentions weird results (11/Feb/11 8:17) and here is one:
<br/>
<br/>
Assume:
<br/>
<br/>
- two cores A and B. Core A has fields a and b, B only has field a
<br/>
- a user query q that finally searches in (a OR b)
<br/>
- q in A matches 0 records in a and n records in b (with n > 0)
<br/>
- q in B matches 0 records in a
<br/>
<br/>
How many records does the search in mindex(A,B) return?
<br/>
<br/>
For mindex(A,B) VF strips the field b, thus we get 0 matches in A and 0 matches in B.
<br/>
<br/>
Adding an index from the user interface might remove records from the result set.
<br/>
---
<br/>
<br/>
This side-effect of field stripping implementation is undocumented.
</td>
</tr>
<tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Project:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
<a style="color:#326ca6;" href="http://vufind.org/jira/browse/VUFIND">VuFind</a>
</td>
</tr> <tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Labels:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
sharding,
mindex
</td>
</tr>
<tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Priority:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
<img src="http://vufind.org/jira/images/icons/priority_critical.gif" height="16" width="16" border="0" align="absmiddle" alt="Critical"> Critical
</td>
</tr> <tr valign="top">
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 10px 10px 0;white-space:nowrap;">
<strong style="font-weight:normal;color:#505050;">Reporter:</strong>
</td>
<td style="color:#000000;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:12px;padding:0 0 10px 0;width:100%;">
<a class="user-hover" rel="dmaus" id="email_dmaus" href="http://vufind.org/jira/secure/ViewProfile.jspa?name=dmaus" style="color:#326ca6;">David Maus</a>
</td>
</tr> </table>
</td>
</tr>
</table>
</td>
</tr>
</table>
</td><!-- End #email-page -->
</tr>
<tr valign="top">
<td style="color:#505050;font-family:Arial,FreeSans,Helvetica,sans-serif;font-size:10px;line-height:14px;padding: 0 16px 16px 16px;text-align:center;">
This message is automatically generated by JIRA.<br />
If you think it was sent incorrectly, please contact your JIRA administrators<br />
For more information on JIRA, see: <a style='color:#326ca6;' href='http://www.atlassian.com/software/jira'>http://www.atlassian.com/software/jira</a>
</td>
</tr>
</table><!-- End #email-wrap -->
</div><!-- End #email-body -->
|
![http://vufind.org/demo/images/vufind.jpg"](http://vufind.org/demo/images/vufind.jpg"){kind=link}
![http://vufind.org/jira/images/icons/bug.gif"](http://vufind.org/jira/images/icons/bug.gif"){kind=link}
![http://vufind.org/jira/images/icons/priority_critical.gif"](http://vufind.org/jira/images/icons/priority_critical.gif"){kind=link}