Menu
▾
▴
bigdata-commit — Commit traffic.
This list is closed, nobody may subscribe to it.
| 2010 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
(139) |
Aug
(94) |
Sep
(232) |
Oct
(143) |
Nov
(138) |
Dec
(55) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2011 |
Jan
(127) |
Feb
(90) |
Mar
(101) |
Apr
(74) |
May
(148) |
Jun
(241) |
Jul
(169) |
Aug
(121) |
Sep
(157) |
Oct
(199) |
Nov
(281) |
Dec
(75) |
| 2012 |
Jan
(107) |
Feb
(122) |
Mar
(184) |
Apr
(73) |
May
(14) |
Jun
(49) |
Jul
(26) |
Aug
(103) |
Sep
(133) |
Oct
(61) |
Nov
(51) |
Dec
(55) |
| 2013 |
Jan
(59) |
Feb
(72) |
Mar
(99) |
Apr
(62) |
May
(92) |
Jun
(19) |
Jul
(31) |
Aug
(138) |
Sep
(47) |
Oct
(83) |
Nov
(95) |
Dec
(111) |
| 2014 |
Jan
(125) |
Feb
(60) |
Mar
(119) |
Apr
(136) |
May
(270) |
Jun
(83) |
Jul
(88) |
Aug
(30) |
Sep
(47) |
Oct
(27) |
Nov
(23) |
Dec
|
| 2015 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(3) |
Oct
|
Nov
|
Dec
|
| 2016 |
Jan
|
Feb
|
Mar
(4) |
Apr
(1) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: <jer...@us...> - 2014-05-09 23:08:38
|
Revision: 8259
http://sourceforge.net/p/bigdata/code/8259
Author: jeremy_carroll
Date: 2014-05-09 23:08:34 +0000 (Fri, 09 May 2014)
Log Message:
-----------
javadoc changes
Modified Paths:
--------------
branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/ConfigurableAnalyzerFactory.java
branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/DefaultAnalyzerFactory.java
Modified: branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/ConfigurableAnalyzerFactory.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/ConfigurableAnalyzerFactory.java 2014-05-09 22:39:19 UTC (rev 8258)
+++ branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/ConfigurableAnalyzerFactory.java 2014-05-09 23:08:34 UTC (rev 8259)
@@ -66,6 +66,7 @@
* Supported classes included all the natural language specific classes from Lucene, and also:
* <ul>
* <li>{@link PatternAnalyzer}
+ * <li>{@link TermCompletionAnalyzer}
* <li>{@link KeywordAnalyzer}
* <li>{@link SimpleAnalyzer}
* <li>{@link StopAnalyzer}
@@ -76,7 +77,6 @@
* <ul>
* <li>no arguments
* <li>{@link Version}
- * <li>{@link Set} (of strings, the stop words)
* <li>{@link Version}, {@link Set}
* </ul>
* is usable. If the class has a static method named <code>getDefaultStopSet()</code> then this is assumed
@@ -89,10 +89,6 @@
* abbreviate to <code>c.b.s.C</code> in this documentation.
* Properties from {@link Options} apply to the factory.
* <p>
- *
- * If there are no such properties at all then the property {@link Options#NATURAL_LANGUAGE_SUPPORT} is set to true,
- * and the behavior of this class is the same as the legacy {@link DefaultAnalyzerFactory}.
- * <p>
* Other properties, from {@link AnalyzerOptions} start with
* <code>c.b.s.C.analyzer.<em>language-range</em></code> where <code><em>language-range</em></code> conforms
* with the extended language range construct from RFC 4647, section 2.2.
@@ -103,7 +99,7 @@
* If no analyzer is specified for the language range <code>*</code> then the {@link StandardAnalyzer} is used.
* <p>
* Given any specific language, then the analyzer matching the longest configured language range,
- * measured in number of subtags is used {@link #getAnalyzer(String, boolean)}
+ * measured in number of subtags is returned by {@link #getAnalyzer(String, boolean)}
* In the event of a tie, the alphabetically first language range is used.
* The algorithm to find a match is "Extended Filtering" as defined in section 3.3.2 of RFC 4647.
* <p>
@@ -132,11 +128,11 @@
/**
* This is an implementation of RFC 4647 language range,
- * targetted at some of the context of bigdata, and only
+ * targetted at the specific needs within bigdata, and only
* supporting the extended filtering specified in section 3.3.2
* <p>
* Language ranges are comparable so that
- * sorting an array and then matching a language tage against each
+ * sorting an array and then matching a language tag against each
* member of the array in sequence will give the longest match.
* i.e. the longer ranges come first.
* @author jeremycarroll
Modified: branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/DefaultAnalyzerFactory.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/DefaultAnalyzerFactory.java 2014-05-09 22:39:19 UTC (rev 8258)
+++ branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/DefaultAnalyzerFactory.java 2014-05-09 23:08:34 UTC (rev 8259)
@@ -51,18 +51,15 @@
import com.bigdata.btree.keys.KeyBuilder;
/**
- * This is the default implementation but could be regarded as legacy since
+ * This is the default implementation but should be regarded as legacy since
* it fails to use the correct {@link Analyzer} for almost all languages (other than
- * English). It uses the correct natural language analyzer for literals tagged with
+ * English). It uses the correct natural language analyzer only for literals tagged with
+ * certain three letter ISO 639 codes:
* "por", "deu", "ger", "zho", "chi", "jpn", "kor", "ces", "cze", "dut", "nld", "gre", "ell",
- * "fra", "fre", "rus" and "tha".
- * This codes do not work if they are used with subtags, e.g. "ger-AT" is treated as English.
- * No two letter code works correctly: note that the W3C and
+ * "fra", "fre", "rus" and "tha". All other tags are treated as English.
+ * These codes do not work if they are used with subtags, e.g. "ger-AT" is treated as English.
+ * No two letter code, other than "en" works correctly: note that the W3C and
* IETF recommend the use of the two letter forms instead of the three letter forms.
- * <p>
- * Default implementation registers a bunch of {@link Analyzer}s for various
- * language codes and then serves the appropriate {@link Analyzer} based on
- * the specified language code.
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
* @deprecated Using {@link ConfigurableAnalyzerFactory} with
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <jer...@us...> - 2014-05-09 22:39:23
|
Revision: 8258
http://sourceforge.net/p/bigdata/code/8258
Author: jeremy_carroll
Date: 2014-05-09 22:39:19 +0000 (Fri, 09 May 2014)
Log Message:
-----------
Documentation and formatting etc.
Modified Paths:
--------------
branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/ConfigurableAnalyzerFactory.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractAnalyzerFactoryTest.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestUnconfiguredAnalyzerFactory.java
Modified: branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/ConfigurableAnalyzerFactory.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/ConfigurableAnalyzerFactory.java 2014-05-09 22:39:10 UTC (rev 8257)
+++ branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/ConfigurableAnalyzerFactory.java 2014-05-09 22:39:19 UTC (rev 8258)
@@ -95,8 +95,10 @@
* <p>
* Other properties, from {@link AnalyzerOptions} start with
* <code>c.b.s.C.analyzer.<em>language-range</em></code> where <code><em>language-range</em></code> conforms
- * with the extended language range construct from RFC 4647, section 2.2. These are used to specify
- * an analyzer for the given language range.
+ * with the extended language range construct from RFC 4647, section 2.2.
+ * There is an issue that bigdata does not allow '*' in property names, and we use the character '_' to
+ * substitute for '*' in extended language ranges in property names.
+ * These are used to specify an analyzer for the given language range.
* <p>
* If no analyzer is specified for the language range <code>*</code> then the {@link StandardAnalyzer} is used.
* <p>
@@ -113,6 +115,8 @@
* <dd>This uses whitespace to tokenize</dd>
* <dt>{@link PatternAnalyzer}</dt>
* <dd>This uses a regular expression to tokenize</dd>
+ * <dt>{@link TermCompletionAnalyzer}</dt>
+ * <dd>This uses up to three regular expressions to specify multiple tokens for each word, to address term completion use cases.</dd>
* <dt>{@link EmptyAnalyzer}</dt>
* <dd>This suppresses the functionality, by treating every expression as a stop word.</dd>
* </dl>
@@ -126,11 +130,26 @@
public class ConfigurableAnalyzerFactory implements IAnalyzerFactory {
final private static transient Logger log = Logger.getLogger(ConfigurableAnalyzerFactory.class);
- static class LanguageRange implements Comparable<LanguageRange> {
+ /**
+ * This is an implementation of RFC 4647 language range,
+ * targetted at some of the context of bigdata, and only
+ * supporting the extended filtering specified in section 3.3.2
+ * <p>
+ * Language ranges are comparable so that
+ * sorting an array and then matching a language tage against each
+ * member of the array in sequence will give the longest match.
+ * i.e. the longer ranges come first.
+ * @author jeremycarroll
+ *
+ */
+ public static class LanguageRange implements Comparable<LanguageRange> {
private final String range[];
private final String full;
-
+ /**
+ * Note range must be in lower case, this is not verified.
+ * @param range
+ */
public LanguageRange(String range) {
this.range = range.split("-");
full = range;
@@ -174,12 +193,22 @@
return full.hashCode();
}
+ /**
+ * This implements the algoirthm of section 3.3.2 of RFC 4647
+ * as modified with the observation about private use tags
+ * in <a href="http://lists.w3.org/Archives/Public/www-international/2014AprJun/0084">
+ * this message</a>.
+ *
+ *
+ * @param langTag The RFC 5646 Language tag in lower case
+ * @return The result of the algorithm
+ */
public boolean extendedFilterMatch(String langTag) {
return extendedFilterMatch(langTag.toLowerCase(Locale.ROOT).split("-"));
}
// See RFC 4647, 3.3.2
- public boolean extendedFilterMatch(String[] language) {
+ boolean extendedFilterMatch(String[] language) {
// RFC 4647 step 2
if (!matchSubTag(language[0], range[0])) {
return false;
@@ -227,13 +256,14 @@
*/
public interface Options {
/**
- * By setting this option to true, then the behavior of the legacy {@link DefaultAnalyzerFactory}
- * is added, and may be overridden by the settings of the user.
+ * By setting this option to true, then all the known Lucene Analyzers for natural
+ * languages are used for a range of language tags.
+ * These settings may then be overridden by the settings of the user.
* Specifically the following properties are loaded, prior to loading the
* user's specification (with <code>c.b.s.C</code> expanding to
* <code>com.bigdata.search.ConfigurableAnalyzerFactory</code>)
<pre>
-c.b.s.C.analyzer.*.like=eng
+c.b.s.C.analyzer._.like=eng
c.b.s.C.analyzer.por.analyzerClass=org.apache.lucene.analysis.br.BrazilianAnalyzer
c.b.s.C.analyzer.pt.like=por
c.b.s.C.analyzer.zho.analyzerClass=org.apache.lucene.analysis.cn.ChineseAnalyzer
@@ -281,7 +311,9 @@
/**
* If specified this is the fully qualified name of a subclass of {@link Analyzer}
* that has appropriate constructors.
- * Either this or {@link #LIKE} or {@link #PATTERN} must be specified for each language range.
+ * This is set implicitly if some of the options below are selected (for example {@link #PATTERN}).
+ * For each configured language range, if it is not set, either explicitly or implicitly, then
+ * {@link #LIKE} must be specified.
*/
String ANALYZER_CLASS = "analyzerClass";
@@ -399,24 +431,64 @@
private static final String LUCENE_STANDARD_ANALYZER =
"com.bigdata.search.ConfigurableAnalyzerFactory.analyzer.*.analyzerClass=org.apache.lucene.analysis.standard.StandardAnalyzer\n";
+ /**
+ * This comment describes the implementation of {@link ConfigurableAnalyzerFactory}.
+ * The only method in the interface is {@link ConfigurableAnalyzerFactory#getAnalyzer(String, boolean)},
+ * a map is used from language tag to {@link AnalyzerPair}, where the pair contains
+ * an {@link Analyzer} both with and without stopwords configured (some times these two analyzers are identical,
+ * if, for example, stop words are not supported or not required).
+ * <p>
+ * If there is no entry for the language tag in the map {@link ConfigurableAnalyzerFactory#langTag2AnalyzerPair},
+ * then one is created, by walking down the array {@link ConfigurableAnalyzerFactory#config} of AnalyzerPairs
+ * until a matching one is found.
+ * <p>
+ * The bulk of the code in this class is invoked from the constructor in order to set up this
+ * {@link ConfigurableAnalyzerFactory#config} array. For example, all of the subclasses of {@link AnalyzerPair}s,
+ * are simply to call the appropriate constructor in the appropriate way: the difficulty is that many subclasses
+ * of {@link Analyzer} have constructors with different signatures, and our code needs to navigate each sort.
+ * @author jeremycarroll
+ *
+ */
private static class AnalyzerPair implements Comparable<AnalyzerPair>{
- private final LanguageRange range;
+ final LanguageRange range;
private final Analyzer withStopWords;
private final Analyzer withoutStopWords;
+ public Analyzer getAnalyzer(boolean filterStopwords) {
+ return filterStopwords ? withStopWords : withoutStopWords;
+ }
+
+ public boolean extendedFilterMatch(String[] language) {
+ return range.extendedFilterMatch(language);
+ }
+
AnalyzerPair(String range, Analyzer withStopWords, Analyzer withOutStopWords) {
this.range = new LanguageRange(range);
this.withStopWords = withStopWords;
this.withoutStopWords = withOutStopWords;
}
+ /**
+ * This clone constructor implements {@link AnalyzerOptions#LIKE}.
+ * @param range
+ * @param copyMe
+ */
AnalyzerPair(String range, AnalyzerPair copyMe) {
this.range = new LanguageRange(range);
this.withStopWords = copyMe.withStopWords;
this.withoutStopWords = copyMe.withoutStopWords;
-
}
+ /**
+ * If we have a constructor, with arguments including a populated
+ * stop word set, then we can use it to make both the withStopWords
+ * analyzer, and the withoutStopWords analyzer.
+ * @param range
+ * @param cons A Constructor including a {@link java.util.Set} argument
+ * for the stop words.
+ * @param params The arguments to pass to the constructor including a populated stopword set.
+ * @throws Exception
+ */
AnalyzerPair(String range, Constructor<? extends Analyzer> cons, Object ... params) throws Exception {
this(range, cons.newInstance(params), cons.newInstance(useEmptyStopWordSet(params)));
}
@@ -435,9 +507,6 @@
return rslt;
}
- public Analyzer getAnalyzer(boolean filterStopwords) {
- return filterStopwords ? withStopWords : withoutStopWords;
- }
@Override
public String toString() {
return range.full + "=(" + withStopWords.getClass().getSimpleName() +")";
@@ -447,30 +516,38 @@
public int compareTo(AnalyzerPair o) {
return range.compareTo(o.range);
}
-
- public boolean extendedFilterMatch(String[] language) {
- return range.extendedFilterMatch(language);
- }
}
+ /**
+ * Used for Analyzer classes with a constructor with signature (Version, Set).
+ * @author jeremycarroll
+ *
+ */
private static class VersionSetAnalyzerPair extends AnalyzerPair {
public VersionSetAnalyzerPair(ConfigOptionsToAnalyzer lro,
Class<? extends Analyzer> cls) throws Exception {
super(lro.languageRange, getConstructor(cls, Version.class, Set.class), Version.LUCENE_CURRENT, lro.getStopWords());
}
}
-
+
+ /**
+ * Used for Analyzer classes which do not support stopwords and have a constructor with signature (Version).
+ * @author jeremycarroll
+ *
+ */
private static class VersionAnalyzerPair extends AnalyzerPair {
-
public VersionAnalyzerPair(String range, Class<? extends Analyzer> cls) throws Exception {
super(range, getConstructor(cls, Version.class).newInstance(Version.LUCENE_CURRENT));
}
}
-
+ /**
+ * Special case code for {@link PatternAnalyzer}
+ * @author jeremycarroll
+ *
+ */
private static class PatternAnalyzerPair extends AnalyzerPair {
-
public PatternAnalyzerPair(ConfigOptionsToAnalyzer lro, Pattern pattern) throws Exception {
super(lro.languageRange, getConstructor(PatternAnalyzer.class,Version.class,Pattern.class,Boolean.TYPE,Set.class),
Version.LUCENE_CURRENT,
@@ -485,6 +562,16 @@
* This class is initialized with the config options, using the {@link #setProperty(String, String)}
* method, for a particular language range and works out which pair of {@link Analyzer}s
* to use for that language range.
+ * <p>
+ * Instances of this class are only alive during the execution of
+ * {@link ConfigurableAnalyzerFactory#ConfigurableAnalyzerFactory(FullTextIndex)},
+ * the life-cycle is:
+ * <ol>
+ * <li>The relveant config properties are applied, and are used to populate the fields.
+ * <li>The fields are validated
+ * <li>An {@link AnalyzerPair} is constructed
+ * </ol>
+ *
* @author jeremycarroll
*
*/
@@ -545,6 +632,10 @@
return ( stopwords == null && pattern == null ) || AnalyzerOptions.STOPWORDS_VALUE_DEFAULT.equals(stopwords);
}
+ /**
+ * The first step in the life-cycle, used to initialize the fields.
+ * @return true if the property was recognized.
+ */
public boolean setProperty(String shortProperty, String value) {
if (shortProperty.equals(AnalyzerOptions.LIKE) ) {
like = value;
@@ -568,6 +659,9 @@
return true;
}
+ /**
+ * The second phase of the life-cycle, used for sanity checking.
+ */
public void validate() {
if (pattern != null ) {
if ( className != null && className != PatternAnalyzer.class.getName()) {
@@ -608,6 +702,10 @@
}
+ /**
+ * The third and final phase of the life-cyle used for identifying
+ * the AnalyzerPair.
+ */
private AnalyzerPair construct() throws Exception {
if (className == null) {
return null;
@@ -660,6 +758,29 @@
throw new RuntimeException("Bad option: cannot find constructor for class " + className + " for language range " + languageRange);
}
+ /**
+ * Also part of the third phase of the life-cycle, following the {@link AnalyzerOptions#LIKE}
+ * properties.
+ * @param depth
+ * @param max
+ * @param analyzers
+ * @return
+ */
+ AnalyzerPair followLikesToAnalyzerPair(int depth, int max,
+ Map<String, ConfigOptionsToAnalyzer> analyzers) {
+ if (result == null) {
+ if (depth == max) {
+ throw new RuntimeException("Bad configuration: - 'like' loop for language range " + languageRange);
+ }
+ ConfigOptionsToAnalyzer next = analyzers.get(like);
+ if (next == null) {
+ throw new RuntimeException("Bad option: - 'like' not found for language range " + languageRange+ " (not found: '"+ like +"')");
+ }
+ result = new AnalyzerPair(languageRange, next.followLikesToAnalyzerPair(depth+1, max, analyzers));
+ }
+ return result;
+ }
+
protected Class<? extends Analyzer> getAnalyzerClass() {
return getAnalyzerClass(className);
}
@@ -678,22 +799,6 @@
void setAnalyzerPair(AnalyzerPair ap) {
result = ap;
}
-
- AnalyzerPair followLikesToAnalyzerPair(int depth, int max,
- Map<String, ConfigOptionsToAnalyzer> analyzers) {
- if (result == null) {
- if (depth == max) {
- throw new RuntimeException("Bad configuration: - 'like' loop for language range " + languageRange);
- }
- ConfigOptionsToAnalyzer next = analyzers.get(like);
- if (next == null) {
- throw new RuntimeException("Bad option: - 'like' not found for language range " + languageRange+ " (not found: '"+ like +"')");
- }
- result = new AnalyzerPair(languageRange, next.followLikesToAnalyzerPair(depth+1, max, analyzers));
- }
- return result;
- }
-
}
private final AnalyzerPair config[];
@@ -712,7 +817,13 @@
private final FullTextIndex<?> fullTextIndex;
+ /**
+ * Builds a new ConfigurableAnalyzerFactory.
+ * @param fullTextIndex
+ */
public ConfigurableAnalyzerFactory(final FullTextIndex<?> fullTextIndex) {
+ // A description of the operation of this method is found on AnalyzerPair and
+ // ConfigOptionsToAnalyzer.
// despite our name, we actually make all the analyzers now, and getAnalyzer method is merely a lookup.
if (fullTextIndex == null)
@@ -837,9 +948,18 @@
protected Properties initProperties() {
final Properties parentProperties = fullTextIndex.getProperties();
Properties myProps;
- if (Boolean.valueOf(parentProperties.getProperty(Options.NATURAL_LANGUAGE_SUPPORT, Options.DEFAULT_NATURAL_LAMGUAGE_SUPPORT))) {
+ if (Boolean.valueOf(parentProperties.getProperty(
+ Options.NATURAL_LANGUAGE_SUPPORT,
+ Options.DEFAULT_NATURAL_LAMGUAGE_SUPPORT))) {
+
myProps = loadPropertyString(ALL_LUCENE_NATURAL_LANGUAGES);
+
+ } else if (hasPropertiesForStarLanguageRange(parentProperties)){
+
+ myProps = new Properties();
+
} else {
+
myProps = loadPropertyString(LUCENE_STANDARD_ANALYZER);
}
@@ -867,6 +987,17 @@
}
}
+ private boolean hasPropertiesForStarLanguageRange(Properties from) {
+ Enumeration<?> en = from.propertyNames();
+ while (en.hasMoreElements()) {
+ String prop = (String)en.nextElement();
+ if (prop.startsWith(Options.ANALYZER+"_.")
+ || prop.startsWith(Options.ANALYZER+"*.")) {
+ return true;
+ }
+ }
+ return false;
+ }
@Override
public Analyzer getAnalyzer(String languageCode, boolean filterStopwords) {
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractAnalyzerFactoryTest.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractAnalyzerFactoryTest.java 2014-05-09 22:39:10 UTC (rev 8257)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractAnalyzerFactoryTest.java 2014-05-09 22:39:19 UTC (rev 8258)
@@ -1,3 +1,29 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2014. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on May 9, 2014
+ */
package com.bigdata.search;
public abstract class AbstractAnalyzerFactoryTest extends AbstractSearchTest {
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java 2014-05-09 22:39:10 UTC (rev 8257)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java 2014-05-09 22:39:19 UTC (rev 8258)
@@ -59,7 +59,8 @@
String analyzer = ConfigurableAnalyzerFactory.Options.ANALYZER;
return new String[]{
FullTextIndex.Options.ANALYZER_FACTORY_CLASS, ConfigurableAnalyzerFactory.class.getName(),
- analyzer+"*."+AnalyzerOptions.ANALYZER_CLASS, EmptyAnalyzer.class.getName(),
+ analyzer+"_."+AnalyzerOptions.LIKE, "x-empty",
+ analyzer+"x-empty."+AnalyzerOptions.ANALYZER_CLASS, EmptyAnalyzer.class.getName(),
analyzer+"x-terms."+AnalyzerOptions.PATTERN, "\\W+",
analyzer+"x-splits."+AnalyzerOptions.ANALYZER_CLASS, TermCompletionAnalyzer.class.getName(),
analyzer+"x-splits."+AnalyzerOptions.STOPWORDS, AnalyzerOptions.STOPWORDS_VALUE_NONE,
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestUnconfiguredAnalyzerFactory.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestUnconfiguredAnalyzerFactory.java 2014-05-09 22:39:10 UTC (rev 8257)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestUnconfiguredAnalyzerFactory.java 2014-05-09 22:39:19 UTC (rev 8258)
@@ -1,3 +1,29 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2014. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on May 7, 2014
+ */
package com.bigdata.search;
public class TestUnconfiguredAnalyzerFactory extends AbstractAnalyzerFactoryTest {
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <jer...@us...> - 2014-05-09 22:39:13
|
Revision: 8257
http://sourceforge.net/p/bigdata/code/8257
Author: jeremy_carroll
Date: 2014-05-09 22:39:10 +0000 (Fri, 09 May 2014)
Log Message:
-----------
Added extra test to check that by default we use StandardAnalyzer for everything; refactored a bit
Modified Paths:
--------------
branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/ConfigurableAnalyzerFactory.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractAnalyzerFactoryTest.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractSearchTest.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestAll.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAsDefaultAnalyzerFactory.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestDefaultAnalyzerFactory.java
Added Paths:
-----------
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractDefaultAnalyzerFactoryTest.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestUnconfiguredAnalyzerFactory.java
Modified: branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/ConfigurableAnalyzerFactory.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/ConfigurableAnalyzerFactory.java 2014-05-09 19:07:09 UTC (rev 8256)
+++ branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/ConfigurableAnalyzerFactory.java 2014-05-09 22:39:10 UTC (rev 8257)
@@ -366,7 +366,7 @@
}
- private static final String DEFAULT_PROPERTIES =
+ private static final String ALL_LUCENE_NATURAL_LANGUAGES =
"com.bigdata.search.ConfigurableAnalyzerFactory.analyzer.*.like=eng\n" +
"com.bigdata.search.ConfigurableAnalyzerFactory.analyzer.por.analyzerClass=org.apache.lucene.analysis.br.BrazilianAnalyzer\n" +
"com.bigdata.search.ConfigurableAnalyzerFactory.analyzer.pt.like=por\n" +
@@ -396,6 +396,9 @@
"com.bigdata.search.ConfigurableAnalyzerFactory.analyzer.eng.analyzerClass=org.apache.lucene.analysis.standard.StandardAnalyzer\n" +
"com.bigdata.search.ConfigurableAnalyzerFactory.analyzer.en.like=eng\n";
+ private static final String LUCENE_STANDARD_ANALYZER =
+ "com.bigdata.search.ConfigurableAnalyzerFactory.analyzer.*.analyzerClass=org.apache.lucene.analysis.standard.StandardAnalyzer\n";
+
private static class AnalyzerPair implements Comparable<AnalyzerPair>{
private final LanguageRange range;
private final Analyzer withStopWords;
@@ -703,6 +706,7 @@
* strategy so the code will still work on the {@link #MAX_LANG_CACHE_SIZE}+1 th entry.
*/
private static final int MAX_LANG_CACHE_SIZE = 500;
+
private String defaultLanguage;
private final FullTextIndex<?> fullTextIndex;
@@ -833,25 +837,20 @@
protected Properties initProperties() {
final Properties parentProperties = fullTextIndex.getProperties();
Properties myProps;
- if (Boolean.getBoolean(parentProperties.getProperty(Options.NATURAL_LANGUAGE_SUPPORT, Options.DEFAULT_NATURAL_LAMGUAGE_SUPPORT))) {
- myProps = defaultProperties();
+ if (Boolean.valueOf(parentProperties.getProperty(Options.NATURAL_LANGUAGE_SUPPORT, Options.DEFAULT_NATURAL_LAMGUAGE_SUPPORT))) {
+ myProps = loadPropertyString(ALL_LUCENE_NATURAL_LANGUAGES);
} else {
- myProps = new Properties();
+ myProps = loadPropertyString(LUCENE_STANDARD_ANALYZER);
}
copyRelevantProperties(fullTextIndex.getProperties(), myProps);
-
- if (myProps.isEmpty()) {
- return defaultProperties();
- } else {
- return myProps;
- }
+ return myProps;
}
- protected Properties defaultProperties() {
+ Properties loadPropertyString(String props) {
Properties rslt = new Properties();
try {
- rslt.load(new StringReader(DEFAULT_PROPERTIES));
+ rslt.load(new StringReader(props));
} catch (IOException e) {
throw new RuntimeException("Impossible - well clearly not!", e);
}
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractAnalyzerFactoryTest.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractAnalyzerFactoryTest.java 2014-05-09 19:07:09 UTC (rev 8256)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractAnalyzerFactoryTest.java 2014-05-09 22:39:10 UTC (rev 8257)
@@ -1,153 +1,20 @@
-/**
-
-Copyright (C) SYSTAP, LLC 2006-2014. All rights reserved.
-
-Contact:
- SYSTAP, LLC
- 4501 Tower Road
- Greensboro, NC 27410
- lic...@bi...
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published by
-the Free Software Foundation; version 2 of the License.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software
-Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-*/
-/*
- * Created on May 7, 2014
- */
package com.bigdata.search;
-import java.io.IOException;
-
-
public abstract class AbstractAnalyzerFactoryTest extends AbstractSearchTest {
- public AbstractAnalyzerFactoryTest() {
+ public AbstractAnalyzerFactoryTest() {
}
-
- public AbstractAnalyzerFactoryTest(String arg0) {
- super(arg0);
+
+ public AbstractAnalyzerFactoryTest(String arg0) {
+ super(arg0);
}
-
- @Override
- public void setUp() throws Exception {
- super.setUp();
- init(getExtraProperties());
- }
-
-
- abstract String[] getExtraProperties();
-
- public void testEnglishFilterStopWords() throws IOException {
- for (String lang: new String[]{ "eng", null, "" }) { //$NON-NLS-1$ //$NON-NLS-2$
- comparisonTest(lang,
- true,
- "The test to end all tests! Forever.", //$NON-NLS-1$
- "test end all tests forever" //$NON-NLS-1$
- );
- }
- }
- public void testEnglishNoFilter() throws IOException {
- for (String lang: new String[]{ "eng", null, "" }) { //$NON-NLS-1$ //$NON-NLS-2$
- comparisonTest(lang,
- false,
- "The test to end all tests! Forever.", //$NON-NLS-1$
- "the test to end all tests forever" //$NON-NLS-1$
- );
- }
- }
-
- // Note we careful use a three letter language code for german.
- // 'de' is more standard, but the DefaultAnalyzerFactory does not
- // implement 'de' correctly.
- public void testGermanFilterStopWords() throws IOException {
- comparisonTest("ger", //$NON-NLS-1$
- true,
- NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.10") + //$NON-NLS-1$
- NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.11"), //$NON-NLS-1$
- NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.12") //$NON-NLS-1$
- );
-
- }
- // Note we careful use a three letter language code for Russian.
- // 'ru' is more standard, but the DefaultAnalyzerFactory does not
- // implement 'ru' correctly.
- public void testRussianFilterStopWords() throws IOException {
- comparisonTest("rus", //$NON-NLS-1$
- true,
- // I hope this is not offensive text.
- NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.14") + //$NON-NLS-1$
- NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.15"), //$NON-NLS-1$
- NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.16") //$NON-NLS-1$
- );
-
- }
- public void testGermanNoStopWords() throws IOException {
- comparisonTest("ger", //$NON-NLS-1$
- false,
- NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.18") + //$NON-NLS-1$
- NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.19"), //$NON-NLS-1$
- NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.20") //$NON-NLS-1$
- );
-
- }
- public void testRussianNoStopWords() throws IOException {
- comparisonTest("rus", //$NON-NLS-1$
- false,
- NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.22") + //$NON-NLS-1$
- NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.23"), //$NON-NLS-1$
- NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.24") //$NON-NLS-1$
- );
-
- }
- public void testJapanese() throws IOException {
- for (boolean filterStopWords: new Boolean[]{true, false}) {
- comparisonTest("jpn", //$NON-NLS-1$
- filterStopWords,
- NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.26"), //$NON-NLS-1$
- NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.27") + //$NON-NLS-1$
- NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.28") + //$NON-NLS-1$
- NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.29")); //$NON-NLS-1$
- }
- }
- public void testConfiguredLanguages() {
- checkConfig("BrazilianAnalyzer", "por", "pt"); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$
- checkConfig("ChineseAnalyzer", "zho", "chi", "zh"); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$ //$NON-NLS-4$
- checkConfig("CJKAnalyzer", "jpn", "ja", "kor", "ko"); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$ //$NON-NLS-4$ //$NON-NLS-5$
- checkConfig("CzechAnalyzer", "ces", "cze", "cs"); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$ //$NON-NLS-4$
- checkConfig("DutchAnalyzer", "dut", "nld", "nl"); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$ //$NON-NLS-4$
- checkConfig("GermanAnalyzer", "deu", "ger", "de"); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$ //$NON-NLS-4$
- checkConfig("GreekAnalyzer", "gre", "ell", "el"); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$ //$NON-NLS-4$
- checkConfig("RussianAnalyzer", "rus", "ru"); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$
- checkConfig("ThaiAnalyzer", "th", "tha"); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$
- checkConfig("StandardAnalyzer", "en", "eng", "", null); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$ //$NON-NLS-4$
- }
-
- private void checkConfig(String classname, String ...langs) {
- checkConfig(isBroken(), classname, langs);
-
+ @Override
+ public void setUp() throws Exception {
+ super.setUp();
+ init(getExtraProperties());
}
- protected void checkConfig(boolean threeLetterOnly, String classname, String ...langs) {
- for (String lang:langs) {
- // The DefaultAnalyzerFactory only works for language tags of length exactly three.
- if ((!threeLetterOnly) || (lang != null && lang.length()==3))
- {
- assertEquals(classname, getAnalyzer(lang,true).getClass().getSimpleName());
- if (!threeLetterOnly) assertEquals(classname, getAnalyzer(lang+"-x-foobar",true).getClass().getSimpleName()); //$NON-NLS-1$
- }
- }
-
- }
- abstract boolean isBroken() ;
+ abstract String[] getExtraProperties();
+
}
Copied: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractDefaultAnalyzerFactoryTest.java (from rev 8256, branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractAnalyzerFactoryTest.java)
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractDefaultAnalyzerFactoryTest.java (rev 0)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractDefaultAnalyzerFactoryTest.java 2014-05-09 22:39:10 UTC (rev 8257)
@@ -0,0 +1,133 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2014. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on May 7, 2014
+ */
+package com.bigdata.search;
+
+import java.io.IOException;
+
+
+public abstract class AbstractDefaultAnalyzerFactoryTest extends AbstractAnalyzerFactoryTest {
+
+ public AbstractDefaultAnalyzerFactoryTest() {
+ }
+
+ public AbstractDefaultAnalyzerFactoryTest(String arg0) {
+ super(arg0);
+ }
+
+ public void testEnglishFilterStopWords() throws IOException {
+ for (String lang: new String[]{ "eng", null, "" }) { //$NON-NLS-1$ //$NON-NLS-2$
+ comparisonTest(lang,
+ true,
+ "The test to end all tests! Forever.", //$NON-NLS-1$
+ "test end all tests forever" //$NON-NLS-1$
+ );
+ }
+ }
+ public void testEnglishNoFilter() throws IOException {
+ for (String lang: new String[]{ "eng", null, "" }) { //$NON-NLS-1$ //$NON-NLS-2$
+ comparisonTest(lang,
+ false,
+ "The test to end all tests! Forever.", //$NON-NLS-1$
+ "the test to end all tests forever" //$NON-NLS-1$
+ );
+ }
+ }
+
+ // Note we careful use a three letter language code for german.
+ // 'de' is more standard, but the DefaultAnalyzerFactory does not
+ // implement 'de' correctly.
+ public void testGermanFilterStopWords() throws IOException {
+ comparisonTest("ger", //$NON-NLS-1$
+ true,
+ NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.10") + //$NON-NLS-1$
+ NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.11"), //$NON-NLS-1$
+ NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.12") //$NON-NLS-1$
+ );
+
+ }
+
+ // Note we careful use a three letter language code for Russian.
+ // 'ru' is more standard, but the DefaultAnalyzerFactory does not
+ // implement 'ru' correctly.
+ public void testRussianFilterStopWords() throws IOException {
+ comparisonTest("rus", //$NON-NLS-1$
+ true,
+ // I hope this is not offensive text.
+ NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.14") + //$NON-NLS-1$
+ NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.15"), //$NON-NLS-1$
+ NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.16") //$NON-NLS-1$
+ );
+
+ }
+ public void testGermanNoStopWords() throws IOException {
+ comparisonTest("ger", //$NON-NLS-1$
+ false,
+ NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.18") + //$NON-NLS-1$
+ NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.19"), //$NON-NLS-1$
+ NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.20") //$NON-NLS-1$
+ );
+
+ }
+ public void testRussianNoStopWords() throws IOException {
+ comparisonTest("rus", //$NON-NLS-1$
+ false,
+ NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.22") + //$NON-NLS-1$
+ NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.23"), //$NON-NLS-1$
+ NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.24") //$NON-NLS-1$
+ );
+
+ }
+ public void testJapanese() throws IOException {
+ for (boolean filterStopWords: new Boolean[]{true, false}) {
+ comparisonTest("jpn", //$NON-NLS-1$
+ filterStopWords,
+ NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.26"), //$NON-NLS-1$
+ NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.27") + //$NON-NLS-1$
+ NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.28") + //$NON-NLS-1$
+ NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.29")); //$NON-NLS-1$
+ }
+ }
+ public void testConfiguredLanguages() {
+ checkConfig("BrazilianAnalyzer", "por", "pt"); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$
+ checkConfig("ChineseAnalyzer", "zho", "chi", "zh"); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$ //$NON-NLS-4$
+ checkConfig("CJKAnalyzer", "jpn", "ja", "kor", "ko"); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$ //$NON-NLS-4$ //$NON-NLS-5$
+ checkConfig("CzechAnalyzer", "ces", "cze", "cs"); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$ //$NON-NLS-4$
+ checkConfig("DutchAnalyzer", "dut", "nld", "nl"); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$ //$NON-NLS-4$
+ checkConfig("GermanAnalyzer", "deu", "ger", "de"); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$ //$NON-NLS-4$
+ checkConfig("GreekAnalyzer", "gre", "ell", "el"); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$ //$NON-NLS-4$
+ checkConfig("RussianAnalyzer", "rus", "ru"); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$
+ checkConfig("ThaiAnalyzer", "th", "tha"); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$
+ checkConfig("StandardAnalyzer", "en", "eng", "", null); //$NON-NLS-1$ //$NON-NLS-2$ //$NON-NLS-3$ //$NON-NLS-4$
+ }
+
+ @Override
+ protected void checkConfig(String classname, String ...langs) {
+ checkConfig(isBroken(), classname, langs);
+
+ }
+ abstract boolean isBroken() ;
+}
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractSearchTest.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractSearchTest.java 2014-05-09 19:07:09 UTC (rev 8256)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractSearchTest.java 2014-05-09 22:39:10 UTC (rev 8257)
@@ -135,13 +135,28 @@
private void compareTokenStream(Analyzer a, String text, String expected[]) throws IOException {
TokenStream s = a.tokenStream(null, new StringReader(text));
int ix = 0;
- while (s.incrementToken()) {
- final TermAttribute term = s.getAttribute(TermAttribute.class);
- final String word = term.term();
- assertTrue(ix < expected.length);
- assertEquals(expected[ix++], word);
- }
- assertEquals(ix, expected.length);
+ while (s.incrementToken()) {
+ final TermAttribute term = s.getAttribute(TermAttribute.class);
+ final String word = term.term();
+ assertTrue(ix < expected.length);
+ assertEquals(expected[ix++], word);
+ }
+ assertEquals(ix, expected.length);
}
+ protected void checkConfig(boolean threeLetterOnly, String classname, String ...langs) {
+ for (String lang:langs) {
+ // The DefaultAnalyzerFactory only works for language tags of length exactly three.
+ if ((!threeLetterOnly) || (lang != null && lang.length()==3)) {
+ assertEquals(classname, getAnalyzer(lang,true).getClass().getSimpleName());
+ if (!threeLetterOnly) {
+ assertEquals(classname, getAnalyzer(lang+"-x-foobar",true).getClass().getSimpleName()); //$NON-NLS-1$
+ }
+ }
+ }
+ }
+ protected void checkConfig(String classname, String ...langs) {
+ checkConfig(false, classname, langs);
+ }
+
}
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestAll.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestAll.java 2014-05-09 19:07:09 UTC (rev 8256)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestAll.java 2014-05-09 22:39:10 UTC (rev 8257)
@@ -115,6 +115,7 @@
// behavior of DefaultAnalyzerFactory
suite.addTestSuite(TestConfigurableAsDefaultAnalyzerFactory.class);
suite.addTestSuite(TestConfigurableAnalyzerFactory.class);
+ suite.addTestSuite(TestUnconfiguredAnalyzerFactory.class);
return suite;
}
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java 2014-05-09 19:07:09 UTC (rev 8256)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java 2014-05-09 22:39:10 UTC (rev 8257)
@@ -45,7 +45,7 @@
* @author jeremycarroll
*
*/
-public class TestConfigurableAnalyzerFactory extends AbstractSearchTest {
+public class TestConfigurableAnalyzerFactory extends AbstractAnalyzerFactoryTest {
public TestConfigurableAnalyzerFactory() {
}
@@ -54,12 +54,8 @@
super(arg0);
}
- public void setUp() throws Exception {
- super.setUp();
- init(getExtraProperties());
- }
-
- private String[] getExtraProperties() {
+ @Override
+ String[] getExtraProperties() {
String analyzer = ConfigurableAnalyzerFactory.Options.ANALYZER;
return new String[]{
FullTextIndex.Options.ANALYZER_FACTORY_CLASS, ConfigurableAnalyzerFactory.class.getName(),
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAsDefaultAnalyzerFactory.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAsDefaultAnalyzerFactory.java 2014-05-09 19:07:09 UTC (rev 8256)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAsDefaultAnalyzerFactory.java 2014-05-09 22:39:10 UTC (rev 8257)
@@ -26,7 +26,7 @@
*/
package com.bigdata.search;
-public class TestConfigurableAsDefaultAnalyzerFactory extends AbstractAnalyzerFactoryTest {
+public class TestConfigurableAsDefaultAnalyzerFactory extends AbstractDefaultAnalyzerFactoryTest {
public TestConfigurableAsDefaultAnalyzerFactory() {
}
@@ -37,7 +37,9 @@
@Override
String[] getExtraProperties() {
- return new String[]{FullTextIndex.Options.ANALYZER_FACTORY_CLASS, ConfigurableAnalyzerFactory.class.getName()};
+ return new String[]{FullTextIndex.Options.ANALYZER_FACTORY_CLASS, ConfigurableAnalyzerFactory.class.getName(),
+ ConfigurableAnalyzerFactory.Options.NATURAL_LANGUAGE_SUPPORT, "true"
+ };
}
@Override
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestDefaultAnalyzerFactory.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestDefaultAnalyzerFactory.java 2014-05-09 19:07:09 UTC (rev 8256)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestDefaultAnalyzerFactory.java 2014-05-09 22:39:10 UTC (rev 8257)
@@ -26,7 +26,7 @@
*/
package com.bigdata.search;
-public class TestDefaultAnalyzerFactory extends AbstractAnalyzerFactoryTest {
+public class TestDefaultAnalyzerFactory extends AbstractDefaultAnalyzerFactoryTest {
public TestDefaultAnalyzerFactory() {
}
Added: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestUnconfiguredAnalyzerFactory.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestUnconfiguredAnalyzerFactory.java (rev 0)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestUnconfiguredAnalyzerFactory.java 2014-05-09 22:39:10 UTC (rev 8257)
@@ -0,0 +1,24 @@
+package com.bigdata.search;
+
+public class TestUnconfiguredAnalyzerFactory extends AbstractAnalyzerFactoryTest {
+
+ public TestUnconfiguredAnalyzerFactory() {
+ }
+
+ public TestUnconfiguredAnalyzerFactory(String arg0) {
+ super(arg0);
+ }
+
+ @Override
+ String[] getExtraProperties() {
+ return new String[]{
+ FullTextIndex.Options.ANALYZER_FACTORY_CLASS, ConfigurableAnalyzerFactory.class.getName(),
+ };
+ }
+
+ public void testConfiguredLanguages() {
+ checkConfig("StandardAnalyzer", "por", "pt", "zho", "chi", "zh", "jpn", "ja", "kor", "ko", "ces", "cze", "cs", "dut", "nld", "nl",
+ "deu", "ger", "de", "gre", "ell", "el", "rus", "ru", "th", "tha", "en", "eng", "", null);
+ }
+
+}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <jer...@us...> - 2014-05-09 19:07:12
|
Revision: 8256
http://sourceforge.net/p/bigdata/code/8256
Author: jeremy_carroll
Date: 2014-05-09 19:07:09 +0000 (Fri, 09 May 2014)
Log Message:
-----------
Addressing trac 915 by documenting the current behavior and deprecating DefaultAnalyzerFactory and suggestion the use of ConfigurableAnalyzerFactory instead
Modified Paths:
--------------
branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/ConfigurableAnalyzerFactory.java
branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/DefaultAnalyzerFactory.java
Modified: branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/ConfigurableAnalyzerFactory.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/ConfigurableAnalyzerFactory.java 2014-05-09 19:07:02 UTC (rev 8255)
+++ branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/ConfigurableAnalyzerFactory.java 2014-05-09 19:07:09 UTC (rev 8256)
@@ -90,7 +90,7 @@
* Properties from {@link Options} apply to the factory.
* <p>
*
- * If there are no such properties at all then the property {@link Options#INCLUDE_DEFAULTS} is set to true,
+ * If there are no such properties at all then the property {@link Options#NATURAL_LANGUAGE_SUPPORT} is set to true,
* and the behavior of this class is the same as the legacy {@link DefaultAnalyzerFactory}.
* <p>
* Other properties, from {@link AnalyzerOptions} start with
@@ -117,7 +117,7 @@
* <dd>This suppresses the functionality, by treating every expression as a stop word.</dd>
* </dl>
* there are in addition the language specific analyzers that are included
- * by using the option {@link Options#INCLUDE_DEFAULTS}
+ * by using the option {@link Options#NATURAL_LANGUAGE_SUPPORT}
*
*
* @author jeremycarroll
@@ -265,18 +265,13 @@
*
*
*/
- String INCLUDE_DEFAULTS = ConfigurableAnalyzerFactory.class.getName() + ".includeDefaults";
+ String NATURAL_LANGUAGE_SUPPORT = ConfigurableAnalyzerFactory.class.getName() + ".includeDefaults";
/**
* This is the prefix to all properties configuring the individual analyzers.
*/
String ANALYZER = ConfigurableAnalyzerFactory.class.getName() + ".analyzer.";
-/**
- * If there is no configuration at all, then the defaults are included,
- * but any configuration at all totally replaces the defaults, unless
- * {@link #INCLUDE_DEFAULTS}
- * is explicitly set to true.
- */
- String DEFAULT_INCLUDE_DEFAULTS = "false";
+
+ String DEFAULT_NATURAL_LAMGUAGE_SUPPORT = "false";
}
/**
* Options understood by analyzers created by {@link ConfigurableAnalyzerFactory}.
@@ -810,7 +805,7 @@
while (en.hasMoreElements()) {
String prop = (String)en.nextElement();
- if (prop.equals(Options.INCLUDE_DEFAULTS)) continue;
+ if (prop.equals(Options.NATURAL_LANGUAGE_SUPPORT)) continue;
if (prop.startsWith(Options.ANALYZER)) {
String languageRangeAndProperty[] = prop.substring(Options.ANALYZER.length()).replaceAll("_","*").split("[.]");
if (languageRangeAndProperty.length == 2) {
@@ -838,7 +833,7 @@
protected Properties initProperties() {
final Properties parentProperties = fullTextIndex.getProperties();
Properties myProps;
- if (Boolean.getBoolean(parentProperties.getProperty(Options.INCLUDE_DEFAULTS, Options.DEFAULT_INCLUDE_DEFAULTS))) {
+ if (Boolean.getBoolean(parentProperties.getProperty(Options.NATURAL_LANGUAGE_SUPPORT, Options.DEFAULT_NATURAL_LAMGUAGE_SUPPORT))) {
myProps = defaultProperties();
} else {
myProps = new Properties();
Modified: branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/DefaultAnalyzerFactory.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/DefaultAnalyzerFactory.java 2014-05-09 19:07:02 UTC (rev 8255)
+++ branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/DefaultAnalyzerFactory.java 2014-05-09 19:07:09 UTC (rev 8256)
@@ -29,7 +29,6 @@
import java.util.Collections;
import java.util.HashMap;
-import java.util.HashSet;
import java.util.Locale;
import java.util.Map;
import java.util.Set;
@@ -52,11 +51,24 @@
import com.bigdata.btree.keys.KeyBuilder;
/**
+ * This is the default implementation but could be regarded as legacy since
+ * it fails to use the correct {@link Analyzer} for almost all languages (other than
+ * English). It uses the correct natural language analyzer for literals tagged with
+ * "por", "deu", "ger", "zho", "chi", "jpn", "kor", "ces", "cze", "dut", "nld", "gre", "ell",
+ * "fra", "fre", "rus" and "tha".
+ * This codes do not work if they are used with subtags, e.g. "ger-AT" is treated as English.
+ * No two letter code works correctly: note that the W3C and
+ * IETF recommend the use of the two letter forms instead of the three letter forms.
+ * <p>
* Default implementation registers a bunch of {@link Analyzer}s for various
* language codes and then serves the appropriate {@link Analyzer} based on
* the specified language code.
*
* @author <a href="mailto:tho...@us...">Bryan Thompson</a>
+ * @deprecated Using {@link ConfigurableAnalyzerFactory} with
+ * the {@link ConfigurableAnalyzerFactory.Options#NATURAL_LANGUAGE_SUPPORT}
+ * uses the appropriate natural language analyzers for the two letter codes
+ * and for tags which include sub-tags.
* @version $Id$
*/
public class DefaultAnalyzerFactory implements IAnalyzerFactory {
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <jer...@us...> - 2014-05-09 19:07:05
|
Revision: 8255
http://sourceforge.net/p/bigdata/code/8255
Author: jeremy_carroll
Date: 2014-05-09 19:07:02 +0000 (Fri, 09 May 2014)
Log Message:
-----------
minor polishing, a few more tests
Modified Paths:
--------------
branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/TermCompletionAnalyzer.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractSearchTest.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java
Modified: branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/TermCompletionAnalyzer.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/TermCompletionAnalyzer.java 2014-05-09 18:10:14 UTC (rev 8254)
+++ branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/TermCompletionAnalyzer.java 2014-05-09 19:07:02 UTC (rev 8255)
@@ -144,11 +144,10 @@
*
* - the subword boundaries are identified in {@link #next()}
* We then set up {@link #found} to contain the most
- * recently found subword, with afterDiscard containing
- * the same word as found with the {@link #discard} pattern
- * applied. {@link #afterDiscard} is not equal to found; if there
- * is nothing to discard then it is null.
+ * recently found subword.
*
+ * - the soft hyphen discarding is processed in {@link #maybeDiscardHyphens()}
+ *
* - if we are not {@link #alwaysDiscard}ing then {@link #afterDiscard}
* can be set to null to return the non-discarded version on the next cycle.
*
@@ -216,14 +215,14 @@
afterDiscard = null;
if (charPos + 1 < currentWord.length && softMatcher.find(charPos+1)) {
charPos = softMatcher.end();
- considerMatch();
+ maybeDiscardHyphens();
return true;
} else {
return nextWord();
}
}
- void considerMatch() {
+ void maybeDiscardHyphens() {
found = CharBuffer.wrap(currentWord, charPos, currentWord.length - charPos);
Matcher discarding = discard.matcher(found);
if (discarding.find()) {
@@ -240,7 +239,7 @@
termAtt.resizeTermBuffer(currentWord.length);
charPos = 0;
softMatcher = subWordBoundary.matcher(words[currentWordIx]);
- considerMatch();
+ maybeDiscardHyphens();
return true;
}
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractSearchTest.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractSearchTest.java 2014-05-09 18:10:14 UTC (rev 8254)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractSearchTest.java 2014-05-09 19:07:02 UTC (rev 8255)
@@ -102,13 +102,13 @@
return getNdx().getAnalyzer(lang, filterStopWords);
}
- protected void comparisonTest(String lang, boolean stopWordsSignificant, String text, String spaceSeparated)
+ protected void comparisonTest(String lang, boolean filterStopWords, String text, String spaceSeparated)
throws IOException {
if (spaceSeparated == null) {
- String rslt = getTokenStream(getAnalyzer(lang, stopWordsSignificant), text);
+ String rslt = getTokenStream(getAnalyzer(lang, filterStopWords), text);
throw new RuntimeException("Got \"" + rslt+ "\"");
}
- compareTokenStream(getAnalyzer(lang, stopWordsSignificant), text,
+ compareTokenStream(getAnalyzer(lang, filterStopWords), text,
split(spaceSeparated)); //$NON-NLS-1$
}
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java 2014-05-09 18:10:14 UTC (rev 8254)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java 2014-05-09 19:07:02 UTC (rev 8255)
@@ -35,6 +35,16 @@
import com.bigdata.search.ConfigurableAnalyzerFactory.AnalyzerOptions;
+/**
+ * Unit tests for {@link ConfigurableAnalyzerFactory}.
+ * We use the same setup, as defined in {@link #getExtraProperties()}
+ * for all the tests. Some of the tests check whether bad combinations
+ * of options are detected and reported correctly.
+ * Others check that some input, in a particular language is
+ * tokenized as expected.
+ * @author jeremycarroll
+ *
+ */
public class TestConfigurableAnalyzerFactory extends AbstractSearchTest {
public TestConfigurableAnalyzerFactory() {
@@ -68,8 +78,8 @@
analyzer+"x-hyphen2."+AnalyzerOptions.WORD_BOUNDARY, " ",
analyzer+"x-hyphen2."+AnalyzerOptions.ALWAYS_REMOVE_SOFT_HYPHENS, "true",
analyzer+"x-keywords."+AnalyzerOptions.ANALYZER_CLASS, KeywordAnalyzer.class.getName(),
- analyzer+"ru-x-de."+AnalyzerOptions.ANALYZER_CLASS, RussianAnalyzer.class.getName(),
- analyzer+"ru-x-de."+AnalyzerOptions.STOPWORDS, GermanAnalyzer.class.getName(),
+ analyzer+"en-x-de."+AnalyzerOptions.ANALYZER_CLASS, StandardAnalyzer.class.getName(),
+ analyzer+"en-x-de."+AnalyzerOptions.STOPWORDS, GermanAnalyzer.class.getName(),
};
}
@@ -142,6 +152,25 @@
);
}
+
+ public void testStopWordSwitch() throws IOException {
+ // en-x-de is an English Analyzer using german stopwords!
+ comparisonTest("en-x-de",
+ true,
+ "The fast car arrived slowly.",
+ "the fast car arrived slowly"
+ );
+ comparisonTest("en-x-de",
+ true,
+ "The fast car die arrived slowly.",
+ "the fast car arrived slowly"
+ );
+ comparisonTest("en-x-de",
+ false,
+ "The fast car die arrived slowly.",
+ "the fast car die arrived slowly"
+ );
+ }
public void testSyapseExample1() throws IOException {
comparisonTest("x-splits",
true,
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <jer...@us...> - 2014-05-09 18:10:17
|
Revision: 8254
http://sourceforge.net/p/bigdata/code/8254
Author: jeremy_carroll
Date: 2014-05-09 18:10:14 +0000 (Fri, 09 May 2014)
Log Message:
-----------
Added test for term completion, with bug fix!
Modified Paths:
--------------
branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/TermCompletionAnalyzer.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java
Modified: branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/TermCompletionAnalyzer.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/TermCompletionAnalyzer.java 2014-05-09 17:44:11 UTC (rev 8253)
+++ branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/TermCompletionAnalyzer.java 2014-05-09 18:10:14 UTC (rev 8254)
@@ -216,16 +216,20 @@
afterDiscard = null;
if (charPos + 1 < currentWord.length && softMatcher.find(charPos+1)) {
charPos = softMatcher.end();
- found = CharBuffer.wrap(currentWord, charPos, currentWord.length - charPos);
- Matcher discarding = discard.matcher(found);
- if (discarding.find()) {
- afterDiscard = discarding.replaceAll("");
- }
+ considerMatch();
return true;
} else {
return nextWord();
}
}
+
+ void considerMatch() {
+ found = CharBuffer.wrap(currentWord, charPos, currentWord.length - charPos);
+ Matcher discarding = discard.matcher(found);
+ if (discarding.find()) {
+ afterDiscard = discarding.replaceAll("");
+ }
+ }
private boolean nextWord() {
currentWordIx++;
@@ -235,8 +239,8 @@
currentWord = words[currentWordIx].toCharArray();
termAtt.resizeTermBuffer(currentWord.length);
charPos = 0;
- found = CharBuffer.wrap(currentWord, charPos, currentWord.length - charPos);
softMatcher = subWordBoundary.matcher(words[currentWordIx]);
+ considerMatch();
return true;
}
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java 2014-05-09 17:44:11 UTC (rev 8253)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java 2014-05-09 18:10:14 UTC (rev 8254)
@@ -59,6 +59,14 @@
analyzer+"x-splits."+AnalyzerOptions.STOPWORDS, AnalyzerOptions.STOPWORDS_VALUE_NONE,
analyzer+"x-splits."+AnalyzerOptions.WORD_BOUNDARY, " ",
analyzer+"x-splits."+AnalyzerOptions.SUB_WORD_BOUNDARY, "(?<!\\p{L}|\\p{N})(?=\\p{L}|\\p{N})|(?<!\\p{Lu})(?=\\p{Lu})|(?<=\\p{N})(?=\\p{L})",
+ analyzer+"x-hyphen."+AnalyzerOptions.SUB_WORD_BOUNDARY, "[-.]",
+ analyzer+"x-hyphen."+AnalyzerOptions.SOFT_HYPHENS, "-",
+ analyzer+"x-hyphen."+AnalyzerOptions.WORD_BOUNDARY, " ",
+ analyzer+"x-hyphen."+AnalyzerOptions.ALWAYS_REMOVE_SOFT_HYPHENS, "false",
+ analyzer+"x-hyphen2."+AnalyzerOptions.SUB_WORD_BOUNDARY, "[-.]",
+ analyzer+"x-hyphen2."+AnalyzerOptions.SOFT_HYPHENS, "-",
+ analyzer+"x-hyphen2."+AnalyzerOptions.WORD_BOUNDARY, " ",
+ analyzer+"x-hyphen2."+AnalyzerOptions.ALWAYS_REMOVE_SOFT_HYPHENS, "true",
analyzer+"x-keywords."+AnalyzerOptions.ANALYZER_CLASS, KeywordAnalyzer.class.getName(),
analyzer+"ru-x-de."+AnalyzerOptions.ANALYZER_CLASS, RussianAnalyzer.class.getName(),
analyzer+"ru-x-de."+AnalyzerOptions.STOPWORDS, GermanAnalyzer.class.getName(),
@@ -190,5 +198,21 @@
);
}
+ public void testSyapseExample8() throws IOException {
+ comparisonTest("x-hyphen",
+ true,
+ "\u00b1-ACE3.1.1 ab-bc.cd-de",
+ "\u00b1ACE3.1.1 \u00b1-ACE3.1.1 ACE3.1.1 1.1 1 abbc.cdde ab-bc.cd-de bc.cdde bc.cd-de cdde cd-de de"
+ );
+
+ }
+ public void testSyapseExample9() throws IOException {
+ comparisonTest("x-hyphen2",
+ true,
+ "\u00b1-ACE3.1.1 ab-bc.cd-de",
+ "\u00b1ACE3.1.1 ACE3.1.1 1.1 1 abbc.cdde bc.cdde cdde de"
+ );
+
+ }
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <jer...@us...> - 2014-05-09 17:44:13
|
Revision: 8253
http://sourceforge.net/p/bigdata/code/8253
Author: jeremy_carroll
Date: 2014-05-09 17:44:11 +0000 (Fri, 09 May 2014)
Log Message:
-----------
Fix to trac 874 - JoinFilter placement and Unions
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/StaticAnalysis.java
branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/StaticAnalysis_CanJoin.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/StaticAnalysis.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/StaticAnalysis.java 2014-05-09 17:44:02 UTC (rev 8252)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/StaticAnalysis.java 2014-05-09 17:44:11 UTC (rev 8253)
@@ -1081,7 +1081,7 @@
*/
// MUST : JOIN GROUP
- private Set<IVariable<?>> getDefinitelyProducedBindings(
+ Set<IVariable<?>> getDefinitelyProducedBindings(
final JoinGroupNode node, final Set<IVariable<?>> vars,
final boolean recursive) {
// Note: always report what is bound when we enter a group. The caller
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/StaticAnalysis_CanJoin.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/StaticAnalysis_CanJoin.java 2014-05-09 17:44:02 UTC (rev 8252)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-rdf/src/java/com/bigdata/rdf/sparql/ast/StaticAnalysis_CanJoin.java 2014-05-09 17:44:11 UTC (rev 8253)
@@ -51,7 +51,7 @@
* @version $Id: StaticAnalysis_CanJoin.java 5378 2011-10-20 20:37:08Z
* thompsonbry $
*/
-public class StaticAnalysis_CanJoin extends StaticAnalysisBase {
+public abstract class StaticAnalysis_CanJoin extends StaticAnalysisBase {
private static final Logger log = Logger.getLogger(StaticAnalysis.class);
@@ -392,12 +392,16 @@
// the constraints for the current predicate in the join path.
final List<FilterNode> constraints = new LinkedList<FilterNode>();
- /*
- * Visit the variables used by the predicate (and bound by it since
- * it is not an optional predicate) and add them into the total set
- * of variables which are bound at this point in the join path.
- */
- getSpannedVariables((BOp) p, boundVars);
+
+// /*
+// * Visit the variables used by the predicate (and bound by it since
+// * it is not an optional predicate) and add them into the total set
+// * of variables which are bound at this point in the join path.
+// */
+// getSpannedVariables((BOp) p, boundVars);
+ // above does not work if p is a Union nor, I suspect, a Minus - jjc
+ // tring this next line as an alternative - jjc.
+ getDefinitelyProducedBindings(p, boundVars, true);
if (joinGraphConstraints != null) {
@@ -479,5 +483,43 @@
return ret;
}
+
+ /**
+ * Return the set of variables which MUST be bound for solutions after the
+ * evaluation of this group. A group will produce "MUST" bindings for
+ * variables from its statement patterns and a LET based on an expression
+ * whose variables are known bound.
+ * <p>
+ * The returned collection reflects "bottom-up" evaluation semantics. This
+ * method does NOT consider variables which are already bound on entry to
+ * the group.
+ * <p>
+ * Note: When invoked for an OPTIONAL or MINUS join group, the variables
+ * which would become bound during the evaluation of the join group are
+ * reported. Caller's who wish to NOT have variables reported for OPTIONAL
+ * or MINUS groups MUST NOT invoke this method for those groups.
+ * <p>
+ * Note: The recursive analysis does not throw out variables when part of
+ * the tree will provably fail to bind anything. It is the role of query
+ * optimizers to identify those situations and prune the AST appropriately.
+ * <p>
+ * The class hierarchy is a little untidy at this point.
+ * This method is defined in the only subclass of this abstract class.
+ * Initially it was thought to not be needed here.
+ *
+ * @param node
+ * The node to be analyzed.
+ * @param vars
+ * Where to store the "MUST" bound variables.
+ * @param recursive
+ * When <code>true</code>, the child groups will be recursively
+ * analyzed. When <code>false</code>, only <i>this</i> group will
+ * be analyzed.
+ *
+ * @return The argument.
+ */
+ public abstract Set<IVariable<?>> getDefinitelyProducedBindings(
+ final IBindingProducerNode node, final Set<IVariable<?>> vars,
+ final boolean recursive);
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <jer...@us...> - 2014-05-09 17:44:05
|
Revision: 8252
http://sourceforge.net/p/bigdata/code/8252
Author: jeremy_carroll
Date: 2014-05-09 17:44:02 +0000 (Fri, 09 May 2014)
Log Message:
-----------
organized imports
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/search/TestKeyBuilder.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/search/TestLanguageRange.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/search/TestPrefixSearch.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/search/TestSearch.java
branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/search/TestSearchRestartSafe.java
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/search/TestKeyBuilder.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/search/TestKeyBuilder.java 2014-05-09 17:43:16 UTC (rev 8251)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/search/TestKeyBuilder.java 2014-05-09 17:44:02 UTC (rev 8252)
@@ -33,13 +33,9 @@
import com.bigdata.btree.BytesUtil;
import com.bigdata.btree.ITupleSerializer;
import com.bigdata.btree.IndexMetadata;
-import com.bigdata.btree.keys.DefaultKeyBuilderFactory;
import com.bigdata.btree.keys.IKeyBuilder;
import com.bigdata.btree.keys.KeyBuilder;
import com.bigdata.btree.keys.StrengthEnum;
-import com.bigdata.journal.IIndexManager;
-import com.bigdata.journal.ITx;
-import com.bigdata.journal.ProxyTestCase;
import com.bigdata.search.FullTextIndex.Options;
/**
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/search/TestLanguageRange.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/search/TestLanguageRange.java 2014-05-09 17:43:16 UTC (rev 8251)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/search/TestLanguageRange.java 2014-05-09 17:44:02 UTC (rev 8252)
@@ -26,10 +26,10 @@
*/
package com.bigdata.search;
+import junit.framework.TestCase2;
+
import com.bigdata.search.ConfigurableAnalyzerFactory.LanguageRange;
-import junit.framework.TestCase2;
-
public class TestLanguageRange extends TestCase2 {
public TestLanguageRange() {
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/search/TestPrefixSearch.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/search/TestPrefixSearch.java 2014-05-09 17:43:16 UTC (rev 8251)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/search/TestPrefixSearch.java 2014-05-09 17:44:02 UTC (rev 8252)
@@ -29,12 +29,8 @@
package com.bigdata.search;
import java.io.StringReader;
-import java.util.Properties;
import java.util.concurrent.TimeUnit;
-import com.bigdata.journal.IIndexManager;
-import com.bigdata.journal.ITx;
-import com.bigdata.journal.ProxyTestCase;
import com.bigdata.rdf.lexicon.ITextIndexer.FullTextQuery;
/**
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/search/TestSearch.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/search/TestSearch.java 2014-05-09 17:43:16 UTC (rev 8251)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/search/TestSearch.java 2014-05-09 17:44:02 UTC (rev 8252)
@@ -33,9 +33,6 @@
import java.util.Properties;
import java.util.concurrent.TimeUnit;
-import com.bigdata.journal.IIndexManager;
-import com.bigdata.journal.ITx;
-import com.bigdata.journal.ProxyTestCase;
import com.bigdata.rdf.lexicon.ITextIndexer.FullTextQuery;
/**
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/search/TestSearchRestartSafe.java
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/search/TestSearchRestartSafe.java 2014-05-09 17:43:16 UTC (rev 8251)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata/src/test/com/bigdata/search/TestSearchRestartSafe.java 2014-05-09 17:44:02 UTC (rev 8252)
@@ -29,12 +29,9 @@
package com.bigdata.search;
import java.io.StringReader;
-import java.util.Properties;
import java.util.concurrent.TimeUnit;
-import com.bigdata.journal.IIndexManager;
import com.bigdata.journal.ITx;
-import com.bigdata.journal.ProxyTestCase;
import com.bigdata.rdf.lexicon.ITextIndexer.FullTextQuery;
/**
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <jer...@us...> - 2014-05-09 17:43:20
|
Revision: 8251
http://sourceforge.net/p/bigdata/code/8251
Author: jeremy_carroll
Date: 2014-05-09 17:43:16 +0000 (Fri, 09 May 2014)
Log Message:
-----------
Got tests working again, and cleaned up somewhat
Modified Paths:
--------------
branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/ConfigurableAnalyzerFactory.java
branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/TermCompletionAnalyzer.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractSearchTest.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java
Modified: branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/ConfigurableAnalyzerFactory.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/ConfigurableAnalyzerFactory.java 2014-05-09 17:43:05 UTC (rev 8250)
+++ branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/ConfigurableAnalyzerFactory.java 2014-05-09 17:43:16 UTC (rev 8251)
@@ -326,12 +326,48 @@
String STOPWORDS_VALUE_NONE = "none";
/**
- * If this property is present then the analyzer being used is a
- * {@link PatternAnalyzer} and the value is the pattern to use.
+ * The value of the pattern parameter to
+ * {@link PatternAnalyzer#PatternAnalyzer(Version, Pattern, boolean, Set)}
* (Note the {@link Pattern#UNICODE_CHARACTER_CLASS} flag is enabled).
* It is an error if a different analyzer class is specified.
*/
String PATTERN = "pattern";
+ /**
+ * The value of the wordBoundary parameter to
+ * {@link TermCompletionAnalyzer#TermCompletionAnalyzer(Pattern, Pattern, Pattern, boolean)}
+ * (Note the {@link Pattern#UNICODE_CHARACTER_CLASS} flag is enabled).
+ * It is an error if a different analyzer class is specified.
+ */
+ String WORD_BOUNDARY = "wordBoundary";
+ /**
+ * The value of the subWordBoundary parameter to
+ * {@link TermCompletionAnalyzer#TermCompletionAnalyzer(Pattern, Pattern, Pattern, boolean)}
+ * (Note the {@link Pattern#UNICODE_CHARACTER_CLASS} flag is enabled).
+ * It is an error if a different analyzer class is specified.
+ */
+ String SUB_WORD_BOUNDARY = "subWordBoundary";
+ /**
+ * The value of the softHyphens parameter to
+ * {@link TermCompletionAnalyzer#TermCompletionAnalyzer(Pattern, Pattern, Pattern, boolean)}
+ * (Note the {@link Pattern#UNICODE_CHARACTER_CLASS} flag is enabled).
+ * It is an error if a different analyzer class is specified.
+ */
+ String SOFT_HYPHENS = "softHypens";
+ /**
+ * The value of the alwaysRemoveSoftHypens parameter to
+ * {@link TermCompletionAnalyzer#TermCompletionAnalyzer(Pattern, Pattern, Pattern, boolean)}
+ * (Note the {@link Pattern#UNICODE_CHARACTER_CLASS} flag is enabled).
+ * It is an error if a different analyzer class is specified.
+ */
+ String ALWAYS_REMOVE_SOFT_HYPHENS = "alwaysRemoveSoftHypens";
+
+ boolean DEFAULT_ALWAYS_REMOVE_SOFT_HYPHENS = false;
+
+ /**
+ * The default sub-word boundary is a pattern that never matches,
+ * i.e. there are no sub-word boundaries.
+ */
+ Pattern DEFAULT_SUB_WORD_BOUNDARY = Pattern.compile("(?!)");
}
@@ -382,16 +418,7 @@
this.withoutStopWords = copyMe.withoutStopWords;
}
-
- public Analyzer getAnalyzer(boolean filterStopwords) {
- return filterStopwords ? withStopWords : withoutStopWords;
- }
- @Override
- public String toString() {
- return range.full + "=(" + withStopWords.getClass().getSimpleName() +")";
- }
-
AnalyzerPair(String range, Constructor<? extends Analyzer> cons, Object ... params) throws Exception {
this(range, cons.newInstance(params), cons.newInstance(useEmptyStopWordSet(params)));
}
@@ -409,7 +436,16 @@
}
return rslt;
}
+
+ public Analyzer getAnalyzer(boolean filterStopwords) {
+ return filterStopwords ? withStopWords : withoutStopWords;
+ }
@Override
+ public String toString() {
+ return range.full + "=(" + withStopWords.getClass().getSimpleName() +")";
+ }
+
+ @Override
public int compareTo(AnalyzerPair o) {
return range.compareTo(o.range);
}
@@ -437,10 +473,10 @@
private static class PatternAnalyzerPair extends AnalyzerPair {
- public PatternAnalyzerPair(ConfigOptionsToAnalyzer lro, String pattern) throws Exception {
+ public PatternAnalyzerPair(ConfigOptionsToAnalyzer lro, Pattern pattern) throws Exception {
super(lro.languageRange, getConstructor(PatternAnalyzer.class,Version.class,Pattern.class,Boolean.TYPE,Set.class),
Version.LUCENE_CURRENT,
- Pattern.compile(pattern, Pattern.UNICODE_CHARACTER_CLASS),
+ pattern,
true,
lro.getStopWords());
}
@@ -459,9 +495,13 @@
String like;
String className;
String stopwords;
- String pattern;
+ Pattern pattern;
final String languageRange;
AnalyzerPair result;
+ Pattern wordBoundary;
+ Pattern subWordBoundary;
+ Pattern softHyphens;
+ Boolean alwaysRemoveSoftHyphens;
public ConfigOptionsToAnalyzer(String languageRange) {
this.languageRange = languageRange;
@@ -515,7 +555,15 @@
} else if (shortProperty.equals(AnalyzerOptions.STOPWORDS) ) {
stopwords = value;
} else if (shortProperty.equals(AnalyzerOptions.PATTERN) ) {
- pattern = value;
+ pattern = Pattern.compile(value,Pattern.UNICODE_CHARACTER_CLASS);
+ } else if (shortProperty.equals(AnalyzerOptions.WORD_BOUNDARY) ) {
+ wordBoundary = Pattern.compile(value,Pattern.UNICODE_CHARACTER_CLASS);
+ } else if (shortProperty.equals(AnalyzerOptions.SUB_WORD_BOUNDARY) ) {
+ subWordBoundary = Pattern.compile(value,Pattern.UNICODE_CHARACTER_CLASS);
+ } else if (shortProperty.equals(AnalyzerOptions.SOFT_HYPHENS) ) {
+ softHyphens = Pattern.compile(value,Pattern.UNICODE_CHARACTER_CLASS);
+ } else if (shortProperty.equals(AnalyzerOptions.ALWAYS_REMOVE_SOFT_HYPHENS) ) {
+ alwaysRemoveSoftHyphens = Boolean.valueOf(value);
} else {
return false;
}
@@ -529,6 +577,27 @@
}
className = PatternAnalyzer.class.getName();
}
+ if (this.wordBoundary != null ) {
+ if ( className != null && className != TermCompletionAnalyzer.class.getName()) {
+ throw new RuntimeException("Bad Option: Language range "+languageRange + " with pattern propety for class "+ className);
+ }
+ className = TermCompletionAnalyzer.class.getName();
+
+ if ( subWordBoundary == null ) {
+ subWordBoundary = AnalyzerOptions.DEFAULT_SUB_WORD_BOUNDARY;
+ }
+ if ( alwaysRemoveSoftHyphens != null && softHyphens == null ) {
+ throw new RuntimeException("Bad option: Language range "+languageRange + ": must specify softHypens when setting alwaysRemoveSoftHyphens");
+ }
+ if (softHyphens != null && alwaysRemoveSoftHyphens == null) {
+ alwaysRemoveSoftHyphens = AnalyzerOptions.DEFAULT_ALWAYS_REMOVE_SOFT_HYPHENS;
+ }
+
+ } else if ( subWordBoundary != null || softHyphens != null || alwaysRemoveSoftHyphens != null ||
+ TermCompletionAnalyzer.class.getName().equals(className) ) {
+ throw new RuntimeException("Bad option: Language range "+languageRange + ": must specify wordBoundary for TermCompletionAnalyzer");
+ }
+
if (PatternAnalyzer.class.getName().equals(className) && pattern == null ) {
throw new RuntimeException("Bad Option: Language range "+languageRange + " must specify pattern for PatternAnalyzer.");
}
@@ -547,8 +616,23 @@
}
if (pattern != null) {
return new PatternAnalyzerPair(this, pattern);
-
- }
+ }
+ if (softHyphens != null) {
+ return new AnalyzerPair(
+ languageRange,
+ new TermCompletionAnalyzer(
+ wordBoundary,
+ subWordBoundary,
+ softHyphens,
+ alwaysRemoveSoftHyphens));
+ }
+ if (wordBoundary != null) {
+ return new AnalyzerPair(
+ languageRange,
+ new TermCompletionAnalyzer(
+ wordBoundary,
+ subWordBoundary));
+ }
final Class<? extends Analyzer> cls = getAnalyzerClass();
if (hasConstructor(cls, Version.class, Set.class)) {
Modified: branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/TermCompletionAnalyzer.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/TermCompletionAnalyzer.java 2014-05-09 17:43:05 UTC (rev 8250)
+++ branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/TermCompletionAnalyzer.java 2014-05-09 17:43:16 UTC (rev 8251)
@@ -81,8 +81,8 @@
*/
public class TermCompletionAnalyzer extends Analyzer {
- private final Pattern wordBoundary; // = Pattern.compile(" ", Pattern.UNICODE_CHARACTER_CLASS);
- private final Pattern subWordBoundary; // = Pattern.compile("(?<!\\p{L}|\\p{N})(?=\\p{L}|\\p{N})|(?<!\\p{Lu})(?=\\p{Lu})|(?<=\\p{N})(?=\\p{L})", Pattern.UNICODE_CHARACTER_CLASS);
+ private final Pattern wordBoundary;
+ private final Pattern subWordBoundary;
private final Pattern discard;
private final boolean alwaysDiscard;
@@ -90,24 +90,25 @@
/**
* Divide the input into words and short tokens
* as with {@link #TermCompletionAnalyzer(Pattern, Pattern)}.
- * If alsoWithSoftHypens is true then output each token,
- * and in any case output each token with every
- * match to softHyphenEtc deleted.
+ * Each term is generated, and then an additional term
+ * is generated with softHypens (defined by the pattern),
+ * removed. If the alwaysRemoveSoftHypens flag is true,
+ * then the first term (before the removal) is suppressed.
*
* @param wordBoundary The definition of space (e.g. " ")
* @param subWordBoundary Also index after matches to this (e.g. "-")
- * @param softHyphenEtc Discard these characters from matches
- * @param alsoWithSoftHyphens If true the discard step is optional.
+ * @param softHyphens Discard these characters from matches
+ * @param alwaysRemoveSoftHypens If false the discard step is optional.
*/
public TermCompletionAnalyzer(Pattern wordBoundary,
Pattern subWordBoundary,
- Pattern softHyphenEtc,
- boolean alsoWithSoftHyphens) {
+ Pattern softHyphens,
+ boolean alwaysRemoveSoftHypens) {
this.wordBoundary = wordBoundary;
this.subWordBoundary = subWordBoundary;
- if (softHyphenEtc != null) {
- discard = softHyphenEtc;
- alwaysDiscard = !alsoWithSoftHyphens;
+ if (softHyphens != null) {
+ discard = softHyphens;
+ alwaysDiscard = alwaysRemoveSoftHypens;
} else {
discard = Pattern.compile("(?!)"); // never matches
alwaysDiscard = true;
@@ -115,9 +116,10 @@
}
/**
* Divide the input into words, separated by the wordBoundary,
- * and return a token for the whole word, and then for the
- * remainder of the word after each successive match of the
- * subWordBoundary.
+ * and return a token for each whole word, and then
+ * generate further tokens for each word by removing prefixes
+ * up to and including each successive match of
+ * subWordBoundary
* @param wordBoundary
* @param subWordBoundary
*/
@@ -189,8 +191,9 @@
afterDiscard.getChars(0, lg, termAtt.termBuffer(), 0);
termAtt.setTermLength(lg);
} else {
- found.get(termAtt.termBuffer());
- termAtt.setTermLength(found.length());
+ int lg = found.length();
+ found.get(termAtt.termBuffer(), 0, lg);
+ termAtt.setTermLength(lg);
}
return true;
} else {
@@ -211,7 +214,7 @@
}
}
afterDiscard = null;
- if (charPos +1 < currentWord.length && softMatcher.find(charPos+1)) {
+ if (charPos + 1 < currentWord.length && softMatcher.find(charPos+1)) {
charPos = softMatcher.end();
found = CharBuffer.wrap(currentWord, charPos, currentWord.length - charPos);
Matcher discarding = discard.matcher(found);
@@ -232,6 +235,7 @@
currentWord = words[currentWordIx].toCharArray();
termAtt.resizeTermBuffer(currentWord.length);
charPos = 0;
+ found = CharBuffer.wrap(currentWord, charPos, currentWord.length - charPos);
softMatcher = subWordBoundary.matcher(words[currentWordIx]);
return true;
}
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractSearchTest.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractSearchTest.java 2014-05-09 17:43:05 UTC (rev 8250)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractSearchTest.java 2014-05-09 17:43:16 UTC (rev 8251)
@@ -139,7 +139,7 @@
final TermAttribute term = s.getAttribute(TermAttribute.class);
final String word = term.term();
assertTrue(ix < expected.length);
- assertEquals(word, expected[ix++]);
+ assertEquals(expected[ix++], word);
}
assertEquals(ix, expected.length);
}
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java 2014-05-09 17:43:05 UTC (rev 8250)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java 2014-05-09 17:43:16 UTC (rev 8251)
@@ -57,6 +57,8 @@
analyzer+"x-terms."+AnalyzerOptions.PATTERN, "\\W+",
analyzer+"x-splits."+AnalyzerOptions.ANALYZER_CLASS, TermCompletionAnalyzer.class.getName(),
analyzer+"x-splits."+AnalyzerOptions.STOPWORDS, AnalyzerOptions.STOPWORDS_VALUE_NONE,
+ analyzer+"x-splits."+AnalyzerOptions.WORD_BOUNDARY, " ",
+ analyzer+"x-splits."+AnalyzerOptions.SUB_WORD_BOUNDARY, "(?<!\\p{L}|\\p{N})(?=\\p{L}|\\p{N})|(?<!\\p{Lu})(?=\\p{Lu})|(?<=\\p{N})(?=\\p{L})",
analyzer+"x-keywords."+AnalyzerOptions.ANALYZER_CLASS, KeywordAnalyzer.class.getName(),
analyzer+"ru-x-de."+AnalyzerOptions.ANALYZER_CLASS, RussianAnalyzer.class.getName(),
analyzer+"ru-x-de."+AnalyzerOptions.STOPWORDS, GermanAnalyzer.class.getName(),
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <jer...@us...> - 2014-05-09 17:43:08
|
Revision: 8250
http://sourceforge.net/p/bigdata/code/8250
Author: jeremy_carroll
Date: 2014-05-09 17:43:05 +0000 (Fri, 09 May 2014)
Log Message:
-----------
Documented and cleaned up interfaces.
Modified Paths:
--------------
branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/TermCompletionAnalyzer.java
Modified: branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/TermCompletionAnalyzer.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/TermCompletionAnalyzer.java 2014-05-09 17:42:56 UTC (rev 8249)
+++ branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/TermCompletionAnalyzer.java 2014-05-09 17:43:05 UTC (rev 8250)
@@ -29,66 +29,201 @@
import java.io.IOException;
import java.io.Reader;
import java.io.StringReader;
+import java.nio.CharBuffer;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.TokenStream;
+import org.apache.lucene.analysis.KeywordAnalyzer;
import org.apache.lucene.analysis.tokenattributes.TermAttribute;
+/**
+ * An analyzer intended for the term-completion use case; particularly
+ * for technical vocabularies and concept schemes.
+ *
+ * <p>
+ * This analyzer generates several index terms for each word in the input.
+ * These are intended to match short sequences (e.g. three or more) characters
+ * of user-input, to then give the user a drop-down list of matching terms.
+ * <p>
+ * This can be set up to address issues like matching <q>half-time</q> when the user types
+ * <q>tim</q> or if the user types <q>halft</q> (treating the hyphen as a soft hyphen); or
+ * to match <q>TermCompletionAnalyzer</q> when the user types <q>Ana</q>
+ * <p>
+ * In contrast, the Lucene Analyzers are mainly geared around the free text search use
+ * case.
+ * <p>
+ * The intended use cases will typical involve a prefix query of the form:
+ * <pre>
+ * ?t bds:search "prefix*" .
+ * </pre>
+ * to find all literals in the selected graphs, which are indexed by a term starting in <q>prefix</q>,
+ * so the problem this class addresses is finding the appropriate index terms to allow
+ * matching, at sensible points, mid-way through words (such as at hyphens).
+ * <p>
+ * To get maximum effectiveness it maybe best to use private language subtags (see RFC 5647),
+ * e.g. <code>"x-term"</code>
+ * which are mapped to this class by {@link ConfigurableAnalyzerFactory} for
+ * the data being loaded into the store, and linked to some very simple process
+ * like {@link KeywordAnalyzer} for queries which are tagged with a different language tag
+ * that is only used for <code>bds:search</code>, e.g. <code>"x-query"</code>.
+ * The above prefix query then becomes:
+ * <pre>
+ * ?t bds:search "prefix*"@x-query .
+ * </pre>
+ *
+ *
+ *
+ * @author jeremycarroll
+ *
+ */
public class TermCompletionAnalyzer extends Analyzer {
- Pattern hard = Pattern.compile(" ", Pattern.UNICODE_CHARACTER_CLASS);
- Pattern soft = Pattern.compile("(?<!\\p{L}|\\p{N})(?=\\p{L}|\\p{N})|(?<!\\p{Lu})(?=\\p{Lu})|(?<=\\p{N})(?=\\p{L})", Pattern.UNICODE_CHARACTER_CLASS);
+ private final Pattern wordBoundary; // = Pattern.compile(" ", Pattern.UNICODE_CHARACTER_CLASS);
+ private final Pattern subWordBoundary; // = Pattern.compile("(?<!\\p{L}|\\p{N})(?=\\p{L}|\\p{N})|(?<!\\p{Lu})(?=\\p{Lu})|(?<=\\p{N})(?=\\p{L})", Pattern.UNICODE_CHARACTER_CLASS);
- public TermCompletionAnalyzer() {
+ private final Pattern discard;
+ private final boolean alwaysDiscard;
+
+ /**
+ * Divide the input into words and short tokens
+ * as with {@link #TermCompletionAnalyzer(Pattern, Pattern)}.
+ * If alsoWithSoftHypens is true then output each token,
+ * and in any case output each token with every
+ * match to softHyphenEtc deleted.
+ *
+ * @param wordBoundary The definition of space (e.g. " ")
+ * @param subWordBoundary Also index after matches to this (e.g. "-")
+ * @param softHyphenEtc Discard these characters from matches
+ * @param alsoWithSoftHyphens If true the discard step is optional.
+ */
+ public TermCompletionAnalyzer(Pattern wordBoundary,
+ Pattern subWordBoundary,
+ Pattern softHyphenEtc,
+ boolean alsoWithSoftHyphens) {
+ this.wordBoundary = wordBoundary;
+ this.subWordBoundary = subWordBoundary;
+ if (softHyphenEtc != null) {
+ discard = softHyphenEtc;
+ alwaysDiscard = !alsoWithSoftHyphens;
+ } else {
+ discard = Pattern.compile("(?!)"); // never matches
+ alwaysDiscard = true;
+ }
}
+ /**
+ * Divide the input into words, separated by the wordBoundary,
+ * and return a token for the whole word, and then for the
+ * remainder of the word after each successive match of the
+ * subWordBoundary.
+ * @param wordBoundary
+ * @param subWordBoundary
+ */
+ public TermCompletionAnalyzer(Pattern wordBoundary,
+ Pattern subWordBoundary) {
+ this(wordBoundary, subWordBoundary, null, true);
+ }
+
+ @Override
+ public TokenStream tokenStream(String ignoredFieldName, Reader reader) {
+ return new TermCompletionTokenStream((StringReader)reader);
+ }
+
+ /**
+ * This classes has three processes going on
+ * all driven from the {@link #increment()} method.
+ *
+ * One process is that of iterating over the words in the input:
+ * - the words are identified in the constructor, and the iteration
+ * is performed by {@link #nextWord()}
+ *
+ * - the subword boundaries are identified in {@link #next()}
+ * We then set up {@link #found} to contain the most
+ * recently found subword, with afterDiscard containing
+ * the same word as found with the {@link #discard} pattern
+ * applied. {@link #afterDiscard} is not equal to found; if there
+ * is nothing to discard then it is null.
+ *
+ * - if we are not {@link #alwaysDiscard}ing then {@link #afterDiscard}
+ * can be set to null to return the non-discarded version on the next cycle.
+ *
+ */
private class TermCompletionTokenStream extends TokenStream {
- final int length;
final String[] words;
+ final TermAttribute termAtt;
+
+
+
char currentWord[] = new char[]{};
Matcher softMatcher;
int currentWordIx = -1;
+
+
int charPos = 0;
- final TermAttribute termAtt;
+ private String afterDiscard;
+ private CharBuffer found;
+
public TermCompletionTokenStream(StringReader reader) {
termAtt = addAttribute(TermAttribute.class);
try {
reader.mark(Integer.MAX_VALUE);
- length = (int) reader.skip(Integer.MAX_VALUE);
+ int length = (int) reader.skip(Integer.MAX_VALUE);
reader.reset();
char fileContent[] = new char[length];
reader.read(fileContent);
- words = hard.split(new String(fileContent));
+ words = wordBoundary.split(new String(fileContent));
} catch (IOException e) {
throw new RuntimeException("Impossible",e);
}
}
+
@Override
public boolean incrementToken() throws IOException {
if ( next() ) {
- int lg = currentWord.length - charPos;
- System.arraycopy(currentWord, charPos, termAtt.termBuffer(), 0, lg );
- termAtt.setTermLength(lg);
+ if (afterDiscard != null) {
+ int lg = afterDiscard.length();
+ afterDiscard.getChars(0, lg, termAtt.termBuffer(), 0);
+ termAtt.setTermLength(lg);
+ } else {
+ found.get(termAtt.termBuffer());
+ termAtt.setTermLength(found.length());
+ }
return true;
} else {
return false;
}
}
+
private boolean next() {
if (currentWordIx >= words.length) {
return false;
}
+ if (!alwaysDiscard) {
+ // Last match was the discarded version,
+ // now do the non-discard version.
+ if (afterDiscard != null) {
+ afterDiscard = null;
+ return true;
+ }
+ }
+ afterDiscard = null;
if (charPos +1 < currentWord.length && softMatcher.find(charPos+1)) {
charPos = softMatcher.end();
+ found = CharBuffer.wrap(currentWord, charPos, currentWord.length - charPos);
+ Matcher discarding = discard.matcher(found);
+ if (discarding.find()) {
+ afterDiscard = discarding.replaceAll("");
+ }
return true;
} else {
return nextWord();
}
}
+
private boolean nextWord() {
currentWordIx++;
if (currentWordIx >= words.length) {
@@ -97,15 +232,10 @@
currentWord = words[currentWordIx].toCharArray();
termAtt.resizeTermBuffer(currentWord.length);
charPos = 0;
- softMatcher = soft.matcher(words[currentWordIx]);
+ softMatcher = subWordBoundary.matcher(words[currentWordIx]);
return true;
}
}
-
- @Override
- public TokenStream tokenStream(String ignoredFieldName, Reader reader) {
- return new TermCompletionTokenStream((StringReader)reader);
- }
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <jer...@us...> - 2014-05-09 17:43:00
|
Revision: 8249
http://sourceforge.net/p/bigdata/code/8249
Author: jeremy_carroll
Date: 2014-05-09 17:42:56 +0000 (Fri, 09 May 2014)
Log Message:
-----------
copyright and tidying up
Modified Paths:
--------------
branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/TermCompletionAnalyzer.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractSearchTest.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/NonEnglishExamples.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java
Modified: branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/TermCompletionAnalyzer.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/TermCompletionAnalyzer.java 2014-05-09 17:42:44 UTC (rev 8248)
+++ branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/TermCompletionAnalyzer.java 2014-05-09 17:42:56 UTC (rev 8249)
@@ -1,3 +1,29 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2014. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on May 8, 2014 by Jeremy J. Carroll, Syapse Inc.
+ */
package com.bigdata.search;
import java.io.IOException;
@@ -3,5 +29,4 @@
import java.io.Reader;
import java.io.StringReader;
-import java.nio.CharBuffer;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
@@ -10,7 +35,6 @@
import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.tokenattributes.TermAttribute;
-import org.apache.lucene.util.Attribute;
public class TermCompletionAnalyzer extends Analyzer {
@@ -19,7 +43,6 @@
Pattern soft = Pattern.compile("(?<!\\p{L}|\\p{N})(?=\\p{L}|\\p{N})|(?<!\\p{Lu})(?=\\p{Lu})|(?<=\\p{N})(?=\\p{L})", Pattern.UNICODE_CHARACTER_CLASS);
public TermCompletionAnalyzer() {
- // TODO Auto-generated constructor stub
}
private class TermCompletionTokenStream extends TokenStream {
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractSearchTest.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractSearchTest.java 2014-05-09 17:42:44 UTC (rev 8248)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractSearchTest.java 2014-05-09 17:42:56 UTC (rev 8249)
@@ -122,7 +122,6 @@
protected String getTokenStream(Analyzer a, String text) throws IOException {
StringBuffer sb = new StringBuffer();
TokenStream s = a.tokenStream(null, new StringReader(text));
- int ix = 0;
while (s.incrementToken()) {
final TermAttribute term = s.getAttribute(TermAttribute.class);
if (sb.length()!=0) {
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/NonEnglishExamples.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/NonEnglishExamples.java 2014-05-09 17:42:44 UTC (rev 8248)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/NonEnglishExamples.java 2014-05-09 17:42:56 UTC (rev 8249)
@@ -1,3 +1,29 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2014. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on May 7, 2014 by Jeremy J. Carroll, Syapse Inc.
+ */
package com.bigdata.search;
import java.util.MissingResourceException;
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java 2014-05-09 17:42:44 UTC (rev 8248)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java 2014-05-09 17:42:56 UTC (rev 8249)
@@ -28,13 +28,10 @@
import java.io.IOException;
-import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.KeywordAnalyzer;
-import org.apache.lucene.analysis.cjk.CJKAnalyzer;
import org.apache.lucene.analysis.de.GermanAnalyzer;
import org.apache.lucene.analysis.ru.RussianAnalyzer;
import org.apache.lucene.analysis.standard.StandardAnalyzer;
-import org.apache.lucene.util.Version;
import com.bigdata.search.ConfigurableAnalyzerFactory.AnalyzerOptions;
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <jer...@us...> - 2014-05-09 17:42:49
|
Revision: 8248
http://sourceforge.net/p/bigdata/code/8248
Author: jeremy_carroll
Date: 2014-05-09 17:42:44 +0000 (Fri, 09 May 2014)
Log Message:
-----------
First version of TermCompletionAnalyzer, and also tests now passing
Modified Paths:
--------------
branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/ConfigurableAnalyzerFactory.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractAnalyzerFactoryTest.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractSearchTest.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestAll.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAsDefaultAnalyzerFactory.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestDefaultAnalyzerFactory.java
Added Paths:
-----------
branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/TermCompletionAnalyzer.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java
Modified: branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/ConfigurableAnalyzerFactory.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/ConfigurableAnalyzerFactory.java 2014-05-09 17:07:05 UTC (rev 8247)
+++ branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/ConfigurableAnalyzerFactory.java 2014-05-09 17:42:44 UTC (rev 8248)
@@ -331,7 +331,7 @@
* (Note the {@link Pattern#UNICODE_CHARACTER_CLASS} flag is enabled).
* It is an error if a different analyzer class is specified.
*/
- String PATTERN = ".pattern";
+ String PATTERN = "pattern";
}
@@ -474,7 +474,7 @@
*/
public Set<?> getStopWords() {
- if (AnalyzerOptions.STOPWORDS_VALUE_NONE.equals(stopwords))
+ if (doNotUseStopWords())
return Collections.EMPTY_SET;
if (useDefaultStopWords()) {
@@ -484,6 +484,10 @@
return getStopWordsForClass(stopwords);
}
+ boolean doNotUseStopWords() {
+ return AnalyzerOptions.STOPWORDS_VALUE_NONE.equals(stopwords) || (stopwords == null && pattern != null);
+ }
+
protected Set<?> getStopWordsForClass(String clazzName) {
Class<? extends Analyzer> analyzerClass = getAnalyzerClass(clazzName);
try {
@@ -500,7 +504,7 @@
}
protected boolean useDefaultStopWords() {
- return stopwords == null || AnalyzerOptions.STOPWORDS_VALUE_DEFAULT.equals(stopwords);
+ return ( stopwords == null && pattern == null ) || AnalyzerOptions.STOPWORDS_VALUE_DEFAULT.equals(stopwords);
}
public boolean setProperty(String shortProperty, String value) {
@@ -550,8 +554,13 @@
if (hasConstructor(cls, Version.class, Set.class)) {
// RussianAnalyzer is missing any way to access stop words.
- if (RussianAnalyzer.class.equals(cls) && useDefaultStopWords()) {
- return new AnalyzerPair(languageRange, new RussianAnalyzer(Version.LUCENE_CURRENT), new RussianAnalyzer(Version.LUCENE_CURRENT, Collections.EMPTY_SET));
+ if (RussianAnalyzer.class.equals(cls)) {
+ if (useDefaultStopWords()) {
+ return new AnalyzerPair(languageRange, new RussianAnalyzer(Version.LUCENE_CURRENT), new RussianAnalyzer(Version.LUCENE_CURRENT, Collections.EMPTY_SET));
+ }
+ if (doNotUseStopWords()) {
+ return new AnalyzerPair(languageRange, new RussianAnalyzer(Version.LUCENE_CURRENT, Collections.EMPTY_SET));
+ }
}
return new VersionSetAnalyzerPair(this, cls);
}
@@ -719,7 +728,7 @@
String prop = (String)en.nextElement();
if (prop.equals(Options.INCLUDE_DEFAULTS)) continue;
if (prop.startsWith(Options.ANALYZER)) {
- String languageRangeAndProperty[] = prop.substring(Options.ANALYZER.length()).split("[.]");
+ String languageRangeAndProperty[] = prop.substring(Options.ANALYZER.length()).replaceAll("_","*").split("[.]");
if (languageRangeAndProperty.length == 2) {
String languageRange = languageRangeAndProperty[0].toLowerCase(Locale.US); // Turkish "I" could create a problem
Added: branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/TermCompletionAnalyzer.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/TermCompletionAnalyzer.java (rev 0)
+++ branches/TEXT_ANALYZERS/bigdata/src/java/com/bigdata/search/TermCompletionAnalyzer.java 2014-05-09 17:42:44 UTC (rev 8248)
@@ -0,0 +1,88 @@
+package com.bigdata.search;
+
+import java.io.IOException;
+import java.io.Reader;
+import java.io.StringReader;
+import java.nio.CharBuffer;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.apache.lucene.analysis.Analyzer;
+import org.apache.lucene.analysis.TokenStream;
+import org.apache.lucene.analysis.tokenattributes.TermAttribute;
+import org.apache.lucene.util.Attribute;
+
+
+public class TermCompletionAnalyzer extends Analyzer {
+
+ Pattern hard = Pattern.compile(" ", Pattern.UNICODE_CHARACTER_CLASS);
+ Pattern soft = Pattern.compile("(?<!\\p{L}|\\p{N})(?=\\p{L}|\\p{N})|(?<!\\p{Lu})(?=\\p{Lu})|(?<=\\p{N})(?=\\p{L})", Pattern.UNICODE_CHARACTER_CLASS);
+
+ public TermCompletionAnalyzer() {
+ // TODO Auto-generated constructor stub
+ }
+
+ private class TermCompletionTokenStream extends TokenStream {
+
+ final int length;
+ final String[] words;
+ char currentWord[] = new char[]{};
+ Matcher softMatcher;
+ int currentWordIx = -1;
+ int charPos = 0;
+ final TermAttribute termAtt;
+ public TermCompletionTokenStream(StringReader reader) {
+ termAtt = addAttribute(TermAttribute.class);
+ try {
+ reader.mark(Integer.MAX_VALUE);
+ length = (int) reader.skip(Integer.MAX_VALUE);
+ reader.reset();
+ char fileContent[] = new char[length];
+ reader.read(fileContent);
+ words = hard.split(new String(fileContent));
+ } catch (IOException e) {
+ throw new RuntimeException("Impossible",e);
+ }
+ }
+ @Override
+ public boolean incrementToken() throws IOException {
+ if ( next() ) {
+ int lg = currentWord.length - charPos;
+ System.arraycopy(currentWord, charPos, termAtt.termBuffer(), 0, lg );
+ termAtt.setTermLength(lg);
+ return true;
+ } else {
+ return false;
+ }
+ }
+ private boolean next() {
+ if (currentWordIx >= words.length) {
+ return false;
+ }
+ if (charPos +1 < currentWord.length && softMatcher.find(charPos+1)) {
+ charPos = softMatcher.end();
+ return true;
+ } else {
+ return nextWord();
+ }
+ }
+ private boolean nextWord() {
+ currentWordIx++;
+ if (currentWordIx >= words.length) {
+ return false;
+ }
+ currentWord = words[currentWordIx].toCharArray();
+ termAtt.resizeTermBuffer(currentWord.length);
+ charPos = 0;
+ softMatcher = soft.matcher(words[currentWordIx]);
+ return true;
+ }
+
+ }
+
+
+ @Override
+ public TokenStream tokenStream(String ignoredFieldName, Reader reader) {
+ return new TermCompletionTokenStream((StringReader)reader);
+ }
+}
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractAnalyzerFactoryTest.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractAnalyzerFactoryTest.java 2014-05-09 17:07:05 UTC (rev 8247)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractAnalyzerFactoryTest.java 2014-05-09 17:42:44 UTC (rev 8248)
@@ -27,11 +27,7 @@
package com.bigdata.search;
import java.io.IOException;
-import java.io.StringReader;
-import org.apache.lucene.analysis.Analyzer;
-import org.apache.lucene.analysis.TokenStream;
-import org.apache.lucene.analysis.tokenattributes.TermAttribute;
public abstract class AbstractAnalyzerFactoryTest extends AbstractSearchTest {
@@ -42,37 +38,16 @@
super(arg0);
}
+ @Override
public void setUp() throws Exception {
super.setUp();
- init(getExtraProperties());
+ init(getExtraProperties());
}
+
+
abstract String[] getExtraProperties();
- private Analyzer getAnalyzer(String lang, boolean filterStopWords) {
- return getNdx().getAnalyzer(lang, filterStopWords);
- }
-
- private void comparisonTest(String lang,
- boolean stopWordsSignificant,
- String text,
- String spaceSeparated) throws IOException {
- compareTokenStream(getAnalyzer(lang, stopWordsSignificant), text,
- spaceSeparated.split(" ")); //$NON-NLS-1$
- }
- private void compareTokenStream(Analyzer a, String text, String expected[]) throws IOException {
- TokenStream s = a.tokenStream(null, new StringReader(text));
- int ix = 0;
- while (s.incrementToken()) {
- final TermAttribute term = s.getAttribute(TermAttribute.class);
- final String word = term.term();
- assertTrue(ix < expected.length);
- assertEquals(word, expected[ix++]);
- }
- assertEquals(ix, expected.length);
- }
-
-
- public void testEnglishFilterStopWords() throws IOException {
+ public void testEnglishFilterStopWords() throws IOException {
for (String lang: new String[]{ "eng", null, "" }) { //$NON-NLS-1$ //$NON-NLS-2$
comparisonTest(lang,
true,
@@ -159,14 +134,20 @@
}
private void checkConfig(String classname, String ...langs) {
+ checkConfig(isBroken(), classname, langs);
+
+ }
+ protected void checkConfig(boolean threeLetterOnly, String classname, String ...langs) {
for (String lang:langs) {
// The DefaultAnalyzerFactory only works for language tags of length exactly three.
-// if (lang != null && lang.length()==3)
+ if ((!threeLetterOnly) || (lang != null && lang.length()==3))
{
assertEquals(classname, getAnalyzer(lang,true).getClass().getSimpleName());
- assertEquals(classname, getAnalyzer(lang+NonEnglishExamples.getString("AbstractAnalyzerFactoryTest.0"),true).getClass().getSimpleName()); //$NON-NLS-1$
+ if (!threeLetterOnly) assertEquals(classname, getAnalyzer(lang+"-x-foobar",true).getClass().getSimpleName()); //$NON-NLS-1$
}
}
}
+
+ abstract boolean isBroken() ;
}
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractSearchTest.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractSearchTest.java 2014-05-09 17:07:05 UTC (rev 8247)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/AbstractSearchTest.java 2014-05-09 17:42:44 UTC (rev 8248)
@@ -26,8 +26,14 @@
*/
package com.bigdata.search;
+import java.io.IOException;
+import java.io.StringReader;
import java.util.Properties;
+import org.apache.lucene.analysis.Analyzer;
+import org.apache.lucene.analysis.TokenStream;
+import org.apache.lucene.analysis.tokenattributes.TermAttribute;
+
import com.bigdata.journal.IIndexManager;
import com.bigdata.journal.ITx;
import com.bigdata.journal.ProxyTestCase;
@@ -62,7 +68,7 @@
}
FullTextIndex<Long> createFullTextIndex(String namespace, String ...propertyValuePairs) {
- return createFullTextIndex(namespace, getProperties(), propertyValuePairs);
+ return createFullTextIndex(namespace, (Properties)getProperties().clone(), propertyValuePairs);
}
public void tearDown() throws Exception {
@@ -92,4 +98,51 @@
return properties;
}
+ protected Analyzer getAnalyzer(String lang, boolean filterStopWords) {
+ return getNdx().getAnalyzer(lang, filterStopWords);
+ }
+
+ protected void comparisonTest(String lang, boolean stopWordsSignificant, String text, String spaceSeparated)
+ throws IOException {
+ if (spaceSeparated == null) {
+ String rslt = getTokenStream(getAnalyzer(lang, stopWordsSignificant), text);
+ throw new RuntimeException("Got \"" + rslt+ "\"");
+ }
+ compareTokenStream(getAnalyzer(lang, stopWordsSignificant), text,
+ split(spaceSeparated)); //$NON-NLS-1$
+ }
+
+ private String[] split(String spaceSeparated) {
+ if (spaceSeparated.length()==0) {
+ return new String[0];
+ }
+ return spaceSeparated.split(" ");
+ }
+
+ protected String getTokenStream(Analyzer a, String text) throws IOException {
+ StringBuffer sb = new StringBuffer();
+ TokenStream s = a.tokenStream(null, new StringReader(text));
+ int ix = 0;
+ while (s.incrementToken()) {
+ final TermAttribute term = s.getAttribute(TermAttribute.class);
+ if (sb.length()!=0) {
+ sb.append(" ");
+ }
+ sb.append(term.term());
+ }
+ return sb.toString();
+ }
+
+ private void compareTokenStream(Analyzer a, String text, String expected[]) throws IOException {
+ TokenStream s = a.tokenStream(null, new StringReader(text));
+ int ix = 0;
+ while (s.incrementToken()) {
+ final TermAttribute term = s.getAttribute(TermAttribute.class);
+ final String word = term.term();
+ assertTrue(ix < expected.length);
+ assertEquals(word, expected[ix++]);
+ }
+ assertEquals(ix, expected.length);
+ }
+
}
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestAll.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestAll.java 2014-05-09 17:07:05 UTC (rev 8247)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestAll.java 2014-05-09 17:42:44 UTC (rev 8248)
@@ -114,6 +114,7 @@
// which is intended to be the same as the intended
// behavior of DefaultAnalyzerFactory
suite.addTestSuite(TestConfigurableAsDefaultAnalyzerFactory.class);
+ suite.addTestSuite(TestConfigurableAnalyzerFactory.class);
return suite;
}
Added: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java (rev 0)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAnalyzerFactory.java 2014-05-09 17:42:44 UTC (rev 8248)
@@ -0,0 +1,195 @@
+/**
+
+Copyright (C) SYSTAP, LLC 2006-2014. All rights reserved.
+
+Contact:
+ SYSTAP, LLC
+ 4501 Tower Road
+ Greensboro, NC 27410
+ lic...@bi...
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+*/
+/*
+ * Created on May 7, 2014
+ */
+package com.bigdata.search;
+
+import java.io.IOException;
+
+import org.apache.lucene.analysis.Analyzer;
+import org.apache.lucene.analysis.KeywordAnalyzer;
+import org.apache.lucene.analysis.cjk.CJKAnalyzer;
+import org.apache.lucene.analysis.de.GermanAnalyzer;
+import org.apache.lucene.analysis.ru.RussianAnalyzer;
+import org.apache.lucene.analysis.standard.StandardAnalyzer;
+import org.apache.lucene.util.Version;
+
+import com.bigdata.search.ConfigurableAnalyzerFactory.AnalyzerOptions;
+
+public class TestConfigurableAnalyzerFactory extends AbstractSearchTest {
+
+ public TestConfigurableAnalyzerFactory() {
+ }
+
+ public TestConfigurableAnalyzerFactory(String arg0) {
+ super(arg0);
+ }
+
+ public void setUp() throws Exception {
+ super.setUp();
+ init(getExtraProperties());
+ }
+
+ private String[] getExtraProperties() {
+ String analyzer = ConfigurableAnalyzerFactory.Options.ANALYZER;
+ return new String[]{
+ FullTextIndex.Options.ANALYZER_FACTORY_CLASS, ConfigurableAnalyzerFactory.class.getName(),
+ analyzer+"*."+AnalyzerOptions.ANALYZER_CLASS, EmptyAnalyzer.class.getName(),
+ analyzer+"x-terms."+AnalyzerOptions.PATTERN, "\\W+",
+ analyzer+"x-splits."+AnalyzerOptions.ANALYZER_CLASS, TermCompletionAnalyzer.class.getName(),
+ analyzer+"x-splits."+AnalyzerOptions.STOPWORDS, AnalyzerOptions.STOPWORDS_VALUE_NONE,
+ analyzer+"x-keywords."+AnalyzerOptions.ANALYZER_CLASS, KeywordAnalyzer.class.getName(),
+ analyzer+"ru-x-de."+AnalyzerOptions.ANALYZER_CLASS, RussianAnalyzer.class.getName(),
+ analyzer+"ru-x-de."+AnalyzerOptions.STOPWORDS, GermanAnalyzer.class.getName(),
+ };
+ }
+
+ private void badCombo(String errorMessage, String ... props) {
+ // Check that some combination of properties on a language create an error
+ String myProps[] = new String[props.length+4];
+ int i=0;
+ for (; i<props.length;i+=2) {
+ myProps[i] = ConfigurableAnalyzerFactory.Options.ANALYZER + "x-testme." + props[i];
+ myProps[i+1] = props[i+1];
+ }
+ myProps[i] = ConfigurableAnalyzerFactory.Options.ANALYZER + "_." + AnalyzerOptions.ANALYZER_CLASS;
+ myProps[i+1] = EmptyAnalyzer.class.getName();
+ myProps[i+2] = FullTextIndex.Options.ANALYZER_FACTORY_CLASS;
+ myProps[i+3] = ConfigurableAnalyzerFactory.class.getName();
+ try {
+ this.createFullTextIndex("test-in-error"+getName(), myProps);
+ }
+ catch (RuntimeException e) {
+ Throwable t = e;
+ while (t.getCause() != null) {
+ t = t.getCause();
+ }
+ assertTrue(t.getMessage(),t.getMessage().contains(errorMessage));
+ return;
+ }
+ fail("No error detected");
+ }
+ public void testBadLike() {
+ badCombo("en-us-x-banana",AnalyzerOptions.LIKE,"en-us-x-banana");
+ }
+ public void testMissingClass() {
+ badCombo("exactly one",AnalyzerOptions.STOPWORDS,AnalyzerOptions.STOPWORDS_VALUE_DEFAULT);
+
+ }
+ public void testLikeAndClass() {
+ badCombo("exactly one",AnalyzerOptions.LIKE,"*", AnalyzerOptions.ANALYZER_CLASS, EmptyAnalyzer.class.getName());
+ }
+ public void testLikeAndStopwords() {
+ badCombo("stopwords",AnalyzerOptions.LIKE,"*", AnalyzerOptions.STOPWORDS,AnalyzerOptions.STOPWORDS_VALUE_DEFAULT);
+ }
+ public void testCantAlwaysHaveStopWords() {
+ badCombo("not supported",
+ AnalyzerOptions.ANALYZER_CLASS, EmptyAnalyzer.class.getName(),
+ AnalyzerOptions.STOPWORDS,StandardAnalyzer.class.getName()
+ );
+
+ }
+ public void testCantAlwaysHaveDefaultStopWords() {
+ badCombo("not supported",
+ AnalyzerOptions.ANALYZER_CLASS, EmptyAnalyzer.class.getName(),
+ AnalyzerOptions.STOPWORDS,AnalyzerOptions.STOPWORDS_VALUE_DEFAULT
+ );
+
+ }
+ public void testCantFindRussianStopWords() {
+ badCombo("find",
+ AnalyzerOptions.ANALYZER_CLASS, GermanAnalyzer.class.getName(),
+ AnalyzerOptions.STOPWORDS,RussianAnalyzer.class.getName()
+ );
+
+ }
+
+
+ public void testEmptyAnalyzer() throws IOException {
+ comparisonTest("en",
+ false,
+ "The fast car arrived slowly.",
+ ""
+ );
+
+ }
+ public void testSyapseExample1() throws IOException {
+ comparisonTest("x-splits",
+ true,
+ "ADENOCARCINOMA OF LUNG, SOMATIC [ERBB2, INS/DUP, NT2322]",
+ "ADENOCARCINOMA OF LUNG, SOMATIC [ERBB2, ERBB2, INS/DUP, DUP, NT2322]"
+ );
+
+ }
+ public void testSyapseExample2() throws IOException {
+ comparisonTest("x-splits",
+ true,
+ "\u2265\u2265\u22653-11.13-11.1",
+ "\u2265\u2265\u22653-11.13-11.1 3-11.13-11.1 11.13-11.1 13-11.1 11.1 1"
+ );
+
+ }
+ public void testSyapseExample4() throws IOException {
+ comparisonTest("x-splits",
+ true,
+ "\u00b1-ACE3.1.1",
+ "\u00b1-ACE3.1.1 ACE3.1.1 1.1 1"
+ );
+
+ }
+ public void testSyapseExample3() throws IOException {
+ comparisonTest("x-splits",
+ true,
+ "2,2,3-trimethylbutane",
+ "2,2,3-trimethylbutane 2,3-trimethylbutane 3-trimethylbutane trimethylbutane"
+ );
+
+ }
+ public void testSyapseExample5() throws IOException {
+ comparisonTest("x-splits",
+ true,
+ "CD8_alpha-low Langerhans cell",
+ "CD8_alpha-low alpha-low low Langerhans cell"
+ );
+
+ }
+ public void testSyapseExample6() throws IOException {
+ comparisonTest("x-splits",
+ true,
+ "6-Monoacetylmorphine:Mass Content:Point in time:Meconium:Quantitative",
+ "6-Monoacetylmorphine:Mass Monoacetylmorphine:Mass Mass Content:Point Point in time:Meconium:Quantitative Meconium:Quantitative Quantitative"
+ );
+
+ }
+ public void testSyapseExample7() throws IOException {
+ comparisonTest("x-splits",
+ true,
+ "N,N-dimethyl",
+ "N,N-dimethyl N-dimethyl dimethyl"
+ );
+
+ }
+
+}
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAsDefaultAnalyzerFactory.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAsDefaultAnalyzerFactory.java 2014-05-09 17:07:05 UTC (rev 8247)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestConfigurableAsDefaultAnalyzerFactory.java 2014-05-09 17:42:44 UTC (rev 8248)
@@ -40,4 +40,9 @@
return new String[]{FullTextIndex.Options.ANALYZER_FACTORY_CLASS, ConfigurableAnalyzerFactory.class.getName()};
}
+ @Override
+ boolean isBroken() {
+ return false;
+ }
+
}
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestDefaultAnalyzerFactory.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestDefaultAnalyzerFactory.java 2014-05-09 17:07:05 UTC (rev 8247)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestDefaultAnalyzerFactory.java 2014-05-09 17:42:44 UTC (rev 8248)
@@ -40,4 +40,27 @@
return new String[0];
}
+ /**
+ * The DefaultAnalyzerFactory has bizarre behavior concerning
+ * language specific settings.
+ * The three letter ISO 639-1 language tags for the languages
+ * for which Lucene has Analyzers use those Analyzers; whereas the two digit ISO
+ * language tags, which are the ones recommended by the IETF and the W3C,
+ * all use the StandardAnalyzer (English). Also a language tag with a subtag
+ * uses the StandardAnalyzer, even if it is a recognized three letter ISO code.
+ */
+ @Override
+ boolean isBroken() {
+ return true;
+ }
+
+ /**
+ * Given legacy concerns, we should preserve the incorrect behavior!
+ */
+ public void testIsBroken() {
+ checkConfig(false, "StandardAnalyzer",
+ "en", "eng", "", null, "ru",
+ "pt", "zh", "por-br", "cs", "dut-za", "nl", "de", "gre-at", "el", "th");
+ }
+
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mrp...@us...> - 2014-05-09 17:07:09
|
Revision: 8247
http://sourceforge.net/p/bigdata/code/8247
Author: mrpersonick
Date: 2014-05-09 17:07:05 +0000 (Fri, 09 May 2014)
Log Message:
-----------
commented out refs to bigdata/blueprints integration
Modified Paths:
--------------
branches/BLUEPRINTS/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/BlueprintsServlet.java
Modified: branches/BLUEPRINTS/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/BlueprintsServlet.java
===================================================================
--- branches/BLUEPRINTS/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/BlueprintsServlet.java 2014-05-09 16:41:58 UTC (rev 8246)
+++ branches/BLUEPRINTS/bigdata-sails/src/java/com/bigdata/rdf/sail/webapp/BlueprintsServlet.java 2014-05-09 17:07:05 UTC (rev 8247)
@@ -31,11 +31,9 @@
import org.apache.log4j.Logger;
-import com.bigdata.blueprints.BigdataGraphBulkLoad;
import com.bigdata.rdf.sail.BigdataSailRepositoryConnection;
import com.bigdata.rdf.sail.webapp.client.MiniMime;
import com.bigdata.rdf.store.AbstractTripleStore;
-import com.tinkerpop.blueprints.util.io.graphml.GraphMLReader;
/**
* Helper servlet for the blueprints layer.
@@ -117,14 +115,15 @@
conn = getBigdataRDFContext()
.getUnisolatedConnection(namespace);
- final BigdataGraphBulkLoad graph = new BigdataGraphBulkLoad(conn);
+// final BigdataGraphBulkLoad graph = new BigdataGraphBulkLoad(conn);
+//
+// GraphMLReader.inputGraph(graph, req.getInputStream());
+//
+// graph.commit();
+//
+// final long nmodified = graph.getMutationCountLastCommit();
+ final long nmodified = 0;
- GraphMLReader.inputGraph(graph, req.getInputStream());
-
- graph.commit();
-
- final long nmodified = graph.getMutationCountLastCommit();
-
final long elapsed = System.currentTimeMillis() - begin;
reportModifiedCount(resp, nmodified, elapsed);
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <mrp...@us...> - 2014-05-09 16:42:00
|
Revision: 8246
http://sourceforge.net/p/bigdata/code/8246
Author: mrpersonick
Date: 2014-05-09 16:41:58 +0000 (Fri, 09 May 2014)
Log Message:
-----------
trying to find a clean build.xml starting point
Modified Paths:
--------------
branches/BLUEPRINTS/build.xml
Modified: branches/BLUEPRINTS/build.xml
===================================================================
--- branches/BLUEPRINTS/build.xml 2014-05-09 11:55:20 UTC (rev 8245)
+++ branches/BLUEPRINTS/build.xml 2014-05-09 16:41:58 UTC (rev 8246)
@@ -62,9 +62,6 @@
<fileset dir="${bigdata.dir}/bigdata-sails/lib">
<include name="**/*.jar" />
</fileset>
- <fileset dir="${bigdata.dir}/bigdata-blueprints/lib">
- <include name="blueprints-core-${blueprints.version}.jar" />
- </fileset>
<fileset dir="${bigdata.dir}/bigdata-gom/lib">
<include name="**/*.jar" />
</fileset>
@@ -232,7 +229,6 @@
<src path="${bigdata.dir}/bigdata-jini/src/java" />
<src path="${bigdata.dir}/bigdata-rdf/src/java" />
<src path="${bigdata.dir}/bigdata-sails/src/java" />
- <src path="${bigdata.dir}/bigdata-blueprints/src/java" />
<src path="${bigdata.dir}/bigdata-gom/src/java" />
<src path="${bigdata.dir}/bigdata-ganglia/src/java" />
<src path="${bigdata.dir}/bigdata-gas/src/java" />
@@ -264,10 +260,6 @@
<exclude name="**/*.java" />
<exclude name="**/package.html" />
</fileset>
- <fileset dir="${bigdata.dir}/bigdata-blueprints/src/java">
- <exclude name="**/*.java" />
- <exclude name="**/package.html" />
- </fileset>
<fileset dir="${bigdata.dir}/bigdata-gom/src/java">
<exclude name="**/*.java" />
<exclude name="**/package.html" />
@@ -318,7 +310,6 @@
<fileset dir="${bigdata.dir}/bigdata-rdf/src/samples" />
<fileset dir="${bigdata.dir}/bigdata-sails/src/java" />
<fileset dir="${bigdata.dir}/bigdata-sails/src/samples" />
- <fileset dir="${bigdata.dir}/bigdata-blueprints/src/java" />
<fileset dir="${bigdata.dir}/bigdata-gom/src/java" />
<fileset dir="${bigdata.dir}/bigdata-gom/src/samples" />
<fileset dir="${bigdata.dir}/ctc-striterators/src/java" />
@@ -376,7 +367,6 @@
<fileset dir="bigdata-jini/src/java" />
<fileset dir="bigdata-rdf/src/java" />
<fileset dir="bigdata-sails/src/java" />
- <fileset dir="bigdata-blueprints/src/java" />
<fileset dir="bigdata-gom/src/java" />
</jar>
<bnd output="${build.dir}/bundles/com.bigata-${osgi.version}.jar" classpath="${build.dir}/classes" eclipse="false" failok="false" exceptions="true" files="${basedir}/osgi/bigdata.bnd" />
@@ -416,7 +406,6 @@
<packageset dir="${bigdata.dir}/bigdata-rdf/src/samples" />
<packageset dir="${bigdata.dir}/bigdata-sails/src/java" />
<packageset dir="${bigdata.dir}/bigdata-sails/src/samples" />
- <packageset dir="${bigdata.dir}/bigdata-blueprints/src/java" />
<packageset dir="${bigdata.dir}/bigdata-gom/src/java" />
<packageset dir="${bigdata.dir}/bigdata-gom/src/samples" />
<packageset dir="${bigdata.dir}/bigdata-gas/src/java" />
@@ -457,9 +446,6 @@
<fileset dir="${bigdata.dir}/bigdata-sails/lib">
<include name="**/*.jar" />
</fileset>
- <fileset dir="${bigdata.dir}/bigdata-blueprints/lib">
- <include name="blueprints-core-${blueprints.version}.jar" />
- </fileset>
<fileset dir="${bigdata.dir}/bigdata-gom/lib">
<include name="**/*.jar" />
</fileset>
@@ -572,7 +558,6 @@
<fileset dir="${bigdata.dir}/bigdata" includes="LEGAL/*"/>
<fileset dir="${bigdata.dir}/bigdata-rdf" includes="LEGAL/*"/>
<fileset dir="${bigdata.dir}/bigdata-sails" includes="LEGAL/*"/>
- <fileset dir="${bigdata.dir}/bigdata-blueprints" includes="LEGAL/*"/>
<fileset dir="${bigdata.dir}/bigdata-gom" includes="LEGAL/*"/>
<fileset dir="${bigdata.dir}/bigdata-jini" includes="LEGAL/*"/>
<!-- bigdata jar plus some dependencies as filtered by autojar.
@@ -682,7 +667,6 @@
<include name="bigdata-jini/LEGAL/*" />
<include name="bigdata-rdf/LEGAL/*" />
<include name="bigdata-sails/LEGAL/*" />
- <include name="bigdata-blueprints/LEGAL/*" />
<include name="bigdata-gom/LEGAL/*" />
</fileset>
</copy>
@@ -949,7 +933,6 @@
<property name="bigdata-jini.lib" location="${bigdata.dir}/bigdata-jini/lib/jini/lib" />
<property name="bigdata-rdf.lib" location="${bigdata.dir}/bigdata-rdf/lib" />
<property name="bigdata-sails.lib" location="${bigdata.dir}/bigdata-sails/lib" />
- <property name="bigdata-blueprints.lib" location="${bigdata.dir}/bigdata-blueprints/lib" />
<property name="bigdata-gom.lib" location="${bigdata.dir}/bigdata-gom/lib" />
<property name="bigdata-jetty.lib" location="${bigdata.dir}/bigdata/lib/jetty" />
<property name="bigdata-http.lib" location="${bigdata.dir}/bigdata-sails/lib/httpcomponents" />
@@ -993,11 +976,6 @@
<!-- GOM library -->
<!-- Note: Nothing yet for GOM -->
- <!-- Blueprints library -->
-
- <copy file="${bigdata-blueprints.lib}/blueprints-core-${blueprints.version}.jar"
- tofile="${dist.lib}/blueprints-core.jar" />
-
<!-- jetty library -->
<copy file="${bigdata-jetty.lib}/jetty-continuation-${jetty.version}.jar"
tofile="${dist.lib}/jetty-continuation.jar" />
@@ -1406,9 +1384,6 @@
<copy toDir="${build.dir}/bigdata-sails/src">
<fileset dir="${bigdata.dir}/bigdata-sails/src" />
</copy>
- <copy toDir="${build.dir}/bigdata-blueprints/src">
- <fileset dir="${bigdata.dir}/bigdata-blueprints/src" />
- </copy>
<copy toDir="${build.dir}/bigdata-gom/src">
<fileset dir="${bigdata.dir}/bigdata-gom/src" />
</copy>
@@ -1449,10 +1424,6 @@
<copy toDir="${build.dir}/bigdata-sails/lib">
<fileset dir="${bigdata.dir}/bigdata-sails/lib" />
</copy>
- <mkdir dir="${build.dir}/bigdata-blueprints/lib" />
- <copy toDir="${build.dir}/bigdata-blueprints/lib">
- <fileset dir="${bigdata.dir}/bigdata-blueprints/lib" />
- </copy>
<mkdir dir="${build.dir}/src" />
<mkdir dir="${build.dir}/src/resources" />
@@ -1506,7 +1477,6 @@
<include name="bigdata-jini/src/**" />
<include name="bigdata-rdf/src/**" />
<include name="bigdata-sails/src/**" />
- <include name="bigdata-blueprints/src/**" />
<include name="bigdata-gom/src/**" />
<include name="bigdata-war/src/**" />
<include name="ctc-striterators/src/**" />
@@ -1578,7 +1548,6 @@
<include name="bigdata-jini/LEGAL/*" />
<include name="bigdata-rdf/LEGAL/*" />
<include name="bigdata-sails/LEGAL/*" />
- <include name="bigdata-blueprints/LEGAL/*" />
<include name="bigdata-gom/LEGAL/*" />
</fileset>
</copy>
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tob...@us...> - 2014-05-09 11:55:24
|
Revision: 8245
http://sourceforge.net/p/bigdata/code/8245
Author: tobycraig
Date: 2014-05-09 11:55:20 +0000 (Fri, 09 May 2014)
Log Message:
-----------
Added support for syntax highlighting to update panel, switching mode according to selected type/format
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/index.html
branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/workbench.js
Added Paths:
-----------
branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-modes/javascript.js
branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-modes/ntriples.js
branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-modes/turtle.js
branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-modes/xml.js
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/index.html
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/index.html 2014-05-09 11:33:52 UTC (rev 8244)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/index.html 2014-05-09 11:55:20 UTC (rev 8245)
@@ -221,7 +221,11 @@
<script src="/bigdata/html/js/vendor/jquery.hotkeys.js"></script>
<script src="/bigdata/html/js/vendor/codemirror.js"></script>
<script src="/bigdata/html/js/vendor/cm-addons/placeholder.js"></script>
+ <script src="/bigdata/html/js/vendor/cm-modes/javascript.js"></script>
+ <script src="/bigdata/html/js/vendor/cm-modes/ntriples.js"></script>
<script src="/bigdata/html/js/vendor/cm-modes/sparql.js"></script>
+ <script src="/bigdata/html/js/vendor/cm-modes/turtle.js"></script>
+ <script src="/bigdata/html/js/vendor/cm-modes/xml.js"></script>
<script src="/bigdata/html/js/workbench.js"></script>
</body>
</html>
Added: branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-modes/javascript.js
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-modes/javascript.js (rev 0)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-modes/javascript.js 2014-05-09 11:55:20 UTC (rev 8245)
@@ -0,0 +1,660 @@
+// TODO actually recognize syntax of TypeScript constructs
+
+(function(mod) {
+ if (typeof exports == "object" && typeof module == "object") // CommonJS
+ mod(require("../../lib/codemirror"));
+ else if (typeof define == "function" && define.amd) // AMD
+ define(["../../lib/codemirror"], mod);
+ else // Plain browser env
+ mod(CodeMirror);
+})(function(CodeMirror) {
+"use strict";
+
+CodeMirror.defineMode("javascript", function(config, parserConfig) {
+ var indentUnit = config.indentUnit;
+ var statementIndent = parserConfig.statementIndent;
+ var jsonldMode = parserConfig.jsonld;
+ var jsonMode = parserConfig.json || jsonldMode;
+ var isTS = parserConfig.typescript;
+
+ // Tokenizer
+
+ var keywords = function(){
+ function kw(type) {return {type: type, style: "keyword"};}
+ var A = kw("keyword a"), B = kw("keyword b"), C = kw("keyword c");
+ var operator = kw("operator"), atom = {type: "atom", style: "atom"};
+
+ var jsKeywords = {
+ "if": kw("if"), "while": A, "with": A, "else": B, "do": B, "try": B, "finally": B,
+ "return": C, "break": C, "continue": C, "new": C, "delete": C, "throw": C, "debugger": C,
+ "var": kw("var"), "const": kw("var"), "let": kw("var"),
+ "function": kw("function"), "catch": kw("catch"),
+ "for": kw("for"), "switch": kw("switch"), "case": kw("case"), "default": kw("default"),
+ "in": operator, "typeof": operator, "instanceof": operator,
+ "true": atom, "false": atom, "null": atom, "undefined": atom, "NaN": atom, "Infinity": atom,
+ "this": kw("this"), "module": kw("module"), "class": kw("class"), "super": kw("atom"),
+ "yield": C, "export": kw("export"), "import": kw("import"), "extends": C
+ };
+
+ // Extend the 'normal' keywords with the TypeScript language extensions
+ if (isTS) {
+ var type = {type: "variable", style: "variable-3"};
+ var tsKeywords = {
+ // object-like things
+ "interface": kw("interface"),
+ "extends": kw("extends"),
+ "constructor": kw("constructor"),
+
+ // scope modifiers
+ "public": kw("public"),
+ "private": kw("private"),
+ "protected": kw("protected"),
+ "static": kw("static"),
+
+ // types
+ "string": type, "number": type, "bool": type, "any": type
+ };
+
+ for (var attr in tsKeywords) {
+ jsKeywords[attr] = tsKeywords[attr];
+ }
+ }
+
+ return jsKeywords;
+ }();
+
+ var isOperatorChar = /[+\-*&%=<>!?|~^]/;
+ var isJsonldKeyword = /^@(context|id|value|language|type|container|list|set|reverse|index|base|vocab|graph)"/;
+
+ function readRegexp(stream) {
+ var escaped = false, next, inSet = false;
+ while ((next = stream.next()) != null) {
+ if (!escaped) {
+ if (next == "/" && !inSet) return;
+ if (next == "[") inSet = true;
+ else if (inSet && next == "]") inSet = false;
+ }
+ escaped = !escaped && next == "\\";
+ }
+ }
+
+ // Used as scratch variables to communicate multiple values without
+ // consing up tons of objects.
+ var type, content;
+ function ret(tp, style, cont) {
+ type = tp; content = cont;
+ return style;
+ }
+ function tokenBase(stream, state) {
+ var ch = stream.next();
+ if (ch == '"' || ch == "'") {
+ state.tokenize = tokenString(ch);
+ return state.tokenize(stream, state);
+ } else if (ch == "." && stream.match(/^\d+(?:[eE][+\-]?\d+)?/)) {
+ return ret("number", "number");
+ } else if (ch == "." && stream.match("..")) {
+ return ret("spread", "meta");
+ } else if (/[\[\]{}\(\),;\:\.]/.test(ch)) {
+ return ret(ch);
+ } else if (ch == "=" && stream.eat(">")) {
+ return ret("=>", "operator");
+ } else if (ch == "0" && stream.eat(/x/i)) {
+ stream.eatWhile(/[\da-f]/i);
+ return ret("number", "number");
+ } else if (/\d/.test(ch)) {
+ stream.match(/^\d*(?:\.\d*)?(?:[eE][+\-]?\d+)?/);
+ return ret("number", "number");
+ } else if (ch == "/") {
+ if (stream.eat("*")) {
+ state.tokenize = tokenComment;
+ return tokenComment(stream, state);
+ } else if (stream.eat("/")) {
+ stream.skipToEnd();
+ return ret("comment", "comment");
+ } else if (state.lastType == "operator" || state.lastType == "keyword c" ||
+ state.lastType == "sof" || /^[\[{}\(,;:]$/.test(state.lastType)) {
+ readRegexp(stream);
+ stream.eatWhile(/[gimy]/); // 'y' is "sticky" option in Mozilla
+ return ret("regexp", "string-2");
+ } else {
+ stream.eatWhile(isOperatorChar);
+ return ret("operator", "operator", stream.current());
+ }
+ } else if (ch == "`") {
+ state.tokenize = tokenQuasi;
+ return tokenQuasi(stream, state);
+ } else if (ch == "#") {
+ stream.skipToEnd();
+ return ret("error", "error");
+ } else if (isOperatorChar.test(ch)) {
+ stream.eatWhile(isOperatorChar);
+ return ret("operator", "operator", stream.current());
+ } else {
+ stream.eatWhile(/[\w\$_]/);
+ var word = stream.current(), known = keywords.propertyIsEnumerable(word) && keywords[word];
+ return (known && state.lastType != ".") ? ret(known.type, known.style, word) :
+ ret("variable", "variable", word);
+ }
+ }
+
+ function tokenString(quote) {
+ return function(stream, state) {
+ var escaped = false, next;
+ if (jsonldMode && stream.peek() == "@" && stream.match(isJsonldKeyword)){
+ state.tokenize = tokenBase;
+ return ret("jsonld-keyword", "meta");
+ }
+ while ((next = stream.next()) != null) {
+ if (next == quote && !escaped) break;
+ escaped = !escaped && next == "\\";
+ }
+ if (!escaped) state.tokenize = tokenBase;
+ return ret("string", "string");
+ };
+ }
+
+ function tokenComment(stream, state) {
+ var maybeEnd = false, ch;
+ while (ch = stream.next()) {
+ if (ch == "/" && maybeEnd) {
+ state.tokenize = tokenBase;
+ break;
+ }
+ maybeEnd = (ch == "*");
+ }
+ return ret("comment", "comment");
+ }
+
+ function tokenQuasi(stream, state) {
+ var escaped = false, next;
+ while ((next = stream.next()) != null) {
+ if (!escaped && (next == "`" || next == "$" && stream.eat("{"))) {
+ state.tokenize = tokenBase;
+ break;
+ }
+ escaped = !escaped && next == "\\";
+ }
+ return ret("quasi", "string-2", stream.current());
+ }
+
+ var brackets = "([{}])";
+ // This is a crude lookahead trick to try and notice that we're
+ // parsing the argument patterns for a fat-arrow function before we
+ // actually hit the arrow token. It only works if the arrow is on
+ // the same line as the arguments and there's no strange noise
+ // (comments) in between. Fallback is to only notice when we hit the
+ // arrow, and not declare the arguments as locals for the arrow
+ // body.
+ function findFatArrow(stream, state) {
+ if (state.fatArrowAt) state.fatArrowAt = null;
+ var arrow = stream.string.indexOf("=>", stream.start);
+ if (arrow < 0) return;
+
+ var depth = 0, sawSomething = false;
+ for (var pos = arrow - 1; pos >= 0; --pos) {
+ var ch = stream.string.charAt(pos);
+ var bracket = brackets.indexOf(ch);
+ if (bracket >= 0 && bracket < 3) {
+ if (!depth) { ++pos; break; }
+ if (--depth == 0) break;
+ } else if (bracket >= 3 && bracket < 6) {
+ ++depth;
+ } else if (/[$\w]/.test(ch)) {
+ sawSomething = true;
+ } else if (sawSomething && !depth) {
+ ++pos;
+ break;
+ }
+ }
+ if (sawSomething && !depth) state.fatArrowAt = pos;
+ }
+
+ // Parser
+
+ var atomicTypes = {"atom": true, "number": true, "variable": true, "string": true, "regexp": true, "this": true, "jsonld-keyword": true};
+
+ function JSLexical(indented, column, type, align, prev, info) {
+ this.indented = indented;
+ this.column = column;
+ this.type = type;
+ this.prev = prev;
+ this.info = info;
+ if (align != null) this.align = align;
+ }
+
+ function inScope(state, varname) {
+ for (var v = state.localVars; v; v = v.next)
+ if (v.name == varname) return true;
+ for (var cx = state.context; cx; cx = cx.prev) {
+ for (var v = cx.vars; v; v = v.next)
+ if (v.name == varname) return true;
+ }
+ }
+
+ function parseJS(state, style, type, content, stream) {
+ var cc = state.cc;
+ // Communicate our context to the combinators.
+ // (Less wasteful than consing up a hundred closures on every call.)
+ cx.state = state; cx.stream = stream; cx.marked = null, cx.cc = cc;
+
+ if (!state.lexical.hasOwnProperty("align"))
+ state.lexical.align = true;
+
+ while(true) {
+ var combinator = cc.length ? cc.pop() : jsonMode ? expression : statement;
+ if (combinator(type, content)) {
+ while(cc.length && cc[cc.length - 1].lex)
+ cc.pop()();
+ if (cx.marked) return cx.marked;
+ if (type == "variable" && inScope(state, content)) return "variable-2";
+ return style;
+ }
+ }
+ }
+
+ // Combinator utils
+
+ var cx = {state: null, column: null, marked: null, cc: null};
+ function pass() {
+ for (var i = arguments.length - 1; i >= 0; i--) cx.cc.push(arguments[i]);
+ }
+ function cont() {
+ pass.apply(null, arguments);
+ return true;
+ }
+ function register(varname) {
+ function inList(list) {
+ for (var v = list; v; v = v.next)
+ if (v.name == varname) return true;
+ return false;
+ }
+ var state = cx.state;
+ if (state.context) {
+ cx.marked = "def";
+ if (inList(state.localVars)) return;
+ state.localVars = {name: varname, next: state.localVars};
+ } else {
+ if (inList(state.globalVars)) return;
+ if (parserConfig.globalVars)
+ state.globalVars = {name: varname, next: state.globalVars};
+ }
+ }
+
+ // Combinators
+
+ var defaultVars = {name: "this", next: {name: "arguments"}};
+ function pushcontext() {
+ cx.state.context = {prev: cx.state.context, vars: cx.state.localVars};
+ cx.state.localVars = defaultVars;
+ }
+ function popcontext() {
+ cx.state.localVars = cx.state.context.vars;
+ cx.state.context = cx.state.context.prev;
+ }
+ function pushlex(type, info) {
+ var result = function() {
+ var state = cx.state, indent = state.indented;
+ if (state.lexical.type == "stat") indent = state.lexical.indented;
+ state.lexical = new JSLexical(indent, cx.stream.column(), type, null, state.lexical, info);
+ };
+ result.lex = true;
+ return result;
+ }
+ function poplex() {
+ var state = cx.state;
+ if (state.lexical.prev) {
+ if (state.lexical.type == ")")
+ state.indented = state.lexical.indented;
+ state.lexical = state.lexical.prev;
+ }
+ }
+ poplex.lex = true;
+
+ function expect(wanted) {
+ function exp(type) {
+ if (type == wanted) return cont();
+ else if (wanted == ";") return pass();
+ else return cont(exp);
+ };
+ return exp;
+ }
+
+ function statement(type, value) {
+ if (type == "var") return cont(pushlex("vardef", value.length), vardef, expect(";"), poplex);
+ if (type == "keyword a") return cont(pushlex("form"), expression, statement, poplex);
+ if (type == "keyword b") return cont(pushlex("form"), statement, poplex);
+ if (type == "{") return cont(pushlex("}"), block, poplex);
+ if (type == ";") return cont();
+ if (type == "if") {
+ if (cx.state.lexical.info == "else" && cx.state.cc[cx.state.cc.length - 1] == poplex)
+ cx.state.cc.pop()();
+ return cont(pushlex("form"), expression, statement, poplex, maybeelse);
+ }
+ if (type == "function") return cont(functiondef);
+ if (type == "for") return cont(pushlex("form"), forspec, statement, poplex);
+ if (type == "variable") return cont(pushlex("stat"), maybelabel);
+ if (type == "switch") return cont(pushlex("form"), expression, pushlex("}", "switch"), expect("{"),
+ block, poplex, poplex);
+ if (type == "case") return cont(expression, expect(":"));
+ if (type == "default") return cont(expect(":"));
+ if (type == "catch") return cont(pushlex("form"), pushcontext, expect("("), funarg, expect(")"),
+ statement, poplex, popcontext);
+ if (type == "module") return cont(pushlex("form"), pushcontext, afterModule, popcontext, poplex);
+ if (type == "class") return cont(pushlex("form"), className, objlit, poplex);
+ if (type == "export") return cont(pushlex("form"), afterExport, poplex);
+ if (type == "import") return cont(pushlex("form"), afterImport, poplex);
+ return pass(pushlex("stat"), expression, expect(";"), poplex);
+ }
+ function expression(type) {
+ return expressionInner(type, false);
+ }
+ function expressionNoComma(type) {
+ return expressionInner(type, true);
+ }
+ function expressionInner(type, noComma) {
+ if (cx.state.fatArrowAt == cx.stream.start) {
+ var body = noComma ? arrowBodyNoComma : arrowBody;
+ if (type == "(") return cont(pushcontext, pushlex(")"), commasep(pattern, ")"), poplex, expect("=>"), body, popcontext);
+ else if (type == "variable") return pass(pushcontext, pattern, expect("=>"), body, popcontext);
+ }
+
+ var maybeop = noComma ? maybeoperatorNoComma : maybeoperatorComma;
+ if (atomicTypes.hasOwnProperty(type)) return cont(maybeop);
+ if (type == "function") return cont(functiondef, maybeop);
+ if (type == "keyword c") return cont(noComma ? maybeexpressionNoComma : maybeexpression);
+ if (type == "(") return cont(pushlex(")"), maybeexpression, comprehension, expect(")"), poplex, maybeop);
+ if (type == "operator" || type == "spread") return cont(noComma ? expressionNoComma : expression);
+ if (type == "[") return cont(pushlex("]"), arrayLiteral, poplex, maybeop);
+ if (type == "{") return contCommasep(objprop, "}", null, maybeop);
+ if (type == "quasi") { return pass(quasi, maybeop); }
+ return cont();
+ }
+ function maybeexpression(type) {
+ if (type.match(/[;\}\)\],]/)) return pass();
+ return pass(expression);
+ }
+ function maybeexpressionNoComma(type) {
+ if (type.match(/[;\}\)\],]/)) return pass();
+ return pass(expressionNoComma);
+ }
+
+ function maybeoperatorComma(type, value) {
+ if (type == ",") return cont(expression);
+ return maybeoperatorNoComma(type, value, false);
+ }
+ function maybeoperatorNoComma(type, value, noComma) {
+ var me = noComma == false ? maybeoperatorComma : maybeoperatorNoComma;
+ var expr = noComma == false ? expression : expressionNoComma;
+ if (value == "=>") return cont(pushcontext, noComma ? arrowBodyNoComma : arrowBody, popcontext);
+ if (type == "operator") {
+ if (/\+\+|--/.test(value)) return cont(me);
+ if (value == "?") return cont(expression, expect(":"), expr);
+ return cont(expr);
+ }
+ if (type == "quasi") { return pass(quasi, me); }
+ if (type == ";") return;
+ if (type == "(") return contCommasep(expressionNoComma, ")", "call", me);
+ if (type == ".") return cont(property, me);
+ if (type == "[") return cont(pushlex("]"), maybeexpression, expect("]"), poplex, me);
+ }
+ function quasi(type, value) {
+ if (type != "quasi") return pass();
+ if (value.slice(value.length - 2) != "${") return cont(quasi);
+ return cont(expression, continueQuasi);
+ }
+ function continueQuasi(type) {
+ if (type == "}") {
+ cx.marked = "string-2";
+ cx.state.tokenize = tokenQuasi;
+ return cont(quasi);
+ }
+ }
+ function arrowBody(type) {
+ findFatArrow(cx.stream, cx.state);
+ if (type == "{") return pass(statement);
+ return pass(expression);
+ }
+ function arrowBodyNoComma(type) {
+ findFatArrow(cx.stream, cx.state);
+ if (type == "{") return pass(statement);
+ return pass(expressionNoComma);
+ }
+ function maybelabel(type) {
+ if (type == ":") return cont(poplex, statement);
+ return pass(maybeoperatorComma, expect(";"), poplex);
+ }
+ function property(type) {
+ if (type == "variable") {cx.marked = "property"; return cont();}
+ }
+ function objprop(type, value) {
+ if (type == "variable") {
+ cx.marked = "property";
+ if (value == "get" || value == "set") return cont(getterSetter);
+ } else if (type == "number" || type == "string") {
+ cx.marked = jsonldMode ? "property" : (type + " property");
+ } else if (type == "[") {
+ return cont(expression, expect("]"), afterprop);
+ }
+ if (atomicTypes.hasOwnProperty(type)) return cont(afterprop);
+ }
+ function getterSetter(type) {
+ if (type != "variable") return pass(afterprop);
+ cx.marked = "property";
+ return cont(functiondef);
+ }
+ function afterprop(type) {
+ if (type == ":") return cont(expressionNoComma);
+ if (type == "(") return pass(functiondef);
+ }
+ function commasep(what, end) {
+ function proceed(type) {
+ if (type == ",") {
+ var lex = cx.state.lexical;
+ if (lex.info == "call") lex.pos = (lex.pos || 0) + 1;
+ return cont(what, proceed);
+ }
+ if (type == end) return cont();
+ return cont(expect(end));
+ }
+ return function(type) {
+ if (type == end) return cont();
+ return pass(what, proceed);
+ };
+ }
+ function contCommasep(what, end, info) {
+ for (var i = 3; i < arguments.length; i++)
+ cx.cc.push(arguments[i]);
+ return cont(pushlex(end, info), commasep(what, end), poplex);
+ }
+ function block(type) {
+ if (type == "}") return cont();
+ return pass(statement, block);
+ }
+ function maybetype(type) {
+ if (isTS && type == ":") return cont(typedef);
+ }
+ function typedef(type) {
+ if (type == "variable"){cx.marked = "variable-3"; return cont();}
+ }
+ function vardef() {
+ return pass(pattern, maybetype, maybeAssign, vardefCont);
+ }
+ function pattern(type, value) {
+ if (type == "variable") { register(value); return cont(); }
+ if (type == "[") return contCommasep(pattern, "]");
+ if (type == "{") return contCommasep(proppattern, "}");
+ }
+ function proppattern(type, value) {
+ if (type == "variable" && !cx.stream.match(/^\s*:/, false)) {
+ register(value);
+ return cont(maybeAssign);
+ }
+ if (type == "variable") cx.marked = "property";
+ return cont(expect(":"), pattern, maybeAssign);
+ }
+ function maybeAssign(_type, value) {
+ if (value == "=") return cont(expressionNoComma);
+ }
+ function vardefCont(type) {
+ if (type == ",") return cont(vardef);
+ }
+ function maybeelse(type, value) {
+ if (type == "keyword b" && value == "else") return cont(pushlex("form", "else"), statement, poplex);
+ }
+ function forspec(type) {
+ if (type == "(") return cont(pushlex(")"), forspec1, expect(")"), poplex);
+ }
+ function forspec1(type) {
+ if (type == "var") return cont(vardef, expect(";"), forspec2);
+ if (type == ";") return cont(forspec2);
+ if (type == "variable") return cont(formaybeinof);
+ return pass(expression, expect(";"), forspec2);
+ }
+ function formaybeinof(_type, value) {
+ if (value == "in" || value == "of") { cx.marked = "keyword"; return cont(expression); }
+ return cont(maybeoperatorComma, forspec2);
+ }
+ function forspec2(type, value) {
+ if (type == ";") return cont(forspec3);
+ if (value == "in" || value == "of") { cx.marked = "keyword"; return cont(expression); }
+ return pass(expression, expect(";"), forspec3);
+ }
+ function forspec3(type) {
+ if (type != ")") cont(expression);
+ }
+ function functiondef(type, value) {
+ if (value == "*") {cx.marked = "keyword"; return cont(functiondef);}
+ if (type == "variable") {register(value); return cont(functiondef);}
+ if (type == "(") return cont(pushcontext, pushlex(")"), commasep(funarg, ")"), poplex, statement, popcontext);
+ }
+ function funarg(type) {
+ if (type == "spread") return cont(funarg);
+ return pass(pattern, maybetype);
+ }
+ function className(type, value) {
+ if (type == "variable") {register(value); return cont(classNameAfter);}
+ }
+ function classNameAfter(_type, value) {
+ if (value == "extends") return cont(expression);
+ }
+ function objlit(type) {
+ if (type == "{") return contCommasep(objprop, "}");
+ }
+ function afterModule(type, value) {
+ if (type == "string") return cont(statement);
+ if (type == "variable") { register(value); return cont(maybeFrom); }
+ }
+ function afterExport(_type, value) {
+ if (value == "*") { cx.marked = "keyword"; return cont(maybeFrom, expect(";")); }
+ if (value == "default") { cx.marked = "keyword"; return cont(expression, expect(";")); }
+ return pass(statement);
+ }
+ function afterImport(type) {
+ if (type == "string") return cont();
+ return pass(importSpec, maybeFrom);
+ }
+ function importSpec(type, value) {
+ if (type == "{") return contCommasep(importSpec, "}");
+ if (type == "variable") register(value);
+ return cont();
+ }
+ function maybeFrom(_type, value) {
+ if (value == "from") { cx.marked = "keyword"; return cont(expression); }
+ }
+ function arrayLiteral(type) {
+ if (type == "]") return cont();
+ return pass(expressionNoComma, maybeArrayComprehension);
+ }
+ function maybeArrayComprehension(type) {
+ if (type == "for") return pass(comprehension, expect("]"));
+ if (type == ",") return cont(commasep(expressionNoComma, "]"));
+ return pass(commasep(expressionNoComma, "]"));
+ }
+ function comprehension(type) {
+ if (type == "for") return cont(forspec, comprehension);
+ if (type == "if") return cont(expression, comprehension);
+ }
+
+ // Interface
+
+ return {
+ startState: function(basecolumn) {
+ var state = {
+ tokenize: tokenBase,
+ lastType: "sof",
+ cc: [],
+ lexical: new JSLexical((basecolumn || 0) - indentUnit, 0, "block", false),
+ localVars: parserConfig.localVars,
+ context: parserConfig.localVars && {vars: parserConfig.localVars},
+ indented: 0
+ };
+ if (parserConfig.globalVars && typeof parserConfig.globalVars == "object")
+ state.globalVars = parserConfig.globalVars;
+ return state;
+ },
+
+ token: function(stream, state) {
+ if (stream.sol()) {
+ if (!state.lexical.hasOwnProperty("align"))
+ state.lexical.align = false;
+ state.indented = stream.indentation();
+ findFatArrow(stream, state);
+ }
+ if (state.tokenize != tokenComment && stream.eatSpace()) return null;
+ var style = state.tokenize(stream, state);
+ if (type == "comment") return style;
+ state.lastType = type == "operator" && (content == "++" || content == "--") ? "incdec" : type;
+ return parseJS(state, style, type, content, stream);
+ },
+
+ indent: function(state, textAfter) {
+ if (state.tokenize == tokenComment) return CodeMirror.Pass;
+ if (state.tokenize != tokenBase) return 0;
+ var firstChar = textAfter && textAfter.charAt(0), lexical = state.lexical;
+ // Kludge to prevent 'maybelse' from blocking lexical scope pops
+ if (!/^\s*else\b/.test(textAfter)) for (var i = state.cc.length - 1; i >= 0; --i) {
+ var c = state.cc[i];
+ if (c == poplex) lexical = lexical.prev;
+ else if (c != maybeelse) break;
+ }
+ if (lexical.type == "stat" && firstChar == "}") lexical = lexical.prev;
+ if (statementIndent && lexical.type == ")" && lexical.prev.type == "stat")
+ lexical = lexical.prev;
+ var type = lexical.type, closing = firstChar == type;
+
+ if (type == "vardef") return lexical.indented + (state.lastType == "operator" || state.lastType == "," ? lexical.info + 1 : 0);
+ else if (type == "form" && firstChar == "{") return lexical.indented;
+ else if (type == "form") return lexical.indented + indentUnit;
+ else if (type == "stat")
+ return lexical.indented + (state.lastType == "operator" || state.lastType == "," ? statementIndent || indentUnit : 0);
+ else if (lexical.info == "switch" && !closing && parserConfig.doubleIndentSwitch != false)
+ return lexical.indented + (/^(?:case|default)\b/.test(textAfter) ? indentUnit : 2 * indentUnit);
+ else if (lexical.align) return lexical.column + (closing ? 0 : 1);
+ else return lexical.indented + (closing ? 0 : indentUnit);
+ },
+
+ electricChars: ":{}",
+ blockCommentStart: jsonMode ? null : "/*",
+ blockCommentEnd: jsonMode ? null : "*/",
+ lineComment: jsonMode ? null : "//",
+ fold: "brace",
+
+ helperType: jsonMode ? "json" : "javascript",
+ jsonldMode: jsonldMode,
+ jsonMode: jsonMode
+ };
+});
+
+CodeMirror.registerHelper("wordChars", "javascript", /[\\w$]/);
+
+CodeMirror.defineMIME("text/javascript", "javascript");
+CodeMirror.defineMIME("text/ecmascript", "javascript");
+CodeMirror.defineMIME("application/javascript", "javascript");
+CodeMirror.defineMIME("application/ecmascript", "javascript");
+CodeMirror.defineMIME("application/json", {name: "javascript", json: true});
+CodeMirror.defineMIME("application/x-json", {name: "javascript", json: true});
+CodeMirror.defineMIME("application/ld+json", {name: "javascript", jsonld: true});
+CodeMirror.defineMIME("text/typescript", { name: "javascript", typescript: true });
+CodeMirror.defineMIME("application/typescript", { name: "javascript", typescript: true });
+
+});
Added: branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-modes/ntriples.js
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-modes/ntriples.js (rev 0)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-modes/ntriples.js 2014-05-09 11:55:20 UTC (rev 8245)
@@ -0,0 +1,183 @@
+/**********************************************************
+* This script provides syntax highlighting support for
+* the Ntriples format.
+* Ntriples format specification:
+* http://www.w3.org/TR/rdf-testcases/#ntriples
+***********************************************************/
+
+/*
+ The following expression defines the defined ASF grammar transitions.
+
+ pre_subject ->
+ {
+ ( writing_subject_uri | writing_bnode_uri )
+ -> pre_predicate
+ -> writing_predicate_uri
+ -> pre_object
+ -> writing_object_uri | writing_object_bnode |
+ (
+ writing_object_literal
+ -> writing_literal_lang | writing_literal_type
+ )
+ -> post_object
+ -> BEGIN
+ } otherwise {
+ -> ERROR
+ }
+*/
+
+(function(mod) {
+ if (typeof exports == "object" && typeof module == "object") // CommonJS
+ mod(require("../../lib/codemirror"));
+ else if (typeof define == "function" && define.amd) // AMD
+ define(["../../lib/codemirror"], mod);
+ else // Plain browser env
+ mod(CodeMirror);
+})(function(CodeMirror) {
+"use strict";
+
+CodeMirror.defineMode("ntriples", function() {
+
+ var Location = {
+ PRE_SUBJECT : 0,
+ WRITING_SUB_URI : 1,
+ WRITING_BNODE_URI : 2,
+ PRE_PRED : 3,
+ WRITING_PRED_URI : 4,
+ PRE_OBJ : 5,
+ WRITING_OBJ_URI : 6,
+ WRITING_OBJ_BNODE : 7,
+ WRITING_OBJ_LITERAL : 8,
+ WRITING_LIT_LANG : 9,
+ WRITING_LIT_TYPE : 10,
+ POST_OBJ : 11,
+ ERROR : 12
+ };
+ function transitState(currState, c) {
+ var currLocation = currState.location;
+ var ret;
+
+ // Opening.
+ if (currLocation == Location.PRE_SUBJECT && c == '<') ret = Location.WRITING_SUB_URI;
+ else if(currLocation == Location.PRE_SUBJECT && c == '_') ret = Location.WRITING_BNODE_URI;
+ else if(currLocation == Location.PRE_PRED && c == '<') ret = Location.WRITING_PRED_URI;
+ else if(currLocation == Location.PRE_OBJ && c == '<') ret = Location.WRITING_OBJ_URI;
+ else if(currLocation == Location.PRE_OBJ && c == '_') ret = Location.WRITING_OBJ_BNODE;
+ else if(currLocation == Location.PRE_OBJ && c == '"') ret = Location.WRITING_OBJ_LITERAL;
+
+ // Closing.
+ else if(currLocation == Location.WRITING_SUB_URI && c == '>') ret = Location.PRE_PRED;
+ else if(currLocation == Location.WRITING_BNODE_URI && c == ' ') ret = Location.PRE_PRED;
+ else if(currLocation == Location.WRITING_PRED_URI && c == '>') ret = Location.PRE_OBJ;
+ else if(currLocation == Location.WRITING_OBJ_URI && c == '>') ret = Location.POST_OBJ;
+ else if(currLocation == Location.WRITING_OBJ_BNODE && c == ' ') ret = Location.POST_OBJ;
+ else if(currLocation == Location.WRITING_OBJ_LITERAL && c == '"') ret = Location.POST_OBJ;
+ else if(currLocation == Location.WRITING_LIT_LANG && c == ' ') ret = Location.POST_OBJ;
+ else if(currLocation == Location.WRITING_LIT_TYPE && c == '>') ret = Location.POST_OBJ;
+
+ // Closing typed and language literal.
+ else if(currLocation == Location.WRITING_OBJ_LITERAL && c == '@') ret = Location.WRITING_LIT_LANG;
+ else if(currLocation == Location.WRITING_OBJ_LITERAL && c == '^') ret = Location.WRITING_LIT_TYPE;
+
+ // Spaces.
+ else if( c == ' ' &&
+ (
+ currLocation == Location.PRE_SUBJECT ||
+ currLocation == Location.PRE_PRED ||
+ currLocation == Location.PRE_OBJ ||
+ currLocation == Location.POST_OBJ
+ )
+ ) ret = currLocation;
+
+ // Reset.
+ else if(currLocation == Location.POST_OBJ && c == '.') ret = Location.PRE_SUBJECT;
+
+ // Error
+ else ret = Location.ERROR;
+
+ currState.location=ret;
+ }
+
+ return {
+ startState: function() {
+ return {
+ location : Location.PRE_SUBJECT,
+ uris : [],
+ anchors : [],
+ bnodes : [],
+ langs : [],
+ types : []
+ };
+ },
+ token: function(stream, state) {
+ var ch = stream.next();
+ if(ch == '<') {
+ transitState(state, ch);
+ var parsedURI = '';
+ stream.eatWhile( function(c) { if( c != '#' && c != '>' ) { parsedURI += c; return true; } return false;} );
+ state.uris.push(parsedURI);
+ if( stream.match('#', false) ) return 'variable';
+ stream.next();
+ transitState(state, '>');
+ return 'variable';
+ }
+ if(ch == '#') {
+ var parsedAnchor = '';
+ stream.eatWhile(function(c) { if(c != '>' && c != ' ') { parsedAnchor+= c; return true; } return false;});
+ state.anchors.push(parsedAnchor);
+ return 'variable-2';
+ }
+ if(ch == '>') {
+ transitState(state, '>');
+ return 'variable';
+ }
+ if(ch == '_') {
+ transitState(state, ch);
+ var parsedBNode = '';
+ stream.eatWhile(function(c) { if( c != ' ' ) { parsedBNode += c; return true; } return false;});
+ state.bnodes.push(parsedBNode);
+ stream.next();
+ transitState(state, ' ');
+ return 'builtin';
+ }
+ if(ch == '"') {
+ transitState(state, ch);
+ stream.eatWhile( function(c) { return c != '"'; } );
+ stream.next();
+ if( stream.peek() != '@' && stream.peek() != '^' ) {
+ transitState(state, '"');
+ }
+ return 'string';
+ }
+ if( ch == '@' ) {
+ transitState(state, '@');
+ var parsedLang = '';
+ stream.eatWhile(function(c) { if( c != ' ' ) { parsedLang += c; return true; } return false;});
+ state.langs.push(parsedLang);
+ stream.next();
+ transitState(state, ' ');
+ return 'string-2';
+ }
+ if( ch == '^' ) {
+ stream.next();
+ transitState(state, '^');
+ var parsedType = '';
+ stream.eatWhile(function(c) { if( c != '>' ) { parsedType += c; return true; } return false;} );
+ state.types.push(parsedType);
+ stream.next();
+ transitState(state, '>');
+ return 'variable';
+ }
+ if( ch == ' ' ) {
+ transitState(state, ch);
+ }
+ if( ch == '.' ) {
+ transitState(state, ch);
+ }
+ }
+ };
+});
+
+CodeMirror.defineMIME("text/n-triples", "ntriples");
+
+});
Added: branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-modes/turtle.js
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-modes/turtle.js (rev 0)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-modes/turtle.js 2014-05-09 11:55:20 UTC (rev 8245)
@@ -0,0 +1,157 @@
+(function(mod) {
+ if (typeof exports == "object" && typeof module == "object") // CommonJS
+ mod(require("../../lib/codemirror"));
+ else if (typeof define == "function" && define.amd) // AMD
+ define(["../../lib/codemirror"], mod);
+ else // Plain browser env
+ mod(CodeMirror);
+})(function(CodeMirror) {
+"use strict";
+
+CodeMirror.defineMode("turtle", function(config) {
+ var indentUnit = config.indentUnit;
+ var curPunc;
+
+ function wordRegexp(words) {
+ return new RegExp("^(?:" + words.join("|") + ")$", "i");
+ }
+ var ops = wordRegexp([]);
+ var keywords = wordRegexp(["@prefix", "@base", "a"]);
+ var operatorChars = /[*+\-<>=&|]/;
+
+ function tokenBase(stream, state) {
+ var ch = stream.next();
+ curPunc = null;
+ if (ch == "<" && !stream.match(/^[\s\u00a0=]/, false)) {
+ stream.match(/^[^\s\u00a0>]*>?/);
+ return "atom";
+ }
+ else if (ch == "\"" || ch == "'") {
+ state.tokenize = tokenLiteral(ch);
+ return state.tokenize(stream, state);
+ }
+ else if (/[{}\(\),\.;\[\]]/.test(ch)) {
+ curPunc = ch;
+ return null;
+ }
+ else if (ch == "#") {
+ stream.skipToEnd();
+ return "comment";
+ }
+ else if (operatorChars.test(ch)) {
+ stream.eatWhile(operatorChars);
+ return null;
+ }
+ else if (ch == ":") {
+ return "operator";
+ } else {
+ stream.eatWhile(/[_\w\d]/);
+ if(stream.peek() == ":") {
+ return "variable-3";
+ } else {
+ var word = stream.current();
+
+ if(keywords.test(word)) {
+ return "meta";
+ }
+
+ if(ch >= "A" && ch <= "Z") {
+ return "comment";
+ } else {
+ return "keyword";
+ }
+ }
+ var word = stream.current();
+ if (ops.test(word))
+ return null;
+ else if (keywords.test(word))
+ return "meta";
+ else
+ return "variable";
+ }
+ }
+
+ function tokenLiteral(quote) {
+ return function(stream, state) {
+ var escaped = false, ch;
+ while ((ch = stream.next()) != null) {
+ if (ch == quote && !escaped) {
+ state.tokenize = tokenBase;
+ break;
+ }
+ escaped = !escaped && ch == "\\";
+ }
+ return "string";
+ };
+ }
+
+ function pushContext(state, type, col) {
+ state.context = {prev: state.context, indent: state.indent, col: col, type: type};
+ }
+ function popContext(state) {
+ state.indent = state.context.indent;
+ state.context = state.context.prev;
+ }
+
+ return {
+ startState: function() {
+ return {tokenize: tokenBase,
+ context: null,
+ indent: 0,
+ col: 0};
+ },
+
+ token: function(stream, state) {
+ if (stream.sol()) {
+ if (state.context && state.context.align == null) state.context.align = false;
+ state.indent = stream.indentation();
+ }
+ if (stream.eatSpace()) return null;
+ var style = state.tokenize(stream, state);
+
+ if (style != "comment" && state.context && state.context.align == null && state.context.type != "pattern") {
+ state.context.align = true;
+ }
+
+ if (curPunc == "(") pushContext(state, ")", stream.column());
+ else if (curPunc == "[") pushContext(state, "]", stream.column());
+ else if (curPunc == "{") pushContext(state, "}", stream.column());
+ else if (/[\]\}\)]/.test(curPunc)) {
+ while (state.context && state.context.type == "pattern") popContext(state);
+ if (state.context && curPunc == state.context.type) popContext(state);
+ }
+ else if (curPunc == "." && state.context && state.context.type == "pattern") popContext(state);
+ else if (/atom|string|variable/.test(style) && state.context) {
+ if (/[\}\]]/.test(state.context.type))
+ pushContext(state, "pattern", stream.column());
+ else if (state.context.type == "pattern" && !state.context.align) {
+ state.context.align = true;
+ state.context.col = stream.column();
+ }
+ }
+
+ return style;
+ },
+
+ indent: function(state, textAfter) {
+ var firstChar = textAfter && textAfter.charAt(0);
+ var context = state.context;
+ if (/[\]\}]/.test(firstChar))
+ while (context && context.type == "pattern") context = context.prev;
+
+ var closing = context && firstChar == context.type;
+ if (!context)
+ return 0;
+ else if (context.type == "pattern")
+ return context.col;
+ else if (context.align)
+ return context.col + (closing ? 0 : 1);
+ else
+ return context.indent + (closing ? 0 : indentUnit);
+ }
+ };
+});
+
+CodeMirror.defineMIME("text/turtle", "turtle");
+
+});
Added: branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-modes/xml.js
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-modes/xml.js (rev 0)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-modes/xml.js 2014-05-09 11:55:20 UTC (rev 8245)
@@ -0,0 +1,381 @@
+(function(mod) {
+ if (typeof exports == "object" && typeof module == "object") // CommonJS
+ mod(require("../../lib/codemirror"));
+ else if (typeof define == "function" && define.amd) // AMD
+ define(["../../lib/codemirror"], mod);
+ else // Plain browser env
+ mod(CodeMirror);
+})(function(CodeMirror) {
+"use strict";
+
+CodeMirror.defineMode("xml", function(config, parserConfig) {
+ var indentUnit = config.indentUnit;
+ var multilineTagIndentFactor = parserConfig.multilineTagIndentFactor || 1;
+ var multilineTagIndentPastTag = parserConfig.multilineTagIndentPastTag;
+ if (multilineTagIndentPastTag == null) multilineTagIndentPastTag = true;
+
+ var Kludges = parserConfig.htmlMode ? {
+ autoSelfClosers: {'area': true, 'base': true, 'br': true, 'col': true, 'command': true,
+ 'embed': true, 'frame': true, 'hr': true, 'img': true, 'input': true,
+ 'keygen': true, 'link': true, 'meta': true, 'param': true, 'source': true,
+ 'track': true, 'wbr': true},
+ implicitlyClosed: {'dd': true, 'li': true, 'optgroup': true, 'option': true, 'p': true,
+ 'rp': true, 'rt': true, 'tbody': true, 'td': true, 'tfoot': true,
+ 'th': true, 'tr': true},
+ contextGrabbers: {
+ 'dd': {'dd': true, 'dt': true},
+ 'dt': {'dd': true, 'dt': true},
+ 'li': {'li': true},
+ 'option': {'option': true, 'optgroup': true},
+ 'optgroup': {'optgroup': true},
+ 'p': {'address': true, 'article': true, 'aside': true, 'blockquote': true, 'dir': true,
+ 'div': true, 'dl': true, 'fieldset': true, 'footer': true, 'form': true,
+ 'h1': true, 'h2': true, 'h3': true, 'h4': true, 'h5': true, 'h6': true,
+ 'header': true, 'hgroup': true, 'hr': true, 'menu': true, 'nav': true, 'ol': true,
+ 'p': true, 'pre': true, 'section': true, 'table': true, 'ul': true},
+ 'rp': {'rp': true, 'rt': true},
+ 'rt': {'rp': true, 'rt': true},
+ 'tbody': {'tbody': true, 'tfoot': true},
+ 'td': {'td': true, 'th': true},
+ 'tfoot': {'tbody': true},
+ 'th': {'td': true, 'th': true},
+ 'thead': {'tbody': true, 'tfoot': true},
+ 'tr': {'tr': true}
+ },
+ doNotIndent: {"pre": true},
+ allowUnquoted: true,
+ allowMissing: true,
+ caseFold: true
+ } : {
+ autoSelfClosers: {},
+ implicitlyClosed: {},
+ contextGrabbers: {},
+ doNotIndent: {},
+ allowUnquoted: false,
+ allowMissing: false,
+ caseFold: false
+ };
+ var alignCDATA = parserConfig.alignCDATA;
+
+ // Return variables for tokenizers
+ var type, setStyle;
+
+ function inText(stream, state) {
+ function chain(parser) {
+ state.tokenize = parser;
+ return parser(stream, state);
+ }
+
+ var ch = stream.next();
+ if (ch == "<") {
+ if (stream.eat("!")) {
+ if (stream.eat("[")) {
+ if (stream.match("CDATA[")) return chain(inBlock("atom", "]]>"));
+ else return null;
+ } else if (stream.match("--")) {
+ return chain(inBlock("comment", "-->"));
+ } else if (stream.match("DOCTYPE", true, true)) {
+ stream.eatWhile(/[\w\._\-]/);
+ return chain(doctype(1));
+ } else {
+ return null;
+ }
+ } else if (stream.eat("?")) {
+ stream.eatWhile(/[\w\._\-]/);
+ state.tokenize = inBlock("meta", "?>");
+ return "meta";
+ } else {
+ type = stream.eat("/") ? "closeTag" : "openTag";
+ state.tokenize = inTag;
+ return "tag bracket";
+ }
+ } else if (ch == "&") {
+ var ok;
+ if (stream.eat("#")) {
+ if (stream.eat("x")) {
+ ok = stream.eatWhile(/[a-fA-F\d]/) && stream.eat(";");
+ } else {
+ ok = stream.eatWhile(/[\d]/) && stream.eat(";");
+ }
+ } else {
+ ok = stream.eatWhile(/[\w\.\-:]/) && stream.eat(";");
+ }
+ return ok ? "atom" : "error";
+ } else {
+ stream.eatWhile(/[^&<]/);
+ return null;
+ }
+ }
+
+ function inTag(stream, state) {
+ var ch = stream.next();
+ if (ch == ">" || (ch == "/" && stream.eat(">"))) {
+ state.tokenize = inText;
+ type = ch == ">" ? "endTag" : "selfcloseTag";
+ return "tag bracket";
+ } else if (ch == "=") {
+ type = "equals";
+ return null;
+ } else if (ch == "<") {
+ state.tokenize = inText;
+ state.state = baseState;
+ state.tagName = state.tagStart = null;
+ var next = state.tokenize(stream, state);
+ return next ? next + " error" : "error";
+ } else if (/[\'\"]/.test(ch)) {
+ state.tokenize = inAttribute(ch);
+ state.stringStartCol = stream.column();
+ return state.tokenize(stream, state);
+ } else {
+ stream.match(/^[^\s\u00a0=<>\"\']*[^\s\u00a0=<>\"\'\/]/);
+ return "word";
+ }
+ }
+
+ function inAttribute(quote) {
+ var closure = function(stream, state) {
+ while (!stream.eol()) {
+ if (stream.next() == quote) {
+ state.tokenize = inTag;
+ break;
+ }
+ }
+ return "string";
+ };
+ closure.isInAttribute = true;
+ return closure;
+ }
+
+ function inBlock(style, terminator) {
+ return function(stream, state) {
+ while (!stream.eol()) {
+ if (stream.match(terminator)) {
+ state.tokenize = inText;
+ break;
+ }
+ stream.next();
+ }
+ return style;
+ };
+ }
+ function doctype(depth) {
+ return function(stream, state) {
+ var ch;
+ while ((ch = stream.next()) != null) {
+ if (ch == "<") {
+ state.tokenize = doctype(depth + 1);
+ return state.tokenize(stream, state);
+ } else if (ch == ">") {
+ if (depth == 1) {
+ state.tokenize = inText;
+ break;
+ } else {
+ state.tokenize = doctype(depth - 1);
+ return state.tokenize(stream, state);
+ }
+ }
+ }
+ return "meta";
+ };
+ }
+
+ function Context(state, tagName, startOfLine) {
+ this.prev = state.context;
+ this.tagName = tagName;
+ this.indent = state.indented;
+ this.startOfLine = startOfLine;
+ if (Kludges.doNotIndent.hasOwnProperty(tagName) || (state.context && state.context.noIndent))
+ this.noIndent = true;
+ }
+ function popContext(state) {
+ if (state.context) state.context = state.context.prev;
+ }
+ function maybePopContext(state, nextTagName) {
+ var parentTagName;
+ while (true) {
+ if (!state.context) {
+ return;
+ }
+ parentTagName = state.context.tagName;
+ if (!Kludges.contextGrabbers.hasOwnProperty(parentTagName) ||
+ !Kludges.contextGrabbers[parentTagName].hasOwnProperty(nextTagName)) {
+ return;
+ }
+ popContext(state);
+ }
+ }
+
+ function baseState(type, stream, state) {
+ if (type == "openTag") {
+ state.tagStart = stream.column();
+ return tagNameState;
+ } else if (type == "closeTag") {
+ return closeTagNameState;
+ } else {
+ return baseState;
+ }
+ }
+ function tagNameState(type, stream, state) {
+ if (type == "word") {
+ state.tagName = stream.current();
+ setStyle = "tag";
+ return attrState;
+ } else {
+ setStyle = "error";
+ return tagNameState;
+ }
+ }
+ function closeTagNameState(type, stream, state) {
+ if (type == "word") {
+ var tagName = stream.current();
+ if (state.context && state.context.tagName != tagName &&
+ Kludges.implicitlyClosed.hasOwnProperty(state.context.tagName))
+ popContext(state);
+ if (state.context && state.context.tagName == tagName) {
+ setStyle = "tag";
+ return closeState;
+ } else {
+ setStyle = "tag error";
+ return closeStateErr;
+ }
+ } else {
+ setStyle = "error";
+ return closeStateErr;
+ }
+ }
+
+ function closeState(type, _stream, state) {
+ if (type != "endTag") {
+ setStyle = "error";
+ return closeState;
+ }
+ popContext(state);
+ return baseState;
+ }
+ function closeStateErr(type, stream, state) {
+ setStyle = "error";
+ return closeState(type, stream, state);
+ }
+
+ function attrState(type, _stream, state) {
+ if (type == "word") {
+ setStyle = "attribute";
+ return attrEqState;
+ } else if (type == "endTag" || type == "selfcloseTag") {
+ var tagName = state.tagName, tagStart = state.tagStart;
+ state.tagName = state.tagStart = null;
+ if (type == "selfcloseTag" ||
+ Kludges.autoSelfClosers.hasOwnProperty(tagName)) {
+ maybePopContext(state, tagName);
+ } else {
+ maybePopContext(state, tagName);
+ state.context = new Context(state, tagName, tagStart == state.indented);
+ }
+ return baseState;
+ }
+ setStyle = "error";
+ return attrState;
+ }
+ function attrEqState(type, stream, state) {
+ if (type == "equals") return attrValueState;
+ if (!Kludges.allowMissing) setStyle = "error";
+ return attrState(type, stream, state);
+ }
+ function attrValueState(type, stream, state) {
+ if (type == "string") return attrContinuedState;
+ if (type == "word" && Kludges.allowUnquoted) {setStyle = "string"; return attrState;}
+ setStyle = "error";
+ return attrState(type, stream, state);
+ }
+ function attrContinuedState(type, stream, state) {
+ if (type == "string") return attrContinuedState;
+ return attrState(type, stream, state);
+ }
+
+ return {
+ startState: function() {
+ return {tokenize: inText,
+ state: baseState,
+ indented: 0,
+ tagName: null, tagStart: null,
+ context: null};
+ },
+
+ token: function(stream, state) {
+ if (!state.tagName && stream.sol())
+ state.indented = stream.indentation();
+
+ if (stream.eatSpace()) return null;
+ type = null;
+ var style = state.tokenize(stream, state);
+ if ((style || type) && style != "comment") {
+ setStyle = null;
+ state.state = state.state(type || style, stream, state);
+ if (setStyle)
+ style = setStyle == "error" ? style + " error" : setStyle;
+ }
+ return style;
+ },
+
+ indent: function(state, textAfter, fullLine) {
+ var context = state.context;
+ // Indent multi-line strings (e.g. css).
+ if (state.tokenize.isInAttribute) {
+ if (state.tagStart == state.indented)
+ return state.stringStartCol + 1;
+ else
+ return state.indented + indentUnit;
+ }
+ if (context && context.noIndent) return CodeMirror.Pass;
+ if (state.tokenize != inTag && state.tokenize != inText)
+ return fullLine ? fullLine.match(/^(\s*)/)[0].length : 0;
+ // Indent the starts of attribute names.
+ if (state.tagName) {
+ if (multilineTagIndentPastTag)
+ return state.tagStart + state.tagName.length + 2;
+ else
+ return state.tagStart + indentUnit * multilineTagIndentFactor;
+ }
+ if (alignCDATA && /<!\[CDATA\[/.test(textAfter)) return 0;
+ var tagAfter = textAfter && /^<(\/)?([\w_:\.-]*)/.exec(textAfter);
+ if (tagAfter && tagAfter[1]) { // Closing tag spotted
+ while (context) {
+ if (context.tagName == tagAfter[2]) {
+ context = context.prev;
+ break;
+ } else if (Kludges.implicitlyClosed.hasOwnProperty(context.tagName)) {
+ context = context.prev;
+ } else {
+ break;
+ }
+ }
+ } else if (tagAfter) { // Opening tag spotted
+ while (context) {
+ var grabbers = Kludges.contextGrabbers[context.tagName];
+ if (grabbers && grabbers.hasOwnProperty(tagAfter[2]))
+ context = context.prev;
+ else
+ break;
+ }
+ }
+ while (context && !context.startOfLine)
+ context = context.prev;
+ if (context) return context.indent + indentUnit;
+ else return 0;
+ },
+
+ electricInput: /<\/[\s\w:]+>$/,
+ blockCommentStart: "<!--",
+ blockCommentEnd: "-->",
+
+ configuration: parserConfig.htmlMode ? "html" : "xml",
+ helperType: parserConfig.htmlMode ? "html" : "xml"
+ };
+});
+
+CodeMirror.defineMIME("text/xml", "xml");
+CodeMirror.defineMIME("application/xml", "xml");
+if (!CodeMirror.mimeModes.hasOwnProperty("text/html"))
+ CodeMirror.defineMIME("text/html", {name: "xml", htmlMode: true});
+
+});
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/workbench.js
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/workbench.js 2014-05-09 11:33:52 UTC (rev 8244)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/workbench.js 2014-05-09 11:55:20 UTC (rev 8245)
@@ -349,16 +349,31 @@
if(type == 'rdf') {
$('#rdf-type').val(format);
}
- showUpdateOptions(type);
+ setUpdateSettings(type);
}
-$('#update-type').change(function() { showUpdateOptions(this.value) });
+$('#update-type').change(function() { setUpdateSettings(this.value); });
+$('#rdf-type').change(function() { setUpdateMode('rdf'); });
-function showUpdateOptions(type) {
+function setUpdateSettings(type) {
$('#rdf-type, label[for="rdf-type"]').attr('disabled', type != 'rdf');
$('#update-tab .advanced-features input').attr('disabled', type != 'sparql');
+ setUpdateMode(type);
}
+function setUpdateMode(type) {
+ var mode = '';
+ if(type == 'sparql') {
+ mode = 'sparql';
+ } else if(type == 'rdf') {
+ type = $('#rdf-type').val();
+ if(type in rdf_modes) {
+ mode = rdf_modes[type];
+ }
+ }
+ UPDATE_EDITOR.setOption('mode', mode);
+}
+
// .xml is used for both RDF and TriX, assume it's RDF
// We could check the parent element to see which it is
var rdf_types = {'nq': 'n-quads',
@@ -383,6 +398,9 @@
'trix': 'application/trix',
'turtle': 'application/x-turtle'};
+// key is value of RDF type selector, value is name of CodeMirror mode
+var rdf_modes = {'n-triples': 'ntriples', 'rdf/xml': 'xml', 'json': 'json', 'turtle': 'turtle'};
+
var sparql_update_commands = ['INSERT', 'DELETE', 'LOAD', 'CLEAR'];
$('#update-file').change(handleFile);
@@ -395,7 +413,7 @@
$('#update-update').click(submitUpdate);
-UPDATE_EDITOR = CodeMirror.fromTextArea($('#update-box')[0], {lineNumbers: true});
+UPDATE_EDITOR = CodeMirror.fromTextArea($('#update-box')[0], {lineNumbers: true, mode: 'sparql'});
function submitUpdate(e) {
// Updates are submitted as a regular form for SPARQL updates in monitor mode, and via AJAX for non-monitor SPARQL, RDF & file path updates.
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tob...@us...> - 2014-05-09 11:33:55
|
Revision: 8244
http://sourceforge.net/p/bigdata/code/8244
Author: tobycraig
Date: 2014-05-09 11:33:52 +0000 (Fri, 09 May 2014)
Log Message:
-----------
Added SPARQL syntax highlighting for query panel
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/index.html
branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/workbench.js
Added Paths:
-----------
branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-modes/
branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-modes/sparql.js
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/index.html
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/index.html 2014-05-09 11:14:35 UTC (rev 8243)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/index.html 2014-05-09 11:33:52 UTC (rev 8244)
@@ -221,6 +221,7 @@
<script src="/bigdata/html/js/vendor/jquery.hotkeys.js"></script>
<script src="/bigdata/html/js/vendor/codemirror.js"></script>
<script src="/bigdata/html/js/vendor/cm-addons/placeholder.js"></script>
+ <script src="/bigdata/html/js/vendor/cm-modes/sparql.js"></script>
<script src="/bigdata/html/js/workbench.js"></script>
</body>
</html>
Added: branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-modes/sparql.js
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-modes/sparql.js (rev 0)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-modes/sparql.js 2014-05-09 11:33:52 UTC (rev 8244)
@@ -0,0 +1,157 @@
+(function(mod) {
+ if (typeof exports == "object" && typeof module == "object") // CommonJS
+ mod(require("../../lib/codemirror"));
+ else if (typeof define == "function" && define.amd) // AMD
+ define(["../../lib/codemirror"], mod);
+ else // Plain browser env
+ mod(CodeMirror);
+})(function(CodeMirror) {
+"use strict";
+
+CodeMirror.defineMode("sparql", function(config) {
+ var indentUnit = config.indentUnit;
+ var curPunc;
+
+ function wordRegexp(words) {
+ return new RegExp("^(?:" + words.join("|") + ")$", "i");
+ }
+ var ops = wordRegexp(["str", "lang", "langmatches", "datatype", "bound", "sameterm", "isiri", "isuri",
+ "isblank", "isliteral", "a"]);
+ var keywords = wordRegexp(["base", "prefix", "select", "distinct", "reduced", "construct", "describe",
+ "ask", "from", "named", "where", "order", "limit", "offset", "filter", "optional",
+ "graph", "by", "asc", "desc", "as", "having", "undef", "values", "group",
+ "minus", "in", "not", "service", "silent", "using", "insert", "delete", "union",
+ "data", "copy", "to", "move", "add", "create", "drop", "clear", "load"]);
+ var operatorChars = /[*+\-<>=&|]/;
+
+ function tokenBase(stream, state) {
+ var ch = stream.next();
+ curPunc = null;
+ if (ch == "$" || ch == "?") {
+ stream.match(/^[\w\d]*/);
+ return "variable-2";
+ }
+ else if (ch == "<" && !stream.match(/^[\s\u00a0=]/, false)) {
+ stream.match(/^[^\s\u00a0>]*>?/);
+ return "atom";
+ }
+ else if (ch == "\"" || ch == "'") {
+ state.tokenize = tokenLiteral(ch);
+ return state.tokenize(stream, state);
+ }
+ else if (/[{}\(\),\.;\[\]]/.test(ch)) {
+ curPunc = ch;
+ return null;
+ }
+ else if (ch == "#") {
+ stream.skipToEnd();
+ return "comment";
+ }
+ else if (operatorChars.test(ch)) {
+ stream.eatWhile(operatorChars);
+ return null;
+ }
+ else if (ch == ":") {
+ stream.eatWhile(/[\w\d\._\-]/);
+ return "atom";
+ }
+ else {
+ stream.eatWhile(/[_\w\d]/);
+ if (stream.eat(":")) {
+ stream.eatWhile(/[\w\d_\-]/);
+ return "atom";
+ }
+ var word = stream.current();
+ if (ops.test(word))
+ return null;
+ else if (keywords.test(word))
+ return "keyword";
+ else
+ return "variable";
+ }
+ }
+
+ function tokenLiteral(quote) {
+ return function(stream, state) {
+ var escaped = false, ch;
+ while ((ch = stream.next()) != null) {
+ if (ch == quote && !escaped) {
+ state.tokenize = tokenBase;
+ break;
+ }
+ escaped = !escaped && ch == "\\";
+ }
+ return "string";
+ };
+ }
+
+ function pushContext(state, type, col) {
+ state.context = {prev: state.context, indent: state.indent, col: col, type: type};
+ }
+ function popContext(state) {
+ state.indent = state.context.indent;
+ state.context = state.context.prev;
+ }
+
+ return {
+ startState: function() {
+ return {tokenize: tokenBase,
+ context: null,
+ indent: 0,
+ col: 0};
+ },
+
+ token: function(stream, state) {
+ if (stream.sol()) {
+ if (state.context && state.context.align == null) state.context.align = false;
+ state.indent = stream.indentation();
+ }
+ if (stream.eatSpace()) return null;
+ var style = state.tokenize(stream, state);
+
+ if (style != "comment" && state.context && state.context.align == null && state.context.type != "pattern") {
+ state.context.align = true;
+ }
+
+ if (curPunc == "(") pushContext(state, ")", stream.column());
+ else if (curPunc == "[") pushContext(state, "]", stream.column());
+ else if (curPunc == "{") pushContext(state, "}", stream.column());
+ else if (/[\]\}\)]/.test(curPunc)) {
+ while (state.context && state.context.type == "pattern") popContext(state);
+ if (state.context && curPunc == state.context.type) popContext(state);
+ }
+ else if (curPunc == "." && state.context && state.context.type == "pattern") popContext(state);
+ else if (/atom|string|variable/.test(style) && state.context) {
+ if (/[\}\]]/.test(state.context.type))
+ pushContext(state, "pattern", stream.column());
+ else if (state.context.type == "pattern" && !state.context.align) {
+ state.context.align = true;
+ state.context.col = stream.column();
+ }
+ }
+
+ return style;
+ },
+
+ indent: function(state, textAfter) {
+ var firstChar = textAfter && textAfter.charAt(0);
+ var context = state.context;
+ if (/[\]\}]/.test(firstChar))
+ while (context && context.type == "pattern") context = context.prev;
+
+ var closing = context && firstChar == context.type;
+ if (!context)
+ return 0;
+ else if (context.type == "pattern")
+ return context.col;
+ else if (context.align)
+ return context.col + (closing ? 0 : 1);
+ else
+ return context.indent + (closing ? 0 : indentUnit);
+ }
+ };
+});
+
+CodeMirror.defineMIME("application/x-sparql-query", "sparql");
+
+});
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/workbench.js
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/workbench.js 2014-05-09 11:14:35 UTC (rev 8243)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/workbench.js 2014-05-09 11:33:52 UTC (rev 8244)
@@ -510,7 +510,7 @@
}
});
-QUERY_EDITOR = CodeMirror.fromTextArea($('#query-box')[0], {lineNumbers: true});
+QUERY_EDITOR = CodeMirror.fromTextArea($('#query-box')[0], {lineNumbers: true, mode: 'sparql'});
function submitQuery(e) {
e.preventDefault();
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tob...@us...> - 2014-05-09 11:14:38
|
Revision: 8243
http://sourceforge.net/p/bigdata/code/8243
Author: tobycraig
Date: 2014-05-09 11:14:35 +0000 (Fri, 09 May 2014)
Log Message:
-----------
Fixed incorrect CSS selector
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/css/style.css
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/css/style.css
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/css/style.css 2014-05-09 00:11:10 UTC (rev 8242)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/css/style.css 2014-05-09 11:14:35 UTC (rev 8243)
@@ -209,7 +209,7 @@
color: #ededed;
}
-#query-form, #load-box-container {
+#query-form, #update-box-container {
clear: both;
}
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <jer...@us...> - 2014-05-09 00:11:14
|
Revision: 8242
http://sourceforge.net/p/bigdata/code/8242
Author: jeremy_carroll
Date: 2014-05-09 00:11:10 +0000 (Fri, 09 May 2014)
Log Message:
-----------
organized imports
Modified Paths:
--------------
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestKeyBuilder.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestLanguageRange.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestPrefixSearch.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestSearch.java
branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestSearchRestartSafe.java
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestKeyBuilder.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestKeyBuilder.java 2014-05-08 23:50:43 UTC (rev 8241)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestKeyBuilder.java 2014-05-09 00:11:10 UTC (rev 8242)
@@ -33,13 +33,9 @@
import com.bigdata.btree.BytesUtil;
import com.bigdata.btree.ITupleSerializer;
import com.bigdata.btree.IndexMetadata;
-import com.bigdata.btree.keys.DefaultKeyBuilderFactory;
import com.bigdata.btree.keys.IKeyBuilder;
import com.bigdata.btree.keys.KeyBuilder;
import com.bigdata.btree.keys.StrengthEnum;
-import com.bigdata.journal.IIndexManager;
-import com.bigdata.journal.ITx;
-import com.bigdata.journal.ProxyTestCase;
import com.bigdata.search.FullTextIndex.Options;
/**
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestLanguageRange.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestLanguageRange.java 2014-05-08 23:50:43 UTC (rev 8241)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestLanguageRange.java 2014-05-09 00:11:10 UTC (rev 8242)
@@ -26,10 +26,10 @@
*/
package com.bigdata.search;
+import junit.framework.TestCase2;
+
import com.bigdata.search.ConfigurableAnalyzerFactory.LanguageRange;
-import junit.framework.TestCase2;
-
public class TestLanguageRange extends TestCase2 {
public TestLanguageRange() {
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestPrefixSearch.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestPrefixSearch.java 2014-05-08 23:50:43 UTC (rev 8241)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestPrefixSearch.java 2014-05-09 00:11:10 UTC (rev 8242)
@@ -29,12 +29,8 @@
package com.bigdata.search;
import java.io.StringReader;
-import java.util.Properties;
import java.util.concurrent.TimeUnit;
-import com.bigdata.journal.IIndexManager;
-import com.bigdata.journal.ITx;
-import com.bigdata.journal.ProxyTestCase;
import com.bigdata.rdf.lexicon.ITextIndexer.FullTextQuery;
/**
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestSearch.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestSearch.java 2014-05-08 23:50:43 UTC (rev 8241)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestSearch.java 2014-05-09 00:11:10 UTC (rev 8242)
@@ -33,9 +33,6 @@
import java.util.Properties;
import java.util.concurrent.TimeUnit;
-import com.bigdata.journal.IIndexManager;
-import com.bigdata.journal.ITx;
-import com.bigdata.journal.ProxyTestCase;
import com.bigdata.rdf.lexicon.ITextIndexer.FullTextQuery;
/**
Modified: branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestSearchRestartSafe.java
===================================================================
--- branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestSearchRestartSafe.java 2014-05-08 23:50:43 UTC (rev 8241)
+++ branches/TEXT_ANALYZERS/bigdata/src/test/com/bigdata/search/TestSearchRestartSafe.java 2014-05-09 00:11:10 UTC (rev 8242)
@@ -29,12 +29,9 @@
package com.bigdata.search;
import java.io.StringReader;
-import java.util.Properties;
import java.util.concurrent.TimeUnit;
-import com.bigdata.journal.IIndexManager;
import com.bigdata.journal.ITx;
-import com.bigdata.journal.ProxyTestCase;
import com.bigdata.rdf.lexicon.ITextIndexer.FullTextQuery;
/**
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <jer...@us...> - 2014-05-08 23:50:46
|
Revision: 8241
http://sourceforge.net/p/bigdata/code/8241
Author: jeremy_carroll
Date: 2014-05-08 23:50:43 +0000 (Thu, 08 May 2014)
Log Message:
-----------
New branch for work on using different Lucene Analyzers for bds:search
Added Paths:
-----------
branches/TEXT_ANALYZERS/
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <tob...@us...> - 2014-05-08 23:17:43
|
Revision: 8240
http://sourceforge.net/p/bigdata/code/8240
Author: tobycraig
Date: 2014-05-08 23:17:39 +0000 (Thu, 08 May 2014)
Log Message:
-----------
Added basic CodeMirror editor
Modified Paths:
--------------
branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/css/style.css
branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/index.html
branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/workbench.js
Added Paths:
-----------
branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/css/vendor/
branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/css/vendor/codemirror.css
branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-addons/
branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-addons/placeholder.js
branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/codemirror.js
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/css/style.css
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/css/style.css 2014-05-08 23:17:07 UTC (rev 8239)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/css/style.css 2014-05-08 23:17:39 UTC (rev 8240)
@@ -209,6 +209,10 @@
color: #ededed;
}
+#query-form, #load-box-container {
+ clear: both;
+}
+
#large-file-message {
display: none;
margin: 5px 0;
@@ -221,6 +225,15 @@
box-sizing: border-box;
}
+.CodeMirror {
+ margin: 5px 0;
+ border: 1px solid #e1e1e1;
+}
+
+.CodeMirror-placeholder {
+ font-style: italic;
+}
+
hr {
background: #929292;
border: none;
@@ -271,36 +284,6 @@
float: right;
}
-#update-box, #query-box {
- background-color: transparent;
- padding: 2px;
- border-width: 1px;
- border-color: #e1e1e1;
-}
-
-/* these should have the same typography so the error highlighting matches up with the query text */
-#update-box, #update-errors, #query-box, #query-errors {
- font-family: sans-serif;
- font-size: 90%;
- line-height: normal;
-}
-
-#update-errors, #query-errors {
- position: absolute;
- z-index: -1;
- padding: 8px 3px;
- color: transparent;
- white-space: pre;
-}
-
-.error-line {
- background-color: lightgreen;
-}
-
-.error-character {
- background-color: red;
-}
-
#running-queries li {
margin: 10px 0;
}
Added: branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/css/vendor/codemirror.css
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/css/vendor/codemirror.css (rev 0)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/css/vendor/codemirror.css 2014-05-08 23:17:39 UTC (rev 8240)
@@ -0,0 +1,272 @@
+/* BASICS */
+
+.CodeMirror {
+ /* Set height, width, borders, and global font properties here */
+ font-family: monospace;
+ height: 300px;
+}
+.CodeMirror-scroll {
+ /* Set scrolling behaviour here */
+ overflow: auto;
+}
+
+/* PADDING */
+
+.CodeMirror-lines {
+ padding: 4px 0; /* Vertical padding around content */
+}
+.CodeMirror pre {
+ padding: 0 4px; /* Horizontal padding of content */
+}
+
+.CodeMirror-scrollbar-filler, .CodeMirror-gutter-filler {
+ background-color: white; /* The little square between H and V scrollbars */
+}
+
+/* GUTTER */
+
+.CodeMirror-gutters {
+ border-right: 1px solid #ddd;
+ background-color: #f7f7f7;
+ white-space: nowrap;
+}
+.CodeMirror-linenumbers {}
+.CodeMirror-linenumber {
+ padding: 0 3px 0 5px;
+ min-width: 20px;
+ text-align: right;
+ color: #999;
+ -moz-box-sizing: content-box;
+ box-sizing: content-box;
+}
+
+/* CURSOR */
+
+.CodeMirror div.CodeMirror-cursor {
+ border-left: 1px solid black;
+}
+/* Shown when moving in bi-directional text */
+.CodeMirror div.CodeMirror-secondarycursor {
+ border-left: 1px solid silver;
+}
+.CodeMirror.cm-keymap-fat-cursor div.CodeMirror-cursor {
+ width: auto;
+ border: 0;
+ background: #7e7;
+}
+/* Can style cursor different in overwrite (non-insert) mode */
+div.CodeMirror-overwrite div.CodeMirror-cursor {}
+
+.cm-tab { display: inline-block; }
+
+.CodeMirror-ruler {
+ border-left: 1px solid #ccc;
+ position: absolute;
+}
+
+/* DEFAULT THEME */
+
+.cm-s-default .cm-keyword {color: #708;}
+.cm-s-default .cm-atom {color: #219;}
+.cm-s-default .cm-number {color: #164;}
+.cm-s-default .cm-def {color: #00f;}
+.cm-s-default .cm-variable,
+.cm-s-default .cm-punctuation,
+.cm-s-default .cm-property,
+.cm-s-default .cm-operator {}
+.cm-s-default .cm-variable-2 {color: #05a;}
+.cm-s-default .cm-variable-3 {color: #085;}
+.cm-s-default .cm-comment {color: #a50;}
+.cm-s-default .cm-string {color: #a11;}
+.cm-s-default .cm-string-2 {color: #f50;}
+.cm-s-default .cm-meta {color: #555;}
+.cm-s-default .cm-qualifier {color: #555;}
+.cm-s-default .cm-builtin {color: #30a;}
+.cm-s-default .cm-bracket {color: #997;}
+.cm-s-default .cm-tag {color: #170;}
+.cm-s-default .cm-attribute {color: #00c;}
+.cm-s-default .cm-header {color: blue;}
+.cm-s-default .cm-quote {color: #090;}
+.cm-s-default .cm-hr {color: #999;}
+.cm-s-default .cm-link {color: #00c;}
+
+.cm-negative {color: #d44;}
+.cm-positive {color: #292;}
+.cm-header, .cm-strong {font-weight: bold;}
+.cm-em {font-style: italic;}
+.cm-link {text-decoration: underline;}
+
+.cm-s-default .cm-error {color: #f00;}
+.cm-invalidchar {color: #f00;}
+
+div.CodeMirror span.CodeMirror-matchingbracket {color: #0f0;}
+div.CodeMirror span.CodeMirror-nonmatchingbracket {color: #f22;}
+.CodeMirror-activeline-background {background: #e8f2ff;}
+
+/* STOP */
+
+/* The rest of this file contains styles related to the mechanics of
+ the editor. You probably shouldn't touch them. */
+
+.CodeMirror {
+ line-height: 1;
+ position: relative;
+ overflow: hidden;
+ background: white;
+ color: black;
+}
+
+.CodeMirror-scroll {
+ /* 30px is the magic margin used to hide the element's real scrollbars */
+ /* See overflow: hidden in .CodeMirror */
+ margin-bottom: -30px; margin-right: -30px;
+ padding-bottom: 30px;
+ height: 100%;
+ outline: none; /* Prevent dragging from highlighting the element */
+ position: relative;
+ -moz-box-sizing: content-box;
+ box-sizing: content-box;
+}
+.CodeMirror-sizer {
+ position: relative;
+ border-right: 30px solid transparent;
+ -moz-box-sizing: content-box;
+ box-sizing: content-box;
+}
+
+/* The fake, visible scrollbars. Used to force redraw during scrolling
+ before actuall scrolling happens, thus preventing shaking and
+ flickering artifacts. */
+.CodeMirror-vscrollbar, .CodeMirror-hscrollbar, .CodeMirror-scrollbar-filler, .CodeMirror-gutter-filler {
+ position: absolute;
+ z-index: 6;
+ display: none;
+}
+.CodeMirror-vscrollbar {
+ right: 0; top: 0;
+ overflow-x: hidden;
+ overflow-y: scroll;
+}
+.CodeMirror-hscrollbar {
+ bottom: 0; left: 0;
+ overflow-y: hidden;
+ overflow-x: scroll;
+}
+.CodeMirror-scrollbar-filler {
+ right: 0; bottom: 0;
+}
+.CodeMirror-gutter-filler {
+ left: 0; bottom: 0;
+}
+
+.CodeMirror-gutters {
+ position: absolute; left: 0; top: 0;
+ padding-bottom: 30px;
+ z-index: 3;
+}
+.CodeMirror-gutter {
+ white-space: normal;
+ height: 100%;
+ -moz-box-sizing: content-box;
+ box-sizing: content-box;
+ padding-bottom: 30px;
+ margin-bottom: -32px;
+ display: inline-block;
+ /* Hack to make IE7 behave */
+ *zoom:1;
+ *display:inline;
+}
+.CodeMirror-gutter-elt {
+ position: absolute;
+ cursor: default;
+ z-index: 4;
+}
+
+.CodeMirror-lines {
+ cursor: text;
+}
+.CodeMirror pre {
+ /* Reset some styles that the rest of the page might have set */
+ -moz-border-radius: 0; -webkit-border-radius: 0; border-radius: 0;
+ border-width: 0;
+ background: transparent;
+ font-family: inherit;
+ font-size: inherit;
+ margin: 0;
+ white-space: pre;
+ word-wrap: normal;
+ line-height: inherit;
+ color: inherit;
+ z-index: 2;
+ position: relative;
+ overflow: visible;
+}
+.CodeMirror-wrap pre {
+ word-wrap: break-word;
+ white-space: pre-wrap;
+ word-break: normal;
+}
+
+.CodeMirror-linebackground {
+ position: absolute;
+ left: 0; right: 0; top: 0; bottom: 0;
+ z-index: 0;
+}
+
+.CodeMirror-linewidget {
+ position: relative;
+ z-index: 2;
+ overflow: auto;
+}
+
+.CodeMirror-widget {}
+
+.CodeMirror-wrap .CodeMirror-scroll {
+ overflow-x: hidden;
+}
+
+.CodeMirror-measure {
+ position: absolute;
+ width: 100%;
+ height: 0;
+ overflow: hidden;
+ visibility: hidden;
+}
+.CodeMirror-measure pre { position: static; }
+
+.CodeMirror div.CodeMirror-cursor {
+ position: absolute;
+ border-right: none;
+ width: 0;
+}
+
+div.CodeMirror-cursors {
+ visibility: hidden;
+ position: relative;
+ z-index: 1;
+}
+.CodeMirror-focused div.CodeMirror-cursors {
+ visibility: visible;
+}
+
+.CodeMirror-selected { background: #d9d9d9; }
+.CodeMirror-focused .CodeMirror-selected { background: #d7d4f0; }
+.CodeMirror-crosshair { cursor: crosshair; }
+
+.cm-searching {
+ background: #ffa;
+ background: rgba(255, 255, 0, .4);
+}
+
+/* IE7 hack to prevent it from returning funny offsetTops on the spans */
+.CodeMirror span { *vertical-align: text-bottom; }
+
+/* Used to force a border model for a node */
+.cm-force-border { padding-right: .1px; }
+
+@media print {
+ /* Hide the cursor when printing */
+ .CodeMirror div.CodeMirror-cursors {
+ visibility: hidden;
+ }
+}
Property changes on: branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/css/vendor/codemirror.css
___________________________________________________________________
Added: svn:executable
## -0,0 +1 ##
+*
\ No newline at end of property
Modified: branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/index.html
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/index.html 2014-05-08 23:17:07 UTC (rev 8239)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/index.html 2014-05-08 23:17:39 UTC (rev 8240)
@@ -7,6 +7,7 @@
<!-- meta charset="utf-8" -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" >
<title>Bigdata Workbench</title>
+ <link rel="stylesheet" href="/bigdata/html/css/vendor/codemirror.css">
<link rel="stylesheet" href="/bigdata/html/css/style.css">
<!-- junit test marker: index.html -->
</head>
@@ -36,8 +37,7 @@
<div class="namespace-shortcuts">
</div>
- <div>
- <div id="update-errors"></div>
+ <div id="update-box-container">
<textarea id="update-box" placeholder="(Type in or drag a file containing RDF data, a SPARQL update or a file path or URL)"></textarea>
</div>
<p id="large-file-message">Your file <span id="filename"></span> is too large to display here, but will be uploaded as normal. <a href="#" id="clear-file">Remove file</a></p>
@@ -89,7 +89,6 @@
</div>
<form id="query-form">
- <div id="query-errors"></div>
<textarea id="query-box" name="query" placeholder="(Input a SPARQL query)"></textarea>
<a href="#" class="advanced-features-toggle">Advanced features</a>
@@ -220,6 +219,8 @@
<script src="//ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script>window.jQuery || document.write('<script src="/bigdata/html/js/vendor/jquery.min.js"><\/script>')</script>
<script src="/bigdata/html/js/vendor/jquery.hotkeys.js"></script>
+ <script src="/bigdata/html/js/vendor/codemirror.js"></script>
+ <script src="/bigdata/html/js/vendor/cm-addons/placeholder.js"></script>
<script src="/bigdata/html/js/workbench.js"></script>
</body>
</html>
Added: branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-addons/placeholder.js
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-addons/placeholder.js (rev 0)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-addons/placeholder.js 2014-05-08 23:17:39 UTC (rev 8240)
@@ -0,0 +1,55 @@
+(function(mod) {
+ if (typeof exports == "object" && typeof module == "object") // CommonJS
+ mod(require("../../lib/codemirror"));
+ else if (typeof define == "function" && define.amd) // AMD
+ define(["../../lib/codemirror"], mod);
+ else // Plain browser env
+ mod(CodeMirror);
+})(function(CodeMirror) {
+ CodeMirror.defineOption("placeholder", "", function(cm, val, old) {
+ var prev = old && old != CodeMirror.Init;
+ if (val && !prev) {
+ cm.on("blur", onBlur);
+ cm.on("change", onChange);
+ onChange(cm);
+ } else if (!val && prev) {
+ cm.off("blur", onBlur);
+ cm.off("change", onChange);
+ clearPlaceholder(cm);
+ var wrapper = cm.getWrapperElement();
+ wrapper.className = wrapper.className.replace(" CodeMirror-empty", "");
+ }
+
+ if (val && !cm.hasFocus()) onBlur(cm);
+ });
+
+ function clearPlaceholder(cm) {
+ if (cm.state.placeholder) {
+ cm.state.placeholder.parentNode.removeChild(cm.state.placeholder);
+ cm.state.placeholder = null;
+ }
+ }
+ function setPlaceholder(cm) {
+ clearPlaceholder(cm);
+ var elt = cm.state.placeholder = document.createElement("pre");
+ elt.style.cssText = "height: 0; overflow: visible";
+ elt.className = "CodeMirror-placeholder";
+ elt.appendChild(document.createTextNode(cm.getOption("placeholder")));
+ cm.display.lineSpace.insertBefore(elt, cm.display.lineSpace.firstChild);
+ }
+
+ function onBlur(cm) {
+ if (isEmpty(cm)) setPlaceholder(cm);
+ }
+ function onChange(cm) {
+ var wrapper = cm.getWrapperElement(), empty = isEmpty(cm);
+ wrapper.className = wrapper.className.replace(" CodeMirror-empty", "") + (empty ? " CodeMirror-empty" : "");
+
+ if (empty) setPlaceholder(cm);
+ else clearPlaceholder(cm);
+ }
+
+ function isEmpty(cm) {
+ return (cm.lineCount() === 1) && (cm.getLine(0) === "");
+ }
+});
Property changes on: branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/cm-addons/placeholder.js
___________________________________________________________________
Added: svn:executable
## -0,0 +1 ##
+*
\ No newline at end of property
Added: branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/codemirror.js
===================================================================
--- branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/codemirror.js (rev 0)
+++ branches/BIGDATA_RELEASE_1_3_0/bigdata-war/src/html/js/vendor/codemirror.js 2014-05-08 23:17:39 UTC (rev 8240)
@@ -0,0 +1,7526 @@
+// This is CodeMirror (http://codemirror.net), a code editor
+// implemented in JavaScript on top of the browser's DOM.
+//
+// You can find some technical background for some of the code below
+// at http://marijnhaverbeke.nl/blog/#cm-internals .
+
+(function(mod) {
+ if (typeof exports == "object" && typeof module == "object") // CommonJS
+ module.exports = mod();
+ else if (typeof define == "function" && define.amd) // AMD
+ return define([], mod);
+ else // Plain browser env
+ this.CodeMirror = mod();
+})(function() {
+ "use strict";
+
+ // BROWSER SNIFFING
+
+ // Kludges for bugs and behavior differences that can't be feature
+ // detected are enabled based on userAgent etc sniffing.
+
+ var gecko = /gecko\/\d/i.test(navigator.userAgent);
+ // ie_uptoN means Internet Explorer version N or lower
+ var ie_upto10 = /MSIE \d/.test(navigator.userAgent);
+ var ie_upto7 = ie_upto10 && (document.documentMode == null || document.documentMode < 8);
+ var ie_upto8 = ie_upto10 && (document.documentMode == null || document.documentMode < 9);
+ var ie_upto9 = ie_upto10 && (document.documentMode == null || document.documentMode < 10);
+ var ie_11up = /Trident\/([7-9]|\d{2,})\./.test(navigator.userAgent);
+ var ie = ie_upto10 || ie_11up;
+ var webkit = /WebKit\//.test(navigator.userAgent);
+ var qtwebkit = webkit && /Qt\/\d+\.\d+/.test(navigator.userAgent);
+ var chrome = /Chrome\//.test(navigator.userAgent);
+ var presto = /Opera\//.test(navigator.userAgent);
+ var safari = /Apple Computer/.test(navigator.vendor);
+ var khtml = /KHTML\//.test(navigator.userAgent);
+ var mac_geMountainLion = /Mac OS X 1\d\D([8-9]|\d\d)\D/.test(navigator.userAgent);
+ var phantom = /PhantomJS/.test(navigator.userAgent);
+
+ var ios = /AppleWebKit/.test(navigator.userAgent) && /Mobile\/\w+/.test(navigator.userAgent);
+ // This is woefully incomplete. Suggestions for alternative methods welcome.
+ var mobile = ios || /Android|webOS|BlackBerry|Opera Mini|Opera Mobi|IEMobile/i.test(navigator.userAgent);
+ var mac = ios || /Mac/.test(navigator.platform);
+ var windows = /win/i.test(navigator.platform);
+
+ var presto_version = presto && navigator.userAgent.match(/Version\/(\d*\.\d*)/);
+ if (presto_version) presto_version = Number(presto_version[1]);
+ if (presto_version && presto_version >= 15) { presto = false; webkit = true; }
+ // Some browsers use the wrong event properties to signal cmd/ctrl on OS X
+ var flipCtrlCmd = mac && (qtwebkit || presto && (presto_version == null || presto_version < 12.11));
+ var captureRightClick = gecko || (ie && !ie_upto8);
+
+ // Optimize some code when these features are not used.
+ var sawReadOnlySpans = false, sawCollapsedSpans = false;
+
+ // EDITOR CONSTRUCTOR
+
+ // A CodeMirror instance represents an editor. This is the object
+ // that user code is usually dealing with.
+
+ function CodeMirror(place, options) {
+ if (!(this instanceof CodeMirror)) return new CodeMirror(place, options);
+
+ this.options = options = options || {};
+ // Determine effective options based on given values and defaults.
+ copyObj(defaults, options, false);
+ setGuttersForLineNumbers(options);
+
+ var doc = options.value;
+ if (typeof doc == "string") doc = new Doc(doc, options.mode);
+ this.doc = doc;
+
+ var display = this.display = new Display(place, doc);
+ display.wrapper.CodeMirror = this;
+ updateGutters(this);
+ themeChanged(this);
+ if (options.lineWrapping)
+ this.display.wrapper.className += " CodeMirror-wrap";
+ if (options.autofocus && !mobile) focusInput(this);
+
+ this.state = {
+ keyMaps: [], // stores maps added by addKeyMap
+ overlays: [], // highlighting overlays, as added by addOverlay
+ modeGen: 0, // bumped when mode/overlay changes, used to invalidate highlighting info
+ overwrite: false, focused: false,
+ suppressEdits: false, // used to disable editing during key handlers when in readOnly mode
+ pasteIncoming: false, cutIncoming: false, // help recognize paste/cut edits in readInput
+ draggingText: false,
+ highlight: new Delayed() // stores highlight worker timeout
+ };
+
+ // Override magic textarea content restore that IE sometimes does
+ // on our hidden textarea on reload
+ if (ie_upto10) setTimeout(bind(resetInput, this, true), 20);
+
+ registerEventHandlers(this);
+
+ var cm = this;
+ runInOp(this, function() {
+ cm.curOp.forceUpdate = true;
+ attachDoc(cm, doc);
+
+ if ((options.autofocus && !mobile) || activeElt() == display.input)
+ setTimeout(bind(onFocus, cm), 20);
+ else
+ onBlur(cm);
+
+ for (var opt in optionHandlers) if (optionHandlers.hasOwnProperty(opt))
+ optionHandlers[opt](cm, options[opt], Init);
+ for (var i = 0; i < initHooks.length; ++i) initHooks[i](cm);
+ });
+ }
+
+ // DISPLAY CONSTRUCTOR
+
+ // The display handles the DOM integration, both for input reading
+ // and content drawing. It holds references to DOM nodes and
+ // display-related state.
+
+ function Display(place, doc) {
+ var d = this;
+
+ // The semihidden textarea that is focused when the editor is
+ // focused, and receives input.
+ var input = d.input = elt("textarea", null, null, "position: absolute; padding: 0; width: 1px; height: 1em; outline: none");
+ // The textarea is kept positioned near the cursor to prevent the
+ // fact that it'll be scrolled into view on input from scrolling
+ // our fake cursor out of view. On webkit, when wrap=off, paste is
+ // very slow. So make the area wide instead.
+ if (webkit) input.style.width = "1000px";
+ else input.setAttribute("wrap", "off");
+ // If border: 0; -- iOS fails to open keyboard (issue #1287)
+ if (ios) input.style.border = "1px solid black";
+ input.setAttribute("autocorrect", "off"); input.setAttribute("autocapitalize", "off"); input.setAttribute("spellcheck", "false");
+
+ // Wraps and hides input textarea
+ d.inputDiv = elt("div", [input], null, "overflow: hidden; position: relative; width: 3px; height: 0px;");
+ // The fake scrollbar elements.
+ d.scrollbarH = elt("div", [elt("div", null, null, "height: 100%; min-height: 1px")], "CodeMirror-hscrollbar");
+ d.scrollbarV = elt("div", [elt("div", null, null, "min-width: 1px")], "CodeMirror-vscrollbar");
+ // Covers bottom-right square when both scrollbars are present.
+ d.scrollbarFiller = elt("div", null, "CodeMirror-scrollbar-filler");
+ // Covers bottom of gutter when coverGutterNextToScrollbar is on
+ // and h scrollbar is present.
+ d.gutterFiller = elt("div", null, "CodeMirror-gutter-filler");
+ // Will contain the actual code, positioned to cover the viewport.
+ d.lineDiv = elt("div", null, "CodeMirror-code");
+ // Elements are added to these to represent selection and cursors.
+ d.selectionDiv = elt("div", null, null, "position: relative; z-index: 1");
+ d.cursorDiv = elt("div", null, "CodeMirror-cursors");
+ // A visibility: hidden element used to find the size of things.
+ d.measure = elt("div", null, "CodeMirror-measure");
+ // When lines outside of the viewport are measured, they are drawn in this.
+ d.lineMeasure = elt("div", null, "CodeMirror-measure");
+ // Wraps everything that needs to exist inside the vertically-padded coordinate system
+ d.lineSpace = elt("div", [d.measure, d.lineMeasure, d.selectionDiv, d.cursorDiv, d.lineDiv],
+ null, "position: relative; outline: none");
+ // Moved around its parent to cover visible view.
+ d.mover = elt("div", [elt("div", [d.lineSpace], "CodeMirror-lines")], null, "position: relative");
+ // Set to the height of the document, allowing scrolling.
+ d.sizer = elt("div", [d.mover], "CodeMirror-sizer");
+ // Behavior of elts with overflow: auto and padding is
+ // inconsistent across browsers. This is used to ensure the
+ // scrollable area is big enough.
+ d.heightForcer = elt("div", null, null, "position: absolute; height: " + scrollerCutOff + "px; width: 1px;");
+ // Will contain the gutters, if any.
+ d.gutters = elt("div", null, "CodeMirror-gutters");
+ d.lineGutter = null;
+ // Actual scrollable element.
+ d.scroller = elt("div", [d.sizer, d.heightForcer, d.gutters], "CodeMirror-scroll");
+ d.scroller.setAttribute("tabIndex", "-1");
+ // The element in which the editor lives.
+ d.wrapper = elt("div", [d.inputDiv, d.scrollbarH, d.scrollbarV,
+ d.scrollbarFiller, d.gutterFiller, d.scroller], "CodeMirror");
+
+ // Work around IE7 z-index bug (not perfect, hence IE7 not really being supported)
+ if (ie_upto7) { d.gutters.style.zIndex = -1; d.scroller.style.paddingRight = 0; }
+ // Needed to hide big blue blinking cursor on Mobile Safari
+ if (ios) input.style.width = "0px";
+ if (!webkit) d.scroller.draggable = true;
+ // Needed to handle Tab key in KHTML
+ if (khtml) { d.inputDiv.style.height = "1px"; d.inputDiv.style.position = "absolute"; }
+ // Need to set a minimum width to see the scrollbar on IE7 (but must not set it on IE8).
+ if (ie_upto7) d.scrollbarH.style.minHeight = d.scrollbarV.style.minWidth = "18px";
+
+ if (place.appendChild) place.appendChild(d.wrapper);
+ else place(d.wrapper);
+
+ // Current rendered range (may be bigger than the view window).
+ d.viewFrom = d.viewTo = doc.first;
+ // Information about the rendered lines.
+ d.view = [];
+ // Holds info about a single rendered line when it was rendered
+ // for measurement, while not in view.
+ d.externalMeasured = null;
+ // Empty space (in pixels) above the view
+ d.viewOffset = 0;
+ d.lastSizeC = 0;
+ d.updateLineNumbers = null;
+
+ // Used to only resize the line number gutter when necessary (when
+ // the amount of lines crosses a boundary that makes its width change)
+ d.lineNumWidth = d.lineNumInnerWidth = d.lineNumChars = null;
+ // See readInput and resetInput
+ d.prevInput = "";
+ // Set to true when a non-horizontal-scrolling line widget is
+ // added. As an optimization, line widget aligning is skipped when
+ // this is false.
+ d.alignWidgets = false;
+ // Flag that indicates whether we expect input to appear real soon
+ // now (after some event like 'keypress' or 'input') and are
+ // polling intensively.
+ d.pollingFast = false;
+ // Self-resetting timeout for the poller
+ d.poll = new Delayed();
+
+ d.cachedCharWidth = d.cachedTextHeight = d.cachedPaddingH = null;
+
+ // Tracks when resetInput has punted to just putting a short
+ // string into the textarea instead of the full selection.
+ d.inaccurateSelection = false;
+
+ // Tracks the maximum line length so that the horizontal scrollbar
+ // can be kept static when scrolling.
+ d.maxLine = null;
+ d.maxLineLength = 0;
+ d.maxLineChanged = false;
+
+ // Used for measuring wheel scrolling granularity
+ d.wheelDX = d.wheelDY = d.wheelStartX = d.wheelStartY = null;
+
+ // True when shift is held down.
+ d.shift = false;
+ }
+
+ // STATE UPDATES
+
+ // Used to get the editor into a consistent state again when options change.
+
+ function loadMode(cm) {
+ cm.doc.mode = CodeMirror.getMode(cm.options, cm.doc.modeOption);
+ resetModeState(cm);
+ }
+
+ function resetModeState(cm) {
+ cm.doc.iter(function(line) {
+ if (line.stateAfter) line.stateAfter = null;
+ if (line.styles) line.styles = null;
+ });
+ cm.doc.frontier = cm.doc.first;
+ startWorker(cm, 100);
+ cm.state.modeGen++;
+ if (cm.curOp) regChange(cm);
+ }
+
+ function wrappingChanged(cm) {
+ if (cm.options.lineWrapping) {
+ addClass(cm.display.wrapper, "CodeMirror-wrap");
+ cm.display.sizer.style.minWidth = "";
+ } else {
+ rmClass(cm.display.wrapper, "CodeMirror-wrap");
+ findMaxLine(cm);
+ }
+ estimateLineHeights(cm);
+ regChange(cm);
+ clearCaches(cm);
+ setTimeout(function(){updateScrollbars(cm);}, 100);
+ }
+
+ // Returns a function that estimates the height of a line, to use as
+ // first approximation until the line becomes visible (and is thus
+ // properly measurable).
+ function estimateHeight(cm) {
+ var th = textHeight(cm.display), wrapping = cm.options.lineWrapping;
+ var perLine = wrapping && Math.max(5, cm.display.scroller.clientWidth / charWidth(cm.display) - 3);
+ return function(line) {
+ if (lineIsHidden(cm.doc, line)) return 0;
+
+ var widgetsHeight = 0;
+ if (line.widgets) for (var i = 0; i < line.widgets.length; i++) {
+ if (line.widgets[i].height) widgetsHeight += line.widgets[i].height;
+ }
+
+ if (wrapping)
+ return widgetsHeight + (Math.ceil(line.text.length / perLine) || 1) * th;
+ else
+ return widgetsHeight + th;
+ };
+ }
+
+ function estimateLineHeights(cm) {
+ var doc = cm.doc, est = estimateHeight(cm);
+ doc.iter(function(line) {
+ var estHeight = est(line);
+ if (estHeight != line.height) updateLineHeight(line, estHeight);
+ });
+ }
+
+ function keyMapChanged(cm) {
+ var map = keyMap[cm.options.keyMap], style = map.style;
+ cm.display.wrapper.className = cm.display.wrapper.className.replace(/\s*cm-keymap-\S+/g, "") +
+ (style ? " cm-keymap-" + style : "");
+ }
+
+ function themeChanged(cm) {
+ cm.display.wrapper.className = cm.display.wrapper.className.replace(/\s*cm-s-\S+/g, "") +
+ cm.options.theme.replace(/(^|\s)\s*/g, " cm-s-");
+ clearCaches(cm);
+ }
+
+ function guttersChanged(cm) {
+ updateGutters(cm);
+ regChange(cm);
+ setTimeout(function(){alignHorizontally(cm);}, 20);
+ }
+
+ // Rebuild the gutter elements, ensure the margin to the left of the
+ // code matches their width.
+ function updateGutters(cm) {
+ var gutters = cm.display.gutters, specs = cm.options.gutters;
+ removeChildren(gutters);
+ for (var i = 0; i < specs.length; ++i) {
+ var gutterClass = specs[i];
+ var gElt = gutters.appendChild(elt("div", null, "CodeMirror-gutter " + gutterClass));
+ if (gutterClass == "CodeMirror-linenumbers") {
+ cm.display.lineGutter = gElt;
+ gElt.style.width = (cm.display.lineNumWidth || 1) + "px";
+ }
+ }
+ gutters.style.display = i ? "" : "none";
+ updateGutterSpace(cm);
+ }
+
+ function updateGutterSpace(cm) {
+ var width = cm.display.gutters.offsetWidth;
+ cm.display.sizer.style.marginLeft = width + "px";
+ cm.display.scrollbarH.style.left = cm.options.fixedGutter ? width + "px" : 0;
+ }
+
+ // Compute the character length of a line, taking into account
+ // collapsed ranges (see markText) that might hide parts, and join
+ // other lines onto it.
+ function lineLength(line) {
+ if (line.height == 0) return 0;
+ var len = line.text.length, merged, cur = line;
+ while (merged = collapsedSpanAtStart(cur)) {
+ var found = merged.find(0, true);
+ cur = found.from.line;
+ len += found.from.ch - found.to.ch;
+ }
+ cur = line;
+ while (merged = collapsedSpanAtEnd(cur)) {
+ var found = merged.find(0, true);
+ len -= cur.text.length - found.from.ch;
+ cur = found.to.line;
+ len += cur.text.length - found.to.ch;
+ }
+ return len;
+ }
+
+ // Find the longest line in the document.
+ function findMaxLine(cm) {
+ var d = cm.display, doc = cm.doc;
+ d.maxLine = getLine(doc, doc.first);
+ d.maxLineLength = lineLength(d.maxLine);
+ d.maxLineChanged = true;
+ doc.iter(function(line) {
+ var len = lineLength(line);
+ if (len > d.maxLineLength) {
+ d.maxLineLength = len;
+ d.maxLine = line;
+ }
+ });
+ }
+
+ // Make sure the gutters options contains the element
+ // "CodeMirror-linenumbers" when the lineNumbers option is true.
+ function setGuttersForLineNumbers(options) {
+ var found = indexOf(options.gutters, "CodeMirror-linenumbers");
+ if (found == -1 && options.lineNumbers) {
+ options.gutters = options.gutters.concat(["CodeMirror-linenumbers"]);
+ } else if (found > -1 && !options.lineNumbers) {
+ options.gutters = options.gutters.slice(0);
+ options.gutters.splice(found, 1);
+ }
+ }
+
+ // SCROLLBARS
+
+ // Prepare DOM reads needed to update the scrollbars. Done in one
+ // shot to minimize update/measure roundtrips.
+ function measureForScrollbars(cm) {
+ var scroll = cm.display.scroller;
+ return {
+ clientHeight: scroll.clientHeight,
+ barHeight: cm.display.scrollbarV.clientHeight,
+ scrollWidth: scroll.scrollWidth, clientWidth: scroll.clientWidth,
+ barWidth: cm.display.scrollbarH.clientWidth,
+ docHeight: Math.round(cm.doc.height + paddingVert(cm.display))
+ };
+ }
+
+ // Re-synchronize the fake scrollbars with the actual size of the
+ // content.
+ function updateScrollbars(cm, measure) {
+ if (!measure) measure = measureForScrollbars(cm);
+ var d = cm.display;
+ var scrollHeight = measure.docHeight + scrollerCutOff;
+ var needsH = measure.scrollWidth > measure.clientWidth;
+ var needsV = scrollHeight > measure.clientHeight;
+ if (needsV) {
+ d.scrollbarV.style.display = "block";
+ d.scrollbarV.style.bottom = needsH ? scrollbarWidth(d.measure) + "px" : "0";
+ // A bug in IE8 can cause this value to be negative, so guard it.
+ d.scrollbarV.firstChild.style.height =
+ Math.max(0, scrollHeight - measure.clientHeight + (measure.barHeight || d.scrollbarV.clientHeight)) + "px";
+ } else {
+ d.scrollbarV.style.display = "";
+ d.scrollbarV.firstChild.style.height = "0";
+ }
+ if (needsH) {
+ d.scrollbarH.style.display = "block";
+ d.scrollbarH.style.right = needsV ? scrollbarWidth(d.measure) + "px" : "0";
+ d.scrollbarH.firstChild.style.width =
+ (measure.scrollWidth - measure.clientWidth + (measure.barWidth || d.scrollbarH.clientWidth)) + "px";
+ } else {
+ d.scrollbarH.style.display = "";
+ d.scrollbarH.firstChild.style.width = "0";
+ }
+ if (needsH && needsV) {
+ d.scrollbarFiller.style.display = "block";
+ d.scrollbarFiller.style.height = d.scrollbarFiller.style.width = scrollbarWidth(d.measure) + "px";
+ } else d.scrollbarFiller.style.display = "";
+ if (needsH && cm.options.coverGutterNextToScrollbar && cm.options.fixedGutter) {
+ d.gutterFiller.style.display = "block";
+ d.gutterFiller.style.height = scrollbarWidth(d.measure) + "px";
+ d.gutterFiller.style.width = d.gutters.offsetWidth + "px";
+ } else d.gutterFiller.style.display = "";
+
+ if (!cm.state.checkedOverlayScrollbar && measure.clientHeight > 0) {
+ if (scrollbarWidth(d.measure) === 0) {
+ var w = mac && !mac_geMountainLion ? "12px" : "18px";
+ d.scrollbarV.style.minWidth = d.scrollbarH.style.minHeight = w;
+ var barMouseDown = function(e) {
+ if (e_target(e) != d.scrollbarV && e_target(e) != d.scrollbarH)
+ operation(cm, onMouseDown)(e);
+ };
+ on(d.scrollbarV, "mousedown", barMouseDown);
+ on(d.scrollbarH, "mousedown", barMouseDown);
+ }
+ cm.state.checkedOverlayScrollbar = true;
+ }
+ }
+
+ // Compute the lines that are visible in a given viewport (defaults
+ // the the current scroll position). viewPort may contain top,
+ // height, and ensure (see op.scrollToPos) properties.
+ function visibleLines(display, doc, viewPort) {
+ var top = viewPort && viewPort.top != null ? viewPort.top : display.scroller.scrollTop;
+ top = Math.floor(top - paddingTop(display));
+ var bottom = viewPort && viewPort.bottom != null ? viewPort.bottom : top + display.wrapper.clientHeight;
+
+ var from = lineAtHeight(doc, top), to = lineAtHeight(doc, bottom);
+ // Ensure is a {from: {line, ch}, to: {line, ch}} object, and
+ // forces those lines into the viewport (if possible).
+ if (viewPort && viewPort.ensure) {
+ var ensureFrom = viewPort.ensure.from.line, ensureTo = viewPort.ensure.to.line;
+ if (ensureFrom < from)
+ return {from: ensureFrom,
+ to: lineAtHeight(doc, heightAtLine(getLine(doc, ensureFrom)) + display.wrapper.clientHeight)};
+ if (Math.min(ensureTo, doc.lastLine()) >= to)
+ return {from: lineAtHeight(doc, heightAtLine(getLine(doc, ensureTo)) - display.wrapper.clientHeight),
+ to: ensureTo};
+ }
+ return {from: from, to: to};
+ }
+
+ // LINE NUMBERS
+
+ // Re-align line numbers and gutter marks to compensate for
+ // horizontal scrolling.
+ function alignHorizontally(cm) {
+ var display = cm.display, view = display.view;
+ if (!display.alignWidgets && (!display.gutters.firstChild || !cm.options.fixedGutter)) return;
+ var comp = compensateForHScroll(display) - display.scroller.scrollLeft + cm.doc.scrollLeft;
+ var gutterW = display.gutters.offsetWidth, left = comp + "px";
+ for (var i = 0; i < view.length; i++) if (!view[i].hidden) {
+ if (cm.options.fixedGutter && view[i].gutter)
+ view[i].gutter.style.left = left;
+ var align = view[i].alignable;
+ if (align) for (var j = 0; j < align.length; j++)
+ align[j].style.left = left;
+ }
+ if (cm.options.fixedGutter)
+ display.gutters.style.left = (comp + gutterW) + "px";
+ }
+
+ // Used to ensure that the line number gutter is still the right
+ // size for the current document size. Returns true when an update
+ // is needed.
+ function maybeUpdateLineNumberWidth(cm) {
+ if (!cm.options.lineNumbers) return false;
+ var doc = cm.doc, last = lineNumberFor(cm.options, doc.first + doc.size - 1), display = cm.display;
+ if (last.length != display.lineNumChars) {
+ var test = display.measure.appendChild(elt("div", [elt("div", last)],
+ "CodeMirror-linenumber CodeMirror-gutter-elt"));
+ var innerW = test.firstChild.offsetWidth, padding = test.offsetWidth - innerW;
+ display.lineGutter.style.width = "";
+ display.lineNumInnerWidth = Math.max(innerW, display.lineGutter.offsetWidth - padding);
+ display.lineNumWidth = display.lineNumInnerWidth + padding;
+ display.lineNumChars = display.lineNumInnerWidth ? last.length : -1;
+ display.lineGutter.style.width = display.lineNumWidth + "px";
+ updateGutterSpace(cm);
+ return true;
+ }
+ return false;
+ }
+
+ function lineNumberFor(options, i) {
+ return String(options.lineNumberFormatter(i + options.firstLineNumber));
+ }
+
+ // Computes display.scroller.scrollLeft + display.gutters.offsetWidth,
+ // but using getBoundingClientRect to get a sub-pixel-accurate
+ // result.
+ function compensateForHScroll(display) {
+ return display.scroller.getBoundingClientRect().left - display.sizer.getBoundingClientRect().left;
+ }
+
+ // DISPLAY DRAWING
+
+ // Updates the display, selection, and scrollbars, using the
+ // information in display.view to find out which nodes are no longer
+ // up-to-date. Tries to bail out early when no changes are needed,
+ // unless forced is true.
+ // Returns true if an actual update happened, false otherwise.
+ function updateDisplay(cm, viewPort, forced) {
+ var oldFrom = cm.display.viewFrom, oldTo = cm.display.viewTo, updated;
+ var visible = visibleLines(cm.display, cm.doc, viewPort);
+ for (var first = true;; first = false) {
+ var oldWidth = cm.display.scroller.clientWidth;
+ if (!updateDisplayInner(cm, visible, forced)) break;
+ updated = true;
+
+ // If the max line changed since it was last measured, measure it,
+ // and ensure the document's width matches it.
+ if (cm.display.maxLineChanged && !cm.options.lineWrapping)
+ adjustContentWidth(cm);
+
+ var barMeasure = measureForScrollbars(cm);
+ updateSelection(cm);
+ setDocumentHeight(cm, barMeasure);
+ updateScrollbars(cm, barMeasure);
+ if (webkit && cm.options.lineWrapping)
+ checkForWebkitWidthBug(cm, barMeasure); // (Issue #2420)
+ if (first && cm.options.lineWrapping && oldWidth != cm.display.scroller.clientWidth) {
+ forced = true;
+ continue;
+ }
+ forced = false;
+
+ // Clip forced viewport to actual scrollable area.
+ if (viewPort && viewPort.top != null)
+ viewPort = {top: Math.min(barMeasure.docHeight - scrollerCutOff - barMeasure.clientHeight, viewPort.top)};
+ // Updated line heights might result in the drawn area not
+ // actually covering the viewport. Keep looping until it does.
+ visible = visibleLines(cm.display, cm.doc, viewPort);
+ if (visible.from >= cm.display.viewFrom && visible.to <= cm.display.viewTo)
+ break;
+ }
+
+ cm.display.updateLineNumbers = null;
+ if (updated) {
+ signalLater(cm, "update", cm);
+ if (cm.display.viewFrom != oldFrom || cm.display.viewTo != oldTo)
+ signalLater(cm, "viewportChange", cm, cm.display.viewFrom, cm.display.viewTo);
+ }
+ return updated;
+ }
+
+ // Does the actual updating of the line display. Bails out
+ // (returning false) when there is nothing to be done and forced is
+ // false.
+ function updateDisplayInner(cm, visible, forced) {
+ var display = cm.display, doc = cm.doc;
+ if (!display.wrapper.offsetWidth) {
+ resetView(cm);
+ return;
+ }
+
+ // Bail out if the visible area is already rendered and nothing changed.
+ if (!forced && visible.from >= display.viewFrom && visible.to <= display.viewTo &&
+ countDirtyView(cm) == 0)
+ return;
+
+ if (maybeUpdateLineNumberWidth(cm))
+ resetView(cm);
+ var dims = getDimensions(cm);
+
+ // Compute a suitable new viewport (from & to)
+ var end = doc.first + doc.size;
+ var from = Math.max(visible.from - cm.options.viewportMargin, doc.first);
+ var to = Math.min(end, visible.to + cm.options.viewportMargin);
+ if (display.viewFrom < from && from - display.viewFrom < 20) from = Math.max(doc.first, display.viewFrom);
+ if (display.viewTo > to && display.viewTo - to < 20) to = Math.min(end, display.viewTo);
+ if (sawCollapsedSpans) {
+ from = visualLineNo(cm.doc, from);
+ to = visualLineEndNo(cm.doc, to);
+ }
+
+ var different = from != display.viewFrom || to != display.viewTo ||
+ display.lastSizeC != display.wrapper.clientHeight;
+ adjustView(cm, from, to);
+
+ display.viewOffset = heightAtLine(getLine(cm.doc, display.viewFrom));
+ // Position the mover div to align with the current scroll position
+ cm.display.mover.style.top = display.viewOffset + "px";
+
+ var toUpdate = countDirtyView(cm);
+ if (!different && toUpdate == 0 && !forced) return;
+
+ // For big changes, we hide the enclosing element during the
+ // update, since that speeds up the operations on most browsers.
+ var focused = activeElt();
+ if (toUpdate > 4) display.lineDiv.style.display = "none";
+ patchDisplay(cm, display.updateLineNumbers, dims);
+ if (toUpdate > 4) display.lineDiv.style.display = "";
+ // There might have been a widget with a focused element that got
+ // hidden or updated, if so re-focus it.
+ if (focused && activeElt() != focused && focused.offsetHeight) focused.focus();
+
+ // Prevent selection and cursors from interfering with the scroll
+ // width.
+ removeChildren(display.cursorDiv);
+ removeChildren(display.selectionDiv);
+
+ if (different) {
+ display.lastSizeC = display.wrapper.clientHeight;
+ startWorker(cm, 400);
+ }
+
+ updateHeightsInViewport(cm);
+
+ return true;
+ }
+
+ function adjustContentWidth(cm) {
+ var display = cm.display;
+ var width = measureChar(cm, display.maxLine, display.maxLine.text.length).left;
+ display.maxLineChanged = false;
+ var minWidth = Math.max(0, width + 3);
+ var maxScrollLeft = Math.max(0, display.sizer.offsetLeft + minWidth + scrollerCutOff - display.scroller.clientWidth);
+ display.sizer.style.minWidth = minWidth + "px";
+ if (maxScrollLeft < cm.doc.scrollLeft)
+ setScrollLeft(cm, Math.min(display.scroller.scrollLeft, maxScrollLeft), true);
+ }
+
+ function setDocumentHeight(cm, measure) {
+ cm.display.sizer.style.minHeight = cm.display.heightForcer.style.top = measure.docHeight + "px";
+ cm.display.gutters.style.height = Math.max(measure.docHeight, measure.clientHeight - scrollerCutOff) + "px";
+ }
+
+
+ function checkForWebkitWidthBug(cm, measure) {
+ // Work around Webkit bug where it sometimes reserves space for a
+ // non-existing phantom scrollbar in the scroller (Issue #2420)
+ if (cm.display.sizer.offsetWidth + cm.display.gutters.offsetWidth < cm.display.scroller.clientWidth - 1) {
+ cm.display.sizer.style.minHeight = cm.display.heightForcer.style.top = "0px";
+ cm.display.gutters.style.height = measure.docHeight + "px";
+ }
+ }
+
+ // Read the actual heights of the rendered lines, and update their
+ // stored heights to match.
+ function updateHeightsInViewport(cm) {
+ var display = cm.display;
+ var prevBottom = display.lineDiv.offsetTop;
+ for (var i = 0; i < display.view.length; i++) {
+ var cur = display.view[i], height;
+ if (cur.hidden) continue;
+ if (ie_upto7) {
+ var bot = cur.node.offsetTop + cur.node.offsetHeight;
+ height = bot - prevBottom;
+ prevBottom = bot;
+ } else {
+ var box = cur.node.getBoundingClientRect();
+ height = box.bottom - box.top;
+ }
+ var diff = cur.line.height - height;
+ if (height < 2) height = textHeight(display);
+ if (diff > .001 || diff < -.001) {
+ updateLineHeight(cur.line, height);
+ updateWidgetHeight(cur.line);
+ if (cur.rest) for (var j = 0; j < cur.rest.length; j++)
+ updateWidgetHeight(cur.rest[j]);
+ }
+ }
+ }
+
+ // Read and store the height of line widgets associated with the
+ // given line.
+ function updateWidgetHeight(line) {
+ if (line.widgets) for (var i = 0; i < line.widgets.length; ++i)
+ line.widgets[i].height = line.widgets[i].node.offsetHeight;
+ }
+
+ // Do a bulk-read of the DOM positions and sizes needed to draw the
+ // view, so that we don't interleave reading and writing to the DOM.
+ function getDimensions(cm) {
+ var d = cm.display, left = {}, width = {};
+ for (var n = d.gutters.firstChild, i = 0; n; n = n.nextSibling, ++i) {
+ left[cm.options.gutters[i]] = n.offsetLeft;
+ width[cm.options.gutters[i]] = n.offsetWidth;
+ }
+ return {fixedPos: compensateForHScroll(d),
+ gutterTotalWidth: d.gutters.offsetWidth,
+ gutterLeft: left,
+ gutterWidth: width,
+ wrapperWidth: d.wrapper.clientWidth};
+ }
+
+ // Sync the actual display DOM structure with display.view, removing
+ // nodes for lines that are no longer in view, and creating the ones
+ // that are not there yet, and updating the ones that are out of
+ // date.
+ function patchDisplay(cm, updateNumbersFrom, dims) {
+ var display = cm.display, lineNumbers = cm.options.lineNumbers;
+ var container = display.lineDiv, cur = container.firstChild;
+
+ function rm(node) {
+ var next = node.nextSibling;
+ // Works around a throw-scroll bug in OS X Webkit
+ if (webkit && mac && cm.display.currentWheelTarget == node)
+ node.style.display = "none";
+ else
+ node.parentNode.removeChild(node);
+ return next;
+ }
+
+ var view = display.view, lineN = display.viewFrom;
+ // Loop over the elements in the view, syncing cur (the DOM nodes
+ // in display.lineDiv) with the view as we go.
+ for (var i = 0; i < view.length; i++) {
+ var lineView = view[i];
+ if (lineView.hidden) {
+ } else if (!lineView.node) { // Not drawn yet
+ var node = buildLineElement(cm, lineView, lineN, dims);
+ container.insertBefore(node, cur);
+ } else { // Already drawn
+ while (cur != lineView.node) cur = rm(cur);
+ var updateNumber = lineNumbers && updateNumbersFrom != null &&
+ updateNumbersFrom <= lineN && lineView.lineNumber;
+ if (lineView.changes) {
+ if (indexOf(lineView.changes, "gutter") > -1) updateNumber = false;
+ updateLineForChanges(cm, lineView, lineN, dims);
+ }
+ if (updateNumber) {
+ removeChildren(lineView.lineNumber);
+ lineView.lineNumber.appendChild(document.createTextNode(lineNumberFor(cm.options, lineN)));
+ }
+ cur = lineView.node.nextSibling;
+ }
+ lineN += lineView.size;
+ }
+ while (cur) cur = rm(cur);
+ }
+
+ // When an aspect of a line changes, a string is added to
+ // lineView.changes. This updates the relevant part of the line's
+ // DOM structure.
+ function updateLineForChanges(cm, lineView, lineN, dims) {
+ for (var j = 0; j < lineView.changes.length; j++) {
+ var type = lineView.changes[j];
+ if (type == "text") updateLineText(cm, lineView);
+ else if (type == "gutter") updateLineGutter(cm, lineView, lineN, dims);
+ else if (type == "class") updateLineClasses(lineView);
+ else if (type == "widget") updateLineWidgets(lineView, dims);
+ }
+ lineView.changes = null;
+ }
+
+ // Lines with gutter elements, widgets or a background class need to
+ // be wrapped, and have the extra elements added to the wrapper div
+ function ensureLineWrapped(lineView) {
+ if (lineView.node == lineView.text) {
+ lineView.node = elt("div", null, null, "position: relative");
+ if (lineView.text.parentNode)
+ lineView.text.parentNode.replaceChild(lineView.node, lineView.text);
+ lineView.node.appendChild(lineView.text);
+ if (ie_upto7) lineView.node.style.zIndex = 2;
+ }
+ return lineView.node;
+ }
+
+ function updateLineBackground(lineView) {
+ var cls = lineView.bgClass ? lineView.bgClass + " " + (lineView.line.bgClass || "") : lineView.line.bgClass;
+ if (cls) cls += " CodeMirror-linebackground";
+ if (lineView.background) {
+ if (cls) lineView.background.className = cls;
+ else { lineView.background.parentNode.removeChild(lineView.background); lineView.background = null; }
+ } else if (cls) {
+ var wrap = ensureLineWrapped(lineView);
+ lineView.background = wrap.insertBefore(elt("div", null, cls), wrap.firstChild);
+ }
+ }
+
+ // Wrapper around buildLineContent which will reuse the structure
+ // in display.externalMeasured when possible.
+ function getLineContent(cm, lineView) {
+ var ext = cm.display.externalMeasured;
+ if (ext && ext.line == lineView.line) {
+ cm.display.externalMeasured = null;
+ lineView.measure = ext.measure;
+ return ext.built;
+ }
+ return buildLineContent(cm, lineView);
+ }
+
+ // Redraw the line's text. Interacts with the background and text
+ // classes because the mode may output tokens that influence these
+ // classes.
+ function updateLineText(cm, lineView) {
+ var cls = lineView.text.className;
+ var built = getLineContent(cm, lineView);
+ if (lineView.text == lineView.node) lineView.node = built.pre;
+ lineView.text.parentNode.replaceChild(built.pre, lineView.text);
+ lineView.text = built.pre;
+ if (built.bgClass != lineView.bgClass || built.textClass != lineView.textClass) {
+ lineView.bgClass = built.bgClass;
+ lineView.textClass = built.textClass;
+ updateLineClasses(lineView);
+ } else if (cls) {
+ lineView.text.className = cls;
+ }
+ }
+
+ function updateLineClasses(lineView) {
+ updateLineBackground(lineView);
+ if (lineView.line.wrapClass)
+ ensureLineWrapped(lineView).className = lineView.line.wrapClass;
+ else if (lineView.node != lineView.text)
+ lineView.node.className = "";
+ var textClass = lineView.textClass ? lineView.textClass + " " + (lineView.line.textClass || "") : lineView.line.textClass;
+ lineView.text.className = textClass || "";
+ }
+
+ function updateLineGutter(cm, lineView, lineN, dims) {
+ if (lineView.gutter) {
+ lineView.node.removeChild(lineView.gutter);
+ lineView.gutter = null;
+ }
+ var markers = lineView.line.gutterMarkers;
+ if (cm.options.lineNumbers || markers) {
+ var wrap = ensureLineWrapped(lineView);
+ var gutterWrap = lineView.gutter =
+ wrap.insertBefore(elt("div", null, "CodeMirror-gutter-wrapper", "position: absolute; left: " +
+ (cm.options.fixedGutter ? dims.fixedPos : -dims.gutterTotalWidth) + "px"),
+ lineView.text);
+ if (cm.options.lineNumbers && (!markers || !markers["CodeMirror-linenumbers"]))
+ lineView.lineNumber = gutterWrap.appendChild(
+ elt("div", lineNumberFor(cm.options, lineN),
+ "CodeMirror-linenumber CodeMirror-gutter-elt",
+ "left: " + dims.gutterLeft["CodeMirror-linenumbers"] + "px; width: "
+ + cm.display.lineNumInnerWidth + "px"));
+ if (markers) for (var k = 0; k < cm.options.gutters.length; ++k) {
+ var id = cm.options.gutters[k], found = markers.hasOwnProperty(id) && markers[id];
+ if (found)
+ gutterWrap.appendChild(elt("div", [found], "CodeMirror-gutter-elt", "left: " +
+ dims.gutterLeft[id] + "px; width: " + dims.gutterWidth[id] + "px"));
+ }
+ }
+ }
+
+ function updateLineWidgets(lineView, dims) {
+ if (lineView.alignable) lineView.alignable = null;
+ for (var node = lineView.node.firstChild, next; node; node = next) {
+ var next = node.nextSibling;
+ if (node.className == "CodeMirror-linewidget")
+ lineView.node.removeChild(node);
+ }
+ insertLineWidgets(lineView, dims);
+ }
+
+ // Build a line's DOM representation from scratch
+ function buildLineElement(cm, lineView, lineN, dims) {
+ var built = getLineContent(cm, lineView);
+ lineView.text = lineView.node = built.pre;
+ if (built.bgClass) lineView.bgClass = built.bgClass;
+ if (built.textClass) lineView.textClass = built.textClass;
+
+ updateLineClasses(lineView);
+ updateLineGutter(cm, lineView, lineN, dims);
+ insertLineWidgets(lineView, dims);
+ return lineView.node;
+ }
+
+ // A lineView may contain multiple logical lines (when merged by
+ // collapsed spans). The widgets for all of them need to be drawn.
+ function insertLineWidgets(lineView, dims) {
+ insertLineWidgetsFor(lineView.line, lineView, dims, true);
+ if (lineView.rest) for (var i = 0; i < lineView.rest.length; i++)
+ insertLineWidgetsFor(lineView.rest[i], lineView, dims, false);
+ }
+
+ function insertLineWidgetsFor(line, lineView, dims, allowAbove) {
+ if (!line.widgets) return;
+ var wrap = ensureLineWrapped(lineView);
+ for (var i = 0, ws = line.widgets; i < ws.length; ++i) {
+ var widget = ws[i], node = elt("div", [widget.node], "CodeMirror-linewidget");
+ if (!widget.handleMouseEvents) node.ignoreEvents = true;
+ positionLineWidget(widget, node, lineView, dims);
+ if (allowAbove && widget.above)
+ wrap.insertBefore(node, lineView.gutter || lineView.text);
+ else
+ wrap.appendChild(node);
+ signalLater(widget, "redraw");
+ }
+ }
+
+ function positionLineWidget(widget, node, lineView, dims) {
+ if (widget.noHScroll) {
+ (lineView.alignable || (lineView.alignable = [])).push(node);
+ var width = dims.wrapperWidth;
+ node.style.left = dims.fixedPos + "px";
+ if (!widget.coverGutter) {
+ width -= dims.gutterTotalWidth;
+ node.style.paddingLeft = dims.gutterTotalWidth + "px";
+ }
+ node.style.width = width + "px";
+ }
+ if (widget.coverGutter) {
+ node.style.zIndex = 5;
+ node.style.position = "relative";
+ if (!widget.noHScroll) node.style.marginLeft = -dims.gutterTotalWidth + "px";
+ }
+ }
+
+ // POSITION OBJECT
+
+ // A Pos instance represents a position within the text.
+ var Pos = CodeMirror.Pos = function(line, ch) {
+ if (!(this instanceof Pos)) return new Pos(line, ch);
+ this.line = line; this.ch = ch;
+ };
+
+ // Compare two positions, return 0 if they are the same, a negative
+ // number when a is less, and a positive number otherwise.
+ var cmp = CodeMirror.cmpPos = function(a, b) { return a.line - b.line || a.ch - b.ch; };
+
+ function copyPos(x) {return Pos(x.line, x.ch);}
+ function maxPos(a, b) { return cmp(a, b) < 0 ? b : a; }
+ function minPos(a, b) { return cmp(a, b) < 0 ? a : b; }
+
+ // SELECTION / CURSOR
+
+ // Selection objects are immutable. A new one is created every time
+ // the selection changes. A selection is one or more non-overlapping
+ // (and non-touching) ranges, sorted, and an integer that indicates
+ // which one is the primary selection (the one that's scrolled into
+ // view, that getCursor returns, etc).
+ function Selection(ranges, primIndex) {
+ this.ranges = ranges;
+ this.primIndex = primIndex;
+ }
+
+ Selection.prototype = {
+ primary: function() { return this.ranges[this.primIndex]; },
+ equals: function(other) {
+ if (other == this) return true;
+ if (other.primIndex != this.primIndex || other.ranges.length != this.ranges.length) return false;
+ for (var i = 0; i < this.ranges.length; i++) {
+ var here = this.ranges[i], there = other.ranges[i];
+ if (cmp(here.anchor, there.anchor) != 0 || cmp(here.head, there.head) != 0) return false;
+ }
+ return true;
+ },
+ deepCopy: function() {
+ for (var out = [], i = 0; i < this.ranges.length; i++)
+ out[i] = new Range(copyPos(this.ranges[i].anchor), copyPos(this.ranges[i].head));
+ return new Selection(out, this.primIndex);
+ },
+ somethingSelected: function() {
+ for (var i = 0; i < this.ranges.length; i++)
+ if (!this.ranges[i].empty()) return true;
+ return false;
+ },
+ contains: function(pos, end) {
+ if (!end) end = pos;
+ for (var i = 0; i < this.ranges.length; i++) {
+ var range = this.ranges[i];
+ if (cmp(end, range.from()) >= 0 && cmp(pos, range.to()) <= 0)
+ return i;
+ }
+ return -1;
+ }
+ };
+
+ function Range(anchor, head) {
+ this.anchor = anchor; this.head = head;
+ }
+
+ Range.prototype = {
+ from: function() { return minPos(this.anchor, this.head); },
+ to: function() { return maxPos(this.anchor, this.head); },
+ empty: function() {
+ return this.head.line == this.anchor.line && this.head.ch == this.anchor.ch;
+ }
+ };
+
+ // Take an unsorted, potentially overlapping set of ranges, and
+ // build a selection out of it. 'Consumes' ranges array (modifying
+ // it).
+ function normalizeSelection(ranges, primIndex) {
+ var prim = ranges[primIndex];
+ ranges.sort(function(a, b) { return cmp(a.from(), b.from()); });
+ primIndex = indexOf(ranges, prim);
+ for (var i = 1; i < ranges.length; i++) {
+ var cur = ranges[i], prev = ranges[i - 1];
+ if (cmp(prev.to(), cur.from()) >= 0) {
+ var from = minPos(prev.from(), cur.from()), to = maxPos(prev.to(), cur.to());
+ var inv = prev.empty() ? cur.from() == cur.head : prev.from() == prev.head;
+ if (i <= primIndex) --primIndex;
+ ranges.splice(--i, 2, new Range(inv ? to : from, inv ? from : to));
+ }
+ }
+ return new Selection(ranges, primIndex);
+ }
+
+ function simpleSelection(anchor, head) {
+ return new Selection([new Range(anchor, head || anchor)], 0);
+ }
+
+ // Most of the external API clips given positions to make sure they
+ // actually exist within the document.
+ function clipLine(doc, n) {return Math.max(doc.first, Math.min(n, doc.first + doc.size - 1));}
+ function clipPos(doc, pos) {
+ if (pos.line < doc.first) return Pos(doc.first, 0);
+ var last = doc.first + doc.size - 1;
+ if (pos.line > last) return Pos(last, getLine(doc, last).text.length);
+ return clipToLen(pos, getLine(doc, pos.line).text.length);
+ }
+ function clipToLen(pos, linelen) {
+ var ch = pos.ch;
+ if (ch == null || ch > linelen) return Pos(pos.line, linelen);
+ else if (ch < 0) return Pos(pos.line, 0);
+ else return pos;
+ }
+ function isLine(doc, l) {return l >= doc.first && l < doc.first + doc.size;}
+ function clipPosArray(doc, array) {
+ for (var out = [], i = 0; i < array.length; i++) out[i] = clipPos(doc, array[i]);
+ return out;
+ }
+
+ // SELECTION UPDATES
+
+ // The 'scroll' parameter given to many of these indicated whether
+ // the new cursor position should be scrolled into view after
+ // modifying the selection.
+
+ // If shift is held or the extend flag is set, extends a range to
+ // include a given position (and optionally a second position).
+ // Otherwise, simply returns the range between the given positions.
+ // Used for cursor motion and such.
+ function extendRange(doc, range, head, other) {
+ if (doc.cm && doc.cm.display.shift || doc.extend) {
+ var anchor = range.anchor;
+ if (other) {
+ var posBefore = cmp(head, anchor) < 0;
+ if (posBefore != (cmp(other, anchor) < 0)) {
+ anchor = head;
+ head = other;
+ } else if (posBefore != (cmp(head, other) < 0)) {
+ head = other;
+ }
+ }
+ return new Range(anchor, head);
+ } else {
+ return new Range(other || head, head);
+ }
+ }
+
+ // Extend the primary selection range, discard the rest.
+ function extendSelection(doc, head, other, options) {
+ setSelection(doc, new Selection([extendRange(doc, doc.sel.primary(), head, other)], 0), options);
+ }
+
+ // Extend all selections (pos is an array of selections with length
+ // equal the number of selections)
+ function extendSelections(doc, heads, options) {
+ for (var out = [], i = 0; i < doc.sel.ranges.length; i++)
+ ...
[truncated message content] |
|
From: <mrp...@us...> - 2014-05-08 23:17:09
|
Revision: 8239
http://sourceforge.net/p/bigdata/code/8239
Author: mrpersonick
Date: 2014-05-08 23:17:07 +0000 (Thu, 08 May 2014)
Log Message:
-----------
trying again with build.xml
Modified Paths:
--------------
branches/BLUEPRINTS/build.xml
Modified: branches/BLUEPRINTS/build.xml
===================================================================
--- branches/BLUEPRINTS/build.xml 2014-05-08 20:50:11 UTC (rev 8238)
+++ branches/BLUEPRINTS/build.xml 2014-05-08 23:17:07 UTC (rev 8239)
@@ -62,12 +62,12 @@
<fileset dir="${bigdata.dir}/bigdata-sails/lib">
<include name="**/*.jar" />
</fileset>
+ <fileset dir="${bigdata.dir}/bigdata-blueprints/lib">
+ <include name="blueprints-core-${blueprints.version}.jar" />
+ </fileset>
<fileset dir="${bigdata.dir}/bigdata-gom/lib">
<include name="**/*.jar" />
</fileset>
- <fileset dir="${bigdata.dir}/bigdata-blueprints/lib">
- <include name="**/*.jar" />
- </fileset>
<!--
<fileset dir="${bigdata.dir}/ctc-striterator/lib">
<include name="**/*.jar" />
@@ -231,8 +231,8 @@
<src path="${bigdata.dir}/bigdata/src/java" />
<src path="${bigdata.dir}/bigdata-jini/src/java" />
<src path="${bigdata.dir}/bigdata-rdf/src/java" />
+ <src path="${bigdata.dir}/bigdata-sails/src/java" />
<src path="${bigdata.dir}/bigdata-blueprints/src/java" />
- <src path="${bigdata.dir}/bigdata-sails/src/java" />
<src path="${bigdata.dir}/bigdata-gom/src/java" />
<src path="${bigdata.dir}/bigdata-ganglia/src/java" />
<src path="${bigdata.dir}/bigdata-gas/src/java" />
@@ -318,10 +318,10 @@
<fileset dir="${bigdata.dir}/bigdata-rdf/src/samples" />
<fileset dir="${bigdata.dir}/bigdata-sails/src/java" />
<fileset dir="${bigdata.dir}/bigdata-sails/src/samples" />
+ <fileset dir="${bigdata.dir}/bigdata-blueprints/src/java" />
<fileset dir="${bigdata.dir}/bigdata-gom/src/java" />
<fileset dir="${bigdata.dir}/bigdata-gom/src/samples" />
<fileset dir="${bigdata.dir}/ctc-striterators/src/java" />
- <fileset dir="${bigdata.dir}/bigdata-blueprints/src/java" />
</jar>
</target>
@@ -376,6 +376,7 @@
<fileset dir="bigdata-jini/src/java" />
<fileset dir="bigdata-rdf/src/java" />
<fileset dir="bigdata-sails/src/java" />
+ <fileset dir="bigdata-blueprints/src/java" />
<fileset dir="bigdata-gom/src/java" />
</jar>
<bnd output="${build.dir}/bundles/com.bigata-${osgi.version}.jar" classpath="${build.dir}/classes" eclipse="false" failok="false" exceptions="true" files="${basedir}/osgi/bigdata.bnd" />
@@ -416,7 +417,7 @@
<packageset dir="${bigdata.dir}/bigdata-sails/src/java" />
<packageset dir="${bigdata.dir}/bigdata-sails/src/samples" />
<packageset dir="${bigdata.dir}/bigdata-blueprints/src/java" />
- <packageset dir="${bigdata.dir}/bigdata-gom/src/java" />
+ <packageset dir="${bigdata.dir}/bigdata-gom/src/java" />
<packageset dir="${bigdata.dir}/bigdata-gom/src/samples" />
<packageset dir="${bigdata.dir}/bigdata-gas/src/java" />
<packageset dir="${bigdata.dir}/ctc-striterators/src/java" />
@@ -456,12 +457,12 @@
<fileset dir="${bigdata.dir}/bigdata-sails/lib">
<include name="**/*.jar" />
</fileset>
+ <fileset dir="${bigdata.dir}/bigdata-blueprints/lib">
+ <include name="blueprints-core-${blueprints.version}.jar" />
+ </fileset>
<fileset dir="${bigdata.dir}/bigdata-gom/lib">
<include name="**/*.jar" />
</fileset>
- <fileset dir="${bigdata.dir}/bigdata-blueprints/lib">
- <include name="**/*.jar" />
- </fileset>
</copy>
<!-- Do NOT flatten the jini jars. We need the to preserve the -->
<!-- lib, lib-dl, and lib-ext distinctions. -->
@@ -571,7 +572,7 @@
<fileset dir="${bigdata.dir}/bigdata" includes="LEGAL/*"/>
<fileset dir="${bigdata.dir}/bigdata-rdf" includes="LEGAL/*"/>
<fileset dir="${bigdata.dir}/bigdata-sails" includes="LEGAL/*"/>
- <fileset dir="${bigdata.dir}/bigdata-blueprints" includes="LEGAL/*"/>
+ <fileset dir="${bigdata.dir}/bigdata-blueprints" includes="LEGAL/*"/>
<fileset dir="${bigdata.dir}/bigdata-gom" includes="LEGAL/*"/>
<fileset dir="${bigdata.dir}/bigdata-jini" includes="LEGAL/*"/>
<!-- bigdata jar plus some dependencies as filtered by autojar.
@@ -681,6 +682,7 @@
<include name="bigdata-jini/LEGAL/*" />
<include name="bigdata-rdf/LEGAL/*" />
<include name="bigdata-sails/LEGAL/*" />
+ <include name="bigdata-blueprints/LEGAL/*" />
<include name="bigdata-gom/LEGAL/*" />
</fileset>
</copy>
@@ -947,8 +949,8 @@
<property name="bigdata-jini.lib" location="${bigdata.dir}/bigdata-jini/lib/jini/lib" />
<property name="bigdata-rdf.lib" location="${bigdata.dir}/bigdata-rdf/lib" />
<property name="bigdata-sails.lib" location="${bigdata.dir}/bigdata-sails/lib" />
+ <property name="bigdata-blueprints.lib" location="${bigdata.dir}/bigdata-blueprints/lib" />
<property name="bigdata-gom.lib" location="${bigdata.dir}/bigdata-gom/lib" />
- <property name="bigdata-blueprints.lib" location="${bigdata.dir}/bigdata-blueprints/lib" />
<property name="bigdata-jetty.lib" location="${bigdata.dir}/bigdata/lib/jetty" />
<property name="bigdata-http.lib" location="${bigdata.dir}/bigdata-sails/lib/httpcomponents" />
<property name="bigdata-zookeeper.lib" location="${bigdata.dir}/bigdata-jini/lib/apache" />
@@ -991,10 +993,11 @@
<!-- GOM library -->
<!-- Note: Nothing yet for GOM -->
- <!-- Blueprints library -->
- <copy file="${bigdata-blueprints.lib}/blueprints-core-${blueprints.version}.jar"
- tofile="${dist.lib}/blueprints-core.jar" />
+ <!-- Blueprints library -->
+ <copy file="${bigdata-blueprints.lib}/blueprints-core-${blueprints.version}.jar"
+ tofile="${dist.lib}/blueprints-core.jar" />
+
<!-- jetty library -->
<copy file="${bigdata-jetty.lib}/jetty-continuation-${jetty.version}.jar"
tofile="${dist.lib}/jetty-continuation.jar" />
@@ -1403,12 +1406,12 @@
<copy toDir="${build.dir}/bigdata-sails/src">
<fileset dir="${bigdata.dir}/bigdata-sails/src" />
</copy>
+ <copy toDir="${build.dir}/bigdata-blueprints/src">
+ <fileset dir="${bigdata.dir}/bigdata-blueprints/src" />
+ </copy>
<copy toDir="${build.dir}/bigdata-gom/src">
<fileset dir="${bigdata.dir}/bigdata-gom/src" />
</copy>
- <copy toDir="${build.dir}/bigdata-blueprints/src">
- <fileset dir="${bigdata.dir}/bigdata-blueprints/src" />
- </copy>
<copy toDir="${build.dir}/bigdata-war/src">
<fileset dir="${bigdata.dir}/bigdata-war/src" />
</copy>
@@ -1446,12 +1449,11 @@
<copy toDir="${build.dir}/bigdata-sails/lib">
<fileset dir="${bigdata.dir}/bigdata-sails/lib" />
</copy>
- <mkdir dir="${build.dir}/bigdata-blueprints/lib" />
- <copy toDir="${build.dir}/bigdata-blueprints/lib">
- <fileset dir="${bigdata.dir}/bigdata-blueprints/lib" />
- </copy>
+ <mkdir dir="${build.dir}/bigdata-blueprints/lib" />
+ <copy toDir="${build.dir}/bigdata-blueprints/lib">
+ <fileset dir="${bigdata.dir}/bigdata-blueprints/lib" />
+ </copy>
-
<mkdir dir="${build.dir}/src" />
<mkdir dir="${build.dir}/src/resources" />
<mkdir dir="${build.dir}/src/resources/config" />
@@ -1504,8 +1506,8 @@
<include name="bigdata-jini/src/**" />
<include name="bigdata-rdf/src/**" />
<include name="bigdata-sails/src/**" />
+ <include name="bigdata-blueprints/src/**" />
<include name="bigdata-gom/src/**" />
- <include name="bigdata-blueprints/src/**" />
<include name="bigdata-war/src/**" />
<include name="ctc-striterators/src/**" />
<include name="lgpl-utils/src/**" />
@@ -1516,7 +1518,6 @@
<include name="bigdata-rdf/lib/**" />
<include name="bigdata-sails/lib/**" />
<include name="bigdata-gom/lib/**" />
- <include name="bigdata-blueprints/lib/**" />
<include name="src/**" />
<exclude name="classes/**" />
<exclude name="${version}.jar" />
@@ -1577,8 +1578,8 @@
<include name="bigdata-jini/LEGAL/*" />
<include name="bigdata-rdf/LEGAL/*" />
<include name="bigdata-sails/LEGAL/*" />
+ <include name="bigdata-blueprints/LEGAL/*" />
<include name="bigdata-gom/LEGAL/*" />
- <include name="bigdata-blueprints/LEGAL/*" />
</fileset>
</copy>
@@ -1777,8 +1778,6 @@
<property name="sesame-sparql-test.jar" location="${bigdata-sails.lib}/sesame-sparql-testsuite-${sesame.version}.jar" />
<property name="sesame-store-test.jar" location="${bigdata-sails.lib}/sesame-store-testsuite-${sesame.version}.jar" />
<property name="sesame-rio-test.jar" location="${bigdata-sails.lib}/sesame-rio-testsuite-${sesame.version}.jar" />
- <property name="blueprints-test.jar" location="${bigdata-blueprints.lib}/blueprints-test-${blueprints.version}.jar" />
- <property name="jettison.jar" location="${bigdata-blueprints.lib}/jettison-${jettison.version}.jar" />
<property name="classes.test.dir" location="${classes.dir}/test" />
<mkdir dir="${classes.test.dir}" />
@@ -1789,7 +1788,7 @@
<!-- TODO ${path.separator}${dist.lib}/bigdata-gas.jar -->
<property name="javac.test.classpath"
- value="${classes.dir}${path.separator}${junit.jar}${path.separator}${junit-ext.jar}${path.separator}${sesame-sparql-test.jar}${path.separator}${sesame-store-test.jar}${path.separator}${sesame-rio-test.jar}${path.separator}${dist.lib}/classserver.jar${path.separator}${dist.lib}/highscalelib.jar${path.separator}${dist.lib}/dsiutils.jar${path.separator}${dist.lib}/lgplutils.jar${path.separator}${dist.lib}/fastutil.jar${path.separator}${dist.lib}/bigdata-ganglia.jar${path.separator}${dist.lib}/icu4j.jar${path.separator}${dist.lib}/icu4j-charset.jar${path.separator}${dist.lib}/log4j.jar${path.separator}${dist.lib}/lucene-analyzer.jar${path.separator}${dist.lib}/lucene-core.jar${path.separator}${path.separator}${dist.lib}/openrdf-sesame.jar${path.separator}${dist.lib}/slf4j.jar${path.separator}${dist.lib}/jsk-lib.jar${path.separator}${dist.lib}/jsk-platform.jar${path.separator}${dist.lib}/nxparser.jar${path.separator}${dist.lib}/zookeeper.jar${path.separator}${dist.lib}/jetty-continuation.jar${path.separator}${dist.lib}/jetty-http.jar${path.separator}${dist.lib}/jetty-io.jar${path.separator}${dist.lib}/jetty-jmx.jar${path.separator}${dist.lib}/jetty-jndi.jar${path.separator}${dist.lib}/jetty-server.jar${path.separator}${dist.lib}/jetty-util.jar${path.separator}${dist.lib}/jetty-webapp.jar${path.separator}${dist.lib}/jetty-servlet.jar${path.separator}${dist.lib}/jetty-security.jar${path.separator}${dist.lib}/jetty-xml.jar${path.separator}${dist.lib}/jetty-rewrite.jar${path.separator}${dist.lib}/jetty-client.jar${path.separator}${dist.lib}/jetty-proxy.jar${path.separator}${dist.lib}/servlet-api.jar${path.separator}${dist.lib}/commons-codec.jar${path.separator}${dist.lib}/commons-fileupload.jar${path.separator}${dist.lib}/commons-io.jar${path.separator}${dist.lib}/commons-logging.jar${path.separator}${dist.lib}/httpclient.jar${path.separator}${dist.lib}/httpclient-cache.jar${path.separator}${dist.lib}/httpcore.jar${path.separator}${dist.lib}/httpmime.jar${path.separator}${dist.lib}/blueprints-core.jar${path.separator}${blueprints-test.jar}${path.separator}${jettison.jar}" />
+ value="${classes.dir}${path.separator}${junit.jar}${path.separator}${junit-ext.jar}${path.separator}${sesame-sparql-test.jar}${path.separator}${sesame-store-test.jar}${path.separator}${sesame-rio-test.jar}${path.separator}${dist.lib}/classserver.jar${path.separator}${dist.lib}/highscalelib.jar${path.separator}${dist.lib}/dsiutils.jar${path.separator}${dist.lib}/lgplutils.jar${path.separator}${dist.lib}/fastutil.jar${path.separator}${dist.lib}/bigdata-ganglia.jar${path.separator}${dist.lib}/icu4j.jar${path.separator}${dist.lib}/icu4j-charset.jar${path.separator}${dist.lib}/log4j.jar${path.separator}${dist.lib}/lucene-analyzer.jar${path.separator}${dist.lib}/lucene-core.jar${path.separator}${path.separator}${dist.lib}/openrdf-sesame.jar${path.separator}${dist.lib}/slf4j.jar${path.separator}${dist.lib}/jsk-lib.jar${path.separator}${dist.lib}/jsk-platform.jar${path.separator}${dist.lib}/nxparser.jar${path.separator}${dist.lib}/zookeeper.jar${path.separator}${dist.lib}/jetty-continuation.jar${path.separator}${dist.lib}/jetty-http.jar${path.separator}${dist.lib}/jetty-io.jar${path.separator}${dist.lib}/jetty-jmx.jar${path.separator}${dist.lib}/jetty-jndi.jar${path.separator}${dist.lib}/jetty-server.jar${path.separator}${dist.lib}/jetty-util.jar${path.separator}${dist.lib}/jetty-webapp.jar${path.separator}${dist.lib}/jetty-servlet.jar${path.separator}${dist.lib}/jetty-security.jar${path.separator}${dist.lib}/jetty-xml.jar${path.separator}${dist.lib}/jetty-rewrite.jar${path.separator}${dist.lib}/jetty-client.jar${path.separator}${dist.lib}/jetty-proxy.jar${path.separator}${dist.lib}/servlet-api.jar${path.separator}${dist.lib}/commons-codec.jar${path.separator}${dist.lib}/commons-fileupload.jar${path.separator}${dist.lib}/commons-io.jar${path.separator}${dist.lib}/commons-logging.jar${path.separator}${dist.lib}/httpclient.jar${path.separator}${dist.lib}/httpclient-cache.jar${path.separator}${dist.lib}/httpcore.jar${path.separator}${dist.lib}/httpmime.jar" />
<echo>javac
</echo>
@@ -1835,7 +1834,6 @@
<src path="${bigdata.dir}/bigdata-rdf/src/test" />
<src path="${bigdata.dir}/bigdata-sails/src/test" />
<src path="${bigdata.dir}/bigdata-gom/src/test" />
- <src path="${bigdata.dir}/bigdata-blueprints/src/test" />
<src path="${bigdata.dir}/bigdata-gas/src/test" />
<src path="${bigdata.dir}/bigdata-ganglia/src/test" />
<src path="${bigdata.dir}/ctc-striterators/src/test" />
@@ -1893,9 +1891,6 @@
<fileset dir="${bigdata.dir}/bigdata-jini/src/test">
<exclude name="**/*.java" />
</fileset>
- <fileset dir="${bigdata.dir}/bigdata-blueprints/src/test">
- <exclude name="**/*.java" />
- </fileset>
</jar>
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <dme...@us...> - 2014-05-08 20:50:15
|
Revision: 8238
http://sourceforge.net/p/bigdata/code/8238
Author: dmekonnen
Date: 2014-05-08 20:50:11 +0000 (Thu, 08 May 2014)
Log Message:
-----------
correction to CWD for ant builds
Modified Paths:
--------------
branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/recipes/nss.rb
branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/recipes/tomcat.rb
Modified: branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/recipes/nss.rb
===================================================================
--- branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/recipes/nss.rb 2014-05-08 20:33:22 UTC (rev 8237)
+++ branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/recipes/nss.rb 2014-05-08 20:50:11 UTC (rev 8238)
@@ -33,7 +33,7 @@
execute "build the nss tar ball" do
user 'ubuntu'
group 'ubuntu'
- cwd node['bigdata'][:source]
+ cwd "/home/ubuntu/#{node['bigdata'][:source]}"
command "ant package-brew-nss"
end
else
Modified: branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/recipes/tomcat.rb
===================================================================
--- branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/recipes/tomcat.rb 2014-05-08 20:33:22 UTC (rev 8237)
+++ branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/recipes/tomcat.rb 2014-05-08 20:50:11 UTC (rev 8238)
@@ -39,7 +39,7 @@
execute "build the war file" do
user 'ubuntu'
group 'ubuntu'
- cwd node['bigdata'][:source]
+ cwd "/home/ubuntu/#{node['bigdata'][:source]}"
command "ant war"
end
@@ -47,7 +47,7 @@
# Install the WAR file:
#
remote_file "#{node['tomcat'][:webapp_dir]}/bigdata.war" do
- source "#{node['bigdata'][:source]}/ant-build/bigdata.war"
+ source "file:///home/ubuntu/#{node['bigdata'][:source]}/ant-build/bigdata.war"
owner node['tomcat'][:user]
group node['tomcat'][:group]
end
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <dme...@us...> - 2014-05-08 20:33:26
|
Revision: 8237
http://sourceforge.net/p/bigdata/code/8237
Author: dmekonnen
Date: 2014-05-08 20:33:22 +0000 (Thu, 08 May 2014)
Log Message:
-----------
snpshot of deployment recipes that build from svn
Modified Paths:
--------------
branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/Berksfile
branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/attributes/default.rb
branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/metadata.rb
branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/recipes/nss.rb
branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/recipes/tomcat.rb
branches/DEPLOYMENT_BRANCH_1_3_1/build.xml
Modified: branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/Berksfile
===================================================================
--- branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/Berksfile 2014-05-08 19:10:31 UTC (rev 8236)
+++ branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/Berksfile 2014-05-08 20:33:22 UTC (rev 8237)
@@ -2,6 +2,8 @@
cookbook "apt"
cookbook "java", '~> 1.22.0'
+cookbook "ant"
cookbook "tomcat"
+cookbook "subversion"
metadata
Modified: branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/attributes/default.rb
===================================================================
--- branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/attributes/default.rb 2014-05-08 19:10:31 UTC (rev 8236)
+++ branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/attributes/default.rb 2014-05-08 20:33:22 UTC (rev 8237)
@@ -1,6 +1,4 @@
-# default['bigdata'][:url] = "http://sourceforge.net/projects/bigdata/files/bigdata/1.3.0/bigdata.war/download"
-default['bigdata'][:url] = "http://softlayer-dal.dl.sourceforge.net/project/bigdata/bigdata/1.3.0/bigdata.war"
default['bigdata'][:home] = "/var/lib/bigdata"
# Who runs bigdata?
@@ -9,6 +7,7 @@
default['bigdata'][:properties] = default['bigdata'][:home] + "RWStore.properties"
+default['bigdata'][:source] = "bigdata-code"
case node['bigdata'][:install_type]
when "nss"
@@ -22,14 +21,16 @@
# Where the bigdata-ha.jnl file will live:
default['bigdata'][:data_dir] = node['bigdata'][:home] + "/var/data"
-when "nss_svn"
- default['bigdata'][:url] = "http://bigdata.com/deploy/bigdata-1.3.0.tgz"
-when "tomcat_svn"
- default['bigdata'][:svn_branch] = "https://svn.code.sf.net/p/bigdata/code/branches/BIGDATA_RELEASE_1_3_0"
-else
+
+ if node['bigdata'][:build_from_svn]
+ default['bigdata'][:svn_branch] = "https://svn.code.sf.net/p/bigdata/code/branches/BIGDATA_RELEASE_1_3_0"
+ end
+when "tomcat"
default['tomcat'][:base_version] = 7
default['tomcat'][:java_options] = "-Djava.awt.headless=true -server -Xmx4G -XX:+UseG1GC"
+ default['bigdata'][:url] = "http://softlayer-dal.dl.sourceforge.net/project/bigdata/bigdata/1.3.0/bigdata.war"
+
default['bigdata'][:web_home] = default['tomcat'][:webapp_dir] + "/bigdata"
default['bigdata'][:web_xml] = default['bigdata'][:web_home] + "/WEB-INF/web.xml"
default['bigdata'][:log4j_properties] = default['bigdata'][:web_home] + "/WEB-INF/classes/log4j.properties"
@@ -39,6 +40,10 @@
# Where the log files will live:
default['bigdata'][:log_dir] = node['bigdata'][:home] + "/log"
+
+ if node['bigdata'][:build_from_svn]
+ default['bigdata'][:svn_branch] = "https://svn.code.sf.net/p/bigdata/code/branches/BIGDATA_RELEASE_1_3_0"
+ end
end
Modified: branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/metadata.rb
===================================================================
--- branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/metadata.rb 2014-05-08 19:10:31 UTC (rev 8236)
+++ branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/metadata.rb 2014-05-08 20:33:22 UTC (rev 8237)
@@ -7,4 +7,6 @@
version '0.1.1'
depends 'apt'
depends 'java', '>= 1.22.0'
+depends 'ant'
depends 'tomcat'
+depends 'subversion'
Modified: branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/recipes/nss.rb
===================================================================
--- branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/recipes/nss.rb 2014-05-08 19:10:31 UTC (rev 8236)
+++ branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/recipes/nss.rb 2014-05-08 20:33:22 UTC (rev 8237)
@@ -18,18 +18,39 @@
action :create
end
- #
- # Retrieve the package prepared for Brew:
- #
- remote_file "/tmp/bigdata.tgz" do
- owner node['bigdata'][:user]
- group node['bigdata'][:group]
- source node['bigdata'][:url]
- end
- execute "Extract and relocate the bigdata archive" do
- cwd "/var/lib"
- command "tar xvf /tmp/bigdata.tgz"
+ if node['bigdata'][:build_from_svn]
+ include_recipe "ant"
+ include_recipe "subversion"
+
+ execute "checkout bigdata from svn repo" do
+ user 'ubuntu'
+ group 'ubuntu'
+ cwd "/home/ubuntu"
+ command "svn checkout #{node['bigdata'][:svn_branch]} #{node['bigdata'][:source]}"
+ end
+
+ execute "build the nss tar ball" do
+ user 'ubuntu'
+ group 'ubuntu'
+ cwd node['bigdata'][:source]
+ command "ant package-brew-nss"
+ end
+ else
+ #
+ # Retrieve the package prepared for Brew:
+ #
+ remote_file "/tmp/bigdata.tgz" do
+ owner node['bigdata'][:user]
+ group node['bigdata'][:group]
+ source node['bigdata'][:url]
+ end
+
+ execute "Extract and relocate the bigdata archive" do
+ cwd "/var/lib"
+ command "tar xvf /tmp/bigdata.tgz"
+ end
+
end
Modified: branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/recipes/tomcat.rb
===================================================================
--- branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/recipes/tomcat.rb 2014-05-08 19:10:31 UTC (rev 8236)
+++ branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/recipes/tomcat.rb 2014-05-08 20:33:22 UTC (rev 8237)
@@ -25,16 +25,44 @@
end
- #
- # Install the WAR file:
- #
- remote_file "#{node['tomcat'][:webapp_dir]}/bigdata.war" do
- source node['bigdata'][:url]
- owner node['tomcat'][:user]
- group node['tomcat'][:group]
+ if node['bigdata'][:build_from_svn]
+ include_recipe "ant"
+ include_recipe "subversion"
+
+ execute "checkout bigdata from svn repo" do
+ user 'ubuntu'
+ group 'ubuntu'
+ cwd "/home/ubuntu"
+ command "svn checkout #{node['bigdata'][:svn_branch]} #{node['bigdata'][:source]}"
+ end
+
+ execute "build the war file" do
+ user 'ubuntu'
+ group 'ubuntu'
+ cwd node['bigdata'][:source]
+ command "ant war"
+ end
+
+ #
+ # Install the WAR file:
+ #
+ remote_file "#{node['tomcat'][:webapp_dir]}/bigdata.war" do
+ source "#{node['bigdata'][:source]}/ant-build/bigdata.war"
+ owner node['tomcat'][:user]
+ group node['tomcat'][:group]
+ end
+
+ else
+ #
+ # Install the WAR file:
+ #
+ remote_file "#{node['tomcat'][:webapp_dir]}/bigdata.war" do
+ source node['bigdata'][:url]
+ owner node['tomcat'][:user]
+ group node['tomcat'][:group]
+ end
end
-
#
# Create the JNL home directory
#
Modified: branches/DEPLOYMENT_BRANCH_1_3_1/build.xml
===================================================================
--- branches/DEPLOYMENT_BRANCH_1_3_1/build.xml 2014-05-08 19:10:31 UTC (rev 8236)
+++ branches/DEPLOYMENT_BRANCH_1_3_1/build.xml 2014-05-08 20:33:22 UTC (rev 8237)
@@ -1219,6 +1219,21 @@
src="http://wiki.bigdata.com/wiki/index.php/NanoSparqlServer?printable=yes"
/>
+ <!-- Stage files specific to NSS deployments provided by Brew and Chef. -->
+ <chmod file="${dist.bin}/bigdata" perm="755" />
+ <copy file="${src.resources}/deployment/nss/bin/bigdataNSS"
+ todir="${dist.bin}" />
+ <chmod file="${dist.bin}/bigdata" perm="755" />
+ <copy file="${src.resources}/deployment/nss/bin/startNSS"
+ todir="${dist.bin}" />
+ <chmod file="${dist.bin}/startNSS" perm="755" />
+ <copy file="${src.resources}/deployment/nss/etc/jetty.xml"
+ todir="${dist.var.jetty}/etc" />
+ <copy file="${src.resources}/deployment/nss/WEB-INF/RWStore.properties"
+ todir="${dist.var.jetty}/WEB-INF" />
+ <copy file="${src.resources}/deployment/nss/WEB-INF/classes/log4j.properties"
+ todir="${dist.var.jetty}/WEB-INF/classes" />
+
</target>
<!-- -->
@@ -1298,6 +1313,41 @@
</target>
+ <target name="package-brew-nss" depends="clean, stage"
+ description="Create compressed tar file for Jetty based deployment via Brew and Chef installers.">
+
+ <tar destfile="${bigdata.dir}/REL-NSS.${version}.tgz"
+ compression="gzip">
+
+ <tarfileset dir="${bigdata.dir}/dist">
+ <include name="bigdata/doc/**" />
+ <exclude name="bigdata/doc/HAJournalServer.html" />
+ <include name="bigdata/lib/**" />
+ <exclude name="bigdata/lib/bigdata-ganglia.jar" />
+ <exclude name="bigdata/lib/browser.jar" />
+ <exclude name="bigdata/lib/reggie.jar" />
+ <exclude name="bigdata/lib/zookeeper.jar" />
+ <exclude name="bigdata/lib/jsk-*.jar" />
+ <exclude name="bigdata/lib-dl" />
+ <exclude name="bigdata/lib-ext" />
+ <include name="bigdata/var/jetty/**" />
+ <include name="bigdata/var/config/logging/logging.properties" />
+ <exclude name="bigdata/var/jetty/jetty.xml" />
+ <exclude name="bigdata/var/jetty/html/new.html" />
+ <exclude name="bigdata/var/jetty/html/old.html" />
+ </tarfileset>
+
+ <!-- Add scripts separately, making them executable -->
+
+ <tarfileset dir="${bigdata.dir}/dist" filemode="755">
+ <include name="bigdata/bin/bigdataNSS" />
+ <include name="bigdata/bin/startNSS" />
+ </tarfileset>
+ </tar>
+
+ </target>
+
+
<!-- FIXME DEBUG and add 'depends="javadoc, stage" (should stage stage javadoc?)' -->
<!-- Note: can require 'rpm' and 'rpm-build. -->
<!-- TODO: We do not need both this and "deploy-artifact". -->
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <dme...@us...> - 2014-05-08 19:10:34
|
Revision: 8236
http://sourceforge.net/p/bigdata/code/8236
Author: dmekonnen
Date: 2014-05-08 19:10:31 +0000 (Thu, 08 May 2014)
Log Message:
-----------
format beautification
Modified Paths:
--------------
branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/recipes/nss.rb
Modified: branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/recipes/nss.rb
===================================================================
--- branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/recipes/nss.rb 2014-05-08 19:06:32 UTC (rev 8235)
+++ branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/recipes/nss.rb 2014-05-08 19:10:31 UTC (rev 8236)
@@ -4,18 +4,18 @@
#
if node['bigdata'][:install_type] == "nss"
- group "bigdata" do
+ group node['bigdata'][:group] do
action :create
append true
end
- user "#{node['bigdata'][:user]}" do
- gid "#{node['bigdata'][:group]}"
+ user node['bigdata'][:user] do
+ gid node['bigdata'][:group]
supports :manage_home => true
- shell "/bin/false"
- home "#{node['bigdata'][:home]}"
- system true
- action :create
+ shell "/bin/false"
+ home node['bigdata'][:home]
+ system true
+ action :create
end
#
@@ -28,16 +28,16 @@
end
execute "Extract and relocate the bigdata archive" do
- cwd "/var/lib"
- command "tar xvf /tmp/bigdata.tgz"
+ cwd "/var/lib"
+ command "tar xvf /tmp/bigdata.tgz"
end
execute "change the ownership of the bigdata home directory to bigdata, which strangely is not" do
- user "root"
- group "root"
- cwd "#{node['bigdata'][:home]}"
- command "chown -R #{node['bigdata'][:user]}:#{node['bigdata'][:group]} ."
+ user "root"
+ group "root"
+ cwd node['bigdata'][:home]
+ command "chown -R #{node['bigdata'][:user]}:#{node['bigdata'][:group]} ."
end
link "/etc/init.d/bigdataNSS" do
@@ -80,6 +80,6 @@
#
# supports :status => true, :start => true, :stop => true, :restart => true
supports :start => true, :stop => true, :restart => true
- action [ :start, :enable ]
+ action [ :enable, :start ]
end
end
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
|
From: <dme...@us...> - 2014-05-08 19:06:34
|
Revision: 8235
http://sourceforge.net/p/bigdata/code/8235
Author: dmekonnen
Date: 2014-05-08 19:06:32 +0000 (Thu, 08 May 2014)
Log Message:
-----------
cleanup
Modified Paths:
--------------
branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/Vagrantfile
branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/Vagrantfile.aws
Modified: branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/Vagrantfile
===================================================================
--- branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/Vagrantfile 2014-05-08 18:33:04 UTC (rev 8234)
+++ branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/Vagrantfile 2014-05-08 19:06:32 UTC (rev 8235)
@@ -32,25 +32,22 @@
config.vm.provider :aws do |aws, override|
override.vm.box = "dummy"
- aws.access_key_id = "AKIAJ26S27XQRS5LFXCQ"
- aws.secret_access_key = "BPBric3lzzE9lHV3Hwz+vG9TQ/e1fOugytYz1LFV"
- aws.keypair_name = "systap"
+ aws.access_key_id = ENV['AWS_ACCESS_KEY_ID']
+ aws.secret_access_key = ENV['AWS_SECRET_ACCESS_KEY']
+ aws.keypair_name = ENV['AWS_KEYPAIR_NAME']
- aws.ami = "ami-a73264ce"
+ aws.ami = ENV['AWS_AMI']
- #
- #
- #
- aws.region = "us-east-1"
- aws.instance_type = "t1.micro"
- aws.security_groups = [ "launch-wizard-4" ]
+ aws.region = ENV['AWS_REGION']
+ aws.instance_type = ENV['AWS_INSTANCE_TYPE']
+ aws.security_groups = [ ENV['AWS_SECURITY_GROUPS'], ENV['AWS_SECURITY_GROUP_PRIVATE'] ]
aws.tags = {
- 'Name' => 'Systap Bigdata'
+ 'Name' => ENV['BIGDATA_HOST_NAME']
}
- override.ssh.username = "ubuntu"
- override.ssh.private_key_path = "/Users/dmekonnen/.ssh/systap.pem"
+ override.ssh.username = ENV['AWS_AMI_USERNAME']
+ override.ssh.private_key_path = ENV['AWS_SSH_PRIVATE_KEY']
end
@@ -70,8 +67,8 @@
}
}
- # config.vm.provision :shell, inline: "sudo apt-get update ; sudo curl -L https://www.opscode.com/chef/install.sh | sudo bash"
- config.vm.provision :shell, inline: "sudo apt-get update"
+ config.vm.provision :shell, inline: "sudo apt-get update ; sudo curl -L https://www.opscode.com/chef/install.sh | sudo bash"
+ # config.vm.provision :shell, inline: "sudo apt-get update"
chef.run_list = [
"recipe[bigdata::nss]"
Modified: branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/Vagrantfile.aws
===================================================================
--- branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/Vagrantfile.aws 2014-05-08 18:33:04 UTC (rev 8234)
+++ branches/DEPLOYMENT_BRANCH_1_3_1/bigdata/src/resources/deployment/vagrant/systap-bigdata/Vagrantfile.aws 2014-05-08 19:06:32 UTC (rev 8235)
@@ -10,7 +10,7 @@
# please see the online documentation at vagrantup.com.
config.vm.box = "dummy"
- config.vm.hostname = "systap-bigdata"
+ config.vm.hostname = "bigdata"
config.berkshelf.enabled = true
@@ -23,25 +23,22 @@
# config.berkshelf.except = []
config.vm.provider :aws do |aws, override|
- aws.access_key_id = "AKIAJ26S27XQRS5LFXCQ"
- aws.secret_access_key = "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"
- aws.keypair_name = "systap"
+ aws.access_key_id = ENV['AWS_ACCESS_KEY_ID']
+ aws.secret_access_key = ENV['AWS_SECRET_ACCESS_KEY']
+ aws.keypair_name = ENV['AWS_KEYPAIR_NAME']
- aws.ami = "ami-a73264ce"
+ aws.ami = ENV['AWS_AMI']
- #
- #
- #
- aws.region = "us-east-1"
- aws.instance_type = "t1.micro"
- aws.security_groups = [ "launch-wizard-4" ]
+ aws.region = ENV['AWS_REGION']
+ aws.instance_type = ENV['AWS_INSTANCE_TYPE']
+ aws.security_groups = [ ENV['AWS_SECURITY_GROUPS'], ENV['AWS_SECURITY_GROUP_PRIVATE'] ]
aws.tags = {
- 'Name' => 'Systap Bigdata'
+ 'Name' => ENV['BIGDATA_HOST_NAME']
}
- override.ssh.username = "ubuntu"
- override.ssh.private_key_path = "/Users/dmekonnen/.ssh/systap.pem"
+ override.ssh.username = ENV['AWS_AMI_USERNAME']
+ override.ssh.private_key_path = ENV['AWS_SSH_PRIVATE_KEY']
end
config.vm.provision :chef_solo do |chef|
This was sent by the SourceForge.net collaborative development platform, the world's largest Open Source development site.
|
4 messages has been excluded from this view by a project administrator.
