--- /dev/null
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<html>
+ <head>
+ <META http-equiv="Content-Type" content="text/html; charset=UTF-8">
+ <title>
+ Apache Lucene ICU integration module
+ </title>
+ </head>
+<body>
+<p>
+This module exposes functionality from
+<a href="http://site.icu-project.org/">ICU</a> to Apache Lucene. ICU4J is a Java
+library that enhances Java's internationalization support by improving
+performance, keeping current with the Unicode Standard, and providing richer
+APIs. This module exposes the following functionality:
+</p>
+<ul>
+ <li><a href="#segmentation">Text Segmentation</a>: Tokenizes text based on
+ properties and rules defined in Unicode.</li>
+ <li><a href="#collation">Collation</a>: Compare strings according to the
+ conventions and standards of a particular language, region or country.</li>
+ <li><a href="#normalization">Normalization</a>: Converts text to a unique,
+ equivalent form.</li>
+ <li><a href="#casefolding">Case Folding</a>: Removes case distinctions with
+ Unicode's Default Caseless Matching algorithm.</li>
+ <li><a href="#searchfolding">Search Term Folding</a>: Removes distinctions
+ (such as accent marks) between similar characters for a loose or fuzzy search.</li>
+ <li><a href="#transform">Text Transformation</a>: Transforms Unicode text in
+ a context-sensitive fashion: e.g. mapping Traditional to Simplified Chinese</li>
+</ul>
+<hr/>
+<h1><a name="segmentation">Text Segmentation</a></h1>
+<p>
+Text Segmentation (Tokenization) divides document and query text into index terms
+(typically words). Unicode provides special properties and rules so that this can
+be done in a manner that works well with most languages.
+</p>
+<p>
+Text Segmentation implements the word segmentation specified in
+<a href="http://unicode.org/reports/tr29/">Unicode Text Segmentation</a>.
+Additionally the algorithm can be tailored based on writing system, for example
+text in the Thai script is automatically delegated to a dictionary-based segmentation
+algorithm.
+</p>
+<h2>Use Cases</h2>
+<ul>
+ <li>
+ As a more thorough replacement for StandardTokenizer that works well for
+ most languages.
+ </li>
+</ul>
+<h2>Example Usages</h2>
+<h3>Tokenizing multilanguage text</h3>
+<pre class="prettyprint">
+ /**
+ * This tokenizer will work well in general for most languages.
+ */
+ Tokenizer tokenizer = new ICUTokenizer(reader);
+</pre>
+<hr/>
+<h1><a name="collation">Collation</a></h1>
+<p>
+ <code>ICUCollationKeyFilter</code>
+ converts each token into its binary <code>CollationKey</code> using the
+ provided <code>Collator</code>, and then encode the <code>CollationKey</code>
+ as a String using
+ {@link org.apache.lucene.util.IndexableBinaryStringTools}, to allow it to be
+ stored as an index term.
+</p>
+<p>
+ <code>ICUCollationKeyFilter</code> depends on ICU4J 4.4 to produce the
+ <code>CollationKey</code>s. <code>icu4j-4.4.jar</code>
+ is included in Lucene's Subversion repository at <code>contrib/icu/lib/</code>.
+</p>
+
+<h2>Use Cases</h2>
+
+<ul>
+ <li>
+ Efficient sorting of terms in languages that use non-Unicode character
+ orderings. (Lucene Sort using a Locale can be very slow.)
+ </li>
+ <li>
+ Efficient range queries over fields that contain terms in languages that
+ use non-Unicode character orderings. (Range queries using a Locale can be
+ very slow.)
+ </li>
+ <li>
+ Effective Locale-specific normalization (case differences, diacritics, etc.).
+ ({@link org.apache.lucene.analysis.LowerCaseFilter} and
+ {@link org.apache.lucene.analysis.ASCIIFoldingFilter} provide these services
+ in a generic way that doesn't take into account locale-specific needs.)
+ </li>
+</ul>
+
+<h2>Example Usages</h2>
+
+<h3>Farsi Range Queries</h3>
+<pre class="prettyprint">
+ Collator collator = Collator.getInstance(new Locale("ar"));
+ ICUCollationKeyAnalyzer analyzer = new ICUCollationKeyAnalyzer(collator);
+ RAMDirectory ramDir = new RAMDirectory();
+ IndexWriter writer = new IndexWriter
+ (ramDir, analyzer, true, IndexWriter.MaxFieldLength.LIMITED);
+ Document doc = new Document();
+ doc.add(new Field("content", "\u0633\u0627\u0628",
+ Field.Store.YES, Field.Index.ANALYZED));
+ writer.addDocument(doc);
+ writer.close();
+ IndexSearcher is = new IndexSearcher(ramDir, true);
+
+ // The AnalyzingQueryParser in Lucene's contrib allows terms in range queries
+ // to be passed through an analyzer - Lucene's standard QueryParser does not
+ // allow this.
+ AnalyzingQueryParser aqp = new AnalyzingQueryParser("content", analyzer);
+ aqp.setLowercaseExpandedTerms(false);
+
+ // Unicode order would include U+0633 in [ U+062F - U+0698 ], but Farsi
+ // orders the U+0698 character before the U+0633 character, so the single
+ // indexed Term above should NOT be returned by a ConstantScoreRangeQuery
+ // with a Farsi Collator (or an Arabic one for the case when Farsi is not
+ // supported).
+ ScoreDoc[] result
+ = is.search(aqp.parse("[ \u062F TO \u0698 ]"), null, 1000).scoreDocs;
+ assertEquals("The index Term should not be included.", 0, result.length);
+</pre>
+
+<h3>Danish Sorting</h3>
+<pre class="prettyprint">
+ Analyzer analyzer
+ = new ICUCollationKeyAnalyzer(Collator.getInstance(new Locale("da", "dk")));
+ RAMDirectory indexStore = new RAMDirectory();
+ IndexWriter writer = new IndexWriter
+ (indexStore, analyzer, true, IndexWriter.MaxFieldLength.LIMITED);
+ String[] tracer = new String[] { "A", "B", "C", "D", "E" };
+ String[] data = new String[] { "HAT", "HUT", "H\u00C5T", "H\u00D8T", "HOT" };
+ String[] sortedTracerOrder = new String[] { "A", "E", "B", "D", "C" };
+ for (int i = 0 ; i < data.length ; ++i) {
+ Document doc = new Document();
+ doc.add(new Field("tracer", tracer[i], Field.Store.YES, Field.Index.NO));
+ doc.add(new Field("contents", data[i], Field.Store.NO, Field.Index.ANALYZED));
+ writer.addDocument(doc);
+ }
+ writer.close();
+ Searcher searcher = new IndexSearcher(indexStore, true);
+ Sort sort = new Sort();
+ sort.setSort(new SortField("contents", SortField.STRING));
+ Query query = new MatchAllDocsQuery();
+ ScoreDoc[] result = searcher.search(query, null, 1000, sort).scoreDocs;
+ for (int i = 0 ; i < result.length ; ++i) {
+ Document doc = searcher.doc(result[i].doc);
+ assertEquals(sortedTracerOrder[i], doc.getValues("tracer")[0]);
+ }
+</pre>
+
+<h3>Turkish Case Normalization</h3>
+<pre class="prettyprint">
+ Collator collator = Collator.getInstance(new Locale("tr", "TR"));
+ collator.setStrength(Collator.PRIMARY);
+ Analyzer analyzer = new ICUCollationKeyAnalyzer(collator);
+ RAMDirectory ramDir = new RAMDirectory();
+ IndexWriter writer = new IndexWriter
+ (ramDir, analyzer, true, IndexWriter.MaxFieldLength.LIMITED);
+ Document doc = new Document();
+ doc.add(new Field("contents", "DIGY", Field.Store.NO, Field.Index.ANALYZED));
+ writer.addDocument(doc);
+ writer.close();
+ IndexSearcher is = new IndexSearcher(ramDir, true);
+ QueryParser parser = new QueryParser("contents", analyzer);
+ Query query = parser.parse("d\u0131gy"); // U+0131: dotless i
+ ScoreDoc[] result = is.search(query, null, 1000).scoreDocs;
+ assertEquals("The index Term should be included.", 1, result.length);
+</pre>
+
+<h2>Caveats and Comparisons</h2>
+<p>
+ <strong>WARNING:</strong> Make sure you use exactly the same
+ <code>Collator</code> at index and query time -- <code>CollationKey</code>s
+ are only comparable when produced by
+ the same <code>Collator</code>. Since {@link java.text.RuleBasedCollator}s
+ are not independently versioned, it is unsafe to search against stored
+ <code>CollationKey</code>s unless the following are exactly the same (best
+ practice is to store this information with the index and check that they
+ remain the same at query time):
+</p>
+<ol>
+ <li>JVM vendor</li>
+ <li>JVM version, including patch version</li>
+ <li>
+ The language (and country and variant, if specified) of the Locale
+ used when constructing the collator via
+ {@link java.text.Collator#getInstance(java.util.Locale)}.
+ </li>
+ <li>
+ The collation strength used - see {@link java.text.Collator#setStrength(int)}
+ </li>
+</ol>
+<p>
+ <code>ICUCollationKeyFilter</code> uses ICU4J's <code>Collator</code>, which
+ makes its version available, thus allowing collation to be versioned
+ independently from the JVM. <code>ICUCollationKeyFilter</code> is also
+ significantly faster and generates significantly shorter keys than
+ <code>CollationKeyFilter</code>. See
+ <a href="http://site.icu-project.org/charts/collation-icu4j-sun"
+ >http://site.icu-project.org/charts/collation-icu4j-sun</a> for key
+ generation timing and key length comparisons between ICU4J and
+ <code>java.text.Collator</code> over several languages.
+</p>
+<p>
+ <code>CollationKey</code>s generated by <code>java.text.Collator</code>s are
+ not compatible with those those generated by ICU Collators. Specifically, if
+ you use <code>CollationKeyFilter</code> to generate index terms, do not use
+ <code>ICUCollationKeyFilter</code> on the query side, or vice versa.
+</p>
+<hr/>
+<h1><a name="normalization">Normalization</a></h1>
+<p>
+ <code>ICUNormalizer2Filter</code> normalizes term text to a
+ <a href="http://unicode.org/reports/tr15/">Unicode Normalization Form</a>, so
+ that <a href="http://en.wikipedia.org/wiki/Unicode_equivalence">equivalent</a>
+ forms are standardized to a unique form.
+</p>
+<h2>Use Cases</h2>
+<ul>
+ <li> Removing differences in width for Asian-language text.
+ </li>
+ <li> Standardizing complex text with non-spacing marks so that characters are
+ ordered consistently.
+ </li>
+</ul>
+<h2>Example Usages</h2>
+<h3>Normalizing text to NFC</h3>
+<pre class="prettyprint">
+ /**
+ * Normalizer2 objects are unmodifiable and immutable.
+ */
+ Normalizer2 normalizer = Normalizer2.getInstance(null, "nfc", Normalizer2.Mode.COMPOSE);
+ /**
+ * This filter will normalize to NFC.
+ */
+ TokenStream tokenstream = new ICUNormalizer2Filter(tokenizer, normalizer);
+</pre>
+<hr/>
+<h1><a name="casefolding">Case Folding</a></h1>
+<p>
+Default caseless matching, or case-folding is more than just conversion to
+lowercase. For example, it handles cases such as the Greek sigma, so that
+"Μάϊος" and "ΜΆΪΟΣ" will match correctly.
+</p>
+<p>
+Case-folding is still only an approximation of the language-specific rules
+governing case. If the specific language is known, consider using
+ICUCollationKeyFilter and indexing collation keys instead. This implementation
+performs the "full" case-folding specified in the Unicode standard, and this
+may change the length of the term. For example, the German ß is case-folded
+to the string 'ss'.
+</p>
+<p>
+Case folding is related to normalization, and as such is coupled with it in
+this integration. To perform case-folding, you use normalization with the form
+"nfkc_cf" (which is the default).
+</p>
+<h2>Use Cases</h2>
+<ul>
+ <li>
+ As a more thorough replacement for LowerCaseFilter that has good behavior
+ for most languages.
+ </li>
+</ul>
+<h2>Example Usages</h2>
+<h3>Lowercasing text</h3>
+<pre class="prettyprint">
+ /**
+ * This filter will case-fold and normalize to NFKC.
+ */
+ TokenStream tokenstream = new ICUNormalizer2Filter(tokenizer);
+</pre>
+<hr/>
+<h1><a name="searchfolding">Search Term Folding</a></h1>
+<p>
+Search term folding removes distinctions (such as accent marks) between
+similar characters. It is useful for a fuzzy or loose search.
+</p>
+<p>
+Search term folding implements many of the foldings specified in
+<a href="http://www.unicode.org/reports/tr30/tr30-4.html">Character Foldings</a>
+as a special normalization form. This folding applies NFKC, Case Folding, and
+many character foldings recursively.
+</p>
+<h2>Use Cases</h2>
+<ul>
+ <li>
+ As a more thorough replacement for ASCIIFoldingFilter and LowerCaseFilter
+ that applies the same ideas to many more languages.
+ </li>
+</ul>
+<h2>Example Usages</h2>
+<h3>Removing accents</h3>
+<pre class="prettyprint">
+ /**
+ * This filter will case-fold, remove accents and other distinctions, and
+ * normalize to NFKC.
+ */
+ TokenStream tokenstream = new ICUFoldingFilter(tokenizer);
+</pre>
+<hr/>
+<h1><a name="transform">Text Transformation</a></h1>
+<p>
+ICU provides text-transformation functionality via its Transliteration API. This allows
+you to transform text in a variety of ways, taking context into account.
+</p>
+<p>
+For more information, see the
+<a href="http://userguide.icu-project.org/transforms/general">User's Guide</a>
+and
+<a href="http://userguide.icu-project.org/transforms/general/rules">Rule Tutorial</a>.
+</p>
+<h2>Use Cases</h2>
+<ul>
+ <li>
+ Convert Traditional to Simplified
+ </li>
+ <li>
+ Transliterate between different writing systems: e.g. Romanization
+ </li>
+</ul>
+<h2>Example Usages</h2>
+<h3>Convert Traditional to Simplified</h3>
+<pre class="prettyprint">
+ /**
+ * This filter will map Traditional Chinese to Simplified Chinese
+ */
+ TokenStream tokenstream = new ICUTransformFilter(tokenizer, Transliterator.getInstance("Traditional-Simplified"));
+</pre>
+<h3>Transliterate Serbian Cyrillic to Serbian Latin</h3>
+<pre class="prettyprint">
+ /**
+ * This filter will map Serbian Cyrillic to Serbian Latin according to BGN rules
+ */
+ TokenStream tokenstream = new ICUTransformFilter(tokenizer, Transliterator.getInstance("Serbian-Latin/BGN"));
+</pre>
+<hr/>
+<h1><a name="backcompat">Backwards Compatibility</a></h1>
+<p>
+This module exists to provide up-to-date Unicode functionality that supports
+the most recent version of Unicode (currently 6.0). However, some users who wish
+for stronger backwards compatibility can restrict
+{@link org.apache.lucene.analysis.icu.ICUNormalizer2Filter} to operate on only
+a specific Unicode Version by using a {@link com.ibm.icu.text.FilteredNormalizer2}.
+</p>
+<h2>Example Usages</h2>
+<h3>Restricting normalization to Unicode 5.0</h3>
+<pre class="prettyprint">
+ /**
+ * This filter will do NFC normalization, but will ignore any characters that
+ * did not exist as of Unicode 5.0. Because of the normalization stability policy
+ * of Unicode, this is an easy way to force normalization to a specific version.
+ */
+ Normalizer2 normalizer = Normalizer2.getInstance(null, "nfc", Normalizer2.Mode.COMPOSE);
+ UnicodeSet set = new UnicodeSet("[:age=5.0:]");
+ // see FilteredNormalizer2 docs, the set should be frozen or performance will suffer
+ set.freeze();
+ FilteredNormalizer2 unicode50 = new FilteredNormalizer2(normalizer, set);
+ TokenStream tokenstream = new ICUNormalizer2Filter(tokenizer, unicode50);
+</pre>
+</body>
+</html>
fnp / pylucene.git / blobdiff