--- /dev/null
+<!doctype html public "-//w3c//dtd html 4.0 transitional//en">
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<html>
+<head>
+ <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
+</head>
+<body>
+<p>The <code>org.apache.lucene.analysis.standard</code> package contains three
+ fast grammar-based tokenizers constructed with JFlex:</p>
+<ul>
+ <li><code><a href="StandardTokenizer.html">StandardTokenizer</a></code>:
+ as of Lucene 3.1, implements the Word Break rules from the Unicode Text
+ Segmentation algorithm, as specified in
+ <a href="http://unicode.org/reports/tr29/">Unicode Standard Annex #29</a>.
+ Unlike <code>UAX29URLEmailTokenizer</code>, URLs and email addresses are
+ <b>not</b> tokenized as single tokens, but are instead split up into
+ tokens according to the UAX#29 word break rules.
+ <br/>
+ <code><a href="StandardAnalyzer">StandardAnalyzer</a></code> includes
+ <code>StandardTokenizer</code>,
+ <code><a href="StandardFilter">StandardFilter</a></code>,
+ <code><a href="../../../../../../all/org/apache/lucene/analysis/LowerCaseFilter.html">LowerCaseFilter</a></code>
+ and <code><a href="../../../../../../all/org/apache/lucene/analysis/StopFilter.html">StopFilter</a></code>.
+ When the <code>Version</code> specified in the constructor is lower than
+ 3.1, the <code><a href="ClassicTokenizer.html">ClassicTokenizer</a></code>
+ implementation is invoked.</li>
+ <li><code><a href="ClassicTokenizer.html">ClassicTokenizer</a></code>:
+ this class was formerly (prior to Lucene 3.1) named
+ <code>StandardTokenizer</code>. (Its tokenization rules are not
+ based on the Unicode Text Segmentation algorithm.)
+ <code><a href="ClassicAnalyzer">ClassicAnalyzer</a></code> includes
+ <code>ClassicTokenizer</code>,
+ <code><a href="StandardFilter">StandardFilter</a></code>,
+ <code><a href="../../../../../../all/org/apache/lucene/analysis/LowerCaseFilter.html">LowerCaseFilter</a></code>
+ and <code><a href="../../../../../../all/org/apache/lucene/analysis/StopFilter.html">StopFilter</a></code>.
+ </li>
+ <li><code><a href="UAX29URLEmailTokenizer.html">UAX29URLEmailTokenizer</a></code>:
+ implements the Word Break rules from the Unicode Text Segmentation
+ algorithm, as specified in
+ <a href="http://unicode.org/reports/tr29/">Unicode Standard Annex #29</a>.
+ URLs and email addresses are also tokenized according to the relevant RFCs.
+ </li>
+</ul>
+</body>
+</html>
fnp / pylucene.git / blobdiff