1 <!doctype html public "-//w3c//dtd html 4.0 transitional//en">
3 Licensed to the Apache Software Foundation (ASF) under one or more
4 contributor license agreements. See the NOTICE file distributed with
5 this work for additional information regarding copyright ownership.
6 The ASF licenses this file to You under the Apache License, Version 2.0
7 (the "License"); you may not use this file except in compliance with
8 the License. You may obtain a copy of the License at
10 http://www.apache.org/licenses/LICENSE-2.0
12 Unless required by applicable law or agreed to in writing, software
13 distributed under the License is distributed on an "AS IS" BASIS,
14 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15 See the License for the specific language governing permissions and
16 limitations under the License.
20 <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
23 <p>The <code>org.apache.lucene.analysis.standard</code> package contains three
24 fast grammar-based tokenizers constructed with JFlex:</p>
26 <li><code><a href="StandardTokenizer.html">StandardTokenizer</a></code>:
27 as of Lucene 3.1, implements the Word Break rules from the Unicode Text
28 Segmentation algorithm, as specified in
29 <a href="http://unicode.org/reports/tr29/">Unicode Standard Annex #29</a>.
30 Unlike <code>UAX29URLEmailTokenizer</code>, URLs and email addresses are
31 <b>not</b> tokenized as single tokens, but are instead split up into
32 tokens according to the UAX#29 word break rules.
34 <code><a href="StandardAnalyzer">StandardAnalyzer</a></code> includes
35 <code>StandardTokenizer</code>,
36 <code><a href="StandardFilter">StandardFilter</a></code>,
37 <code><a href="../../../../../../all/org/apache/lucene/analysis/LowerCaseFilter.html">LowerCaseFilter</a></code>
38 and <code><a href="../../../../../../all/org/apache/lucene/analysis/StopFilter.html">StopFilter</a></code>.
39 When the <code>Version</code> specified in the constructor is lower than
40 3.1, the <code><a href="ClassicTokenizer.html">ClassicTokenizer</a></code>
41 implementation is invoked.</li>
42 <li><code><a href="ClassicTokenizer.html">ClassicTokenizer</a></code>:
43 this class was formerly (prior to Lucene 3.1) named
44 <code>StandardTokenizer</code>. (Its tokenization rules are not
45 based on the Unicode Text Segmentation algorithm.)
46 <code><a href="ClassicAnalyzer">ClassicAnalyzer</a></code> includes
47 <code>ClassicTokenizer</code>,
48 <code><a href="StandardFilter">StandardFilter</a></code>,
49 <code><a href="../../../../../../all/org/apache/lucene/analysis/LowerCaseFilter.html">LowerCaseFilter</a></code>
50 and <code><a href="../../../../../../all/org/apache/lucene/analysis/StopFilter.html">StopFilter</a></code>.
52 <li><code><a href="UAX29URLEmailTokenizer.html">UAX29URLEmailTokenizer</a></code>:
53 implements the Word Break rules from the Unicode Text Segmentation
54 algorithm, as specified in
55 <a href="http://unicode.org/reports/tr29/">Unicode Standard Annex #29</a>.
56 URLs and email addresses are also tokenized according to the relevant RFCs.