--- /dev/null
+<!doctype html public "-//w3c//dtd html 4.0 transitional//en">
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<html>
+<head>
+ <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
+</head>
+<body>
+Code to search indices.
+
+<h2>Table Of Contents</h2>
+<p>
+ <ol>
+ <li><a href="#search">Search Basics</a></li>
+ <li><a href="#query">The Query Classes</a></li>
+ <li><a href="#scoring">Changing the Scoring</a></li>
+ </ol>
+</p>
+<a name="search"></a>
+<h2>Search</h2>
+<p>
+Search over indices.
+
+Applications usually call {@link
+org.apache.lucene.search.Searcher#search(Query,int)} or {@link
+org.apache.lucene.search.Searcher#search(Query,Filter,int)}.
+
+ <!-- FILL IN MORE HERE -->
+</p>
+<a name="query"></a>
+<h2>Query Classes</h2>
+<h4>
+ <a href="TermQuery.html">TermQuery</a>
+</h4>
+
+<p>Of the various implementations of
+ <a href="Query.html">Query</a>, the
+ <a href="TermQuery.html">TermQuery</a>
+ is the easiest to understand and the most often used in applications. A <a
+ href="TermQuery.html">TermQuery</a> matches all the documents that contain the
+ specified
+ <a href="../index/Term.html">Term</a>,
+ which is a word that occurs in a certain
+ <a href="../document/Field.html">Field</a>.
+ Thus, a <a href="TermQuery.html">TermQuery</a> identifies and scores all
+ <a href="../document/Document.html">Document</a>s that have a <a
+ href="../document/Field.html">Field</a> with the specified string in it.
+ Constructing a <a
+ href="TermQuery.html">TermQuery</a>
+ is as simple as:
+ <pre>
+ TermQuery tq = new TermQuery(new Term("fieldName", "term"));
+ </pre>In this example, the <a href="Query.html">Query</a> identifies all <a
+ href="../document/Document.html">Document</a>s that have the <a
+ href="../document/Field.html">Field</a> named <tt>"fieldName"</tt>
+ containing the word <tt>"term"</tt>.
+</p>
+<h4>
+ <a href="BooleanQuery.html">BooleanQuery</a>
+</h4>
+
+<p>Things start to get interesting when one combines multiple
+ <a href="TermQuery.html">TermQuery</a> instances into a <a
+ href="BooleanQuery.html">BooleanQuery</a>.
+ A <a href="BooleanQuery.html">BooleanQuery</a> contains multiple
+ <a href="BooleanClause.html">BooleanClause</a>s,
+ where each clause contains a sub-query (<a href="Query.html">Query</a>
+ instance) and an operator (from <a
+ href="BooleanClause.Occur.html">BooleanClause.Occur</a>)
+ describing how that sub-query is combined with the other clauses:
+ <ol>
+
+ <li><p>SHOULD — Use this operator when a clause can occur in the result set, but is not required.
+ If a query is made up of all SHOULD clauses, then every document in the result
+ set matches at least one of these clauses.</p></li>
+
+ <li><p>MUST — Use this operator when a clause is required to occur in the result set. Every
+ document in the result set will match
+ all such clauses.</p></li>
+
+ <li><p>MUST NOT — Use this operator when a
+ clause must not occur in the result set. No
+ document in the result set will match
+ any such clauses.</p></li>
+ </ol>
+ Boolean queries are constructed by adding two or more
+ <a href="BooleanClause.html">BooleanClause</a>
+ instances. If too many clauses are added, a <a href="BooleanQuery.TooManyClauses.html">TooManyClauses</a>
+ exception will be thrown during searching. This most often occurs
+ when a <a href="Query.html">Query</a>
+ is rewritten into a <a href="BooleanQuery.html">BooleanQuery</a> with many
+ <a href="TermQuery.html">TermQuery</a> clauses,
+ for example by <a href="WildcardQuery.html">WildcardQuery</a>.
+ The default setting for the maximum number
+ of clauses 1024, but this can be changed via the
+ static method <a href="BooleanQuery.html#setMaxClauseCount(int)">setMaxClauseCount</a>
+ in <a href="BooleanQuery.html">BooleanQuery</a>.
+</p>
+
+<h4>Phrases</h4>
+
+<p>Another common search is to find documents containing certain phrases. This
+ is handled two different ways:
+ <ol>
+ <li>
+ <p><a href="PhraseQuery.html">PhraseQuery</a>
+ — Matches a sequence of
+ <a href="../index/Term.html">Terms</a>.
+ <a href="PhraseQuery.html">PhraseQuery</a> uses a slop factor to determine
+ how many positions may occur between any two terms in the phrase and still be considered a match.</p>
+ </li>
+ <li>
+ <p><a href="spans/SpanNearQuery.html">SpanNearQuery</a>
+ — Matches a sequence of other
+ <a href="spans/SpanQuery.html">SpanQuery</a>
+ instances. <a href="spans/SpanNearQuery.html">SpanNearQuery</a> allows for
+ much more
+ complicated phrase queries since it is constructed from other <a
+ href="spans/SpanQuery.html">SpanQuery</a>
+ instances, instead of only <a href="TermQuery.html">TermQuery</a>
+ instances.</p>
+ </li>
+ </ol>
+</p>
+
+<h4>
+ <a href="TermRangeQuery.html">TermRangeQuery</a>
+</h4>
+
+<p>The
+ <a href="TermRangeQuery.html">TermRangeQuery</a>
+ matches all documents that occur in the
+ exclusive range of a lower
+ <a href="../index/Term.html">Term</a>
+ and an upper
+ <a href="../index/Term.html">Term</a>.
+ according to {@link java.lang.String#compareTo(String)}. It is not intended
+ for numerical ranges, use <a href="NumericRangeQuery.html">NumericRangeQuery</a> instead.
+
+ For example, one could find all documents
+ that have terms beginning with the letters <tt>a</tt> through <tt>c</tt>. This type of <a
+ href="Query.html">Query</a> is frequently used to
+ find
+ documents that occur in a specific date range.
+</p>
+
+<h4>
+ <a href="NumericRangeQuery.html">NumericRangeQuery</a>
+</h4>
+
+<p>The
+ <a href="NumericRangeQuery.html">NumericRangeQuery</a>
+ matches all documents that occur in a numeric range.
+ For NumericRangeQuery to work, you must index the values
+ using a special <a href="../document/NumericField.html">
+ NumericField</a>.
+</p>
+
+<h4>
+ <a href="PrefixQuery.html">PrefixQuery</a>,
+ <a href="WildcardQuery.html">WildcardQuery</a>
+</h4>
+
+<p>While the
+ <a href="PrefixQuery.html">PrefixQuery</a>
+ has a different implementation, it is essentially a special case of the
+ <a href="WildcardQuery.html">WildcardQuery</a>.
+ The <a href="PrefixQuery.html">PrefixQuery</a> allows an application
+ to identify all documents with terms that begin with a certain string. The <a
+ href="WildcardQuery.html">WildcardQuery</a> generalizes this by allowing
+ for the use of <tt>*</tt> (matches 0 or more characters) and <tt>?</tt> (matches exactly one character) wildcards.
+ Note that the <a href="WildcardQuery.html">WildcardQuery</a> can be quite slow. Also
+ note that
+ <a href="WildcardQuery.html">WildcardQuery</a> should
+ not start with <tt>*</tt> and <tt>?</tt>, as these are extremely slow.
+ To remove this protection and allow a wildcard at the beginning of a term, see method
+ <a href="../queryParser/QueryParser.html#setAllowLeadingWildcard(boolean)">setAllowLeadingWildcard</a> in
+ <a href="../queryParser/QueryParser.html">QueryParser</a>.
+</p>
+<h4>
+ <a href="FuzzyQuery.html">FuzzyQuery</a>
+</h4>
+
+<p>A
+ <a href="FuzzyQuery.html">FuzzyQuery</a>
+ matches documents that contain terms similar to the specified term. Similarity is
+ determined using
+ <a href="http://en.wikipedia.org/wiki/Levenshtein">Levenshtein (edit) distance</a>.
+ This type of query can be useful when accounting for spelling variations in the collection.
+</p>
+<a name="changingSimilarity"></a>
+<h2>Changing Similarity</h2>
+
+<p>Chances are <a href="DefaultSimilarity.html">DefaultSimilarity</a> is sufficient for all
+ your searching needs.
+ However, in some applications it may be necessary to customize your <a
+ href="Similarity.html">Similarity</a> implementation. For instance, some
+ applications do not need to
+ distinguish between shorter and longer documents (see <a
+ href="http://www.gossamer-threads.com/lists/lucene/java-user/38967#38967">a "fair" similarity</a>).</p>
+
+<p>To change <a href="Similarity.html">Similarity</a>, one must do so for both indexing and
+ searching, and the changes must happen before
+ either of these actions take place. Although in theory there is nothing stopping you from changing mid-stream, it
+ just isn't well-defined what is going to happen.
+</p>
+
+<p>To make this change, implement your own <a href="Similarity.html">Similarity</a> (likely
+ you'll want to simply subclass
+ <a href="DefaultSimilarity.html">DefaultSimilarity</a>) and then use the new
+ class by calling
+ <a href="../index/IndexWriter.html#setSimilarity(org.apache.lucene.search.Similarity)">IndexWriter.setSimilarity</a>
+ before indexing and
+ <a href="Searcher.html#setSimilarity(org.apache.lucene.search.Similarity)">Searcher.setSimilarity</a>
+ before searching.
+</p>
+
+<p>
+ If you are interested in use cases for changing your similarity, see the Lucene users's mailing list at <a
+ href="http://www.nabble.com/Overriding-Similarity-tf2128934.html">Overriding Similarity</a>.
+ In summary, here are a few use cases:
+ <ol>
+ <li><p><a href="api/org/apache/lucene/misc/SweetSpotSimilarity.html">SweetSpotSimilarity</a> — <a
+ href="api/org/apache/lucene/misc/SweetSpotSimilarity.html">SweetSpotSimilarity</a> gives small increases
+ as the frequency increases a small amount
+ and then greater increases when you hit the "sweet spot", i.e. where you think the frequency of terms is
+ more significant.</p></li>
+ <li><p>Overriding tf — In some applications, it doesn't matter what the score of a document is as long as a
+ matching term occurs. In these
+ cases people have overridden Similarity to return 1 from the tf() method.</p></li>
+ <li><p>Changing Length Normalization — By overriding <a
+ href="Similarity.html#lengthNorm(java.lang.String,%20int)">lengthNorm</a>,
+ it is possible to discount how the length of a field contributes
+ to a score. In <a href="DefaultSimilarity.html">DefaultSimilarity</a>,
+ lengthNorm = 1 / (numTerms in field)^0.5, but if one changes this to be
+ 1 / (numTerms in field), all fields will be treated
+ <a href="http://www.gossamer-threads.com/lists/lucene/java-user/38967#38967">"fairly"</a>.</p></li>
+ </ol>
+ In general, Chris Hostetter sums it up best in saying (from <a
+ href="http://www.gossamer-threads.com/lists/lucene/java-user/39125#39125">the Lucene users's mailing list</a>):
+ <blockquote>[One would override the Similarity in] ... any situation where you know more about your data then just
+ that
+ it's "text" is a situation where it *might* make sense to to override your
+ Similarity method.</blockquote>
+</p>
+<a name="scoring"></a>
+<h2>Changing Scoring — Expert Level</h2>
+
+<p>Changing scoring is an expert level task, so tread carefully and be prepared to share your code if
+ you want help.
+</p>
+
+<p>With the warning out of the way, it is possible to change a lot more than just the Similarity
+ when it comes to scoring in Lucene. Lucene's scoring is a complex mechanism that is grounded by
+ <span >three main classes</span>:
+ <ol>
+ <li>
+ <a href="Query.html">Query</a> — The abstract object representation of the
+ user's information need.</li>
+ <li>
+ <a href="Weight.html">Weight</a> — The internal interface representation of
+ the user's Query, so that Query objects may be reused.</li>
+ <li>
+ <a href="Scorer.html">Scorer</a> — An abstract class containing common
+ functionality for scoring. Provides both scoring and explanation capabilities.</li>
+ </ol>
+ Details on each of these classes, and their children, can be found in the subsections below.
+</p>
+<h4>The Query Class</h4>
+ <p>In some sense, the
+ <a href="Query.html">Query</a>
+ class is where it all begins. Without a Query, there would be
+ nothing to score. Furthermore, the Query class is the catalyst for the other scoring classes as it
+ is often responsible
+ for creating them or coordinating the functionality between them. The
+ <a href="Query.html">Query</a> class has several methods that are important for
+ derived classes:
+ <ol>
+ <li>createWeight(Searcher searcher) — A
+ <a href="Weight.html">Weight</a> is the internal representation of the
+ Query, so each Query implementation must
+ provide an implementation of Weight. See the subsection on <a
+ href="#The Weight Interface">The Weight Interface</a> below for details on implementing the Weight
+ interface.</li>
+ <li>rewrite(IndexReader reader) — Rewrites queries into primitive queries. Primitive queries are:
+ <a href="TermQuery.html">TermQuery</a>,
+ <a href="BooleanQuery.html">BooleanQuery</a>, <span
+ >and other queries that implement Query.html#createWeight(Searcher searcher)</span></li>
+ </ol>
+ </p>
+<h4>The Weight Interface</h4>
+ <p>The
+ <a href="Weight.html">Weight</a>
+ interface provides an internal representation of the Query so that it can be reused. Any
+ <a href="Searcher.html">Searcher</a>
+ dependent state should be stored in the Weight implementation,
+ not in the Query class. The interface defines six methods that must be implemented:
+ <ol>
+ <li>
+ <a href="Weight.html#getQuery()">Weight#getQuery()</a> — Pointer to the
+ Query that this Weight represents.</li>
+ <li>
+ <a href="Weight.html#getValue()">Weight#getValue()</a> — The weight for
+ this Query. For example, the TermQuery.TermWeight value is
+ equal to the idf^2 * boost * queryNorm <!-- DOUBLE CHECK THIS --></li>
+ <li>
+ <a href="Weight.html#sumOfSquaredWeights()">
+ Weight#sumOfSquaredWeights()</a> — The sum of squared weights. For TermQuery, this is (idf *
+ boost)^2</li>
+ <li>
+ <a href="Weight.html#normalize(float)">
+ Weight#normalize(float)</a> — Determine the query normalization factor. The query normalization may
+ allow for comparing scores between queries.</li>
+ <li>
+ <a href="Weight.html#scorer(org.apache.lucene.index.IndexReader, boolean, boolean)">
+ Weight#scorer(IndexReader, boolean, boolean)</a> — Construct a new
+ <a href="Scorer.html">Scorer</a>
+ for this Weight. See
+ <a href="#The Scorer Class">The Scorer Class</a>
+ below for help defining a Scorer. As the name implies, the
+ Scorer is responsible for doing the actual scoring of documents given the Query.
+ </li>
+ <li>
+ <a href="Weight.html#explain(org.apache.lucene.search.Searcher, org.apache.lucene.index.IndexReader, int)">
+ Weight#explain(Searcher, IndexReader, int)</a> — Provide a means for explaining why a given document was
+ scored
+ the way it was.</li>
+ </ol>
+ </p>
+<h4>The Scorer Class</h4>
+ <p>The
+ <a href="Scorer.html">Scorer</a>
+ abstract class provides common scoring functionality for all Scorer implementations and
+ is the heart of the Lucene scoring process. The Scorer defines the following abstract (some of them are not
+ yet abstract, but will be in future versions and should be considered as such now) methods which
+ must be implemented (some of them inherited from <a href="DocIdSetIterator.html">DocIdSetIterator</a> ):
+ <ol>
+ <li>
+ <a href="DocIdSetIterator.html#nextDoc()">DocIdSetIterator#nextDoc()</a> — Advances to the next
+ document that matches this Query, returning true if and only
+ if there is another document that matches.</li>
+ <li>
+ <a href="DocIdSetIterator.html#docID()">DocIdSetIterator#docID()</a> — Returns the id of the
+ <a href="../document/Document.html">Document</a>
+ that contains the match. It is not valid until next() has been called at least once.
+ </li>
+ <li>
+ <a href="Scorer.html#score(org.apache.lucene.search.Collector)">Scorer#score(Collector)</a> —
+ Scores and collects all matching documents using the given Collector.
+ </li>
+ <li>
+ <a href="Scorer.html#score()">Scorer#score()</a> — Return the score of the
+ current document. This value can be determined in any
+ appropriate way for an application. For instance, the
+ <a href="http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/java/org/apache/lucene/search/TermScorer.java?view=log">TermScorer</a>
+ returns the tf * Weight.getValue() * fieldNorm.
+ </li>
+ <li>
+ <a href="DocIdSetIterator.html#advance(int)">DocIdSetIterator#advance(int)</a> — Skip ahead in
+ the document matches to the document whose id is greater than
+ or equal to the passed in value. In many instances, advance can be
+ implemented more efficiently than simply looping through all the matching documents until
+ the target document is identified.</li>
+ </ol>
+ </p>
+<h4>Why would I want to add my own Query?</h4>
+
+ <p>In a nutshell, you want to add your own custom Query implementation when you think that Lucene's
+ aren't appropriate for the
+ task that you want to do. You might be doing some cutting edge research or you need more information
+ back
+ out of Lucene (similar to Doug adding SpanQuery functionality).</p>
+
+</body>
+</html>
fnp / pylucene.git / blobdiff