--- /dev/null
+<html>
+<body>
+
+<p>This module enables search result grouping with Lucene, where hits
+with the same value in the specified single-valued group field are
+grouped together. For example, if you group by the <code>author</code>
+field, then all documents with the same value in the <code>author</code>
+field fall into a single group.</p>
+
+<p>Grouping requires a number of inputs:</p>
+
+ <ul>
+ <li> <code>groupField</code>: this is the field used for grouping.
+ For example, if you use the <code>author</code> field then each
+ group has all books by the same author. Documents that don't
+ have this field are grouped under a single group with
+ a <code>null</code> group value.
+
+ <li> <code>groupSort</code>: how the groups are sorted. For sorting
+ purposes, each group is "represented" by the highest-sorted
+ document according to the <code>groupSort</code> within it. For
+ example, if you specify "price" (ascending) then the first group
+ is the one with the lowest price book within it. Or if you
+ specify relevance group sort, then the first group is the one
+ containing the highest scoring book.
+
+ <li> <code>topNGroups</code>: how many top groups to keep. For
+ example, 10 means the top 10 groups are computed.
+
+ <li> <code>groupOffset</code>: which "slice" of top groups you want to
+ retrieve. For example, 3 means you'll get 7 groups back
+ (assuming <code>topNGroups</code> is 10). This is useful for
+ paging, where you might show 5 groups per page.
+
+ <li> <code>withinGroupSort</code>: how the documents within each group
+ are sorted. This can be different from the group sort.
+
+ <li> <code>maxDocsPerGroup</code>: how many top documents within each
+ group to keep.
+
+ <li> <code>withinGroupOffset</code>: which "slice" of top
+ documents you want to retrieve from each group.
+
+ </ul>
+
+<p>The implementation is two-pass: the first pass ({@link
+ org.apache.lucene.search.grouping.TermFirstPassGroupingCollector})
+ gathers the top groups, and the second pass ({@link
+ org.apache.lucene.search.grouping.TermSecondPassGroupingCollector})
+ gathers documents within those groups. If the search is costly to
+ run you may want to use the {@link
+ org.apache.lucene.search.CachingCollector} class, which
+ caches hits and can (quickly) replay them for the second pass. This
+ way you only run the query once, but you pay a RAM cost to (briefly)
+ hold all hits. Results are returned as a {@link
+ org.apache.lucene.search.grouping.TopGroups} instance.</p>
+
+<p>
+ This module abstracts away what defines group and how it is collected. All grouping collectors
+ are abstract and have currently term based implementations. One can implement
+ collectors that for example group on multiple fields.
+</p>
+
+<p>Known limitations:</p>
+<ul>
+ <li> For the two-pass grouping collector, the group field must be a
+ single-valued indexed field.
+ {@link org.apache.lucene.search.FieldCache} is used to load the {@link org.apache.lucene.search.FieldCache.StringIndex} for this field.
+ <li> Although Solr support grouping by function and this module has abstraction of what a group is, there are currently only
+ implementations for grouping based on terms.
+ <li> Sharding is not directly supported, though is not too
+ difficult, if you can merge the top groups and top documents per
+ group yourself.
+</ul>
+
+<p>Typical usage for the generic two-pass collector looks like this
+ (using the {@link org.apache.lucene.search.CachingCollector}):</p>
+
+<pre class="prettyprint">
+ TermFirstPassGroupingCollector c1 = new TermFirstPassGroupingCollector("author", groupSort, groupOffset+topNGroups);
+
+ boolean cacheScores = true;
+ double maxCacheRAMMB = 4.0;
+ CachingCollector cachedCollector = CachingCollector.create(c1, cacheScores, maxCacheRAMMB);
+ s.search(new TermQuery(new Term("content", searchTerm)), cachedCollector);
+
+ Collection<SearchGroup<BytesRef>> topGroups = c1.getTopGroups(groupOffset, fillFields);
+
+ if (topGroups == null) {
+ // No groups matched
+ return;
+ }
+
+ boolean getScores = true;
+ boolean getMaxScores = true;
+ boolean fillFields = true;
+ TermSecondPassGroupingCollector c2 = new TermSecondPassGroupingCollector("author", topGroups, groupSort, docSort, docOffset+docsPerGroup, getScores, getMaxScores, fillFields);
+
+ //Optionally compute total group count
+ TermAllGroupsCollector allGroupsCollector = null;
+ if (requiredTotalGroupCount) {
+ allGroupsCollector = new TermAllGroupsCollector("author");
+ c2 = MultiCollector.wrap(c2, allGroupsCollector);
+ }
+
+ if (cachedCollector.isCached()) {
+ // Cache fit within maxCacheRAMMB, so we can replay it:
+ cachedCollector.replay(c2);
+ } else {
+ // Cache was too large; must re-execute query:
+ s.search(new TermQuery(new Term("content", searchTerm)), c2);
+ }
+
+ TopGroups<BytesRef> groupsResult = c2.getTopGroups(docOffset);
+ if (requiredTotalGroupCount) {
+ groupsResult = new TopGroups<BytesRef>(groupsResult, allGroupsCollector.getGroupCount());
+ }
+
+ // Render groupsResult...
+</pre>
+
+<p>To use the single-pass <code>BlockGroupingCollector</code>,
+ first, at indexing time, you must ensure all docs in each group
+ are added as a block, and you have some way to find the last
+ document of each group. One simple way to do this is to add a
+ marker binary field:</p>
+
+<pre class="prettyprint">
+ // Create Documents from your source:
+ List<Document> oneGroup = ...;
+
+ Field groupEndField = new Field("groupEnd", "x", Field.Store.NO, Field.Index.NOT_ANALYZED);
+ groupEndField.setOmitTermFreqAndPositions(true);
+ groupEndField.setOmitNorms(true);
+ oneGroup.get(oneGroup.size()-1).add(groupEndField);
+
+ // You can also use writer.updateDocuments(); just be sure you
+ // replace an entire previous doc block with this new one. For
+ // example, each group could have a "groupID" field, with the same
+ // value for all docs in this group:
+ writer.addDocuments(oneGroup);
+</pre>
+
+Then, at search time, do this up front:
+
+<pre class="prettyprint">
+ // Set this once in your app & save away for reusing across all queries:
+ Filter groupEndDocs = new CachingWrapperFilter(new QueryWrapperFilter(new TermQuery(new Term("groupEnd", "x"))));
+</pre>
+
+Finally, do this per search:
+
+<pre class="prettyprint">
+ // Per search:
+ BlockGroupingCollector c = new BlockGroupingCollector(groupSort, groupOffset+topNGroups, needsScores, groupEndDocs);
+ s.search(new TermQuery(new Term("content", searchTerm)), c);
+ TopGroups groupsResult = c.getTopGroups(withinGroupSort, groupOffset, docOffset, docOffset+docsPerGroup, fillFields);
+
+ // Render groupsResult...
+</pre>
+
+Note that the <code>groupValue</code> of each <code>GroupDocs</code>
+will be <code>null</code>, so if you need to present this value you'll
+have to separately retrieve it (for example using stored
+fields, <code>FieldCache</code>, etc.).
+
+<p>Another collector is the <code>TermAllGroupHeadsCollector</code> that can be used to retrieve all most relevant
+ documents per group. Also known as group heads. This can be useful in situations when one wants to compute grouping
+ based facets / statistics on the complete query result. The collector can be executed during the first or second
+ phase.</p>
+
+<pre class="prettyprint">
+ AbstractAllGroupHeadsCollector c = TermAllGroupHeadsCollector.create(groupField, sortWithinGroup);
+ s.search(new TermQuery(new Term("content", searchTerm)), c);
+ // Return all group heads as int array
+ int[] groupHeadsArray = c.retrieveGroupHeads()
+ // Return all group heads as FixedBitSet.
+ int maxDoc = s.maxDoc();
+ FixedBitSet groupHeadsBitSet = c.retrieveGroupHeads(maxDoc)
+</pre>
+
+</body>
+</html>
fnp / pylucene.git / blobdiff