--- /dev/null
+<?xml version="1.0"?>
+
+<document>
+ <header>
+ <title>
+ Apache Lucene - Index File Formats
+ </title>
+ </header>
+
+ <body>
+ <section id="Index File Formats"><title>Index File Formats</title>
+
+ <p>
+ This document defines the index file formats used
+ in this version of Lucene. If you are using a different
+ version of Lucene, please consult the copy of
+ <code>docs/fileformats.html</code>
+ that was distributed
+ with the version you are using.
+ </p>
+
+ <p>
+ Apache Lucene is written in Java, but several
+ efforts are underway to write
+ <a href="http://wiki.apache.org/lucene-java/LuceneImplementations">versions
+ of Lucene in other programming
+ languages</a>. If these versions are to remain compatible with Apache
+ Lucene, then a language-independent definition of the Lucene index
+ format is required. This document thus attempts to provide a
+ complete and independent definition of the Apache Lucene file
+ formats.
+ </p>
+
+ <p>
+ As Lucene evolves, this document should evolve.
+ Versions of Lucene in different programming languages should endeavor
+ to agree on file formats, and generate new versions of this document.
+ </p>
+
+ <p>
+ Compatibility notes are provided in this document,
+ describing how file formats have changed from prior versions.
+ </p>
+
+ <p>
+ In version 2.1, the file format was changed to allow
+ lock-less commits (ie, no more commit lock). The
+ change is fully backwards compatible: you can open a
+ pre-2.1 index for searching or adding/deleting of
+ docs. When the new segments file is saved
+ (committed), it will be written in the new file format
+ (meaning no specific "upgrade" process is needed).
+ But note that once a commit has occurred, pre-2.1
+ Lucene will not be able to read the index.
+ </p>
+
+ <p>
+ In version 2.3, the file format was changed to allow
+ segments to share a single set of doc store (vectors &
+ stored fields) files. This allows for faster indexing
+ in certain cases. The change is fully backwards
+ compatible (in the same way as the lock-less commits
+ change in 2.1).
+ </p>
+
+ <p>
+ In version 2.4, Strings are now written as true UTF-8
+ byte sequence, not Java's modified UTF-8. See issue
+ LUCENE-510 for details.
+ </p>
+
+ <p>
+ In version 2.9, an optional opaque Map<String,String>
+ CommitUserData may be passed to IndexWriter's commit
+ methods (and later retrieved), which is recorded in
+ the segments_N file. See issue LUCENE-1382 for
+ details. Also, diagnostics were added to each segment
+ written recording details about why it was written
+ (due to flush, merge; which OS/JRE was used; etc.).
+ See issue LUCENE-1654 for details.
+ </p>
+
+ <p>
+ In version 3.0, compressed fields are no longer
+ written to the index (they can still be read, but on
+ merge the new segment will write them,
+ uncompressed). See issue LUCENE-1960 for details.
+ </p>
+
+ <p>
+ In version 3.1, segments records the code version
+ that created them. See LUCENE-2720 for details.
+
+ Additionally segments track explicitly whether or
+ not they have term vectors. See LUCENE-2811 for details.
+ </p>
+ <p>
+ In version 3.2, numeric fields are written as natively
+ to stored fields file, previously they were stored in
+ text format only.
+ </p>
+ <p>
+ In version 3.4, fields can omit position data while
+ still indexing term frequencies.
+ </p>
+ </section>
+
+ <section id="Definitions"><title>Definitions</title>
+
+ <p>
+ The fundamental concepts in Lucene are index,
+ document, field and term.
+ </p>
+
+
+ <p>
+ An index contains a sequence of documents.
+ </p>
+
+ <ul>
+ <li>
+ <p>
+ A document is a sequence of fields.
+ </p>
+ </li>
+
+ <li>
+ <p>
+ A field is a named sequence of terms.
+ </p>
+ </li>
+
+ <li>
+ A term is a string.
+ </li>
+ </ul>
+
+ <p>
+ The same string in two different fields is
+ considered a different term. Thus terms are represented as a pair of
+ strings, the first naming the field, and the second naming text
+ within the field.
+ </p>
+
+ <section id="Inverted Indexing"><title>Inverted Indexing</title>
+
+ <p>
+ The index stores statistics about terms in order
+ to make term-based search more efficient. Lucene's
+ index falls into the family of indexes known as an <i>inverted
+ index.</i> This is because it can list, for a term, the documents that contain
+ it. This is the inverse of the natural relationship, in which
+ documents list terms.
+ </p>
+ </section>
+ <section id="Types of Fields">
+ <title>Types of Fields</title>
+ <p>
+ In Lucene, fields may be <i>stored</i>, in which
+ case their text is stored in the index literally, in a non-inverted
+ manner. Fields that are inverted are called <i>indexed</i>. A field
+ may be both stored and indexed.</p>
+
+ <p>The text of a field may be <i>tokenized</i> into terms to be
+ indexed, or the text of a field may be used literally as a term to be indexed.
+ Most fields are
+ tokenized, but sometimes it is useful for certain identifier fields
+ to be indexed literally.
+ </p>
+ <p>See the <a href="api/core/org/apache/lucene/document/Field.html">Field</a> java docs for more information on Fields.</p>
+ </section>
+
+ <section id="Segments"><title>Segments</title>
+
+ <p>
+ Lucene indexes may be composed of multiple sub-indexes, or
+ <i>segments</i>. Each segment is a fully independent index, which could be searched
+ separately. Indexes evolve by:
+ </p>
+
+ <ol>
+ <li>
+ <p>Creating new segments for newly added documents.</p>
+ </li>
+ <li>
+ <p>Merging existing segments.</p>
+ </li>
+ </ol>
+
+ <p>
+ Searches may involve multiple segments and/or multiple indexes, each
+ index potentially composed of a set of segments.
+ </p>
+ </section>
+
+ <section id="Document Numbers"><title>Document Numbers</title>
+
+ <p>
+ Internally, Lucene refers to documents by an integer <i>document
+ number</i>. The first document added to an index is numbered zero, and each
+ subsequent document added gets a number one greater than the previous.
+ </p>
+
+ <p>
+ <br/>
+ </p>
+
+ <p>
+ Note that a document's number may change, so caution should be taken
+ when storing these numbers outside of Lucene. In particular, numbers may
+ change in the following situations:
+ </p>
+
+
+ <ul>
+ <li>
+ <p>
+ The
+ numbers stored in each segment are unique only within the segment,
+ and must be converted before they can be used in a larger context.
+ The standard technique is to allocate each segment a range of
+ values, based on the range of numbers used in that segment. To
+ convert a document number from a segment to an external value, the
+ segment's <i>base</i> document
+ number is added. To convert an external value back to a
+ segment-specific value, the segment is identified by the range that
+ the external value is in, and the segment's base value is
+ subtracted. For example two five document segments might be
+ combined, so that the first segment has a base value of zero, and
+ the second of five. Document three from the second segment would
+ have an external value of eight.
+ </p>
+ </li>
+ <li>
+ <p>
+ When documents are deleted, gaps are created
+ in the numbering. These are eventually removed as the index evolves
+ through merging. Deleted documents are dropped when segments are
+ merged. A freshly-merged segment thus has no gaps in its numbering.
+ </p>
+ </li>
+ </ul>
+
+ </section>
+
+ </section>
+
+ <section id="Overview"><title>Overview</title>
+
+ <p>
+ Each segment index maintains the following:
+ </p>
+ <ul>
+ <li>
+ <p>Field names. This
+ contains the set of field names used in the index.
+
+ </p>
+ </li>
+ <li>
+ <p>Stored Field
+ values. This contains, for each document, a list of attribute-value
+ pairs, where the attributes are field names. These are used to
+ store auxiliary information about the document, such as its title,
+ url, or an identifier to access a
+ database. The set of stored fields are what is returned for each hit
+ when searching. This is keyed by document number.
+ </p>
+ </li>
+ <li>
+ <p>Term dictionary.
+ A dictionary containing all of the terms used in all of the indexed
+ fields of all of the documents. The dictionary also contains the
+ number of documents which contain the term, and pointers to the
+ term's frequency and proximity data.
+ </p>
+ </li>
+
+ <li>
+ <p>Term Frequency
+ data. For each term in the dictionary, the numbers of all the
+ documents that contain that term, and the frequency of the term in
+ that document, unless frequencies are omitted (IndexOptions.DOCS_ONLY)
+ </p>
+ </li>
+
+ <li>
+ <p>Term Proximity
+ data. For each term in the dictionary, the positions that the term
+ occurs in each document. Note that this will
+ not exist if all fields in all documents omit position data.
+ </p>
+ </li>
+
+ <li>
+ <p>Normalization
+ factors. For each field in each document, a value is stored that is
+ multiplied into the score for hits on that field.
+ </p>
+ </li>
+ <li>
+ <p>Term Vectors. For each field in each document, the term vector
+ (sometimes called document vector) may be stored. A term vector consists
+ of term text and term frequency. To add Term Vectors to your index see the
+ <a href="api/core/org/apache/lucene/document/Field.html">Field</a>
+ constructors
+ </p>
+ </li>
+ <li>
+ <p>Deleted documents.
+ An optional file indicating which documents are deleted.
+ </p>
+ </li>
+ </ul>
+
+ <p>Details on each of these are provided in subsequent sections.
+ </p>
+ </section>
+
+ <section id="File Naming"><title>File Naming</title>
+
+ <p>
+ All files belonging to a segment have the same name with varying
+ extensions. The extensions correspond to the different file formats
+ described below. When using the Compound File format (default in 1.4 and greater) these files are
+ collapsed into a single .cfs file (see below for details)
+ </p>
+
+ <p>
+ Typically, all segments
+ in an index are stored in a single directory, although this is not
+ required.
+ </p>
+
+ <p>
+ As of version 2.1 (lock-less commits), file names are
+ never re-used (there is one exception, "segments.gen",
+ see below). That is, when any file is saved to the
+ Directory it is given a never before used filename.
+ This is achieved using a simple generations approach.
+ For example, the first segments file is segments_1,
+ then segments_2, etc. The generation is a sequential
+ long integer represented in alpha-numeric (base 36)
+ form.
+ </p>
+
+ </section>
+ <section id="file-names"><title>Summary of File Extensions</title>
+ <p>The following table summarizes the names and extensions of the files in Lucene:
+ <table>
+ <tr>
+ <th>Name</th>
+ <th>Extension</th>
+ <th>Brief Description</th>
+ </tr>
+ <tr>
+ <td><a href="#Segments File">Segments File</a></td>
+ <td>segments.gen, segments_N</td>
+ <td>Stores information about segments</td>
+ </tr>
+ <tr>
+ <td><a href="#Lock File">Lock File</a></td>
+ <td>write.lock</td>
+ <td>The Write lock prevents multiple IndexWriters from writing to the same file.</td>
+ </tr>
+ <tr>
+ <td><a href="#Compound Files">Compound File</a></td>
+ <td>.cfs</td>
+ <td>An optional "virtual" file consisting of all the other index files for systems
+ that frequently run out of file handles.</td>
+ </tr>
+ <tr>
+ <td><a href="#Compound File">Compound File Entry table</a></td>
+ <td>.cfe</td>
+ <td>The "virtual" compound file's entry table holding all entries in the corresponding .cfs file (Since 3.4)</td>
+ </tr>
+ <tr>
+ <td><a href="#Fields">Fields</a></td>
+ <td>.fnm</td>
+ <td>Stores information about the fields</td>
+ </tr>
+ <tr>
+ <td><a href="#field_index">Field Index</a></td>
+ <td>.fdx</td>
+ <td>Contains pointers to field data</td>
+ </tr>
+ <tr>
+ <td><a href="#field_data">Field Data</a></td>
+ <td>.fdt</td>
+ <td>The stored fields for documents</td>
+ </tr>
+ <tr>
+ <td><a href="#tis">Term Infos</a></td>
+ <td>.tis</td>
+ <td>Part of the term dictionary, stores term info</td>
+ </tr>
+ <tr>
+ <td><a href="#tii">Term Info Index</a></td>
+ <td>.tii</td>
+ <td>The index into the Term Infos file</td>
+ </tr>
+ <tr>
+ <td><a href="#Frequencies">Frequencies</a></td>
+ <td>.frq</td>
+ <td>Contains the list of docs which contain each term along with frequency</td>
+ </tr>
+ <tr>
+ <td><a href="#Positions">Positions</a></td>
+ <td>.prx</td>
+ <td>Stores position information about where a term occurs in the index</td>
+ </tr>
+ <tr>
+ <td><a href="#Normalization Factors">Norms</a></td>
+ <td>.nrm</td>
+ <td>Encodes length and boost factors for docs and fields</td>
+ </tr>
+ <tr>
+ <td><a href="#tvx">Term Vector Index</a></td>
+ <td>.tvx</td>
+ <td>Stores offset into the document data file</td>
+ </tr>
+ <tr>
+ <td><a href="#tvd">Term Vector Documents</a></td>
+ <td>.tvd</td>
+ <td>Contains information about each document that has term vectors</td>
+ </tr>
+ <tr>
+ <td><a href="#tvf">Term Vector Fields</a></td>
+ <td>.tvf</td>
+ <td>The field level info about term vectors</td>
+ </tr>
+ <tr>
+ <td><a href="#Deleted Documents">Deleted Documents</a></td>
+ <td>.del</td>
+ <td>Info about what files are deleted</td>
+ </tr>
+ </table>
+
+ </p>
+ </section>
+
+ <section id="Primitive Types"><title>Primitive Types</title>
+
+ <section id="Byte"><title>Byte</title>
+
+ <p>
+ The most primitive type
+ is an eight-bit byte. Files are accessed as sequences of bytes. All
+ other data types are defined as sequences
+ of bytes, so file formats are byte-order independent.
+ </p>
+
+ </section>
+
+ <section id="UInt32"><title>UInt32</title>
+
+ <p>
+ 32-bit unsigned integers are written as four
+ bytes, high-order bytes first.
+ </p>
+ <p>
+ UInt32 --> <Byte><sup>4</sup>
+ </p>
+
+ </section>
+
+ <section id="Uint64"><title>Uint64</title>
+
+ <p>
+ 64-bit unsigned integers are written as eight
+ bytes, high-order bytes first.
+ </p>
+
+ <p>UInt64 --> <Byte><sup>8</sup>
+ </p>
+
+ </section>
+
+ <section id="VInt"><title>VInt</title>
+
+ <p>
+ A variable-length format for positive integers is
+ defined where the high-order bit of each byte indicates whether more
+ bytes remain to be read. The low-order seven bits are appended as
+ increasingly more significant bits in the resulting integer value.
+ Thus values from zero to 127 may be stored in a single byte, values
+ from 128 to 16,383 may be stored in two bytes, and so on.
+ </p>
+
+ <p>
+ <b>VInt Encoding Example</b>
+ </p>
+
+ <table width="100%" border="0" cellpadding="4" cellspacing="0">
+ <col width="64*"/>
+ <col width="64*"/>
+ <col width="64*"/>
+ <col width="64*"/>
+ <tr valign="TOP">
+ <td width="25%">
+ <p align="RIGHT">
+ <b>Value</b>
+ </p>
+ </td>
+ <td width="25%">
+ <p align="RIGHT">
+ <b>First byte</b>
+ </p>
+ </td>
+ <td width="25%">
+ <p align="RIGHT">
+ <b>Second byte</b>
+ </p>
+ </td>
+ <td width="25%">
+ <p align="RIGHT">
+ <b>Third byte</b>
+ </p>
+ </td>
+ </tr>
+ <tr valign="BOTTOM">
+ <td width="25%" sdval="0" sdnum="1033;0;#,##0">
+ <p align="RIGHT">0
+ </p>
+ </td>
+ <td width="25%" sdval="0" sdnum="1033;0;00000000">
+ <p class="western" align="RIGHT" style="margin-left: 0.11cm;
+ margin-right: 0.01cm">
+ 00000000
+ </p>
+ </td>
+ <td width="25%" sdnum="1033;0;00000000">
+ <p align="RIGHT" style="margin-left: -0.07cm; margin-right:
+ 0.01cm">
+ <br/>
+
+ </p>
+ </td>
+ <td width="25%" sdnum="1033;0;00000000">
+ <p align="RIGHT" style="margin-left: -0.47cm; margin-right:
+ 0.01cm">
+ <br/>
+
+ </p>
+ </td>
+ </tr>
+ <tr valign="BOTTOM">
+ <td width="25%" sdval="1" sdnum="1033;0;#,##0">
+ <p align="RIGHT">1
+ </p>
+ </td>
+ <td width="25%" sdval="1" sdnum="1033;0;00000000">
+ <p class="western" align="RIGHT" style="margin-left: 0.11cm;
+ margin-right: 0.01cm">
+ 00000001
+ </p>
+ </td>
+ <td width="25%" sdnum="1033;0;00000000">
+ <p align="RIGHT" style="margin-left: -0.07cm; margin-right:
+ 0.01cm">
+ <br/>
+
+ </p>
+ </td>
+ <td width="25%" sdnum="1033;0;00000000">
+ <p align="RIGHT" style="margin-left: -0.47cm; margin-right:
+ 0.01cm">
+ <br/>
+
+ </p>
+ </td>
+ </tr>
+ <tr valign="BOTTOM">
+ <td width="25%" sdval="2" sdnum="1033;0;#,##0">
+ <p align="RIGHT">2
+ </p>
+ </td>
+ <td width="25%" sdval="10" sdnum="1033;0;00000000">
+ <p class="western" align="RIGHT" style="margin-left: 0.11cm;
+ margin-right: 0.01cm">
+ 00000010
+ </p>
+ </td>
+ <td width="25%" sdnum="1033;0;00000000">
+ <p align="RIGHT" style="margin-left: -0.07cm; margin-right:
+ 0.01cm">
+ <br/>
+
+ </p>
+ </td>
+ <td width="25%" sdnum="1033;0;00000000">
+ <p align="RIGHT" style="margin-left: -0.47cm; margin-right:
+ 0.01cm">
+ <br/>
+
+ </p>
+ </td>
+ </tr>
+ <tr>
+ <td width="25%" valign="TOP">
+ <p align="RIGHT">...
+ </p>
+ </td>
+ <td width="25%" valign="BOTTOM" sdnum="1033;0;00000000">
+ <p align="RIGHT" style="margin-left: 0.11cm; margin-right:
+ 0.01cm">
+ <br/>
+
+ </p>
+ </td>
+ <td width="25%" valign="BOTTOM" sdnum="1033;0;00000000">
+ <p align="RIGHT" style="margin-left: -0.07cm; margin-right:
+ 0.01cm">
+ <br/>
+
+ </p>
+ </td>
+ <td width="25%" valign="BOTTOM" sdnum="1033;0;00000000">
+ <p align="RIGHT" style="margin-left: -0.47cm; margin-right:
+ 0.01cm">
+ <br/>
+
+ </p>
+ </td>
+ </tr>
+ <tr valign="BOTTOM">
+ <td width="25%" sdval="127" sdnum="1033;0;#,##0">
+ <p align="RIGHT">127
+ </p>
+ </td>
+ <td width="25%" sdval="1111111" sdnum="1033;0;00000000">
+ <p class="western" align="RIGHT" style="margin-left: 0.11cm;
+ margin-right: 0.01cm">
+ 01111111
+ </p>
+ </td>
+ <td width="25%" sdnum="1033;0;00000000">
+ <p align="RIGHT" style="margin-left: -0.07cm; margin-right:
+ 0.01cm">
+ <br/>
+
+ </p>
+ </td>
+ <td width="25%" sdnum="1033;0;00000000">
+ <p align="RIGHT" style="margin-left: -0.47cm; margin-right:
+ 0.01cm">
+ <br/>
+
+ </p>
+ </td>
+ </tr>
+ <tr valign="BOTTOM">
+ <td width="25%" sdval="128" sdnum="1033;0;#,##0">
+ <p align="RIGHT">128
+ </p>
+ </td>
+ <td width="25%" sdval="10000000" sdnum="1033;0;00000000">
+ <p class="western" align="RIGHT" style="margin-left: 0.11cm;
+ margin-right: 0.01cm">
+ 10000000
+ </p>
+ </td>
+ <td width="25%" sdval="1" sdnum="1033;0;00000000">
+ <p class="western" align="RIGHT" style="margin-left: -0.07cm;
+ margin-right: 0.01cm">
+ 00000001
+ </p>
+ </td>
+ <td width="25%" sdnum="1033;0;00000000">
+ <p align="RIGHT" style="margin-left: -0.47cm; margin-right:
+ 0.01cm">
+ <br/>
+
+ </p>
+ </td>
+ </tr>
+ <tr valign="BOTTOM">
+ <td width="25%" sdval="129" sdnum="1033;0;#,##0">
+ <p align="RIGHT">129
+ </p>
+ </td>
+ <td width="25%" sdval="10000001" sdnum="1033;0;00000000">
+ <p class="western" align="RIGHT" style="margin-left: 0.11cm;
+ margin-right: 0.01cm">
+ 10000001
+ </p>
+ </td>
+ <td width="25%" sdval="1" sdnum="1033;0;00000000">
+ <p class="western" align="RIGHT" style="margin-left: -0.07cm;
+ margin-right: 0.01cm">
+ 00000001
+ </p>
+ </td>
+ <td width="25%" sdnum="1033;0;00000000">
+ <p align="RIGHT" style="margin-left: -0.47cm; margin-right:
+ 0.01cm">
+ <br/>
+
+ </p>
+ </td>
+ </tr>
+ <tr valign="BOTTOM">
+ <td width="25%" sdval="130" sdnum="1033;0;#,##0">
+ <p align="RIGHT">130
+ </p>
+ </td>
+ <td width="25%" sdval="10000010" sdnum="1033;0;00000000">
+ <p class="western" align="RIGHT" style="margin-left: 0.11cm;
+ margin-right: 0.01cm">
+ 10000010
+ </p>
+ </td>
+ <td width="25%" sdval="1" sdnum="1033;0;00000000">
+ <p class="western" align="RIGHT" style="margin-left: -0.07cm;
+ margin-right: 0.01cm">
+ 00000001
+ </p>
+ </td>
+ <td width="25%" sdnum="1033;0;00000000">
+ <p align="RIGHT" style="margin-left: -0.47cm; margin-right:
+ 0.01cm">
+ <br/>
+
+ </p>
+ </td>
+ </tr>
+ <tr>
+ <td width="25%" valign="TOP">
+ <p align="RIGHT">...
+ </p>
+ </td>
+ <td width="25%" valign="BOTTOM" sdnum="1033;0;00000000">
+ <p align="RIGHT" style="margin-left: 0.11cm; margin-right:
+ 0.01cm">
+ <br/>
+
+ </p>
+ </td>
+ <td width="25%" valign="BOTTOM" sdnum="1033;0;00000000">
+ <p align="RIGHT" style="margin-left: -0.07cm; margin-right:
+ 0.01cm">
+ <br/>
+
+ </p>
+ </td>
+ <td width="25%" valign="BOTTOM" sdnum="1033;0;00000000">
+ <p align="RIGHT" style="margin-left: -0.47cm; margin-right:
+ 0.01cm">
+ <br/>
+
+ </p>
+ </td>
+ </tr>
+ <tr valign="BOTTOM">
+ <td width="25%" sdval="16383" sdnum="1033;0;#,##0">
+ <p align="RIGHT">16,383
+ </p>
+ </td>
+ <td width="25%" sdval="11111111" sdnum="1033;0;00000000">
+ <p class="western" align="RIGHT" style="margin-left: 0.11cm;
+ margin-right: 0.01cm">
+ 11111111
+ </p>
+ </td>
+ <td width="25%" sdval="1111111" sdnum="1033;0;00000000">
+ <p class="western" align="RIGHT" style="margin-left: -0.07cm;
+ margin-right: 0.01cm">
+ 01111111
+ </p>
+ </td>
+ <td width="25%" sdnum="1033;0;00000000">
+ <p align="RIGHT" style="margin-left: -0.47cm; margin-right:
+ 0.01cm">
+ <br/>
+
+ </p>
+ </td>
+ </tr>
+ <tr valign="BOTTOM">
+ <td width="25%" sdval="16384" sdnum="1033;0;#,##0">
+ <p align="RIGHT">16,384
+ </p>
+ </td>
+ <td width="25%" sdval="10000000" sdnum="1033;0;00000000">
+ <p class="western" align="RIGHT" style="margin-left: 0.11cm;
+ margin-right: 0.01cm">
+ 10000000
+ </p>
+ </td>
+ <td width="25%" sdval="10000000" sdnum="1033;0;00000000">
+ <p class="western" align="RIGHT" style="margin-left: -0.07cm;
+ margin-right: 0.01cm">
+ 10000000
+ </p>
+ </td>
+ <td width="25%" sdval="1" sdnum="1033;0;00000000">
+ <p class="western" align="RIGHT" style="margin-left: -0.47cm;
+ margin-right: 0.01cm">
+ 00000001
+ </p>
+ </td>
+ </tr>
+ <tr valign="BOTTOM">
+ <td width="25%" sdval="16385" sdnum="1033;0;#,##0">
+ <p align="RIGHT">16,385
+ </p>
+ </td>
+ <td width="25%" sdval="10000001" sdnum="1033;0;00000000">
+ <p class="western" align="RIGHT" style="margin-left: 0.11cm;
+ margin-right: 0.01cm">
+ 10000001
+ </p>
+ </td>
+ <td width="25%" sdval="10000000" sdnum="1033;0;00000000">
+ <p class="western" align="RIGHT" style="margin-left: -0.07cm;
+ margin-right: 0.01cm">
+ 10000000
+ </p>
+ </td>
+ <td width="25%" sdval="1" sdnum="1033;0;00000000">
+ <p class="western" align="RIGHT" style="margin-left: -0.47cm;
+ margin-right: 0.01cm">
+ 00000001
+ </p>
+ </td>
+ </tr>
+ <tr>
+ <td width="25%" valign="TOP">
+ <p align="RIGHT">...
+ </p>
+ </td>
+ <td width="25%" valign="BOTTOM" sdnum="1033;0;00000000">
+ <p class="western" align="RIGHT" style="margin-left: 0.11cm;
+ margin-right: 0.01cm">
+ <br/>
+
+ </p>
+ </td>
+ <td width="25%" valign="BOTTOM" sdnum="1033;0;00000000">
+ <p class="western" align="RIGHT" style="margin-left: -0.07cm;
+ margin-right: 0.01cm">
+ <br/>
+
+ </p>
+ </td>
+ <td width="25%" valign="BOTTOM" sdnum="1033;0;00000000">
+ <p class="western" align="RIGHT" style="margin-left: -0.47cm;
+ margin-right: 0.01cm">
+ <br/>
+
+ </p>
+ </td>
+ </tr>
+ </table>
+
+ <p>
+ This provides compression while still being
+ efficient to decode.
+ </p>
+
+ </section>
+
+ <section id="Chars"><title>Chars</title>
+
+ <p>
+ Lucene writes unicode
+ character sequences as UTF-8 encoded bytes.
+ </p>
+
+
+ </section>
+
+ <section id="String"><title>String</title>
+
+ <p>
+ Lucene writes strings as UTF-8 encoded bytes.
+ First the length, in bytes, is written as a VInt,
+ followed by the bytes.
+ </p>
+
+ <p>
+ String --> VInt, Chars
+ </p>
+
+ </section>
+ </section>
+
+ <section id="Compound Types"><title>Compound Types</title>
+ <section id="MapStringString"><title>Map<String,String></title>
+
+ <p>
+ In a couple places Lucene stores a Map
+ String->String.
+ </p>
+
+ <p>
+ Map<String,String> --> Count<String,String><sup>Count</sup>
+ </p>
+
+ </section>
+
+ </section>
+
+ <section id="Per-Index Files"><title>Per-Index Files</title>
+
+ <p>
+ The files in this section exist one-per-index.
+ </p>
+
+ <section id="Segments File"><title>Segments File</title>
+
+ <p>
+ The active segments in the index are stored in the
+ segment info file,
+ <tt>segments_N</tt>.
+ There may
+ be one or more
+ <tt>segments_N</tt>
+ files in the
+ index; however, the one with the largest
+ generation is the active one (when older
+ segments_N files are present it's because they
+ temporarily cannot be deleted, or, a writer is in
+ the process of committing, or a custom
+ <a href="api/core/org/apache/lucene/index/IndexDeletionPolicy.html">IndexDeletionPolicy</a>
+ is in use). This file lists each
+ segment by name, has details about the separate
+ norms and deletion files, and also contains the
+ size of each segment.
+ </p>
+
+ <p>
+ As of 2.1, there is also a file
+ <tt>segments.gen</tt>.
+ This file contains the
+ current generation (the
+ <tt>_N</tt>
+ in
+ <tt>segments_N</tt>)
+ of the index. This is
+ used only as a fallback in case the current
+ generation cannot be accurately determined by
+ directory listing alone (as is the case for some
+ NFS clients with time-based directory cache
+ expiraation). This file simply contains an Int32
+ version header (SegmentInfos.FORMAT_LOCKLESS =
+ -2), followed by the generation recorded as Int64,
+ written twice.
+ </p>
+ <p>
+ <b>3.1</b>
+ Segments --> Format, Version, NameCounter, SegCount, <SegVersion, SegName, SegSize, DelGen, DocStoreOffset, [DocStoreSegment, DocStoreIsCompoundFile], HasSingleNormFile, NumField,
+ NormGen<sup>NumField</sup>,
+ IsCompoundFile, DeletionCount, HasProx, Diagnostics, HasVectors><sup>SegCount</sup>, CommitUserData, Checksum
+ </p>
+
+ <p>
+ Format, NameCounter, SegCount, SegSize, NumField,
+ DocStoreOffset, DeletionCount --> Int32
+ </p>
+
+ <p>
+ Version, DelGen, NormGen, Checksum --> Int64
+ </p>
+
+ <p>
+ SegVersion, SegName, DocStoreSegment --> String
+ </p>
+
+ <p>
+ Diagnostics --> Map<String,String>
+ </p>
+
+ <p>
+ IsCompoundFile, HasSingleNormFile,
+ DocStoreIsCompoundFile, HasProx, HasVectors --> Int8
+ </p>
+
+ <p>
+ CommitUserData --> Map<String,String>
+ </p>
+
+ <p>
+ Format is -9 (SegmentInfos.FORMAT_DIAGNOSTICS).
+ </p>
+
+ <p>
+ Version counts how often the index has been
+ changed by adding or deleting documents.
+ </p>
+
+ <p>
+ NameCounter is used to generate names for new segment files.
+ </p>
+
+ <p>
+ SegVersion is the code version that created the segment.
+ </p>
+
+ <p>
+ SegName is the name of the segment, and is used as the file name prefix
+ for all of the files that compose the segment's index.
+ </p>
+
+ <p>
+ SegSize is the number of documents contained in the segment index.
+ </p>
+
+ <p>
+ DelGen is the generation count of the separate
+ deletes file. If this is -1, there are no
+ separate deletes. If it is 0, this is a pre-2.1
+ segment and you must check filesystem for the
+ existence of _X.del. Anything above zero means
+ there are separate deletes (_X_N.del).
+ </p>
+
+ <p>
+ NumField is the size of the array for NormGen, or
+ -1 if there are no NormGens stored.
+ </p>
+
+ <p>
+ NormGen records the generation of the separate
+ norms files. If NumField is -1, there are no
+ normGens stored and they are all assumed to be 0
+ when the segment file was written pre-2.1 and all
+ assumed to be -1 when the segments file is 2.1 or
+ above. The generation then has the same meaning
+ as delGen (above).
+ </p>
+
+ <p>
+ IsCompoundFile records whether the segment is
+ written as a compound file or not. If this is -1,
+ the segment is not a compound file. If it is 1,
+ the segment is a compound file. Else it is 0,
+ which means we check filesystem to see if _X.cfs
+ exists.
+ </p>
+
+ <p>
+ If HasSingleNormFile is 1, then the field norms are
+ written as a single joined file (with extension
+ <tt>.nrm</tt>); if it is 0 then each field's norms
+ are stored as separate <tt>.fN</tt> files. See
+ "Normalization Factors" below for details.
+ </p>
+
+ <p>
+ DocStoreOffset, DocStoreSegment,
+ DocStoreIsCompoundFile: If DocStoreOffset is -1,
+ this segment has its own doc store (stored fields
+ values and term vectors) files and DocStoreSegment
+ and DocStoreIsCompoundFile are not stored. In
+ this case all files for stored field values
+ (<tt>*.fdt</tt> and <tt>*.fdx</tt>) and term
+ vectors (<tt>*.tvf</tt>, <tt>*.tvd</tt> and
+ <tt>*.tvx</tt>) will be stored with this segment.
+ Otherwise, DocStoreSegment is the name of the
+ segment that has the shared doc store files;
+ DocStoreIsCompoundFile is 1 if that segment is
+ stored in compound file format (as a <tt>.cfx</tt>
+ file); and DocStoreOffset is the starting document
+ in the shared doc store files where this segment's
+ documents begin. In this case, this segment does
+ not store its own doc store files but instead
+ shares a single set of these files with other
+ segments.
+ </p>
+
+ <p>
+ Checksum contains the CRC32 checksum of all bytes
+ in the segments_N file up until the checksum.
+ This is used to verify integrity of the file on
+ opening the index.
+ </p>
+
+ <p>
+ DeletionCount records the number of deleted
+ documents in this segment.
+ </p>
+
+ <p>
+ HasProx is 1 if any fields in this segment have
+ position data (IndexOptions.DOCS_AND_FREQS_AND_POSITIONS); else, it's 0.
+ </p>
+
+ <p>
+ CommitUserData stores an optional user-supplied
+ opaque Map<String,String> that was passed to
+ IndexWriter's commit or prepareCommit, or
+ IndexReader's flush methods.
+ </p>
+ <p>
+ The Diagnostics Map is privately written by
+ IndexWriter, as a debugging aid, for each segment
+ it creates. It includes metadata like the current
+ Lucene version, OS, Java version, why the segment
+ was created (merge, flush, addIndexes), etc.
+ </p>
+
+ <p> HasVectors is 1 if this segment stores term vectors,
+ else it's 0.
+ </p>
+
+ </section>
+
+ <section id="Lock File"><title>Lock File</title>
+
+ <p>
+ The write lock, which is stored in the index
+ directory by default, is named "write.lock". If
+ the lock directory is different from the index
+ directory then the write lock will be named
+ "XXXX-write.lock" where XXXX is a unique prefix
+ derived from the full path to the index directory.
+ When this file is present, a writer is currently
+ modifying the index (adding or removing
+ documents). This lock file ensures that only one
+ writer is modifying the index at a time.
+ </p>
+ </section>
+
+ <section id="Deletable File"><title>Deletable File</title>
+
+ <p>
+ A writer dynamically computes
+ the files that are deletable, instead, so no file
+ is written.
+ </p>
+
+ </section>
+
+ <section id="Compound Files"><title>Compound Files</title>
+
+ <p>Starting with Lucene 1.4 the compound file format became default. This
+ is simply a container for all files described in the next section
+ (except for the .del file).</p>
+ <p>Compound Entry Table (.cfe) --> Version, FileCount, <FileName, DataOffset, DataLength>
+ <sup>FileCount</sup>
+ </p>
+
+ <p>Compound (.cfs) --> FileData <sup>FileCount</sup>
+ </p>
+
+ <p>Version --> Int</p>
+
+ <p>FileCount --> VInt</p>
+
+ <p>DataOffset --> Long</p>
+
+ <p>DataLength --> Long</p>
+
+ <p>FileName --> String</p>
+
+ <p>FileData --> raw file data</p>
+ <p>The raw file data is the data from the individual files named above.</p>
+
+ <p>Starting with Lucene 2.3, doc store files (stored
+ field values and term vectors) can be shared in a
+ single set of files for more than one segment. When
+ compound file is enabled, these shared files will be
+ added into a single compound file (same format as
+ above) but with the extension <tt>.cfx</tt>.
+ </p>
+
+ </section>
+
+ </section>
+
+ <section id="Per-Segment Files"><title>Per-Segment Files</title>
+
+ <p>
+ The remaining files are all per-segment, and are
+ thus defined by suffix.
+ </p>
+ <section id="Fields"><title>Fields</title>
+ <p>
+ <br/>
+ <b>Field Info</b>
+ <br/>
+ </p>
+
+ <p>
+ Field names are
+ stored in the field info file, with suffix .fnm.
+ </p>
+ <p>
+ FieldInfos
+ (.fnm) --> FNMVersion,FieldsCount, <FieldName,
+ FieldBits>
+ <sup>FieldsCount</sup>
+ </p>
+
+ <p>
+ FNMVersion, FieldsCount --> VInt
+ </p>
+
+ <p>
+ FieldName --> String
+ </p>
+
+ <p>
+ FieldBits --> Byte
+ </p>
+
+ <p>
+ <ul>
+ <li>
+ The low-order bit is one for
+ indexed fields, and zero for non-indexed fields.
+ </li>
+ <li>
+ The second lowest-order
+ bit is one for fields that have term vectors stored, and zero for fields
+ without term vectors.
+ </li>
+ <li>If the third lowest-order bit is set (0x04), term positions are stored with the term vectors.</li>
+ <li>If the fourth lowest-order bit is set (0x08), term offsets are stored with the term vectors.</li>
+ <li>If the fifth lowest-order bit is set (0x10), norms are omitted for the indexed field.</li>
+ <li>If the sixth lowest-order bit is set (0x20), payloads are stored for the indexed field.</li>
+ <li>If the seventh lowest-order bit is set (0x40), term frequencies and positions omitted for the indexed field.</li>
+ <li>If the eighth lowest-order bit is set (0x80), positions are omitted for the indexed field.</li>
+ </ul>
+ </p>
+
+ <p>
+ FNMVersion (added in 2.9) is -2 for indexes from 2.9 - 3.3. It is -3 for indexes in Lucene 3.4+
+ </p>
+
+ <p>
+ Fields are numbered by their order in this file. Thus field zero is
+ the
+ first field in the file, field one the next, and so on. Note that,
+ like document numbers, field numbers are segment relative.
+ </p>
+
+
+
+ <p>
+ <br/>
+ <b>Stored Fields</b>
+ <br/>
+ </p>
+
+ <p>
+ Stored fields are represented by two files:
+ </p>
+
+ <ol>
+ <li><a name="field_index"/>
+ <p>
+ The field index, or .fdx file.
+ </p>
+
+ <p>
+ This contains, for each document, a pointer to
+ its field data, as follows:
+ </p>
+
+ <p>
+ FieldIndex
+ (.fdx) -->
+ <FieldValuesPosition>
+ <sup>SegSize</sup>
+ </p>
+ <p>FieldValuesPosition
+ --> Uint64
+ </p>
+ <p>This
+ is used to find the location within the field data file of the
+ fields of a particular document. Because it contains fixed-length
+ data, this file may be easily randomly accessed. The position of
+ document
+ <i>n</i>
+ 's
+ <i></i>
+ field data is the Uint64 at
+ <i>n*8</i>
+ in
+ this file.
+ </p>
+ </li>
+ <li>
+ <p><a name="field_data"/>
+ The field data, or .fdt file.
+
+ </p>
+
+ <p>
+ This contains the stored fields of each document,
+ as follows:
+ </p>
+
+ <p>
+ FieldData (.fdt) -->
+ <DocFieldData>
+ <sup>SegSize</sup>
+ </p>
+ <p>DocFieldData -->
+ FieldCount, <FieldNum, Bits, Value>
+ <sup>FieldCount</sup>
+ </p>
+ <p>FieldCount -->
+ VInt
+ </p>
+ <p>FieldNum -->
+ VInt
+ </p>
+ <p>Bits -->
+ Byte
+ </p>
+ <p>
+ <ul>
+ <li>low order bit is one for tokenized fields</li>
+ <li>second bit is one for fields containing binary data</li>
+ <li>third bit is one for fields with compression option enabled
+ (if compression is enabled, the algorithm used is ZLIB),
+ only available for indexes until Lucene version 2.9.x</li>
+ <li>4th to 6th bit (mask: 0x7<<3) define the type of a
+ numeric field: <ul>
+ <li>all bits in mask are cleared if no numeric field at all</li>
+ <li>1<<3: Value is Int</li>
+ <li>2<<3: Value is Long</li>
+ <li>3<<3: Value is Int as Float (as of Float.intBitsToFloat)</li>
+ <li>4<<3: Value is Long as Double (as of Double.longBitsToDouble)</li>
+ </ul></li>
+ </ul>
+ </p>
+ <p>Value -->
+ String | BinaryValue | Int | Long (depending on Bits)
+ </p>
+ <p>BinaryValue -->
+ ValueSize, <Byte>^ValueSize
+ </p>
+ <p>ValueSize -->
+ VInt
+ </p>
+
+ </li>
+ </ol>
+
+ </section>
+ <section id="Term Dictionary"><title>Term Dictionary</title>
+
+ <p>
+ The term dictionary is represented as two files:
+ </p>
+ <ol>
+ <li><a name="tis"/>
+ <p>
+ The term infos, or tis file.
+ </p>
+
+ <p>
+ TermInfoFile (.tis)-->
+ TIVersion, TermCount, IndexInterval, SkipInterval, MaxSkipLevels, TermInfos
+ </p>
+ <p>TIVersion -->
+ UInt32
+ </p>
+ <p>TermCount -->
+ UInt64
+ </p>
+ <p>IndexInterval -->
+ UInt32
+ </p>
+ <p>SkipInterval -->
+ UInt32
+ </p>
+ <p>MaxSkipLevels -->
+ UInt32
+ </p>
+ <p>TermInfos -->
+ <TermInfo>
+ <sup>TermCount</sup>
+ </p>
+ <p>TermInfo -->
+ <Term, DocFreq, FreqDelta, ProxDelta, SkipDelta>
+ </p>
+ <p>Term -->
+ <PrefixLength, Suffix, FieldNum>
+ </p>
+ <p>Suffix -->
+ String
+ </p>
+ <p>PrefixLength,
+ DocFreq, FreqDelta, ProxDelta, SkipDelta
+ <br/>
+ --> VInt
+ </p>
+ <p>
+ This file is sorted by Term. Terms are
+ ordered first lexicographically (by UTF16
+ character code) by the term's field name,
+ and within that lexicographically (by
+ UTF16 character code) by the term's text.
+ </p>
+ <p>TIVersion names the version of the format
+ of this file and is equal to TermInfosWriter.FORMAT_CURRENT.
+ </p>
+ <p>Term
+ text prefixes are shared. The PrefixLength is the number of initial
+ characters from the previous term which must be pre-pended to a
+ term's suffix in order to form the term's text. Thus, if the
+ previous term's text was "bone" and the term is "boy",
+ the PrefixLength is two and the suffix is "y".
+ </p>
+ <p>FieldNumber
+ determines the term's field, whose name is stored in the .fdt file.
+ </p>
+ <p>DocFreq
+ is the count of documents which contain the term.
+ </p>
+ <p>FreqDelta
+ determines the position of this term's TermFreqs within the .frq
+ file. In particular, it is the difference between the position of
+ this term's data in that file and the position of the previous
+ term's data (or zero, for the first term in the file).
+ </p>
+ <p>ProxDelta
+ determines the position of this term's TermPositions within the .prx
+ file. In particular, it is the difference between the position of
+ this term's data in that file and the position of the previous
+ term's data (or zero, for the first term in the file. For fields
+ that omit position data, this will be 0 since
+ prox information is not stored.
+ </p>
+ <p>SkipDelta determines the position of this
+ term's SkipData within the .frq file. In
+ particular, it is the number of bytes
+ after TermFreqs that the SkipData starts.
+ In other words, it is the length of the
+ TermFreq data. SkipDelta is only stored
+ if DocFreq is not smaller than SkipInterval.
+ </p>
+ </li>
+ <li>
+ <p><a name="tii"/>
+ The term info index, or .tii file.
+ </p>
+
+ <p>
+ This contains every IndexInterval
+ <sup>th</sup>
+ entry from the .tis
+ file, along with its location in the "tis" file. This is
+ designed to be read entirely into memory and used to provide random
+ access to the "tis" file.
+ </p>
+
+ <p>
+ The structure of this file is very similar to the
+ .tis file, with the addition of one item per record, the IndexDelta.
+ </p>
+
+ <p>
+ TermInfoIndex (.tii)-->
+ TIVersion, IndexTermCount, IndexInterval, SkipInterval, MaxSkipLevels, TermIndices
+ </p>
+ <p>TIVersion -->
+ UInt32
+ </p>
+ <p>IndexTermCount -->
+ UInt64
+ </p>
+ <p>IndexInterval -->
+ UInt32
+ </p>
+ <p>SkipInterval -->
+ UInt32
+ </p>
+ <p>TermIndices -->
+ <TermInfo, IndexDelta>
+ <sup>IndexTermCount</sup>
+ </p>
+ <p>IndexDelta -->
+ VLong
+ </p>
+ <p>IndexDelta
+ determines the position of this term's TermInfo within the .tis file. In
+ particular, it is the difference between the position of this term's
+ entry in that file and the position of the previous term's entry.
+ </p>
+ <p>SkipInterval is the fraction of TermDocs stored in skip tables. It is used to accelerate TermDocs.skipTo(int).
+ Larger values result in smaller indexes, greater acceleration, but fewer accelerable cases, while
+ smaller values result in bigger indexes, less acceleration (in case of a small value for MaxSkipLevels) and more
+ accelerable cases.</p>
+ <p>MaxSkipLevels is the max. number of skip levels stored for each term in the .frq file. A low value results in
+ smaller indexes but less acceleration, a larger value results in slighly larger indexes but greater acceleration.
+ See format of .frq file for more information about skip levels.</p>
+ </li>
+ </ol>
+ </section>
+
+ <section id="Frequencies"><title>Frequencies</title>
+
+ <p>
+ The .frq file contains the lists of documents
+ which contain each term, along with the frequency of the term in that
+ document (except when frequencies are omitted: IndexOptions.DOCS_ONLY).
+ </p>
+ <p>FreqFile (.frq) -->
+ <TermFreqs, SkipData>
+ <sup>TermCount</sup>
+ </p>
+ <p>TermFreqs -->
+ <TermFreq>
+ <sup>DocFreq</sup>
+ </p>
+ <p>TermFreq -->
+ DocDelta[, Freq?]
+ </p>
+ <p>SkipData -->
+ <<SkipLevelLength, SkipLevel>
+ <sup>NumSkipLevels-1</sup>, SkipLevel>
+ <SkipDatum>
+ </p>
+ <p>SkipLevel -->
+ <SkipDatum>
+ <sup>DocFreq/(SkipInterval^(Level + 1))</sup>
+ </p>
+ <p>SkipDatum -->
+ DocSkip,PayloadLength?,FreqSkip,ProxSkip,SkipChildLevelPointer?
+ </p>
+ <p>DocDelta,Freq,DocSkip,PayloadLength,FreqSkip,ProxSkip -->
+ VInt
+ </p>
+ <p>SkipChildLevelPointer -->
+ VLong
+ </p>
+ <p>TermFreqs
+ are ordered by term (the term is implicit, from the .tis file).
+ </p>
+ <p>TermFreq
+ entries are ordered by increasing document number.
+ </p>
+ <p>DocDelta: if frequencies are indexed, this determines both
+ the document number and the frequency. In
+ particular, DocDelta/2 is the difference between
+ this document number and the previous document
+ number (or zero when this is the first document in
+ a TermFreqs). When DocDelta is odd, the frequency
+ is one. When DocDelta is even, the frequency is
+ read as another VInt. If frequencies are omitted, DocDelta
+ contains the gap (not multiplied by 2) between
+ document numbers and no frequency information is
+ stored.
+ </p>
+ <p>For example, the TermFreqs for a term which occurs
+ once in document seven and three times in document
+ eleven, with frequencies indexed, would be the following
+ sequence of VInts:
+ </p>
+ <p>15, 8, 3
+ </p>
+ <p> If frequencies were omitted (IndexOptions.DOCS_ONLY) it would be this sequence
+ of VInts instead:
+ </p>
+ <p>
+ 7,4
+ </p>
+ <p>DocSkip records the document number before every
+ SkipInterval
+ <sup>th</sup>
+ document in TermFreqs.
+ If payloads are disabled for the term's field,
+ then DocSkip represents the difference from the
+ previous value in the sequence.
+ If payloads are enabled for the term's field,
+ then DocSkip/2 represents the difference from the
+ previous value in the sequence. If payloads are enabled
+ and DocSkip is odd,
+ then PayloadLength is stored indicating the length
+ of the last payload before the SkipInterval<sup>th</sup>
+ document in TermPositions.
+ FreqSkip and ProxSkip record the position of every
+ SkipInterval
+ <sup>th</sup>
+ entry in FreqFile and
+ ProxFile, respectively. File positions are
+ relative to the start of TermFreqs and Positions,
+ to the previous SkipDatum in the sequence.
+ </p>
+ <p>For example, if DocFreq=35 and SkipInterval=16,
+ then there are two SkipData entries, containing
+ the 15
+ <sup>th</sup>
+ and 31
+ <sup>st</sup>
+ document
+ numbers in TermFreqs. The first FreqSkip names
+ the number of bytes after the beginning of
+ TermFreqs that the 16
+ <sup>th</sup>
+ SkipDatum
+ starts, and the second the number of bytes after
+ that that the 32
+ <sup>nd</sup>
+ starts. The first
+ ProxSkip names the number of bytes after the
+ beginning of Positions that the 16
+ <sup>th</sup>
+ SkipDatum starts, and the second the number of
+ bytes after that that the 32
+ <sup>nd</sup>
+ starts.
+ </p>
+ <p>Each term can have multiple skip levels.
+ The amount of skip levels for a term is NumSkipLevels = Min(MaxSkipLevels, floor(log(DocFreq/log(SkipInterval)))).
+ The number of SkipData entries for a skip level is DocFreq/(SkipInterval^(Level + 1)), whereas the lowest skip
+ level is Level=0. <br></br>
+ Example: SkipInterval = 4, MaxSkipLevels = 2, DocFreq = 35. Then skip level 0 has 8 SkipData entries,
+ containing the 3<sup>rd</sup>, 7<sup>th</sup>, 11<sup>th</sup>, 15<sup>th</sup>, 19<sup>th</sup>, 23<sup>rd</sup>,
+ 27<sup>th</sup>, and 31<sup>st</sup> document numbers in TermFreqs. Skip level 1 has 2 SkipData entries, containing the
+ 15<sup>th</sup> and 31<sup>st</sup> document numbers in TermFreqs. <br></br>
+ The SkipData entries on all upper levels > 0 contain a SkipChildLevelPointer referencing the corresponding SkipData
+ entry in level-1. In the example has entry 15 on level 1 a pointer to entry 15 on level 0 and entry 31 on level 1 a pointer
+ to entry 31 on level 0.
+ </p>
+
+ </section>
+ <section id="Positions"><title>Positions</title>
+
+ <p>
+ The .prx file contains the lists of positions that
+ each term occurs at within documents. Note that
+ fields omitting positional data do not store
+ anything into this file, and if all fields in the
+ index omit positional data then the .prx file will not
+ exist.
+ </p>
+ <p>ProxFile (.prx) -->
+ <TermPositions>
+ <sup>TermCount</sup>
+ </p>
+ <p>TermPositions -->
+ <Positions>
+ <sup>DocFreq</sup>
+ </p>
+ <p>Positions -->
+ <PositionDelta,Payload?>
+ <sup>Freq</sup>
+ </p>
+ <p>Payload -->
+ <PayloadLength?,PayloadData>
+ </p>
+ <p>PositionDelta -->
+ VInt
+ </p>
+ <p>PayloadLength -->
+ VInt
+ </p>
+ <p>PayloadData -->
+ byte<sup>PayloadLength</sup>
+ </p>
+ <p>TermPositions
+ are ordered by term (the term is implicit, from the .tis file).
+ </p>
+ <p>Positions
+ entries are ordered by increasing document number (the document
+ number is implicit from the .frq file).
+ </p>
+ <p>PositionDelta
+ is, if payloads are disabled for the term's field, the difference
+ between the position of the current occurrence in
+ the document and the previous occurrence (or zero, if this is the
+ first occurrence in this document).
+ If payloads are enabled for the term's field, then PositionDelta/2
+ is the difference between the current and the previous position. If
+ payloads are enabled and PositionDelta is odd, then PayloadLength is
+ stored, indicating the length of the payload at the current term position.
+ </p>
+ <p>
+ For example, the TermPositions for a
+ term which occurs as the fourth term in one document, and as the
+ fifth and ninth term in a subsequent document, would be the following
+ sequence of VInts (payloads disabled):
+ </p>
+ <p>4,
+ 5, 4
+ </p>
+ <p>PayloadData
+ is metadata associated with the current term position. If PayloadLength
+ is stored at the current position, then it indicates the length of this
+ Payload. If PayloadLength is not stored, then this Payload has the same
+ length as the Payload at the previous position.
+ </p>
+ </section>
+ <section id="Normalization Factors"><title>Normalization Factors</title>
+
+ <p>There's a single .nrm file containing all norms:
+ </p>
+ <p>AllNorms
+ (.nrm) --> NormsHeader,<Norms>
+ <sup>NumFieldsWithNorms</sup>
+ </p>
+ <p>Norms
+ --> <Byte>
+ <sup>SegSize</sup>
+ </p>
+ <p>NormsHeader
+ --> 'N','R','M',Version
+ </p>
+ <p>Version
+ --> Byte
+ </p>
+ <p>NormsHeader
+ has 4 bytes, last of which is the format version for this file, currently -1.
+ </p>
+ <p>Each
+ byte encodes a floating point value. Bits 0-2 contain the 3-bit
+ mantissa, and bits 3-8 contain the 5-bit exponent.
+ </p>
+ <p>These
+ are converted to an IEEE single float value as follows:
+ </p>
+ <ol>
+ <li>
+ <p>If
+ the byte is zero, use a zero float.
+ </p>
+ </li>
+ <li>
+ <p>Otherwise,
+ set the sign bit of the float to zero;
+ </p>
+ </li>
+ <li>
+ <p>add
+ 48 to the exponent and use this as the float's exponent;
+ </p>
+ </li>
+ <li>
+ <p>map
+ the mantissa to the high-order 3 bits of the float's mantissa; and
+
+ </p>
+ </li>
+ <li>
+ <p>set
+ the low-order 21 bits of the float's mantissa to zero.
+ </p>
+ </li>
+ </ol>
+ <p>A separate norm file is created when the norm values of an existing segment are modified.
+ When field <em>N</em> is modified, a separate norm file <em>.sN</em>
+ is created, to maintain the norm values for that field.
+ </p>
+ <p>Separate norm files are created (when adequate) for both compound and non compound segments.
+ </p>
+
+ </section>
+ <section id="Term Vectors"><title>Term Vectors</title>
+ <p>
+ Term Vector support is an optional on a field by
+ field basis. It consists of 3 files.
+ </p>
+ <ol>
+ <li><a name="tvx"/>
+ <p>The Document Index or .tvx file.</p>
+ <p>For each document, this stores the offset
+ into the document data (.tvd) and field
+ data (.tvf) files.
+ </p>
+ <p>DocumentIndex (.tvx) --> TVXVersion<DocumentPosition,FieldPosition>
+ <sup>NumDocs</sup>
+ </p>
+ <p>TVXVersion --> Int (TermVectorsReader.CURRENT)</p>
+ <p>DocumentPosition --> UInt64 (offset in
+ the .tvd file)</p>
+ <p>FieldPosition --> UInt64 (offset in the
+ .tvf file)</p>
+ </li>
+ <li><a name="tvd"/>
+ <p>The Document or .tvd file.</p>
+ <p>This contains, for each document, the number of fields, a list of the fields with
+ term vector info and finally a list of pointers to the field information in the .tvf
+ (Term Vector Fields) file.</p>
+ <p>
+ Document (.tvd) --> TVDVersion<NumFields, FieldNums, FieldPositions>
+ <sup>NumDocs</sup>
+ </p>
+ <p>TVDVersion --> Int (TermVectorsReader.FORMAT_CURRENT)</p>
+ <p>NumFields --> VInt</p>
+ <p>FieldNums --> <FieldNumDelta>
+ <sup>NumFields</sup>
+ </p>
+ <p>FieldNumDelta --> VInt</p>
+ <p>FieldPositions --> <FieldPositionDelta>
+ <sup>NumFields-1</sup>
+ </p>
+ <p>FieldPositionDelta --> VLong</p>
+ <p>The .tvd file is used to map out the fields that have term vectors stored and
+ where the field information is in the .tvf file.</p>
+ </li>
+ <li><a name="tvf"/>
+ <p>The Field or .tvf file.</p>
+ <p>This file contains, for each field that has a term vector stored, a list of
+ the terms, their frequencies and, optionally, position and offest information.</p>
+ <p>Field (.tvf) --> TVFVersion<NumTerms, Position/Offset, TermFreqs>
+ <sup>NumFields</sup>
+ </p>
+ <p>TVFVersion --> Int (TermVectorsReader.FORMAT_CURRENT)</p>
+ <p>NumTerms --> VInt</p>
+ <p>Position/Offset --> Byte</p>
+ <p>TermFreqs --> <TermText, TermFreq, Positions?, Offsets?>
+ <sup>NumTerms</sup>
+ </p>
+ <p>TermText --> <PrefixLength, Suffix></p>
+ <p>PrefixLength --> VInt</p>
+ <p>Suffix --> String</p>
+ <p>TermFreq --> VInt</p>
+ <p>Positions --> <VInt><sup>TermFreq</sup></p>
+ <p>Offsets --> <VInt, VInt><sup>TermFreq</sup></p>
+ <br/>
+ <p>Notes:</p>
+ <ul>
+ <li>Position/Offset byte stores whether this term vector has position or offset information stored.</li>
+ <li>Term
+ text prefixes are shared. The PrefixLength is the number of initial
+ characters from the previous term which must be pre-pended to a
+ term's suffix in order to form the term's text. Thus, if the
+ previous term's text was "bone" and the term is "boy",
+ the PrefixLength is two and the suffix is "y".
+ </li>
+ <li>Positions are stored as delta encoded VInts. This means we only store the difference of the current position from the last position</li>
+ <li>Offsets are stored as delta encoded VInts. The first VInt is the startOffset, the second is the endOffset.</li>
+ </ul>
+
+
+ </li>
+ </ol>
+ </section>
+
+ <section id="Deleted Documents"><title>Deleted Documents</title>
+
+ <p>The .del file is
+ optional, and only exists when a segment contains deletions.
+ </p>
+
+ <p>Although per-segment, this file is maintained exterior to compound segment files.
+ </p>
+ <p>
+ Deletions
+ (.del) --> [Format],ByteCount,BitCount, Bits | DGaps (depending on Format)
+ </p>
+
+ <p>Format,ByteSize,BitCount -->
+ Uint32
+ </p>
+
+ <p>Bits -->
+ <Byte>
+ <sup>ByteCount</sup>
+ </p>
+
+ <p>DGaps -->
+ <DGap,NonzeroByte>
+ <sup>NonzeroBytesCount</sup>
+ </p>
+
+ <p>DGap -->
+ VInt
+ </p>
+
+ <p>NonzeroByte -->
+ Byte
+ </p>
+
+ <p>Format
+ is Optional. -1 indicates DGaps. Non-negative value indicates Bits, and that Format is excluded.
+ </p>
+
+ <p>ByteCount
+ indicates the number of bytes in Bits. It is typically
+ (SegSize/8)+1.
+ </p>
+
+ <p>
+ BitCount
+ indicates the number of bits that are currently set in Bits.
+ </p>
+
+ <p>Bits
+ contains one bit for each document indexed. When the bit
+ corresponding to a document number is set, that document is marked as
+ deleted. Bit ordering is from least to most significant. Thus, if
+ Bits contains two bytes, 0x00 and 0x02, then document 9 is marked as
+ deleted.
+ </p>
+
+ <p>DGaps
+ represents sparse bit-vectors more efficiently than Bits.
+ It is made of DGaps on indexes of nonzero bytes in Bits,
+ and the nonzero bytes themselves. The number of nonzero bytes
+ in Bits (NonzeroBytesCount) is not stored.
+ </p>
+ <p>For example,
+ if there are 8000 bits and only bits 10,12,32 are set,
+ DGaps would be used:
+ </p>
+ <p>
+ (VInt) 1 , (byte) 20 , (VInt) 3 , (Byte) 1
+ </p>
+ </section>
+ </section>
+
+ <section id="Limitations"><title>Limitations</title>
+
+ <p>
+ When referring to term numbers, Lucene's current
+ implementation uses a Java <code>int</code> to hold the
+ term index, which means the maximum number of unique
+ terms in any single index segment is ~2.1 billion times
+ the term index interval (default 128) = ~274 billion.
+ This is technically not a limitation of the index file
+ format, just of Lucene's current implementation.
+ </p>
+ <p>
+ Similarly, Lucene uses a Java <code>int</code> to refer
+ to document numbers, and the index file format uses an
+ <code>Int32</code> on-disk to store document numbers.
+ This is a limitation of both the index file format and
+ the current implementation. Eventually these should be
+ replaced with either <code>UInt64</code> values, or
+ better yet, <code>VInt</code> values which have no
+ limit.
+ </p>
+
+ </section>
+
+ </body>
+
+</document>
fnp / pylucene.git / blobdiff