6 Apache Lucene - Index File Formats
11 <section id="Index File Formats"><title>Index File Formats</title>
14 This document defines the index file formats used
15 in this version of Lucene. If you are using a different
16 version of Lucene, please consult the copy of
17 <code>docs/fileformats.html</code>
19 with the version you are using.
23 Apache Lucene is written in Java, but several
24 efforts are underway to write
25 <a href="http://wiki.apache.org/lucene-java/LuceneImplementations">versions
26 of Lucene in other programming
27 languages</a>. If these versions are to remain compatible with Apache
28 Lucene, then a language-independent definition of the Lucene index
29 format is required. This document thus attempts to provide a
30 complete and independent definition of the Apache Lucene file
35 As Lucene evolves, this document should evolve.
36 Versions of Lucene in different programming languages should endeavor
37 to agree on file formats, and generate new versions of this document.
41 Compatibility notes are provided in this document,
42 describing how file formats have changed from prior versions.
46 In version 2.1, the file format was changed to allow
47 lock-less commits (ie, no more commit lock). The
48 change is fully backwards compatible: you can open a
49 pre-2.1 index for searching or adding/deleting of
50 docs. When the new segments file is saved
51 (committed), it will be written in the new file format
52 (meaning no specific "upgrade" process is needed).
53 But note that once a commit has occurred, pre-2.1
54 Lucene will not be able to read the index.
58 In version 2.3, the file format was changed to allow
59 segments to share a single set of doc store (vectors &
60 stored fields) files. This allows for faster indexing
61 in certain cases. The change is fully backwards
62 compatible (in the same way as the lock-less commits
67 In version 2.4, Strings are now written as true UTF-8
68 byte sequence, not Java's modified UTF-8. See issue
69 LUCENE-510 for details.
73 In version 2.9, an optional opaque Map<String,String>
74 CommitUserData may be passed to IndexWriter's commit
75 methods (and later retrieved), which is recorded in
76 the segments_N file. See issue LUCENE-1382 for
77 details. Also, diagnostics were added to each segment
78 written recording details about why it was written
79 (due to flush, merge; which OS/JRE was used; etc.).
80 See issue LUCENE-1654 for details.
84 In version 3.0, compressed fields are no longer
85 written to the index (they can still be read, but on
86 merge the new segment will write them,
87 uncompressed). See issue LUCENE-1960 for details.
91 In version 3.1, segments records the code version
92 that created them. See LUCENE-2720 for details.
94 Additionally segments track explicitly whether or
95 not they have term vectors. See LUCENE-2811 for details.
98 In version 3.2, numeric fields are written as natively
99 to stored fields file, previously they were stored in
103 In version 3.4, fields can omit position data while
104 still indexing term frequencies.
108 <section id="Definitions"><title>Definitions</title>
111 The fundamental concepts in Lucene are index,
112 document, field and term.
117 An index contains a sequence of documents.
123 A document is a sequence of fields.
129 A field is a named sequence of terms.
139 The same string in two different fields is
140 considered a different term. Thus terms are represented as a pair of
141 strings, the first naming the field, and the second naming text
145 <section id="Inverted Indexing"><title>Inverted Indexing</title>
148 The index stores statistics about terms in order
149 to make term-based search more efficient. Lucene's
150 index falls into the family of indexes known as an <i>inverted
151 index.</i> This is because it can list, for a term, the documents that contain
152 it. This is the inverse of the natural relationship, in which
153 documents list terms.
156 <section id="Types of Fields">
157 <title>Types of Fields</title>
159 In Lucene, fields may be <i>stored</i>, in which
160 case their text is stored in the index literally, in a non-inverted
161 manner. Fields that are inverted are called <i>indexed</i>. A field
162 may be both stored and indexed.</p>
164 <p>The text of a field may be <i>tokenized</i> into terms to be
165 indexed, or the text of a field may be used literally as a term to be indexed.
167 tokenized, but sometimes it is useful for certain identifier fields
168 to be indexed literally.
170 <p>See the <a href="api/core/org/apache/lucene/document/Field.html">Field</a> java docs for more information on Fields.</p>
173 <section id="Segments"><title>Segments</title>
176 Lucene indexes may be composed of multiple sub-indexes, or
177 <i>segments</i>. Each segment is a fully independent index, which could be searched
178 separately. Indexes evolve by:
183 <p>Creating new segments for newly added documents.</p>
186 <p>Merging existing segments.</p>
191 Searches may involve multiple segments and/or multiple indexes, each
192 index potentially composed of a set of segments.
196 <section id="Document Numbers"><title>Document Numbers</title>
199 Internally, Lucene refers to documents by an integer <i>document
200 number</i>. The first document added to an index is numbered zero, and each
201 subsequent document added gets a number one greater than the previous.
209 Note that a document's number may change, so caution should be taken
210 when storing these numbers outside of Lucene. In particular, numbers may
211 change in the following situations:
219 numbers stored in each segment are unique only within the segment,
220 and must be converted before they can be used in a larger context.
221 The standard technique is to allocate each segment a range of
222 values, based on the range of numbers used in that segment. To
223 convert a document number from a segment to an external value, the
224 segment's <i>base</i> document
225 number is added. To convert an external value back to a
226 segment-specific value, the segment is identified by the range that
227 the external value is in, and the segment's base value is
228 subtracted. For example two five document segments might be
229 combined, so that the first segment has a base value of zero, and
230 the second of five. Document three from the second segment would
231 have an external value of eight.
236 When documents are deleted, gaps are created
237 in the numbering. These are eventually removed as the index evolves
238 through merging. Deleted documents are dropped when segments are
239 merged. A freshly-merged segment thus has no gaps in its numbering.
248 <section id="Overview"><title>Overview</title>
251 Each segment index maintains the following:
256 contains the set of field names used in the index.
262 values. This contains, for each document, a list of attribute-value
263 pairs, where the attributes are field names. These are used to
264 store auxiliary information about the document, such as its title,
265 url, or an identifier to access a
266 database. The set of stored fields are what is returned for each hit
267 when searching. This is keyed by document number.
272 A dictionary containing all of the terms used in all of the indexed
273 fields of all of the documents. The dictionary also contains the
274 number of documents which contain the term, and pointers to the
275 term's frequency and proximity data.
281 data. For each term in the dictionary, the numbers of all the
282 documents that contain that term, and the frequency of the term in
283 that document, unless frequencies are omitted (IndexOptions.DOCS_ONLY)
289 data. For each term in the dictionary, the positions that the term
290 occurs in each document. Note that this will
291 not exist if all fields in all documents omit position data.
297 factors. For each field in each document, a value is stored that is
298 multiplied into the score for hits on that field.
302 <p>Term Vectors. For each field in each document, the term vector
303 (sometimes called document vector) may be stored. A term vector consists
304 of term text and term frequency. To add Term Vectors to your index see the
305 <a href="api/core/org/apache/lucene/document/Field.html">Field</a>
310 <p>Deleted documents.
311 An optional file indicating which documents are deleted.
316 <p>Details on each of these are provided in subsequent sections.
320 <section id="File Naming"><title>File Naming</title>
323 All files belonging to a segment have the same name with varying
324 extensions. The extensions correspond to the different file formats
325 described below. When using the Compound File format (default in 1.4 and greater) these files are
326 collapsed into a single .cfs file (see below for details)
330 Typically, all segments
331 in an index are stored in a single directory, although this is not
336 As of version 2.1 (lock-less commits), file names are
337 never re-used (there is one exception, "segments.gen",
338 see below). That is, when any file is saved to the
339 Directory it is given a never before used filename.
340 This is achieved using a simple generations approach.
341 For example, the first segments file is segments_1,
342 then segments_2, etc. The generation is a sequential
343 long integer represented in alpha-numeric (base 36)
348 <section id="file-names"><title>Summary of File Extensions</title>
349 <p>The following table summarizes the names and extensions of the files in Lucene:
354 <th>Brief Description</th>
357 <td><a href="#Segments File">Segments File</a></td>
358 <td>segments.gen, segments_N</td>
359 <td>Stores information about segments</td>
362 <td><a href="#Lock File">Lock File</a></td>
364 <td>The Write lock prevents multiple IndexWriters from writing to the same file.</td>
367 <td><a href="#Compound Files">Compound File</a></td>
369 <td>An optional "virtual" file consisting of all the other index files for systems
370 that frequently run out of file handles.</td>
373 <td><a href="#Compound File">Compound File Entry table</a></td>
375 <td>The "virtual" compound file's entry table holding all entries in the corresponding .cfs file (Since 3.4)</td>
378 <td><a href="#Fields">Fields</a></td>
380 <td>Stores information about the fields</td>
383 <td><a href="#field_index">Field Index</a></td>
385 <td>Contains pointers to field data</td>
388 <td><a href="#field_data">Field Data</a></td>
390 <td>The stored fields for documents</td>
393 <td><a href="#tis">Term Infos</a></td>
395 <td>Part of the term dictionary, stores term info</td>
398 <td><a href="#tii">Term Info Index</a></td>
400 <td>The index into the Term Infos file</td>
403 <td><a href="#Frequencies">Frequencies</a></td>
405 <td>Contains the list of docs which contain each term along with frequency</td>
408 <td><a href="#Positions">Positions</a></td>
410 <td>Stores position information about where a term occurs in the index</td>
413 <td><a href="#Normalization Factors">Norms</a></td>
415 <td>Encodes length and boost factors for docs and fields</td>
418 <td><a href="#tvx">Term Vector Index</a></td>
420 <td>Stores offset into the document data file</td>
423 <td><a href="#tvd">Term Vector Documents</a></td>
425 <td>Contains information about each document that has term vectors</td>
428 <td><a href="#tvf">Term Vector Fields</a></td>
430 <td>The field level info about term vectors</td>
433 <td><a href="#Deleted Documents">Deleted Documents</a></td>
435 <td>Info about what files are deleted</td>
442 <section id="Primitive Types"><title>Primitive Types</title>
444 <section id="Byte"><title>Byte</title>
447 The most primitive type
448 is an eight-bit byte. Files are accessed as sequences of bytes. All
449 other data types are defined as sequences
450 of bytes, so file formats are byte-order independent.
455 <section id="UInt32"><title>UInt32</title>
458 32-bit unsigned integers are written as four
459 bytes, high-order bytes first.
462 UInt32 --> <Byte><sup>4</sup>
467 <section id="Uint64"><title>Uint64</title>
470 64-bit unsigned integers are written as eight
471 bytes, high-order bytes first.
474 <p>UInt64 --> <Byte><sup>8</sup>
479 <section id="VInt"><title>VInt</title>
482 A variable-length format for positive integers is
483 defined where the high-order bit of each byte indicates whether more
484 bytes remain to be read. The low-order seven bits are appended as
485 increasingly more significant bits in the resulting integer value.
486 Thus values from zero to 127 may be stored in a single byte, values
487 from 128 to 16,383 may be stored in two bytes, and so on.
491 <b>VInt Encoding Example</b>
494 <table width="100%" border="0" cellpadding="4" cellspacing="0">
522 <td width="25%" sdval="0" sdnum="1033;0;#,##0">
526 <td width="25%" sdval="0" sdnum="1033;0;00000000">
527 <p class="western" align="RIGHT" style="margin-left: 0.11cm;
528 margin-right: 0.01cm">
532 <td width="25%" sdnum="1033;0;00000000">
533 <p align="RIGHT" style="margin-left: -0.07cm; margin-right:
539 <td width="25%" sdnum="1033;0;00000000">
540 <p align="RIGHT" style="margin-left: -0.47cm; margin-right:
548 <td width="25%" sdval="1" sdnum="1033;0;#,##0">
552 <td width="25%" sdval="1" sdnum="1033;0;00000000">
553 <p class="western" align="RIGHT" style="margin-left: 0.11cm;
554 margin-right: 0.01cm">
558 <td width="25%" sdnum="1033;0;00000000">
559 <p align="RIGHT" style="margin-left: -0.07cm; margin-right:
565 <td width="25%" sdnum="1033;0;00000000">
566 <p align="RIGHT" style="margin-left: -0.47cm; margin-right:
574 <td width="25%" sdval="2" sdnum="1033;0;#,##0">
578 <td width="25%" sdval="10" sdnum="1033;0;00000000">
579 <p class="western" align="RIGHT" style="margin-left: 0.11cm;
580 margin-right: 0.01cm">
584 <td width="25%" sdnum="1033;0;00000000">
585 <p align="RIGHT" style="margin-left: -0.07cm; margin-right:
591 <td width="25%" sdnum="1033;0;00000000">
592 <p align="RIGHT" style="margin-left: -0.47cm; margin-right:
600 <td width="25%" valign="TOP">
604 <td width="25%" valign="BOTTOM" sdnum="1033;0;00000000">
605 <p align="RIGHT" style="margin-left: 0.11cm; margin-right:
611 <td width="25%" valign="BOTTOM" sdnum="1033;0;00000000">
612 <p align="RIGHT" style="margin-left: -0.07cm; margin-right:
618 <td width="25%" valign="BOTTOM" sdnum="1033;0;00000000">
619 <p align="RIGHT" style="margin-left: -0.47cm; margin-right:
627 <td width="25%" sdval="127" sdnum="1033;0;#,##0">
631 <td width="25%" sdval="1111111" sdnum="1033;0;00000000">
632 <p class="western" align="RIGHT" style="margin-left: 0.11cm;
633 margin-right: 0.01cm">
637 <td width="25%" sdnum="1033;0;00000000">
638 <p align="RIGHT" style="margin-left: -0.07cm; margin-right:
644 <td width="25%" sdnum="1033;0;00000000">
645 <p align="RIGHT" style="margin-left: -0.47cm; margin-right:
653 <td width="25%" sdval="128" sdnum="1033;0;#,##0">
657 <td width="25%" sdval="10000000" sdnum="1033;0;00000000">
658 <p class="western" align="RIGHT" style="margin-left: 0.11cm;
659 margin-right: 0.01cm">
663 <td width="25%" sdval="1" sdnum="1033;0;00000000">
664 <p class="western" align="RIGHT" style="margin-left: -0.07cm;
665 margin-right: 0.01cm">
669 <td width="25%" sdnum="1033;0;00000000">
670 <p align="RIGHT" style="margin-left: -0.47cm; margin-right:
678 <td width="25%" sdval="129" sdnum="1033;0;#,##0">
682 <td width="25%" sdval="10000001" sdnum="1033;0;00000000">
683 <p class="western" align="RIGHT" style="margin-left: 0.11cm;
684 margin-right: 0.01cm">
688 <td width="25%" sdval="1" sdnum="1033;0;00000000">
689 <p class="western" align="RIGHT" style="margin-left: -0.07cm;
690 margin-right: 0.01cm">
694 <td width="25%" sdnum="1033;0;00000000">
695 <p align="RIGHT" style="margin-left: -0.47cm; margin-right:
703 <td width="25%" sdval="130" sdnum="1033;0;#,##0">
707 <td width="25%" sdval="10000010" sdnum="1033;0;00000000">
708 <p class="western" align="RIGHT" style="margin-left: 0.11cm;
709 margin-right: 0.01cm">
713 <td width="25%" sdval="1" sdnum="1033;0;00000000">
714 <p class="western" align="RIGHT" style="margin-left: -0.07cm;
715 margin-right: 0.01cm">
719 <td width="25%" sdnum="1033;0;00000000">
720 <p align="RIGHT" style="margin-left: -0.47cm; margin-right:
728 <td width="25%" valign="TOP">
732 <td width="25%" valign="BOTTOM" sdnum="1033;0;00000000">
733 <p align="RIGHT" style="margin-left: 0.11cm; margin-right:
739 <td width="25%" valign="BOTTOM" sdnum="1033;0;00000000">
740 <p align="RIGHT" style="margin-left: -0.07cm; margin-right:
746 <td width="25%" valign="BOTTOM" sdnum="1033;0;00000000">
747 <p align="RIGHT" style="margin-left: -0.47cm; margin-right:
755 <td width="25%" sdval="16383" sdnum="1033;0;#,##0">
756 <p align="RIGHT">16,383
759 <td width="25%" sdval="11111111" sdnum="1033;0;00000000">
760 <p class="western" align="RIGHT" style="margin-left: 0.11cm;
761 margin-right: 0.01cm">
765 <td width="25%" sdval="1111111" sdnum="1033;0;00000000">
766 <p class="western" align="RIGHT" style="margin-left: -0.07cm;
767 margin-right: 0.01cm">
771 <td width="25%" sdnum="1033;0;00000000">
772 <p align="RIGHT" style="margin-left: -0.47cm; margin-right:
780 <td width="25%" sdval="16384" sdnum="1033;0;#,##0">
781 <p align="RIGHT">16,384
784 <td width="25%" sdval="10000000" sdnum="1033;0;00000000">
785 <p class="western" align="RIGHT" style="margin-left: 0.11cm;
786 margin-right: 0.01cm">
790 <td width="25%" sdval="10000000" sdnum="1033;0;00000000">
791 <p class="western" align="RIGHT" style="margin-left: -0.07cm;
792 margin-right: 0.01cm">
796 <td width="25%" sdval="1" sdnum="1033;0;00000000">
797 <p class="western" align="RIGHT" style="margin-left: -0.47cm;
798 margin-right: 0.01cm">
804 <td width="25%" sdval="16385" sdnum="1033;0;#,##0">
805 <p align="RIGHT">16,385
808 <td width="25%" sdval="10000001" sdnum="1033;0;00000000">
809 <p class="western" align="RIGHT" style="margin-left: 0.11cm;
810 margin-right: 0.01cm">
814 <td width="25%" sdval="10000000" sdnum="1033;0;00000000">
815 <p class="western" align="RIGHT" style="margin-left: -0.07cm;
816 margin-right: 0.01cm">
820 <td width="25%" sdval="1" sdnum="1033;0;00000000">
821 <p class="western" align="RIGHT" style="margin-left: -0.47cm;
822 margin-right: 0.01cm">
828 <td width="25%" valign="TOP">
832 <td width="25%" valign="BOTTOM" sdnum="1033;0;00000000">
833 <p class="western" align="RIGHT" style="margin-left: 0.11cm;
834 margin-right: 0.01cm">
839 <td width="25%" valign="BOTTOM" sdnum="1033;0;00000000">
840 <p class="western" align="RIGHT" style="margin-left: -0.07cm;
841 margin-right: 0.01cm">
846 <td width="25%" valign="BOTTOM" sdnum="1033;0;00000000">
847 <p class="western" align="RIGHT" style="margin-left: -0.47cm;
848 margin-right: 0.01cm">
857 This provides compression while still being
863 <section id="Chars"><title>Chars</title>
866 Lucene writes unicode
867 character sequences as UTF-8 encoded bytes.
873 <section id="String"><title>String</title>
876 Lucene writes strings as UTF-8 encoded bytes.
877 First the length, in bytes, is written as a VInt,
878 followed by the bytes.
882 String --> VInt, Chars
888 <section id="Compound Types"><title>Compound Types</title>
889 <section id="MapStringString"><title>Map<String,String></title>
892 In a couple places Lucene stores a Map
897 Map<String,String> --> Count<String,String><sup>Count</sup>
904 <section id="Per-Index Files"><title>Per-Index Files</title>
907 The files in this section exist one-per-index.
910 <section id="Segments File"><title>Segments File</title>
913 The active segments in the index are stored in the
920 index; however, the one with the largest
921 generation is the active one (when older
922 segments_N files are present it's because they
923 temporarily cannot be deleted, or, a writer is in
924 the process of committing, or a custom
925 <a href="api/core/org/apache/lucene/index/IndexDeletionPolicy.html">IndexDeletionPolicy</a>
926 is in use). This file lists each
927 segment by name, has details about the separate
928 norms and deletion files, and also contains the
929 size of each segment.
933 As of 2.1, there is also a file
934 <tt>segments.gen</tt>.
935 This file contains the
936 current generation (the
940 of the index. This is
941 used only as a fallback in case the current
942 generation cannot be accurately determined by
943 directory listing alone (as is the case for some
944 NFS clients with time-based directory cache
945 expiraation). This file simply contains an Int32
946 version header (SegmentInfos.FORMAT_LOCKLESS =
947 -2), followed by the generation recorded as Int64,
952 Segments --> Format, Version, NameCounter, SegCount, <SegVersion, SegName, SegSize, DelGen, DocStoreOffset, [DocStoreSegment, DocStoreIsCompoundFile], HasSingleNormFile, NumField,
953 NormGen<sup>NumField</sup>,
954 IsCompoundFile, DeletionCount, HasProx, Diagnostics, HasVectors><sup>SegCount</sup>, CommitUserData, Checksum
958 Format, NameCounter, SegCount, SegSize, NumField,
959 DocStoreOffset, DeletionCount --> Int32
963 Version, DelGen, NormGen, Checksum --> Int64
967 SegVersion, SegName, DocStoreSegment --> String
971 Diagnostics --> Map<String,String>
975 IsCompoundFile, HasSingleNormFile,
976 DocStoreIsCompoundFile, HasProx, HasVectors --> Int8
980 CommitUserData --> Map<String,String>
984 Format is -9 (SegmentInfos.FORMAT_DIAGNOSTICS).
988 Version counts how often the index has been
989 changed by adding or deleting documents.
993 NameCounter is used to generate names for new segment files.
997 SegVersion is the code version that created the segment.
1001 SegName is the name of the segment, and is used as the file name prefix
1002 for all of the files that compose the segment's index.
1006 SegSize is the number of documents contained in the segment index.
1010 DelGen is the generation count of the separate
1011 deletes file. If this is -1, there are no
1012 separate deletes. If it is 0, this is a pre-2.1
1013 segment and you must check filesystem for the
1014 existence of _X.del. Anything above zero means
1015 there are separate deletes (_X_N.del).
1019 NumField is the size of the array for NormGen, or
1020 -1 if there are no NormGens stored.
1024 NormGen records the generation of the separate
1025 norms files. If NumField is -1, there are no
1026 normGens stored and they are all assumed to be 0
1027 when the segment file was written pre-2.1 and all
1028 assumed to be -1 when the segments file is 2.1 or
1029 above. The generation then has the same meaning
1034 IsCompoundFile records whether the segment is
1035 written as a compound file or not. If this is -1,
1036 the segment is not a compound file. If it is 1,
1037 the segment is a compound file. Else it is 0,
1038 which means we check filesystem to see if _X.cfs
1043 If HasSingleNormFile is 1, then the field norms are
1044 written as a single joined file (with extension
1045 <tt>.nrm</tt>); if it is 0 then each field's norms
1046 are stored as separate <tt>.fN</tt> files. See
1047 "Normalization Factors" below for details.
1051 DocStoreOffset, DocStoreSegment,
1052 DocStoreIsCompoundFile: If DocStoreOffset is -1,
1053 this segment has its own doc store (stored fields
1054 values and term vectors) files and DocStoreSegment
1055 and DocStoreIsCompoundFile are not stored. In
1056 this case all files for stored field values
1057 (<tt>*.fdt</tt> and <tt>*.fdx</tt>) and term
1058 vectors (<tt>*.tvf</tt>, <tt>*.tvd</tt> and
1059 <tt>*.tvx</tt>) will be stored with this segment.
1060 Otherwise, DocStoreSegment is the name of the
1061 segment that has the shared doc store files;
1062 DocStoreIsCompoundFile is 1 if that segment is
1063 stored in compound file format (as a <tt>.cfx</tt>
1064 file); and DocStoreOffset is the starting document
1065 in the shared doc store files where this segment's
1066 documents begin. In this case, this segment does
1067 not store its own doc store files but instead
1068 shares a single set of these files with other
1073 Checksum contains the CRC32 checksum of all bytes
1074 in the segments_N file up until the checksum.
1075 This is used to verify integrity of the file on
1080 DeletionCount records the number of deleted
1081 documents in this segment.
1085 HasProx is 1 if any fields in this segment have
1086 position data (IndexOptions.DOCS_AND_FREQS_AND_POSITIONS); else, it's 0.
1090 CommitUserData stores an optional user-supplied
1091 opaque Map<String,String> that was passed to
1092 IndexWriter's commit or prepareCommit, or
1093 IndexReader's flush methods.
1096 The Diagnostics Map is privately written by
1097 IndexWriter, as a debugging aid, for each segment
1098 it creates. It includes metadata like the current
1099 Lucene version, OS, Java version, why the segment
1100 was created (merge, flush, addIndexes), etc.
1103 <p> HasVectors is 1 if this segment stores term vectors,
1109 <section id="Lock File"><title>Lock File</title>
1112 The write lock, which is stored in the index
1113 directory by default, is named "write.lock". If
1114 the lock directory is different from the index
1115 directory then the write lock will be named
1116 "XXXX-write.lock" where XXXX is a unique prefix
1117 derived from the full path to the index directory.
1118 When this file is present, a writer is currently
1119 modifying the index (adding or removing
1120 documents). This lock file ensures that only one
1121 writer is modifying the index at a time.
1125 <section id="Deletable File"><title>Deletable File</title>
1128 A writer dynamically computes
1129 the files that are deletable, instead, so no file
1135 <section id="Compound Files"><title>Compound Files</title>
1137 <p>Starting with Lucene 1.4 the compound file format became default. This
1138 is simply a container for all files described in the next section
1139 (except for the .del file).</p>
1140 <p>Compound Entry Table (.cfe) --> Version, FileCount, <FileName, DataOffset, DataLength>
1141 <sup>FileCount</sup>
1144 <p>Compound (.cfs) --> FileData <sup>FileCount</sup>
1147 <p>Version --> Int</p>
1149 <p>FileCount --> VInt</p>
1151 <p>DataOffset --> Long</p>
1153 <p>DataLength --> Long</p>
1155 <p>FileName --> String</p>
1157 <p>FileData --> raw file data</p>
1158 <p>The raw file data is the data from the individual files named above.</p>
1160 <p>Starting with Lucene 2.3, doc store files (stored
1161 field values and term vectors) can be shared in a
1162 single set of files for more than one segment. When
1163 compound file is enabled, these shared files will be
1164 added into a single compound file (same format as
1165 above) but with the extension <tt>.cfx</tt>.
1172 <section id="Per-Segment Files"><title>Per-Segment Files</title>
1175 The remaining files are all per-segment, and are
1176 thus defined by suffix.
1178 <section id="Fields"><title>Fields</title>
1187 stored in the field info file, with suffix .fnm.
1191 (.fnm) --> FNMVersion,FieldsCount, <FieldName,
1193 <sup>FieldsCount</sup>
1197 FNMVersion, FieldsCount --> VInt
1201 FieldName --> String
1205 FieldBits --> Byte
1211 The low-order bit is one for
1212 indexed fields, and zero for non-indexed fields.
1215 The second lowest-order
1216 bit is one for fields that have term vectors stored, and zero for fields
1217 without term vectors.
1219 <li>If the third lowest-order bit is set (0x04), term positions are stored with the term vectors.</li>
1220 <li>If the fourth lowest-order bit is set (0x08), term offsets are stored with the term vectors.</li>
1221 <li>If the fifth lowest-order bit is set (0x10), norms are omitted for the indexed field.</li>
1222 <li>If the sixth lowest-order bit is set (0x20), payloads are stored for the indexed field.</li>
1223 <li>If the seventh lowest-order bit is set (0x40), term frequencies and positions omitted for the indexed field.</li>
1224 <li>If the eighth lowest-order bit is set (0x80), positions are omitted for the indexed field.</li>
1229 FNMVersion (added in 2.9) is -2 for indexes from 2.9 - 3.3. It is -3 for indexes in Lucene 3.4+
1233 Fields are numbered by their order in this file. Thus field zero is
1235 first field in the file, field one the next, and so on. Note that,
1236 like document numbers, field numbers are segment relative.
1243 <b>Stored Fields</b>
1248 Stored fields are represented by two files:
1252 <li><a name="field_index"/>
1254 The field index, or .fdx file.
1258 This contains, for each document, a pointer to
1259 its field data, as follows:
1265 <FieldValuesPosition>
1268 <p>FieldValuesPosition
1272 is used to find the location within the field data file of the
1273 fields of a particular document. Because it contains fixed-length
1274 data, this file may be easily randomly accessed. The position of
1279 field data is the Uint64 at
1286 <p><a name="field_data"/>
1287 The field data, or .fdt file.
1292 This contains the stored fields of each document,
1297 FieldData (.fdt) -->
1298 <DocFieldData>
1301 <p>DocFieldData -->
1302 FieldCount, <FieldNum, Bits, Value>
1303 <sup>FieldCount</sup>
1305 <p>FieldCount -->
1316 <li>low order bit is one for tokenized fields</li>
1317 <li>second bit is one for fields containing binary data</li>
1318 <li>third bit is one for fields with compression option enabled
1319 (if compression is enabled, the algorithm used is ZLIB),
1320 only available for indexes until Lucene version 2.9.x</li>
1321 <li>4th to 6th bit (mask: 0x7<<3) define the type of a
1323 <li>all bits in mask are cleared if no numeric field at all</li>
1324 <li>1<<3: Value is Int</li>
1325 <li>2<<3: Value is Long</li>
1326 <li>3<<3: Value is Int as Float (as of Float.intBitsToFloat)</li>
1327 <li>4<<3: Value is Long as Double (as of Double.longBitsToDouble)</li>
1332 String | BinaryValue | Int | Long (depending on Bits)
1334 <p>BinaryValue -->
1335 ValueSize, <Byte>^ValueSize
1345 <section id="Term Dictionary"><title>Term Dictionary</title>
1348 The term dictionary is represented as two files:
1353 The term infos, or tis file.
1357 TermInfoFile (.tis)-->
1358 TIVersion, TermCount, IndexInterval, SkipInterval, MaxSkipLevels, TermInfos
1366 <p>IndexInterval -->
1369 <p>SkipInterval -->
1372 <p>MaxSkipLevels -->
1377 <sup>TermCount</sup>
1380 <Term, DocFreq, FreqDelta, ProxDelta, SkipDelta>
1383 <PrefixLength, Suffix, FieldNum>
1389 DocFreq, FreqDelta, ProxDelta, SkipDelta
1394 This file is sorted by Term. Terms are
1395 ordered first lexicographically (by UTF16
1396 character code) by the term's field name,
1397 and within that lexicographically (by
1398 UTF16 character code) by the term's text.
1400 <p>TIVersion names the version of the format
1401 of this file and is equal to TermInfosWriter.FORMAT_CURRENT.
1404 text prefixes are shared. The PrefixLength is the number of initial
1405 characters from the previous term which must be pre-pended to a
1406 term's suffix in order to form the term's text. Thus, if the
1407 previous term's text was "bone" and the term is "boy",
1408 the PrefixLength is two and the suffix is "y".
1411 determines the term's field, whose name is stored in the .fdt file.
1414 is the count of documents which contain the term.
1417 determines the position of this term's TermFreqs within the .frq
1418 file. In particular, it is the difference between the position of
1419 this term's data in that file and the position of the previous
1420 term's data (or zero, for the first term in the file).
1423 determines the position of this term's TermPositions within the .prx
1424 file. In particular, it is the difference between the position of
1425 this term's data in that file and the position of the previous
1426 term's data (or zero, for the first term in the file. For fields
1427 that omit position data, this will be 0 since
1428 prox information is not stored.
1430 <p>SkipDelta determines the position of this
1431 term's SkipData within the .frq file. In
1432 particular, it is the number of bytes
1433 after TermFreqs that the SkipData starts.
1434 In other words, it is the length of the
1435 TermFreq data. SkipDelta is only stored
1436 if DocFreq is not smaller than SkipInterval.
1441 The term info index, or .tii file.
1445 This contains every IndexInterval
1448 file, along with its location in the "tis" file. This is
1449 designed to be read entirely into memory and used to provide random
1450 access to the "tis" file.
1454 The structure of this file is very similar to the
1455 .tis file, with the addition of one item per record, the IndexDelta.
1459 TermInfoIndex (.tii)-->
1460 TIVersion, IndexTermCount, IndexInterval, SkipInterval, MaxSkipLevels, TermIndices
1465 <p>IndexTermCount -->
1468 <p>IndexInterval -->
1471 <p>SkipInterval -->
1474 <p>TermIndices -->
1475 <TermInfo, IndexDelta>
1476 <sup>IndexTermCount</sup>
1478 <p>IndexDelta -->
1482 determines the position of this term's TermInfo within the .tis file. In
1483 particular, it is the difference between the position of this term's
1484 entry in that file and the position of the previous term's entry.
1486 <p>SkipInterval is the fraction of TermDocs stored in skip tables. It is used to accelerate TermDocs.skipTo(int).
1487 Larger values result in smaller indexes, greater acceleration, but fewer accelerable cases, while
1488 smaller values result in bigger indexes, less acceleration (in case of a small value for MaxSkipLevels) and more
1489 accelerable cases.</p>
1490 <p>MaxSkipLevels is the max. number of skip levels stored for each term in the .frq file. A low value results in
1491 smaller indexes but less acceleration, a larger value results in slighly larger indexes but greater acceleration.
1492 See format of .frq file for more information about skip levels.</p>
1497 <section id="Frequencies"><title>Frequencies</title>
1500 The .frq file contains the lists of documents
1501 which contain each term, along with the frequency of the term in that
1502 document (except when frequencies are omitted: IndexOptions.DOCS_ONLY).
1504 <p>FreqFile (.frq) -->
1505 <TermFreqs, SkipData>
1506 <sup>TermCount</sup>
1516 <<SkipLevelLength, SkipLevel>
1517 <sup>NumSkipLevels-1</sup>, SkipLevel>
1522 <sup>DocFreq/(SkipInterval^(Level + 1))</sup>
1525 DocSkip,PayloadLength?,FreqSkip,ProxSkip,SkipChildLevelPointer?
1527 <p>DocDelta,Freq,DocSkip,PayloadLength,FreqSkip,ProxSkip -->
1530 <p>SkipChildLevelPointer -->
1534 are ordered by term (the term is implicit, from the .tis file).
1537 entries are ordered by increasing document number.
1539 <p>DocDelta: if frequencies are indexed, this determines both
1540 the document number and the frequency. In
1541 particular, DocDelta/2 is the difference between
1542 this document number and the previous document
1543 number (or zero when this is the first document in
1544 a TermFreqs). When DocDelta is odd, the frequency
1545 is one. When DocDelta is even, the frequency is
1546 read as another VInt. If frequencies are omitted, DocDelta
1547 contains the gap (not multiplied by 2) between
1548 document numbers and no frequency information is
1551 <p>For example, the TermFreqs for a term which occurs
1552 once in document seven and three times in document
1553 eleven, with frequencies indexed, would be the following
1558 <p> If frequencies were omitted (IndexOptions.DOCS_ONLY) it would be this sequence
1564 <p>DocSkip records the document number before every
1567 document in TermFreqs.
1568 If payloads are disabled for the term's field,
1569 then DocSkip represents the difference from the
1570 previous value in the sequence.
1571 If payloads are enabled for the term's field,
1572 then DocSkip/2 represents the difference from the
1573 previous value in the sequence. If payloads are enabled
1575 then PayloadLength is stored indicating the length
1576 of the last payload before the SkipInterval<sup>th</sup>
1577 document in TermPositions.
1578 FreqSkip and ProxSkip record the position of every
1581 entry in FreqFile and
1582 ProxFile, respectively. File positions are
1583 relative to the start of TermFreqs and Positions,
1584 to the previous SkipDatum in the sequence.
1586 <p>For example, if DocFreq=35 and SkipInterval=16,
1587 then there are two SkipData entries, containing
1593 numbers in TermFreqs. The first FreqSkip names
1594 the number of bytes after the beginning of
1595 TermFreqs that the 16
1598 starts, and the second the number of bytes after
1602 ProxSkip names the number of bytes after the
1603 beginning of Positions that the 16
1605 SkipDatum starts, and the second the number of
1606 bytes after that that the 32
1610 <p>Each term can have multiple skip levels.
1611 The amount of skip levels for a term is NumSkipLevels = Min(MaxSkipLevels, floor(log(DocFreq/log(SkipInterval)))).
1612 The number of SkipData entries for a skip level is DocFreq/(SkipInterval^(Level + 1)), whereas the lowest skip
1613 level is Level=0. <br></br>
1614 Example: SkipInterval = 4, MaxSkipLevels = 2, DocFreq = 35. Then skip level 0 has 8 SkipData entries,
1615 containing the 3<sup>rd</sup>, 7<sup>th</sup>, 11<sup>th</sup>, 15<sup>th</sup>, 19<sup>th</sup>, 23<sup>rd</sup>,
1616 27<sup>th</sup>, and 31<sup>st</sup> document numbers in TermFreqs. Skip level 1 has 2 SkipData entries, containing the
1617 15<sup>th</sup> and 31<sup>st</sup> document numbers in TermFreqs. <br></br>
1618 The SkipData entries on all upper levels > 0 contain a SkipChildLevelPointer referencing the corresponding SkipData
1619 entry in level-1. In the example has entry 15 on level 1 a pointer to entry 15 on level 0 and entry 31 on level 1 a pointer
1620 to entry 31 on level 0.
1624 <section id="Positions"><title>Positions</title>
1627 The .prx file contains the lists of positions that
1628 each term occurs at within documents. Note that
1629 fields omitting positional data do not store
1630 anything into this file, and if all fields in the
1631 index omit positional data then the .prx file will not
1634 <p>ProxFile (.prx) -->
1635 <TermPositions>
1636 <sup>TermCount</sup>
1638 <p>TermPositions -->
1643 <PositionDelta,Payload?>
1647 <PayloadLength?,PayloadData>
1649 <p>PositionDelta -->
1652 <p>PayloadLength -->
1655 <p>PayloadData -->
1656 byte<sup>PayloadLength</sup>
1659 are ordered by term (the term is implicit, from the .tis file).
1662 entries are ordered by increasing document number (the document
1663 number is implicit from the .frq file).
1666 is, if payloads are disabled for the term's field, the difference
1667 between the position of the current occurrence in
1668 the document and the previous occurrence (or zero, if this is the
1669 first occurrence in this document).
1670 If payloads are enabled for the term's field, then PositionDelta/2
1671 is the difference between the current and the previous position. If
1672 payloads are enabled and PositionDelta is odd, then PayloadLength is
1673 stored, indicating the length of the payload at the current term position.
1676 For example, the TermPositions for a
1677 term which occurs as the fourth term in one document, and as the
1678 fifth and ninth term in a subsequent document, would be the following
1679 sequence of VInts (payloads disabled):
1685 is metadata associated with the current term position. If PayloadLength
1686 is stored at the current position, then it indicates the length of this
1687 Payload. If PayloadLength is not stored, then this Payload has the same
1688 length as the Payload at the previous position.
1691 <section id="Normalization Factors"><title>Normalization Factors</title>
1693 <p>There's a single .nrm file containing all norms:
1696 (.nrm) --> NormsHeader,<Norms>
1697 <sup>NumFieldsWithNorms</sup>
1704 --> 'N','R','M',Version
1710 has 4 bytes, last of which is the format version for this file, currently -1.
1713 byte encodes a floating point value. Bits 0-2 contain the 3-bit
1714 mantissa, and bits 3-8 contain the 5-bit exponent.
1717 are converted to an IEEE single float value as follows:
1722 the byte is zero, use a zero float.
1727 set the sign bit of the float to zero;
1732 48 to the exponent and use this as the float's exponent;
1737 the mantissa to the high-order 3 bits of the float's mantissa; and
1743 the low-order 21 bits of the float's mantissa to zero.
1747 <p>A separate norm file is created when the norm values of an existing segment are modified.
1748 When field <em>N</em> is modified, a separate norm file <em>.sN</em>
1749 is created, to maintain the norm values for that field.
1751 <p>Separate norm files are created (when adequate) for both compound and non compound segments.
1755 <section id="Term Vectors"><title>Term Vectors</title>
1757 Term Vector support is an optional on a field by
1758 field basis. It consists of 3 files.
1762 <p>The Document Index or .tvx file.</p>
1763 <p>For each document, this stores the offset
1764 into the document data (.tvd) and field
1767 <p>DocumentIndex (.tvx) --> TVXVersion<DocumentPosition,FieldPosition>
1770 <p>TVXVersion --> Int (TermVectorsReader.CURRENT)</p>
1771 <p>DocumentPosition --> UInt64 (offset in
1773 <p>FieldPosition --> UInt64 (offset in the
1777 <p>The Document or .tvd file.</p>
1778 <p>This contains, for each document, the number of fields, a list of the fields with
1779 term vector info and finally a list of pointers to the field information in the .tvf
1780 (Term Vector Fields) file.</p>
1782 Document (.tvd) --> TVDVersion<NumFields, FieldNums, FieldPositions>
1785 <p>TVDVersion --> Int (TermVectorsReader.FORMAT_CURRENT)</p>
1786 <p>NumFields --> VInt</p>
1787 <p>FieldNums --> <FieldNumDelta>
1788 <sup>NumFields</sup>
1790 <p>FieldNumDelta --> VInt</p>
1791 <p>FieldPositions --> <FieldPositionDelta>
1792 <sup>NumFields-1</sup>
1794 <p>FieldPositionDelta --> VLong</p>
1795 <p>The .tvd file is used to map out the fields that have term vectors stored and
1796 where the field information is in the .tvf file.</p>
1799 <p>The Field or .tvf file.</p>
1800 <p>This file contains, for each field that has a term vector stored, a list of
1801 the terms, their frequencies and, optionally, position and offest information.</p>
1802 <p>Field (.tvf) --> TVFVersion<NumTerms, Position/Offset, TermFreqs>
1803 <sup>NumFields</sup>
1805 <p>TVFVersion --> Int (TermVectorsReader.FORMAT_CURRENT)</p>
1806 <p>NumTerms --> VInt</p>
1807 <p>Position/Offset --> Byte</p>
1808 <p>TermFreqs --> <TermText, TermFreq, Positions?, Offsets?>
1811 <p>TermText --> <PrefixLength, Suffix></p>
1812 <p>PrefixLength --> VInt</p>
1813 <p>Suffix --> String</p>
1814 <p>TermFreq --> VInt</p>
1815 <p>Positions --> <VInt><sup>TermFreq</sup></p>
1816 <p>Offsets --> <VInt, VInt><sup>TermFreq</sup></p>
1820 <li>Position/Offset byte stores whether this term vector has position or offset information stored.</li>
1822 text prefixes are shared. The PrefixLength is the number of initial
1823 characters from the previous term which must be pre-pended to a
1824 term's suffix in order to form the term's text. Thus, if the
1825 previous term's text was "bone" and the term is "boy",
1826 the PrefixLength is two and the suffix is "y".
1828 <li>Positions are stored as delta encoded VInts. This means we only store the difference of the current position from the last position</li>
1829 <li>Offsets are stored as delta encoded VInts. The first VInt is the startOffset, the second is the endOffset.</li>
1837 <section id="Deleted Documents"><title>Deleted Documents</title>
1840 optional, and only exists when a segment contains deletions.
1843 <p>Although per-segment, this file is maintained exterior to compound segment files.
1847 (.del) --> [Format],ByteCount,BitCount, Bits | DGaps (depending on Format)
1850 <p>Format,ByteSize,BitCount -->
1856 <sup>ByteCount</sup>
1860 <DGap,NonzeroByte>
1861 <sup>NonzeroBytesCount</sup>
1868 <p>NonzeroByte -->
1873 is Optional. -1 indicates DGaps. Non-negative value indicates Bits, and that Format is excluded.
1877 indicates the number of bytes in Bits. It is typically
1883 indicates the number of bits that are currently set in Bits.
1887 contains one bit for each document indexed. When the bit
1888 corresponding to a document number is set, that document is marked as
1889 deleted. Bit ordering is from least to most significant. Thus, if
1890 Bits contains two bytes, 0x00 and 0x02, then document 9 is marked as
1895 represents sparse bit-vectors more efficiently than Bits.
1896 It is made of DGaps on indexes of nonzero bytes in Bits,
1897 and the nonzero bytes themselves. The number of nonzero bytes
1898 in Bits (NonzeroBytesCount) is not stored.
1901 if there are 8000 bits and only bits 10,12,32 are set,
1902 DGaps would be used:
1905 (VInt) 1 , (byte) 20 , (VInt) 3 , (Byte) 1
1910 <section id="Limitations"><title>Limitations</title>
1913 When referring to term numbers, Lucene's current
1914 implementation uses a Java <code>int</code> to hold the
1915 term index, which means the maximum number of unique
1916 terms in any single index segment is ~2.1 billion times
1917 the term index interval (default 128) = ~274 billion.
1918 This is technically not a limitation of the index file
1919 format, just of Lucene's current implementation.
1922 Similarly, Lucene uses a Java <code>int</code> to refer
1923 to document numbers, and the index file format uses an
1924 <code>Int32</code> on-disk to store document numbers.
1925 This is a limitation of both the index file format and
1926 the current implementation. Eventually these should be
1927 replaced with either <code>UInt64</code> values, or
1928 better yet, <code>VInt</code> values which have no