[KinoSearch] Backwards Compatibility Policy
marvin at rectangular.com
Tue Jan 23 13:59:07 PST 2007
On Jan 22, 2007, at 11:28 PM, Henka wrote:
>> This policy would be put in place as of revision 1.0, which is likely
>> to be 0.20 with a few minor bugfixes after a few weeks of testing.
> As an aside, will 0.20 break backwards compatibility for existing
> I seem to recall that it would.
Yes, it will shatter it. Every current app will fail
catastrophically. Continuity from the current file format to the new
would be too hard and take too long, so it's time to invoke the
"alpha" clause. And once we've committed to that course of action,
doing things halfway doesn't make sense. We don't want subtly
degraded behavior and intermittent failure.
The new format is designed to be more mutable, so we don't have to
resort to such measures again, especially not soon. Below you'll
find a the text of a Lucene JIRA issue I've opened, explaining some
of the reasoning.
Begin forwarded message:
From: "Marvin Humphrey (JIRA)" <jira at apache.org>
Date: January 23, 2007 11:41:49 AM PST
To: java-dev at lucene.apache.org
Subject: [jira] Created: (LUCENE-783) Store all metadata in human-
readable segments file
Reply-To: java-dev at lucene.apache.org
Store all metadata in human-readable segments file
Project: Lucene - Java
Issue Type: Improvement
Reporter: Marvin Humphrey
Various index-reading components in Lucene need metadata in addition
This metadata is presently stored in arbitrary binary headers and
over several files. We should move to concentrate it in a single
this file should be encoded using a human-readable, extensible,
data serialization language -- either XML or YAML.
* Making metadata human-readable makes debugging easier.
makes debugging easier still. Developers benefit from being able
and locate relevant information quickly and with less debug
get a new window through which to peer into the index structure.
* Since metadata is written to a separate file, there would no longer
need to seek back to the beginning of any data file to finish a
solving issue LUCENE-532.
* Special-case parsing code needed for extracting metadata supplied by
different index formats can be pared down. If a value is no longer
necessary, it can just be ignored/discarded.
* Removing headers from the data files simplifies them and makes the
format easier to implement.
* With headers removed, all or nearly all data structures can take the
form of records stacked end to end, so that once a decoder has been
selected, an iterator can read the file from top to tail. To an
this allows us to separate our data-processing algorithms from our
serialization algorithms, decoupling Lucene's code base from its file
format. For instance, instead of further subclassing TermDocs to
"flexible indexing" formats, we might replace it with a
returns a subclass of Posting. The deserialization code would be
contained within the Posting subclass rather than spread out over
subclasses of TermDocs.
* YAML and XML are equally well suited for the task of storing metadata,
but in either case a complete parser would not be needed -- a
of the language will do. KinoSearch 0.20's custom-coded YAML parser
occupies about 600 lines of C -- not too bad, considering how
string handling capabilities are.
KinoSearch mailing list
KinoSearch at rectangular.com
More information about the kinosearch