org.apache.solr.analysis
Class RemoveDuplicatesTokenFilter

java.lang.Object
  extended by org.apache.lucene.analysis.TokenStream
      extended by org.apache.solr.analysis.BufferedTokenStream
          extended by org.apache.solr.analysis.RemoveDuplicatesTokenFilter

public class RemoveDuplicatesTokenFilter
extends BufferedTokenStream

A TokenFilter which filters out Tokens at the same position and Term text as the previous token in the stream.


Constructor Summary
RemoveDuplicatesTokenFilter(org.apache.lucene.analysis.TokenStream input)
           
 
Method Summary
protected  org.apache.lucene.analysis.Token process(org.apache.lucene.analysis.Token t)
          Process a token.
 
Methods inherited from class org.apache.solr.analysis.BufferedTokenStream
next, output, peek, pushBack, read, write
 
Methods inherited from class org.apache.lucene.analysis.TokenStream
close, next, reset
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

RemoveDuplicatesTokenFilter

public RemoveDuplicatesTokenFilter(org.apache.lucene.analysis.TokenStream input)
Method Detail

process

protected org.apache.lucene.analysis.Token process(org.apache.lucene.analysis.Token t)
                                            throws java.io.IOException
Description copied from class: BufferedTokenStream
Process a token. Subclasses may read more tokens from the input stream, write more tokens to the output stream, or simply return the next token to be output. Subclasses may return null if the token is to be dropped. If a subclass writes tokens to the output stream and returns a non-null Token, the returned Token is considered to be at the head of the token output stream.

Specified by:
process in class BufferedTokenStream
Throws:
java.io.IOException


Copyright © 2006 - 2009 The Apache Software Foundation