org.apache.lucene.analysis.ngram
Class EdgeNGramTokenizer
java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.Tokenizer
org.apache.lucene.analysis.ngram.EdgeNGramTokenizer
- All Implemented Interfaces:
- Closeable
public final class EdgeNGramTokenizer
- extends Tokenizer
Tokenizes the input from an edge into n-grams of given size(s).
This Tokenizer
create n-grams from the beginning edge or ending edge of a input token.
MaxGram can't be larger than 1024 because of limitation.
Nested Class Summary |
static class |
EdgeNGramTokenizer.Side
Specifies which side of the input the n-gram should be generated from |
Fields inherited from class org.apache.lucene.analysis.Tokenizer |
input |
Constructor Summary |
EdgeNGramTokenizer(AttributeSource.AttributeFactory factory,
Reader input,
EdgeNGramTokenizer.Side side,
int minGram,
int maxGram)
Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range |
EdgeNGramTokenizer(AttributeSource.AttributeFactory factory,
Reader input,
String sideLabel,
int minGram,
int maxGram)
Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range |
EdgeNGramTokenizer(Reader input,
EdgeNGramTokenizer.Side side,
int minGram,
int maxGram)
Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range |
EdgeNGramTokenizer(Reader input,
String sideLabel,
int minGram,
int maxGram)
Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range |
Method Summary |
void |
end()
|
boolean |
incrementToken()
Returns the next token in the stream, or null at EOS. |
void |
reset()
|
Methods inherited from class org.apache.lucene.util.AttributeSource |
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState |
DEFAULT_SIDE
public static final EdgeNGramTokenizer.Side DEFAULT_SIDE
DEFAULT_MAX_GRAM_SIZE
public static final int DEFAULT_MAX_GRAM_SIZE
- See Also:
- Constant Field Values
DEFAULT_MIN_GRAM_SIZE
public static final int DEFAULT_MIN_GRAM_SIZE
- See Also:
- Constant Field Values
EdgeNGramTokenizer
public EdgeNGramTokenizer(Reader input,
EdgeNGramTokenizer.Side side,
int minGram,
int maxGram)
- Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range
- Parameters:
input
- Reader
holding the input to be tokenizedside
- the EdgeNGramTokenizer.Side
from which to chop off an n-gramminGram
- the smallest n-gram to generatemaxGram
- the largest n-gram to generate
EdgeNGramTokenizer
public EdgeNGramTokenizer(AttributeSource.AttributeFactory factory,
Reader input,
EdgeNGramTokenizer.Side side,
int minGram,
int maxGram)
- Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range
- Parameters:
factory
- AttributeSource.AttributeFactory
to useinput
- Reader
holding the input to be tokenizedside
- the EdgeNGramTokenizer.Side
from which to chop off an n-gramminGram
- the smallest n-gram to generatemaxGram
- the largest n-gram to generate
EdgeNGramTokenizer
public EdgeNGramTokenizer(Reader input,
String sideLabel,
int minGram,
int maxGram)
- Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range
- Parameters:
input
- Reader
holding the input to be tokenizedsideLabel
- the name of the EdgeNGramTokenizer.Side
from which to chop off an n-gramminGram
- the smallest n-gram to generatemaxGram
- the largest n-gram to generate
EdgeNGramTokenizer
public EdgeNGramTokenizer(AttributeSource.AttributeFactory factory,
Reader input,
String sideLabel,
int minGram,
int maxGram)
- Creates EdgeNGramTokenizer that can generate n-grams in the sizes of the given range
- Parameters:
factory
- AttributeSource.AttributeFactory
to useinput
- Reader
holding the input to be tokenizedsideLabel
- the name of the EdgeNGramTokenizer.Side
from which to chop off an n-gramminGram
- the smallest n-gram to generatemaxGram
- the largest n-gram to generate
incrementToken
public boolean incrementToken()
throws IOException
- Returns the next token in the stream, or null at EOS.
- Specified by:
incrementToken
in class TokenStream
- Throws:
IOException
end
public void end()
- Overrides:
end
in class TokenStream
reset
public void reset()
throws IOException
- Overrides:
reset
in class TokenStream
- Throws:
IOException
Copyright © 2000-2013 Apache Software Foundation. All Rights Reserved.