public class MaximumLengthTokenizer extends Tokenizer
tokenVectorMap
Constructor and Description |
---|
MaximumLengthTokenizer(java.lang.String length)
Constructor that sets the maximum token length to be considered
|
Modifier and Type | Method and Description |
---|---|
java.util.List<Token> |
tokenize(java.util.List<Token> tokens)
Eliminates tokens that are longer than the length specified in the constructor
|
getTokenVectorMap, iterator, printTokens, tokenize, toString
public MaximumLengthTokenizer(java.lang.String length)
length
- the string value of the maximum token length to be included