public class MinimumLengthTokenizer extends Tokenizer
tokenVectorMap
Constructor and Description |
---|
MinimumLengthTokenizer(java.lang.String length)
Constructor that sets the minimum token length to be considered
|
Modifier and Type | Method and Description |
---|---|
java.util.List<Token> |
tokenize(java.util.List<Token> tokens)
Eliminates tokens that are shorter than the length specified in the constructor
|
getTokenVectorMap, iterator, printTokens, tokenize, toString
public MinimumLengthTokenizer(java.lang.String length)
length
- the string value of the minimum token length to be included