public class StripPunctuationTokenizer extends Tokenizer
tokenVectorMap
Constructor and Description |
---|
StripPunctuationTokenizer() |
Modifier and Type | Method and Description |
---|---|
java.util.List<Token> |
tokenize(java.util.List<Token> tokens)
Separates tokens based on punctuation and removes punctuation from tokens
|
getTokenVectorMap, iterator, printTokens, tokenize, toString