public class MemoryTokenizer extends Tokenizer
tokenVectorMap
Constructor and Description |
---|
MemoryTokenizer() |
MemoryTokenizer(java.lang.String tokenizers) |
Modifier and Type | Method and Description |
---|---|
java.util.List<Token> |
tokenize(java.lang.String str)
Splits the document into tokens.
|
void |
tokenizeString(java.lang.String str) |
getTokenVectorMap, iterator, printTokens, tokenize, toString
public MemoryTokenizer(java.lang.String tokenizers)
public MemoryTokenizer()
public void tokenizeString(java.lang.String str)
public java.util.List<Token> tokenize(java.lang.String str)
Tokenizer