public class Splitter extends Tokenizer
tokenVectorMap
Constructor and Description |
---|
Splitter(java.lang.String splitter,
java.util.List<Token> list)
Constructor that takes a string indicating the type and percent for the splitter and a list of tokens to split
|
Modifier and Type | Method and Description |
---|---|
java.util.Map<java.lang.String,java.util.List<Token>> |
tokenizeEachEntry(java.util.List<Tokenizer> tokenizers)
Tokenizes each splitter entry with the list of tokenizers; assumes no FileTokenizers are in the list
|
getTokenVectorMap, iterator, printTokens, tokenize, tokenize, toString
public Splitter(java.lang.String splitter, java.util.List<Token> list)
splitter
- the string containing the splitter name and percentlist
- the list of tokens to splitpublic java.util.Map<java.lang.String,java.util.List<Token>> tokenizeEachEntry(java.util.List<Tokenizer> tokenizers)
tokenizers
- the list of tokenizers