NEWLINE_TOKEN, nextToken| Constructor and Description |
|---|
WordSegmentingTokenizer(WordSegmenter segmenter,
Reader r) |
WordSegmentingTokenizer(WordSegmenter segmenter,
Tokenizer<CoreLabel> tokenizer) |
| Modifier and Type | Method and Description |
|---|---|
static TokenizerFactory<HasWord> |
factory(WordSegmenter wordSegmenter) |
protected HasWord |
getNext()
Internally fetches the next token.
|
hasNext, next, peek, remove, tokenizeclone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitforEachRemainingpublic WordSegmentingTokenizer(WordSegmenter segmenter, Reader r)
public WordSegmentingTokenizer(WordSegmenter segmenter, Tokenizer<CoreLabel> tokenizer)
protected HasWord getNext()
AbstractTokenizergetNext in class AbstractTokenizer<HasWord>public static TokenizerFactory<HasWord> factory(WordSegmenter wordSegmenter)