public class CHTBTokenizer extends AbstractTokenizer<String>
NEWLINE_TOKEN, nextToken| Constructor and Description |
|---|
CHTBTokenizer(Reader r)
Constructs a new tokenizer from a Reader.
|
| Modifier and Type | Method and Description |
|---|---|
protected String |
getNext()
Internally fetches the next token.
|
static void |
main(String[] args)
The main() method tokenizes a file in the specified Encoding
and prints it to standard output in the specified Encoding.
|
hasNext, next, peek, remove, tokenizeclone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitforEachRemainingpublic CHTBTokenizer(Reader r)
ConvertEncodingThread, or by specifying
the files encoding explicitly in the Reader with
java.io.InputStreamReader.r - Readerprotected String getNext()
getNext in class AbstractTokenizer<String>public static void main(String[] args) throws IOException
IOException