Overview | Package | Class | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## com.singularsys.jep.configurableparser
Class Tokenizer

    java.lang.Object
      com.singularsys.jep.configurableparser.Tokenizer

    public class Tokenizer

extends java.lang.Object

Breaks the input into a set of tokens.

Field Summary
`protected java.io.BufferedReader` | `br`
`protected java.lang.String` | `currentLine`
`protected int` | `currentLineNumber`
`protected int` | `currentPos`
`static TerminatorToken` | `EOF` A token which represents the end of file/input.
`static TerminatorToken` | `EOL` A token which represents and end of line.
`protected java.util.List<TokenMatcher>` | `matchers`
`protected java.util.List<Token>` | `tokens`

Constructor Summary
`Tokenizer(java.io.BufferedReader br, java.util.List<TokenMatcher> matchers)` Construct a tokenizer for a given input.

Method Summary
`java.util.List<Token>` | `scan()` Scans the input, breaking it down into tokens.
`static java.lang.String` | `toString(java.util.List<Token> toks)`

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Field Detail

### matchers

    protected java.util.List<TokenMatcher> matchers

### currentPos

    protected int currentPos

### currentLine

    protected java.lang.String currentLine

### currentLineNumber

    protected int currentLineNumber

### br

    protected java.io.BufferedReader br

### tokens

    protected java.util.List<Token> tokens

### EOL

    public static TerminatorToken EOL

  A token which represents and end of line.

### EOF

    public static TerminatorToken EOF

  A token which represents the end of file/input. A TokenMatcher can return this symbol if it wishes to terminate processing of the input.

Constructor Detail

### Tokenizer

    public Tokenizer(java.io.BufferedReader br,
                     java.util.List<TokenMatcher> matchers)

  Construct a tokenizer for a given input.

  Parameters:
    `br` - the input to read from.
    `matchers` - a set of TokenMatchers to match the given input.

Method Detail

### scan

    public java.util.List<Token> scan()
                               throws ParseException

  Scans the input, breaking it down into tokens. Returns null if attempts to read past end of the file

  Returns:
    the list of tokens that the input was broken down into
  Throws:
    `ParseException`

### toString

    public static java.lang.String toString(java.util.List<Token> toks)

Overview | Package | Class | Tree | Deprecated | Index | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2008 Singular Systems http://www.singularsys.com/jep