jwo.utils.expression
Class Tokenizer

java.lang.Object
  jwo.utils.expression.Tokenizer

public class Tokenizer

extends Object

A tokenizer that uses regular expressions to parse text. Modified from original code described at www.devx.com/premier/mgznarch/javapro/2001/05may01/ps0105/ps0105.asp.

Version:

2.1, 15th February, 2001, modified 18th July, 2003.

Author:

Daniel Savarese, modified by Jo Wood.

Field Summary

static String

DEFAULT_SEPARATOR

Constructor Summary

Tokenizer()
Creates a default tokenizer with no initial expression to process.

Tokenizer(String expressionText)
Creates a tokenizer using the given initial expression.

Method Summary

void	addToken(String lexeme, Object token) Adds a token to the currently stored regular expression to use for processing.
Token	nextToken() Returns the next token in the input.
void	reset() Resets the list of tokens to process.
void	setInput(String expressionText) Sets the input sequence of characters to process.
void	setSeparator(String separatorText) Sets the separator used to distinguish between tokens.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_SEPARATOR

public static final String DEFAULT_SEPARATOR

See Also:

Constant Field Values

Constructor Detail

Tokenizer

public Tokenizer()

Creates a default tokenizer with no initial expression to process.

Tokenizer

public Tokenizer(String expressionText)

Creates a tokenizer using the given initial expression.

Parameters:

expressionText - Expression to process.

Method Detail

setInput

public void setInput(String expressionText)
              throws PatternSyntaxException

Sets the input sequence of characters to process.

Parameters:

expressionText - Expression to process.

Throws:

PatternSyntaxException

setSeparator

public void setSeparator(String separatorText)
                  throws PatternSyntaxException

Sets the separator used to distinguish between tokens. The default is any whitespace characters ("\\s+").

Parameters:

separatorText - Regular expression identifying token separators.

Throws:

PatternSyntaxException

addToken

public void addToken(String lexeme,
                     Object token)
              throws PatternSyntaxException

Adds a token to the currently stored regular expression to use for processing.

Parameters:

lexeme - The expression lexeme to add.

token - The type of token to add.

Throws:

PatternSyntaxException

reset

public void reset()

Resets the list of tokens to process.

nextToken

public Token nextToken()
                throws TokenNotFoundException

Returns the next token in the input. If at end of input returns null. If no token is found, throws an exception.

Throws:

TokenNotFoundException