Peeking or pushing back into a StringTokenizer?
807588Feb 26 2009 — edited Mar 5 2009Hello all, I am working on creating a Lexer for C using Java, and am having a hard time with the StringTokenizer, which I have to use. To recognize tokens such as "==" and "<=" I know I need either a peek method or a pushback method, however I have no idea how to construct these with a StringTokenizer, without consuming the token. My code is something like this:
class Lexer:
private StringTokenizer st;
method nextToken():
String nextVal;
if (token.equals("="))
{
while (st.hasMoreTokens())
{
nextVal = st.nextToken();
if (!nextVal.equals("="))
{
pushBack(nextVal); //PUSH TOKEN BACK INTO THE STRING TOKENIZER
return new Token(Token.EQUALS);
}
else {
return new Token(Token.REPBY);
}
}
}
The code should work as follows.... if the token read in is a "=" it should check for the token after that. As long as it is not another "=" then return the Equals token: a single "=". If it reaches the Else statement, we know the next token is another "=", so we return the ReplacedBy token: "==". The code works for every "==" but does not for the single "=". This is because it is consumed by the "st.nextToken()" call, so the token after the single "=" is never read. ("=true" would output as just an "=", the "true" is consumed and lost) Anyways, I know the token must be pushed back into the StringTokenizer stream when there is only a single "=", but I am unsure about what the "pushBack" method should be. I was thinking something like this (but of course it doesnt work):
public void pushBack(String token)
{
st = new StringTokenizer(token + st.nextToken());
}
Or, we could use a peek method instead of a pushBack method.... using some sort of temp variable as a buffer.... but again I am unsure how. Any help?