Skip to Main Content

Java Programming

Announcement

For appeals, questions and feedback about Oracle Forums, please email oracle-forums-moderators_us@oracle.com. Technical questions should be asked in the appropriate category. Thank you!

Peeking or pushing back into a StringTokenizer?

807588Feb 26 2009 — edited Mar 5 2009
Hello all, I am working on creating a Lexer for C using Java, and am having a hard time with the StringTokenizer, which I have to use. To recognize tokens such as "==" and "<=" I know I need either a peek method or a pushback method, however I have no idea how to construct these with a StringTokenizer, without consuming the token. My code is something like this:

class Lexer:

private StringTokenizer st;

method nextToken():

String nextVal;

if (token.equals("="))
{
while (st.hasMoreTokens())
{
nextVal = st.nextToken();
if (!nextVal.equals("="))
{
pushBack(nextVal); //PUSH TOKEN BACK INTO THE STRING TOKENIZER
return new Token(Token.EQUALS);
}
else {
return new Token(Token.REPBY);
}
}
}

The code should work as follows.... if the token read in is a "=" it should check for the token after that. As long as it is not another "=" then return the Equals token: a single "=". If it reaches the Else statement, we know the next token is another "=", so we return the ReplacedBy token: "==". The code works for every "==" but does not for the single "=". This is because it is consumed by the "st.nextToken()" call, so the token after the single "=" is never read. ("=true" would output as just an "=", the "true" is consumed and lost) Anyways, I know the token must be pushed back into the StringTokenizer stream when there is only a single "=", but I am unsure about what the "pushBack" method should be. I was thinking something like this (but of course it doesnt work):

public void pushBack(String token)
{
st = new StringTokenizer(token + st.nextToken());
}


Or, we could use a peek method instead of a pushBack method.... using some sort of temp variable as a buffer.... but again I am unsure how. Any help?
Comments
Locked Post
New comments cannot be posted to this locked post.
Post Details
Locked on Apr 2 2009
Added on Feb 26 2009
9 comments
845 views