I'm having a some trouble using StreamTokenizer to split up an input stream into tokens, and then put them into an ArrayList.
My problem is that when the ST encounters a "-", ttype is set to 45. As a consequence, "-"'s are not passed into my ArrayList.
I know that the return of 45 is the ASCII value of "-". The ST API specs state that 0-9, "." and "-" are treated as numerals.
I have tried with and without wordChar settings. With and without parseNumbers()
-69 -> -69.0 (ttype = -2)
69 -> 69.0 (ttype = -2)
69- -> 69.0 + ttype = 45
69-- -> 69.0 + ttype = 45 + ttype = 45
i- -> i- (ttype = -3)
i-- -> i-- (ttype = -3)
i-j -> i-j (ttype = -3)
i - j -> i +ttype=45 + j
Question 1: Why is the ttype = 45 and not -2 (TT_NUMBER)?
Question 2: How can I fix it?
Thanks in advance
This is my code:
// This reads in the file
BufferedReader inFile = new BufferedReader (new
// Creates the stream for the file
StreamTokenizer streamT = new StreamTokenizer(inFile);
streamT.wordChars(33,47); //characters make up words .
streamT.wordChars(58,64); // 48-57 are numerals 58-64 are some punctuation stuff
streamT.wordChars(91,96); // 65-90 are already wordchars 91-96 are more punctuation
streamT.wordChars(123,126);// 97-121 are already wordchars 123-126 are more punctuation
streamT.parseNumbers(); // this says to recognise numbers as numbers.
streamT.eolIsSignificant(true); // the end of line is significant, and returns a different token, not just whitespace
// creates array to hold the seperate tokens from the program
ArrayList programTokens = new ArrayList();
int printIndex = 0; // this is here for debugging
// while the next token isn't the end of file..
while(streamT.nextToken() != StreamTokenizer.TT_EOF)
System.out.println(streamT.ttype); // this is here for debugging
// if the type of variable is a word, then add the word
if(streamT.ttype == StreamTokenizer.TT_WORD)
// if the type of variable is a number, then add the number
else if(streamT.ttype == StreamTokenizer.TT_NUMBER)
// ArrayList stores objects, so I convert my number to a string
String stringInput = Double.toString(streamT.nval);
// if the type of variable is EOL, then print EOL.
else if(streamT.ttype == StreamTokenizer.TT_EOL)
System.out.println(programTokens.get(printIndex)); // this is here for debugging
printIndex++; // this is here for debugging