Skip to content

Most visited

Recently visited

navigation
Added in API level 1

StreamTokenizer

public class StreamTokenizer
extends Object

java.lang.Object
   ↳ java.io.StreamTokenizer


Parses a stream into a set of defined tokens, one at a time. The different types of tokens that can be found are numbers, identifiers, quoted strings, and different comment styles. The class can be used for limited processing of source code of programming languages like Java, although it is nowhere near a full parser.

Summary

Constants

int TT_EOF

The constant representing the end of the stream.

int TT_EOL

The constant representing the end of the line.

int TT_NUMBER

The constant representing a number token.

int TT_WORD

The constant representing a word token.

Fields

public double nval

Contains a number if the current token is a number (ttype == TT_NUMBER).

public String sval

Contains a string if the current token is a word (ttype == TT_WORD).

public int ttype

After calling nextToken(), ttype contains the type of token that has been read.

Public constructors

StreamTokenizer(InputStream is)

This constructor was deprecated in API level 1. Use StreamTokenizer(Reader) instead.

StreamTokenizer(Reader r)

Constructs a new StreamTokenizer with r as source reader.

Public methods

void commentChar(int ch)

Specifies that the character ch shall be treated as a comment character.

void eolIsSignificant(boolean flag)

Specifies whether the end of a line is significant and should be returned as TT_EOF in ttype by this tokenizer.

int lineno()

Returns the current line number.

void lowerCaseMode(boolean flag)

Specifies whether word tokens should be converted to lower case when they are stored in sval.

int nextToken()

Parses the next token from this tokenizer's source stream or reader.

void ordinaryChar(int ch)

Specifies that the character ch shall be treated as an ordinary character by this tokenizer.

void ordinaryChars(int low, int hi)

Specifies that the characters in the range from low to hi shall be treated as an ordinary character by this tokenizer.

void parseNumbers()

Specifies that this tokenizer shall parse numbers.

void pushBack()

Indicates that the current token should be pushed back and returned again the next time nextToken() is called.

void quoteChar(int ch)

Specifies that the character ch shall be treated as a quote character.

void resetSyntax()

Specifies that all characters shall be treated as ordinary characters.

void slashSlashComments(boolean flag)

Specifies whether "slash-slash" (C++-style) comments shall be recognized.

void slashStarComments(boolean flag)

Specifies whether "slash-star" (C-style) comments shall be recognized.

String toString()

Returns the state of this tokenizer in a readable format.

void whitespaceChars(int low, int hi)

Specifies that the characters in the range from low to hi shall be treated as whitespace characters by this tokenizer.

void wordChars(int low, int hi)

Specifies that the characters in the range from low to hi shall be treated as word characters by this tokenizer.

Inherited methods

From class java.lang.Object

Constants

TT_EOF

Added in API level 1
int TT_EOF

The constant representing the end of the stream.

Constant Value: -1 (0xffffffff)

TT_EOL

Added in API level 1
int TT_EOL

The constant representing the end of the line.

Constant Value: 10 (0x0000000a)

TT_NUMBER

Added in API level 1
int TT_NUMBER

The constant representing a number token.

Constant Value: -2 (0xfffffffe)

TT_WORD

Added in API level 1
int TT_WORD

The constant representing a word token.

Constant Value: -3 (0xfffffffd)

Fields

nval

Added in API level 1
double nval

Contains a number if the current token is a number (ttype == TT_NUMBER).

sval

Added in API level 1
String sval

Contains a string if the current token is a word (ttype == TT_WORD).

ttype

Added in API level 1
int ttype

After calling nextToken(), ttype contains the type of token that has been read. When a single character is read, its value converted to an integer is stored in ttype. For a quoted string, the value is the quoted character. Otherwise, its value is one of the following:

  • TT_WORD - the token is a word.
  • TT_NUMBER - the token is a number.
  • TT_EOL - the end of line has been reached. Depends on whether eolIsSignificant is true.
  • TT_EOF - the end of the stream has been reached.

Public constructors

StreamTokenizer

Added in API level 1
StreamTokenizer (InputStream is)

This constructor was deprecated in API level 1.
Use StreamTokenizer(Reader) instead.

Constructs a new StreamTokenizer with is as source input stream. This constructor is deprecated; instead, the constructor that takes a Reader as an argument should be used.

Parameters
is InputStream: the source stream from which to parse tokens.
Throws
NullPointerException if is is null.

StreamTokenizer

Added in API level 1
StreamTokenizer (Reader r)

Constructs a new StreamTokenizer with r as source reader. The tokenizer's initial state is as follows:

  • All byte values 'A' through 'Z', 'a' through 'z', and '\u00A0' through '\u00FF' are considered to be alphabetic.
  • All byte values '\u0000' through '\u0020' are considered to be white space. '/' is a comment character.
  • Single quote '\'' and double quote '"' are string quote characters.
  • Numbers are parsed.
  • End of lines are considered to be white space rather than separate tokens.
  • C-style and C++-style comments are not recognized.

Parameters
r Reader: the source reader from which to parse tokens.

Public methods

commentChar

Added in API level 1
void commentChar (int ch)

Specifies that the character ch shall be treated as a comment character.

Parameters
ch int: the character to be considered a comment character.

eolIsSignificant

Added in API level 1
void eolIsSignificant (boolean flag)

Specifies whether the end of a line is significant and should be returned as TT_EOF in ttype by this tokenizer.

Parameters
flag boolean: true if EOL is significant, false otherwise.

lineno

Added in API level 1
int lineno ()

Returns the current line number.

Returns
int this tokenizer's current line number.

lowerCaseMode

Added in API level 1
void lowerCaseMode (boolean flag)

Specifies whether word tokens should be converted to lower case when they are stored in sval.

Parameters
flag boolean: true if sval should be converted to lower case, false otherwise.

nextToken

Added in API level 1
int nextToken ()

Parses the next token from this tokenizer's source stream or reader. The type of the token is stored in the ttype field, additional information may be stored in the nval or sval fields.

Returns
int the value of ttype.
Throws
IOException if an I/O error occurs while parsing the next token.

ordinaryChar

Added in API level 1
void ordinaryChar (int ch)

Specifies that the character ch shall be treated as an ordinary character by this tokenizer. That is, it has no special meaning as a comment character, word component, white space, string delimiter or number.

Parameters
ch int: the character to be considered an ordinary character.

ordinaryChars

Added in API level 1
void ordinaryChars (int low, 
                int hi)

Specifies that the characters in the range from low to hi shall be treated as an ordinary character by this tokenizer. That is, they have no special meaning as a comment character, word component, white space, string delimiter or number.

Parameters
low int: the first character in the range of ordinary characters.
hi int: the last character in the range of ordinary characters.

parseNumbers

Added in API level 1
void parseNumbers ()

Specifies that this tokenizer shall parse numbers.

pushBack

Added in API level 1
void pushBack ()

Indicates that the current token should be pushed back and returned again the next time nextToken() is called.

quoteChar

Added in API level 1
void quoteChar (int ch)

Specifies that the character ch shall be treated as a quote character.

Parameters
ch int: the character to be considered a quote character.

resetSyntax

Added in API level 1
void resetSyntax ()

Specifies that all characters shall be treated as ordinary characters.

slashSlashComments

Added in API level 1
void slashSlashComments (boolean flag)

Specifies whether "slash-slash" (C++-style) comments shall be recognized. This kind of comment ends at the end of the line.

Parameters
flag boolean: true if // should be recognized as the start of a comment, false otherwise.

slashStarComments

Added in API level 1
void slashStarComments (boolean flag)

Specifies whether "slash-star" (C-style) comments shall be recognized. Slash-star comments cannot be nested and end when a star-slash combination is found.

Parameters
flag boolean: true if /* should be recognized as the start of a comment, false otherwise.

toString

Added in API level 1
String toString ()

Returns the state of this tokenizer in a readable format.

Returns
String the current state of this tokenizer.

whitespaceChars

Added in API level 1
void whitespaceChars (int low, 
                int hi)

Specifies that the characters in the range from low to hi shall be treated as whitespace characters by this tokenizer.

Parameters
low int: the first character in the range of whitespace characters.
hi int: the last character in the range of whitespace characters.

wordChars

Added in API level 1
void wordChars (int low, 
                int hi)

Specifies that the characters in the range from low to hi shall be treated as word characters by this tokenizer. A word consists of a word character followed by zero or more word or number characters.

Parameters
low int: the first character in the range of word characters.
hi int: the last character in the range of word characters.
This site uses cookies to store your preferences for site-specific language and display options.

Hooray!

This class requires API level or higher

This doc is hidden because your selected API level for the documentation is . You can change the documentation API level with the selector above the left navigation.

For more information about specifying the API level your app requires, read Supporting Different Platform Versions.