CIT 594 Assignment 4: Tokenizer
Spring 2012, David Matuszek

Purposes:

• To familiarize you with regular expressions.
• To write a class that will be useful in subsequent assignments.

General Idea:

Write a program that decomposes a String into a series of tokens.

A token is one "thing" in the input string; for instance, a word or a number. Since we will be writing a tokenizer for computer programs, we will also have tokens representing operators, and grouping symbols such as parentheses.

Write your program using Eclipse. Use regular expressions to recognize each token. Provide complete JUnit tests for your program, and document it fully with javadoc comments.

Details:

Write the following classes:

TokenType

The entire definition for this class is:

public enum TokenType {
    INT, FLOAT, NAME, OPERATOR, GROUPING_SYMBOL, STRING, COMMENT, ERROR;
}

Token

A Token has a type and a text. The type is one of the aboveTokenType values. The text is a String containing the exact characters that make up the Token. Types are as follows:

• INT -- an unsigned integer, consisting of one or more digits.
• FLOAT -- an unsigned floating-point number, containing either a decimal point, an exponent, or both.
  • An exponent, if present, consists of the letter e or E, an optional sign, and up to three digits.
• NAME -- begins with a letter or an underscore, followed by zero or more letters, digits, and underscores.
• GROUPING_SYMBOL -- any one of the following six symbols: ( ) [ ] { } .
• OPERATOR -- any of the following:
  • Single characters: ~ ` ! @ $ % ^ & * - + = | \ < , > . / ? (but not a grouping symbol, or #)
  • Two-character sequences: += -= *= /= %= == != <= >= (but not ++ or --).
• STRING -- zero or more characters, enclosed in either single quotes (') or double quotes ("). There are no escaped characters, therefore:
  • a single quote cannot occur within a string demarcated by single quotes, and
  • a double quote cannot occur within a string demarcated by double quotes.
• COMMENT -- A Python-style string, beginning with # and extending to the end of the line.
• ERROR -- anything that isn't one of the above, such as a non-ASCII character.

The Token class should have the following constructor:

• public Token(TokenType type, String text)

the following instance variables:

• public final TokenType type;
• public final String text;

and the following methods:

• public String toString() -- returns a string of the form type:text (no spaces).
• public boolean equals(Object obj)
• public int hashCode()

Tokenizer (implements Iterator)

The Tokenizer will have the following constructor and methods:

• public Tokenizer(String input) -- constructor (sets the String to be tokenized).
• public boolean hasNext() -- returns true if there are more tokens to be returned.
• public Token next() -- returns the next token from the string.
• public void backUp() -- "backs up" one token, so that whatever was returned from the most recent call to next() will be returned again the next time next() is called; only one token is remembered, so if you call backUp() multiple times, the second and subsequent calls don't make any difference.

TokenTest

A JUnit test class for Token. Pretty simple.

TokenizerTest

A JUnit test class for your Tokenizer.

Comments:

You don't need a main method. All your testing can be done via JUnit.

Due date

Your program is due before 6am Tuesday, February 14. Zip up your entire Eclipse project, and submit via Blackboard. Only assignments submitted via Blackboard will be accepted--any assignments emailed to me will be discarded without comment. A late penalty of 5 points per day (out of 100 points) will apply.

Because many of you are interviewing this semester, a limited number of 48-hour extensions will be available. To get an extension, email me before 5pm Friday, stating the reason you need the extension. No extensions will be granted after Friday. If you get an extension and fail to get the project in by the extended due date, late points will be counted from the original due date.