antlr

Class TokenStreamRewriteEngine

Implemented Interfaces:
IASDebugStream, TokenStream

public class TokenStreamRewriteEngine
extends Object
implements TokenStream, IASDebugStream

This token stream tracks the *entire* token stream coming from a lexer, but does not pass on the whitespace (or whatever else you want to discard) to the parser. This class can then be asked for the ith token in the input stream. Useful for dumping out the input stream exactly after doing some augmentation or other manipulations. Tokens are index from 0..n-1 You can insert stuff, replace, and delete chunks. Note that the operations are done lazily--only if you convert the buffer to a String. This is very efficient because you are not moving data around all the time. As the buffer of tokens is converted to strings, the toString() method(s) check to see if there is an operation at the current index. If so, the operation is done and then normal String rendering continues on the buffer. This is like having multiple Turing machine instruction streams (programs) operating on a single input tape. :) Since the operations are done lazily at toString-time, operations do not screw up the token index values. That is, an insert operation at token index i does not change the index values for tokens i+1..n-1. Because operations never actually alter the buffer, you may always get the original token stream back without undoing anything. Since the instructions are queued up, you can easily simulate transactions and roll back any changes if there is an error just by removing instructions. For example, TokenStreamRewriteEngine rewriteEngine = new TokenStreamRewriteEngine(lexer); JavaRecognizer parser = new JavaRecognizer(rewriteEngine); ... rewriteEngine.insertAfter("pass1", t, "foobar");} rewriteEngine.insertAfter("pass2", u, "start");} System.out.println(rewriteEngine.toString("pass1")); System.out.println(rewriteEngine.toString("pass2")); You can also have multiple "instruction streams" and get multiple rewrites from a single pass over the input. Just name the instruction streams and use that name again when printing the buffer. This could be useful for generating a C file and also its header file--all from the same buffer. If you don't use named rewrite streams, a "default" stream is used. Terence Parr, parrt at antlr.org University of San Francisco February 2004

Nested Class Summary

(package private) static class
TokenStreamRewriteEngine.DeleteOp
(package private) static class
TokenStreamRewriteEngine.InsertBeforeOp
(package private) static class
TokenStreamRewriteEngine.ReplaceOp
(package private) static class
TokenStreamRewriteEngine.RewriteOperation

Field Summary

static String
DEFAULT_PROGRAM_NAME
static int
MIN_TOKEN_INDEX
static int
PROGRAM_INIT_SIZE
protected BitSet
discardMask
Which (whitespace) token(s) to throw out
protected int
index
track index of tokens
protected Map
lastRewriteTokenIndexes
Map String (program name) -> Integer index
protected Map
programs
You may have multiple, named streams of rewrite operations.
protected TokenStream
stream
Who do we suck tokens from?
protected List
tokens
Track the incoming list of tokens

Constructor Summary

TokenStreamRewriteEngine(TokenStream upstream)
TokenStreamRewriteEngine(TokenStream upstream, int initialSize)

Method Summary

protected void
addToSortedRewriteList(String programName, TokenStreamRewriteEngine.RewriteOperation op)
protected void
addToSortedRewriteList(TokenStreamRewriteEngine.RewriteOperation op)
If op.index > lastRewriteTokenIndexes, just add to the end.
void
delete(String programName, Token from, Token to)
void
delete(String programName, int from, int to)
void
delete(Token indexT)
void
delete(Token from, Token to)
void
delete(int index)
void
delete(int from, int to)
void
deleteProgram()
void
deleteProgram(String programName)
Reset the program so that no instructions exist
void
discard(int ttype)
String
getEntireText()
Returns the entire text input to the lexer.
int
getLastRewriteTokenIndex()
protected int
getLastRewriteTokenIndex(String programName)
TokenOffsetInfo
getOffsetInfo(Token token)
Returns the offset information for the token
protected List
getProgram(String name)
TokenWithIndex
getToken(int i)
int
getTokenStreamSize()
int
index()
void
insertAfter(String programName, Token t, String text)
void
insertAfter(String programName, int index, String text)
void
insertAfter(Token t, String text)
void
insertAfter(int index, String text)
void
insertBefore(String programName, Token t, String text)
void
insertBefore(String programName, int index, String text)
void
insertBefore(Token t, String text)
void
insertBefore(int index, String text)
Token
nextToken()
void
replace(String programName, Token from, Token to, String text)
void
replace(String programName, int from, int to, String text)
void
replace(Token indexT, String text)
void
replace(Token from, Token to, String text)
void
replace(int index, String text)
void
replace(int from, int to, String text)
void
rollback(String programName, int instructionIndex)
Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream.
void
rollback(int instructionIndex)
protected void
setLastRewriteTokenIndex(String programName, int i)
int
size()
String
toDebugString()
String
toDebugString(int start, int end)
String
toOriginalString()
String
toOriginalString(int start, int end)
String
toString()
String
toString(String programName)
String
toString(String programName, int start, int end)
String
toString(int start, int end)

Field Details

DEFAULT_PROGRAM_NAME

public static final String DEFAULT_PROGRAM_NAME

MIN_TOKEN_INDEX

public static final int MIN_TOKEN_INDEX
Field Value:
0

PROGRAM_INIT_SIZE

public static final int PROGRAM_INIT_SIZE
Field Value:
100

discardMask

protected BitSet discardMask
Which (whitespace) token(s) to throw out

index

protected int index
track index of tokens

lastRewriteTokenIndexes

protected Map lastRewriteTokenIndexes
Map String (program name) -> Integer index

programs

protected Map programs
You may have multiple, named streams of rewrite operations. I'm calling these things "programs." Maps String (name) -> rewrite (List)

stream

protected TokenStream stream
Who do we suck tokens from?

tokens

protected List tokens
Track the incoming list of tokens

Constructor Details

TokenStreamRewriteEngine

public TokenStreamRewriteEngine(TokenStream upstream)

TokenStreamRewriteEngine

public TokenStreamRewriteEngine(TokenStream upstream,
                                int initialSize)

Method Details

addToSortedRewriteList

protected void addToSortedRewriteList(String programName,
                                      TokenStreamRewriteEngine.RewriteOperation op)

addToSortedRewriteList

protected void addToSortedRewriteList(TokenStreamRewriteEngine.RewriteOperation op)
If op.index > lastRewriteTokenIndexes, just add to the end. Otherwise, do linear

delete

public void delete(String programName,
                   Token from,
                   Token to)

delete

public void delete(String programName,
                   int from,
                   int to)

delete

public void delete(Token indexT)

delete

public void delete(Token from,
                   Token to)

delete

public void delete(int index)

delete

public void delete(int from,
                   int to)

deleteProgram

public void deleteProgram()

deleteProgram

public void deleteProgram(String programName)
Reset the program so that no instructions exist

discard

public void discard(int ttype)

getEntireText

public String getEntireText()
Returns the entire text input to the lexer.
Specified by:
getEntireText in interface IASDebugStream
Returns:
The entire text or null, if error occured or System.in was used.

getLastRewriteTokenIndex

public int getLastRewriteTokenIndex()

getLastRewriteTokenIndex

protected int getLastRewriteTokenIndex(String programName)

getOffsetInfo

public TokenOffsetInfo getOffsetInfo(Token token)
Returns the offset information for the token
Specified by:
getOffsetInfo in interface IASDebugStream
Parameters:
token - the token whose information need to be retrieved
Returns:
offset info, or null

getProgram

protected List getProgram(String name)

getToken

public TokenWithIndex getToken(int i)

getTokenStreamSize

public int getTokenStreamSize()

index

public int index()

insertAfter

public void insertAfter(String programName,
                        Token t,
                        String text)

insertAfter

public void insertAfter(String programName,
                        int index,
                        String text)

insertAfter

public void insertAfter(Token t,
                        String text)

insertAfter

public void insertAfter(int index,
                        String text)

insertBefore

public void insertBefore(String programName,
                         Token t,
                         String text)

insertBefore

public void insertBefore(String programName,
                         int index,
                         String text)

insertBefore

public void insertBefore(Token t,
                         String text)

insertBefore

public void insertBefore(int index,
                         String text)

nextToken

public Token nextToken()
            throws TokenStreamException
Specified by:
nextToken in interface TokenStream

replace

public void replace(String programName,
                    Token from,
                    Token to,
                    String text)

replace

public void replace(String programName,
                    int from,
                    int to,
                    String text)

replace

public void replace(Token indexT,
                    String text)

replace

public void replace(Token from,
                    Token to,
                    String text)

replace

public void replace(int index,
                    String text)

replace

public void replace(int from,
                    int to,
                    String text)

rollback

public void rollback(String programName,
                     int instructionIndex)
Rollback the instruction stream for a program so that the indicated instruction (via instructionIndex) is no longer in the stream. UNTESTED!

rollback

public void rollback(int instructionIndex)

setLastRewriteTokenIndex

protected void setLastRewriteTokenIndex(String programName,
                                        int i)

size

public int size()

toDebugString

public String toDebugString()

toDebugString

public String toDebugString(int start,
                            int end)

toOriginalString

public String toOriginalString()

toOriginalString

public String toOriginalString(int start,
                               int end)

toString

public String toString()

toString

public String toString(String programName)

toString

public String toString(String programName,
                       int start,
                       int end)

toString

public String toString(int start,
                       int end)