Examples of TokenRewriteStream

org.antlr.runtime.TokenRewriteStream
Useful for dumping out the input stream after doing some augmentation or other manipulations. You can insert stuff, replace, and delete chunks. Note that the operations are done lazily--only if you convert the buffer to a String. This is very efficient because you are not moving data around all the time. As the buffer of tokens is converted to strings, the toString() method(s) check to see if there is an operation at the current index. If so, the operation is done and then normal String rendering continues on the buffer. This is like having multiple Turing machine instruction streams (programs) operating on a single input tape. :) Since the operations are done lazily at toString-time, operations do not screw up the token index values. That is, an insert operation at token index i does not change the index values for tokens i+1..n-1. Because operations never actually alter the buffer, you may always get the original token stream back without undoing anything. Since the instructions are queued up, you can easily simulate transactions and roll back any changes if there is an error just by removing instructions. For example, CharStream input = new ANTLRFileStream("input"); TLexer lex = new TLexer(input); TokenRewriteStream tokens = new TokenRewriteStream(lex); T parser = new T(tokens); parser.startRule(); Then in the rules, you can execute Token t,u; ... input.insertAfter(t, "text to put after t");} input.insertAfter(u, "text after u");} System.out.println(tokens.toString()); Actually, you have to cast the 'input' to a TokenRewriteStream. :( You can also have multiple "instruction streams" and get multiple rewrites from a single pass over the input. Just name the instruction streams and use that name again when printing the buffer. This could be useful for generating a C file and also its header file--all from the same buffer: tokens.insertAfter("pass1", t, "text to put after t");} tokens.insertAfter("pass2", u, "text after u");} System.out.println(tokens.toString("pass1")); System.out.println(tokens.toString("pass2")); If you don't use named rewrite streams, a "default" stream is used as the first example shows.
org.eclipse.persistence.internal.libraries.antlr.runtime.TokenRewriteStream
Useful for dumping out the input stream after doing some augmentation or other manipulations. You can insert stuff, replace, and delete chunks. Note that the operations are done lazily--only if you convert the buffer to a String. This is very efficient because you are not moving data around all the time. As the buffer of tokens is converted to strings, the toString() method(s) check to see if there is an operation at the current index. If so, the operation is done and then normal String rendering continues on the buffer. This is like having multiple Turing machine instruction streams (programs) operating on a single input tape. :) Since the operations are done lazily at toString-time, operations do not screw up the token index values. That is, an insert operation at token index i does not change the index values for tokens i+1..n-1. Because operations never actually alter the buffer, you may always get the original token stream back without undoing anything. Since the instructions are queued up, you can easily simulate transactions and roll back any changes if there is an error just by removing instructions. For example, CharStream input = new ANTLRFileStream("input"); TLexer lex = new TLexer(input); TokenRewriteStream tokens = new TokenRewriteStream(lex); T parser = new T(tokens); parser.startRule(); Then in the rules, you can execute Token t,u; ... input.insertAfter(t, "text to put after t");} input.insertAfter(u, "text after u");} System.out.println(tokens.toString()); Actually, you have to cast the 'input' to a TokenRewriteStream. :( You can also have multiple "instruction streams" and get multiple rewrites from a single pass over the input. Just name the instruction streams and use that name again when printing the buffer. This could be useful for generating a C file and also its header file--all from the same buffer: tokens.insertAfter("pass1", t, "text to put after t");} tokens.insertAfter("pass2", u, "text after u");} System.out.println(tokens.toString("pass1")); System.out.println(tokens.toString("pass2")); If you don't use named rewrite streams, a "default" stream is used as the first example shows.

Examples of org.antlr.runtime.TokenRewriteStream

   */
  public ASTNode parse(String command, Context ctx) throws ParseException {
    LOG.info("Parsing command: " + command);


    HiveLexerX lexer = new HiveLexerX(new ANTLRNoCaseStringStream(command));
    TokenRewriteStream tokens = new TokenRewriteStream(lexer);
    if (ctx != null) {
      ctx.setTokenRewriteStream(tokens);
    }
    HiveParserX parser = new HiveParserX(tokens);
    parser.setTreeAdaptor(adaptor);

View Full Code Here

Examples of org.antlr.runtime.TokenRewriteStream

    ANTLRStringStream stream = new ANTLRStringStream(code.getSourceCode());
    Integer fileId = mapper.map(code.getFilePath());
    String mappedName = String.valueOf(fileId);
    stream.name = mappedName;
    ES3InstrumentLexer lexer = new ES3InstrumentLexer(stream);
    TokenRewriteStream tokens = new TokenRewriteStream(lexer);
    ES3InstrumentParser parser = new ES3InstrumentParser(tokens);
    parser.setTemplateLib(templates);
    try {
      parser.program();
    } catch (Exception e) {
      throw new InstrumentationException(code.getFilePath(), e);
    }
    List<Integer> executableLines = parser.linesMap.get(mappedName);
    return new InstrumentedCode(fileId,
                                code.getFilePath(),
                                executableLines == null ?
                                    Collections.<Integer>emptyList() : executableLines,
                                tokens.toString());
  }

View Full Code Here

Examples of org.antlr.runtime.TokenRewriteStream

      "A : 'a';\n" +
      "B : 'b';\n" +
      "C : 'c';\n");
    CharStream input = new ANTLRStringStream("abc");
    Interpreter lexEngine = new Interpreter(g, input);
    TokenRewriteStream tokens = new TokenRewriteStream(lexEngine);
    tokens.insertBefore(0, "0");
    String result = tokens.toString();
    String expecting = "0abc";
    assertEquals(expecting, result);
  }

View Full Code Here

Examples of org.antlr.runtime.TokenRewriteStream

      "A : 'a';\n" +
      "B : 'b';\n" +
      "C : 'c';\n");
    CharStream input = new ANTLRStringStream("abc");
    Interpreter lexEngine = new Interpreter(g, input);
    TokenRewriteStream tokens = new TokenRewriteStream(lexEngine);
    tokens.insertAfter(2, "x");
    String result = tokens.toString();
    String expecting = "abcx";
    assertEquals(expecting, result);
  }

View Full Code Here

Examples of org.antlr.runtime.TokenRewriteStream

      "A : 'a';\n" +
      "B : 'b';\n" +
      "C : 'c';\n");
    CharStream input = new ANTLRStringStream("abc");
    Interpreter lexEngine = new Interpreter(g, input);
    TokenRewriteStream tokens = new TokenRewriteStream(lexEngine);
    tokens.fill();
    tokens.insertBefore(1, "x");
    tokens.insertAfter(1, "x");
    String result = tokens.toString();
    String expecting = "axbxc";
    assertEquals(expecting, result);
  }

View Full Code Here

Examples of org.antlr.runtime.TokenRewriteStream

      "A : 'a';\n" +
      "B : 'b';\n" +
      "C : 'c';\n");
    CharStream input = new ANTLRStringStream("abc");
    Interpreter lexEngine = new Interpreter(g, input);
    TokenRewriteStream tokens = new TokenRewriteStream(lexEngine);
    tokens.fill();
    tokens.replace(0, "x");
    String result = tokens.toString();
    String expecting = "xbc";
    assertEquals(expecting, result);
  }

View Full Code Here

Examples of org.antlr.runtime.TokenRewriteStream

      "A : 'a';\n" +
      "B : 'b';\n" +
      "C : 'c';\n");
    CharStream input = new ANTLRStringStream("abc");
    Interpreter lexEngine = new Interpreter(g, input);
    TokenRewriteStream tokens = new TokenRewriteStream(lexEngine);
    tokens.fill();
    tokens.replace(2, "x");
    String result = tokens.toString();
    String expecting = "abx";
    assertEquals(expecting, result);
  }

View Full Code Here

Examples of org.antlr.runtime.TokenRewriteStream

      "A : 'a';\n" +
      "B : 'b';\n" +
      "C : 'c';\n");
    CharStream input = new ANTLRStringStream("abc");
    Interpreter lexEngine = new Interpreter(g, input);
    TokenRewriteStream tokens = new TokenRewriteStream(lexEngine);
    tokens.fill();
    tokens.replace(1, "x");
    String result = tokens.toString();
    String expecting = "axc";
    assertEquals(expecting, result);
  }

View Full Code Here

Examples of org.antlr.runtime.TokenRewriteStream

            "WS : ' '+;\n");
        // Tokens: 0123456789
        // Input:  x = 3 * 0;
        CharStream input = new ANTLRStringStream("x = 3 * 0;");
        Interpreter lexEngine = new Interpreter(g, input);
        TokenRewriteStream tokens = new TokenRewriteStream(lexEngine);
        tokens.fill();
        tokens.replace(4, 8, "0"); // replace 3 * 0 with 0


        String result = tokens.toOriginalString();
        String expecting = "x = 3 * 0;";
        assertEquals(expecting, result);


        result = tokens.toString();
        expecting = "x = 0;";
        assertEquals(expecting, result);


        result = tokens.toString(0,9);
        expecting = "x = 0;";
        assertEquals(expecting, result);


        result = tokens.toString(4,8);
        expecting = "0";
        assertEquals(expecting, result);
    }

View Full Code Here

Examples of org.antlr.runtime.TokenRewriteStream

            "WS : ' '+;\n");
        // Tokens: 012345678901234567
        // Input:  x = 3 * 0 + 2 * 0;
        CharStream input = new ANTLRStringStream("x = 3 * 0 + 2 * 0;");
        Interpreter lexEngine = new Interpreter(g, input);
        TokenRewriteStream tokens = new TokenRewriteStream(lexEngine);
        tokens.fill();


        String result = tokens.toOriginalString();
        String expecting = "x = 3 * 0 + 2 * 0;";
        assertEquals(expecting, result);


        tokens.replace(4, 8, "0"); // replace 3 * 0 with 0
        result = tokens.toString();
        expecting = "x = 0 + 2 * 0;";
        assertEquals(expecting, result);


        result = tokens.toString(0,17);
        expecting = "x = 0 + 2 * 0;";
        assertEquals(expecting, result);


        result = tokens.toString(4,8);
        expecting = "0";
        assertEquals(expecting, result);


        result = tokens.toString(0,8);
        expecting = "x = 0";
        assertEquals(expecting, result);


        result = tokens.toString(12,16);
        expecting = "2 * 0";
        assertEquals(expecting, result);


        tokens.insertAfter(17, "// comment");
        result = tokens.toString(12,18);
        expecting = "2 * 0;// comment";
        assertEquals(expecting, result);


        result = tokens.toString(0,8); // try again after insert at end
        expecting = "x = 0";
        assertEquals(expecting, result);
    }

View Full Code Here

0 1 2 3 4 5

TOP

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.