Recent changes to FirstCodeGenerator

WikiPage FirstCodeGenerator modified by Tom Harwood

Tom Harwood — Wed, 20 Mar 2013 23:16:32 -0000

--- v22
+++ v23
@@ -7,7 +7,7 @@
 The code generator has four subsystems, all contained within `src/jburg/tutorial/first/first.jbg`:

 
-Reduction Selection: This is the primary code generator subsystem; it is formally known as a Bottom-Up Rewrite Machine, or _BURM_. The BURM works from the bottom of the tree (the leaves, i.e., nodes with no child nodes) towards the root.  At each level, the BURM performs a pattern match on the tree to determine the best available reductions to all feasible goal states; these reductions feed the parent node's pattern match to eventually compute an optimum sequence of reductions to rewrite the input tree.
+
Reduction Selection: This is the primary code generator subsystem; it is formally known as a Bottom-Up Rewrite Machine, or BURM. The BURM works from the bottom of the tree (the leaves, i.e., nodes with no child nodes) towards the root.  At each level, the BURM performs a pattern match on the tree to determine the best available reductions to all feasible goal states; these reductions feed the parent node's pattern match to eventually compute an optimum sequence of reductions to rewrite the input tree.

 
Reducer: The reducer is the action code that actually performs the tree rewrite.

WikiPage FirstCodeGenerator modified by Tom Harwood

Tom Harwood — Wed, 20 Mar 2013 21:23:42 -0000

--- v21
+++ v22
@@ -1,10 +1,10 @@
-# First Compiler: Code Generator
+# First Example Compiler: Code Generator

 [TOC]

 ## Overview

-The code generator has four subsystems:
+The code generator has four subsystems, all contained within `src/jburg/tutorial/first/first.jbg`:

 
 Reduction Selection: This is the primary code generator subsystem; it is formally known as a Bottom-Up Rewrite Machine, or _BURM_. The BURM works from the bottom of the tree (the leaves, i.e., nodes with no child nodes) towards the root.  At each level, the BURM performs a pattern match on the tree to determine the best available reductions to all feasible goal states; these reductions feed the parent node's pattern match to eventually compute an optimum sequence of reductions to rewrite the input tree.
@@ -14,126 +14,83 @@
 
Back End Management: This code generator's back end manager is a simple TemplateManager.

 
Semantic Analysis: Even this rudimentary compiler needs some semantic analysis. There are several semantic analysis tasks:
-
+  
    Register Allocation: register allocation in this compiler is very limited.
   
 Stack Management: local variables are defined as stack offsets.  Stack management
        is simplified because the variables are all 4-byte integers.
   
 Data Segment Management: most of the compiler's output is executable program text,
-       but constant declarations (mostly string constants) go into the data section.
-
+       but constant declarations (i.e., string constants) go into the data section.
+  
 

-## Reduction Selection
+## Code Generator Preamble

-### Pattern-Matching Rules
+The preamble consists of various [directives](DirectivesGuide).  Of note are:

-All rewrites start with a pattern match.  Pattern-matching rules are syntactically recognizable by their format:
+    package jburg.tutorial.first;

-    goalState = NODE_TYPE(subgoal subtree): cost
+Declares the code generator's package, consistent with the package of the front end and the driver.

-The key feature that distinguishes a pattern-matching rule from other rules is the `(subgoal subtree)` section. All pattern-matching rules have this; pattern matches at leaf nodes specify their (non-existent) subtrees' goals as `(void)`.
+    header {
+        import org.antlr.runtime.tree.*; // etc.

-#### Goal (a.k.a. nonterminal) States
+Import the classes the code generator uses.  Also note:

-Every rewrite is done to produce a new kind of result; these results are grouped into classes known as _goal states_ or _nonterminal_ states. What a particular goal state means is up to you, as the programmer of the pattern matcher; the pattern matcher does not know anything about a goal state's meaning.
+        import static jburg.tutorial.first.TokenTypes.*;

-Examples of goal states typically found in a code generator:
-* `statement`
-* `intExpression`
-* `intConstant`
-* `stringConstant`
+The build system generates a TokenTypes abstract class from ANTLR's .tokens file; the code generator could refer to these as, e.g., `TokenTypes.PLUS`, but it's easier to read the specification if they're simple identifiers.  The deprecated `implements` directive performed a similar function for the values in the ANTLR2-generated interface.

-### Subtrees and Subgoals
+    INodeType CommonTree;
+    INodeAdapter jburg.burg.inode.Antlr3JavaAdapter;

-Any node that has children must match a pattern that has feasible and sensible goals for all the children.  For example, an ADD node generally does single-mode arithmetic:
+The front end uses ANTLR's builtin tree generation facilities to generate trees made up of CommonTree nodes.  JBurg has a builtin `[INodeAdapter](INodeAdapter)` class that generates appropriate method calls to get the nodes' type, number of childen, and _n_'th child.

-    intExpression = ADD(intExpression lhs, intExpression rhs): cost
+    ReturnType Object;
+    ReturnType intConstant = Integer;
+    ReturnType name = String;

-This pattern will match a node whose operator code is ADD, and which has two children, both of which can be reduced to `intExpression` results.  The result of applying this rule will itself be an `intExpression`.
+The code generator generates MIPS/SPIM assembly using StringTemplate objects, and StringTemplate objects' attributes are of type Object; so, for the most part, the particular type of object that a nonterminal class reduces to doesn't matter.

-The subtrees' subgoals affect their parent's pattern match. For example, suppose we have two rules to add integers, one of which is an add immediate:
+For integer constants and for identifiers, though, it's more convenient to use a representation that's natural, and so these nonterminal types have more specialized return types.

-    intExpression = ADD(intExpression lhs, intExpression rhs): cost1
-    intExpression = ADD(intExpression lhs, intConstant rhs): cost2
+The next block of Java code, which begins with the comment `This "inclass" block is copied verbatim into the generated tree parser/code generator class body`, contains the semantic analysis code and the `TemplateManager` instance that serves as the compiler's back end.

-If a node's second subtree can be reduced to an `intConstant` and the constant is in the range of the target's add immediate range, then the code generator can be programmed to prefer a reduction that emits an `addi` instruction instead of an `add`.
+## Rules

-### Cost Factors
+The first set of [pattern rules](PatternRuleGuide) recognize the ASTs for this language's three statements, and generate code or, in the case of the variable declaration, populate the variable name table:

-How does the code generator select the optimal rule? It totals up the cost factors for each viable alternative, including the cost of the subtrees' subgoals, and selects the least-cost alternative. (In case of a tie, the first rule specified wins.) For the example above, there are two ways to give the add immediate variant a lower cost:
-1. Give the second rule a lower cost than the first rule.
-2. Give the rule that converts an `intConstant` into an `intExpression` a greater aggregate cost.
-
-In practice, both these techniques are likely to be employed.
-
-Cost factors can be supplied in several different ways:
-* As integer literals, e.g., `intExpr = ADD(intExpr terms+): 1`
-* As `JBurg.Constant` values, e.g., `intExpr = ADD(intExpr terms+): OrdinaryInstruction` where `OrdinaryInstruction` has been declared as a JBurg.Constant value in the specification.
-* As _cost functions_, explained below.
-* As references to field values or other values in the compiler's host programming language.
-
-To produce a fast pattern matcher, specify the cost as an int literal or a `JBurg.Constant` wherever possible; this allows JBurg to perform more analysis at compiler-compiler time, which can result in dramatically better pattern matchers.
-
-### Cost Functions
-
-Let's look again at the pattern
-
-    intExpression = ADD(intExpression lhs, intConstant rhs): cost2
-
-We said above that this pattern was feasible (and preferred) if the value of the int constant was within the range of the target's add immediate field, e.g., a signed 16-bit quantity on the MIPS/SPIM platform. The pattern itself does not have any way to specify this constraint, but we can delegate to a method that can check:
-
-    intExpression = ADD(intExpression lhs, intConstant rhs): constantInRange(getNode().getChild(1))
-
-Let's assume that this BURM is working on AST nodes of ANTLR's CommonTree class. The definition of `constantInRange()`, then, will be similar to:
-
-    int constantInRange(CommonTree intConst) {
-        if ( intConst.getType() == INT_LITERAL ) {
-            int value = Integer.parseInt(intCost.getText());
-            return value <= −32768 && value < 32768? 1: Integer.MAX_VALUE;
-        }
-        else
-            return Integer.MAX_VALUE;
+    // Declare a local variable.
+    // This rule returns null, because it allocates
+    // stack space for the local as a side effect.
+    statement = INT_TYPE(name name): 1
+    {
+        this.declareVariable(name);
+        return null;
     }

-In a Java compiler host, `Integer.MAX_VALUE` is an out-of-range value that means the pattern match is not feasible. If any component of the pattern match cost equals `Integer.MAX_VALUE` then the entire pattern match fails; if the aggregate cost of the pattern match would exceed `Integer.MAX_VALUE` without any component equalling `Integer.MAX_VALUE`, then the aggregate cost of the pattern match will be capped at `Integer.MAX_VALUE - 1`.
+Note that the statement-level reductions that emit executable code also call resetTemps(), which resets the temporary register allocator as a side effect.

-### Nonterminal to Nonterminal Rules
+    // Assign a value to a local variable.
+    statement = EQUALS(name lvalue, intExpression rvalue): 1
+    {
+        return resetTemps(templateManager.getTemplate("storeLocal", "offset", offsetOf(lvalue), "rvalue", rvalue));
+    }

-Now let's go back to the integer constant rules. We can reduce an `INT_LITERAL` node to `intConstant` easily enough:
-
-    intConstant = INT_LITERAL(void): 1
-
-And we can then use that rule to build up a pattern that can accept an `intConstant` rewrite of its corresponding child; but suppose we are adding `x + 100000` and we can't use a `addi` instruction? We need to be able to use the `INT_LITERAL` node as an `intExpression`. One way to do that would be to define another pattern match rule:
-
-    intExpression = INT_LITERAL(void): 2
-
-But this duplication is going to become a maintenance problem very quickly, especially since it'll need to be done for every arithmetic operation that has similar variants.  
-
-JBurg provides a facility to continue transforming an already-reduced value, a nonterminal-to-nonterminal transformation.  There are two varieties of these transformational rules.
-
-#### Simple Transformational Rule (Chain Rule)
-
-The simple form of a transformational rule allows one nonterminal state to alias another:
-
-    generalExpression = intExpression;
-
-This form of transformational rule is also called a _chain rule_ in the literature. As the example suggests, it's useful to group various other forms of rules.
-
-#### General Transformational Rule
-
-The second form of transformational rule is more powerful. It defines a generalized conversion from one nonterminal state to another:
-
-    intExpression = intConstant: 1
-    // reducer code to implement conversion, e.g., issue li instruction
-
-## The Reducer
-
-The reducer, in this example compiler, is for the most part a sequence of calls out to the template manager to get appropriate templates. The reducer code is placed inline with the rules, for example:
+Contrast this to an expression-level pattern, which allocates a temp register for the expression's result:

     intExpression = PLUS(intExpression lhs, intExpression rhs):1
     {
         return allocateTemp(templateManager.getTemplate("add", "lhs", lhs, "rhs", rhs));
     }

+Since the number of temporary registers is limited, and the temporary register pool only resets at the statement level, the number of operators in an expression has to be small or the register allocator will run out of registers. This of course would not be the case in a more realistic compiler.
+
+This is an example of a [general transformational rule](TransformationRuleGuide): the code generator will select this rule to transform a `name` -- which is already the result of a successful pattern match on `ID(void)` -- and further transform the name into an intExpression by loading the local variable into a register.
+
+    intExpression = name: 1
+    {
+        return allocateTemp(templateManager.getTemplate("loadLocal", "offset", offsetOf(name)));
+    }
+

 | [First Compiler Intro](FirstCompiler) | [Front End](FrontEnd) | Code Generator | [Templates](TutorialCompilerTemplates) |

WikiPage FirstCodeGenerator modified by Tom Harwood

Tom Harwood — Tue, 19 Mar 2013 15:40:54 -0000

WikiPage FirstCodeGenerator modified by Tom Harwood

Tom Harwood — Tue, 19 Mar 2013 15:33:19 -0000

--- v19
+++ v20
@@ -136,4 +136,4 @@
     }

-| [First Compiler Intro](FirstCompiler) | [Front End](FrontEnd) | Code Generator | [Templates](FirstTemplates) |
+| [First Compiler Intro](FirstCompiler) | [Front End](FrontEnd) | Code Generator | [Templates](TutorialCompilerTemplates) |

WikiPage FirstCodeGenerator modified by Tom Harwood

Tom Harwood — Tue, 19 Mar 2013 15:32:09 -0000

--- v18
+++ v19
@@ -7,7 +7,7 @@
 The code generator has four subsystems:

 
-Reduction Selection: This is the primary code generator subsystem; it is formally known as a [Bottom-Up Rewrite Machine](http://en.wikipedia.org/wiki/BURS), or _BURM_. The BURM works from the bottom of the tree (the leaves, i.e., nodes with no child nodes) towards the root.  At each level, the BURM performs a pattern match on the tree to determine the best available reductions to all feasible goal states; these reductions feed the parent node's pattern match to eventually compute an optimum sequence of reductions to rewrite the input tree.
+
Reduction Selection: This is the primary code generator subsystem; it is formally known as a Bottom-Up Rewrite Machine, or _BURM_. The BURM works from the bottom of the tree (the leaves, i.e., nodes with no child nodes) towards the root.  At each level, the BURM performs a pattern match on the tree to determine the best available reductions to all feasible goal states; these reductions feed the parent node's pattern match to eventually compute an optimum sequence of reductions to rewrite the input tree.

 
Reducer: The reducer is the action code that actually performs the tree rewrite.

@@ -136,4 +136,4 @@
     }

-| [First Compiler Intro](FirstCompiler) | Prev: [Front End](FrontEnd) | Code Generator | Next : [Templates](FirstTemplates) |
+| [First Compiler Intro](FirstCompiler) | [Front End](FrontEnd) | Code Generator | [Templates](FirstTemplates) |

WikiPage FirstCodeGenerator modified by Tom Harwood

Tom Harwood — Tue, 19 Mar 2013 15:17:58 -0000

--- v17
+++ v18
@@ -13,10 +13,13 @@

 Back End Management: This code generator's back end manager is a simple TemplateManager.

-
Semantic Analysis: Even this rudimentary compiler needs some semantic analysis. There are two semantic analysis tasks:
+
Semantic Analysis: Even this rudimentary compiler needs some semantic analysis. There are several semantic analysis tasks:
 
    Register Allocation: register allocation in this compiler is very limited.
-  
 Data Segment Management: most of the compiler's output is executable program text, but constant declarations (mostly string constants) go into the data section.
+  
 Stack Management: local variables are defined as stack offsets.  Stack management
+       is simplified because the variables are all 4-byte integers.
+  
 Data Segment Management: most of the compiler's output is executable program text,
+       but constant declarations (mostly string constants) go into the data section.

WikiPage FirstCodeGenerator modified by Tom Harwood

Tom Harwood — Tue, 19 Mar 2013 02:12:48 -0000

--- v16
+++ v17
@@ -123,7 +123,14 @@
     intExpression = intConstant: 1
     // reducer code to implement conversion, e.g., issue li instruction

+## The Reducer

+The reducer, in this example compiler, is for the most part a sequence of calls out to the template manager to get appropriate templates. The reducer code is placed inline with the rules, for example:
+
+    intExpression = PLUS(intExpression lhs, intExpression rhs):1
+    {
+        return allocateTemp(templateManager.getTemplate("add", "lhs", lhs, "rhs", rhs));
+    }

 | [First Compiler Intro](FirstCompiler) | Prev: [Front End](FrontEnd) | Code Generator | Next : [Templates](FirstTemplates) |

WikiPage FirstCodeGenerator modified by Tom Harwood

Tom Harwood — Tue, 19 Mar 2013 02:10:28 -0000

--- v15
+++ v16
@@ -71,7 +71,7 @@

 To produce a fast pattern matcher, specify the cost as an int literal or a `JBurg.Constant` wherever possible; this allows JBurg to perform more analysis at compiler-compiler time, which can result in dramatically better pattern matchers.

-#### Cost Functions
+### Cost Functions

 Let's look again at the pattern

@@ -94,5 +94,36 @@

 In a Java compiler host, `Integer.MAX_VALUE` is an out-of-range value that means the pattern match is not feasible. If any component of the pattern match cost equals `Integer.MAX_VALUE` then the entire pattern match fails; if the aggregate cost of the pattern match would exceed `Integer.MAX_VALUE` without any component equalling `Integer.MAX_VALUE`, then the aggregate cost of the pattern match will be capped at `Integer.MAX_VALUE - 1`.

+### Nonterminal to Nonterminal Rules
+
+Now let's go back to the integer constant rules. We can reduce an `INT_LITERAL` node to `intConstant` easily enough:
+
+    intConstant = INT_LITERAL(void): 1
+
+And we can then use that rule to build up a pattern that can accept an `intConstant` rewrite of its corresponding child; but suppose we are adding `x + 100000` and we can't use a `addi` instruction? We need to be able to use the `INT_LITERAL` node as an `intExpression`. One way to do that would be to define another pattern match rule:
+
+    intExpression = INT_LITERAL(void): 2
+
+But this duplication is going to become a maintenance problem very quickly, especially since it'll need to be done for every arithmetic operation that has similar variants.  
+
+JBurg provides a facility to continue transforming an already-reduced value, a nonterminal-to-nonterminal transformation.  There are two varieties of these transformational rules.
+
+#### Simple Transformational Rule (Chain Rule)
+
+The simple form of a transformational rule allows one nonterminal state to alias another:
+
+    generalExpression = intExpression;
+
+This form of transformational rule is also called a _chain rule_ in the literature. As the example suggests, it's useful to group various other forms of rules.
+
+#### General Transformational Rule
+
+The second form of transformational rule is more powerful. It defines a generalized conversion from one nonterminal state to another:
+
+    intExpression = intConstant: 1
+    // reducer code to implement conversion, e.g., issue li instruction
+
+
+

 | [First Compiler Intro](FirstCompiler) | Prev: [Front End](FrontEnd) | Code Generator | Next : [Templates](FirstTemplates) |

WikiPage FirstCodeGenerator modified by Tom Harwood

Tom Harwood — Mon, 18 Mar 2013 21:42:34 -0000

--- v14
+++ v15
@@ -1,4 +1,8 @@
 # First Compiler: Code Generator
+
+[TOC]
+
+## Overview

 The code generator has four subsystems:

WikiPage FirstCodeGenerator modified by Tom Harwood

Tom Harwood — Mon, 18 Mar 2013 21:41:45 -0000

--- v13
+++ v14
@@ -89,3 +89,6 @@
     }

 In a Java compiler host, `Integer.MAX_VALUE` is an out-of-range value that means the pattern match is not feasible. If any component of the pattern match cost equals `Integer.MAX_VALUE` then the entire pattern match fails; if the aggregate cost of the pattern match would exceed `Integer.MAX_VALUE` without any component equalling `Integer.MAX_VALUE`, then the aggregate cost of the pattern match will be capped at `Integer.MAX_VALUE - 1`.
+
+
+| [First Compiler Intro](FirstCompiler) | Prev: [Front End](FrontEnd) | Code Generator | Next : [Templates](FirstTemplates) |