jdt-core-home/howto/generate parser/generateParser.html

Parent Directory Parent Directory | Revision Log Revision Log

Revision 1.1 - (view) (download) (as text)

1 : pmulet 1.1 <!doctype html public "-//w3c//dtd html 4.0 transitional//en">
2 :     <html>
3 :     <head>
4 :     <title>How to: Generate the Parser</title>
5 :     <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
6 :     <link rel="stylesheet" href="http://dev.eclipse.org/default_style.css" type="text/css">
7 :     </head>
8 :     <body bgcolor="#FFFFFF" text="#000000">
9 :     <table border=0 cellspacing=5 cellpadding=2 width="100%" >
10 :    
11 :     <tr>
12 :     <td align=LEFT valign=TOP colspan="2" bgcolor="#0080C0"><b><font color="#FFFFFF">
13 :     JDT Core / HowTo: Generate the Parser </font></b></td>
14 :     </tr>
15 :    
16 :     <tr>
17 :     <td ALIGN=RIGHT VALIGN=TOP WIDTH="2%"><img SRC="http://dev.eclipse.org/images/Adarrow.gif" NOSAVE BORDER=0 height=16 width=16></td>
18 :     <td WIDTH="98%"><b>Where to get the parser generator</b><br>
19 :     <blockquote>The parser files and resources are automatically generated using the LPG parser generator. This tools has
20 :     been renamed Jikes Parser generator. You can find more information and latest releases at this <A HREF="http://www-124.ibm.com/developerworks/projects/jikes/">link</A>.
21 :     The latest tool is provided in source format. We don't provide any help for compiling these
22 :     source files. Refer to the link above if you have trouble to get binaries.
23 :     <br>Our grammar is generated using the version 2.30 of LPG. If newer versions fail to generate resources from our
24 :     grammar, please send request to the Jikes Parser Generator team.
25 :     </blockquote>
26 :     </p>
27 :     </td>
28 :     </tr>
29 :     <tr>
30 :     <td ALIGN=RIGHT VALIGN=TOP WIDTH="2%"><img SRC="http://dev.eclipse.org/images/Adarrow.gif" NOSAVE BORDER=0 height=16 width=16></td>
31 :     <td WIDTH="98%"><b>Where to get the grammar</b><br>
32 :     <blockquote>The latest grammar is always located in the <code>grammar()</code> method of the <code>Parser</code> class. Go to the org.eclipse.jdt.core plugins
33 :     directory (<font color="#3366FF">eclipse\plugins\org.eclipse.jdt.core</font> where eclipse is the root of your eclipse installation) and open the
34 :     <font color="#3366FF">jdtcoresrc.zip</font> file. Then search for the <font color="#3366FF">Parser.java</font> inside the
35 :     folder <font color="#3366FF">org\eclipse\jdt\internal\compiler\parser\</font>. You need to look for the method named
36 :     <code>grammar()</code>. Then copy its contents from:
37 :     <PRE>--main options
38 :     %options ACTION, AN=JavaAction.java, GP=java,
39 :     ....
40 :     $end
41 :     -- need a carriage return after the $end
42 :     </PRE>
43 :     into a file called <font color="#3366FF">java.g</font>. It is important to add a carriage return at the end of the last line.
44 :     You can save this file where you want, we will assume from thereon you saved it in <code>d:\temp\</code>.
45 :     </blockquote>
46 :     </p>
47 :     </td>
48 :     </tr>
49 :     <tr>
50 :     <td ALIGN=RIGHT VALIGN=TOP WIDTH="2%"><img SRC="http://dev.eclipse.org/images/Adarrow.gif" NOSAVE BORDER=0 height=16 width=16></td>
51 :     <td WIDTH="98%"><b>What to do with these files and update the parser class...</b><br>
52 :     <blockquote>
53 :     Assuming, the LPG executable (<code>lpg.exe</code> or <code>jikespg.exe</code>) is located inside <code>d:\lpg</code>.
54 :     <ol>
55 :     <li>First in a console, run:
56 :     <PRE>
57 :     d:
58 :     cd \temp
59 :     d:\lpg\lpg.exe java.g
60 :     </PRE>
61 :     </li>
62 :     <li>You will get an output that looks like this:
63 :     <PRE>
64 :     LPG Parser Generator (V2.30) Tue Apr 02 12:49:13 2002
65 :     %OPTIONS ACTION, AN=JavaAction.java, GP=java,
66 :     %OPTIONS FILE-PREFIX=java, ESCAPE=$, PREFIX=TokenName, OUTPUT-SIZE=125 ,
67 :     %OPTIONS NOGOTO-DEFAULT, SINGLE-PRODUCTIONS, LALR=1 , TABLE=TIME ,
68 :     %OPTIONS ERROR_MAPS
69 :     %OPTIONS first follow
70 :     %OPTIONS TRACE=FULL ,
71 :     %OPTIONS VERBOSE
72 :     Options in effect:
73 :     ACTION ACTFILE-NAME=JavaAction.java BLOCKB=/. BLOCKE=./ BYTE CONFLIC
74 :     DEFAULT=5 NODEBUG DEFERRED NOEDIT ERROR-MAPS ESCAPE=$
75 :     FILE-PREFIX=java FIRST FOLLOW GENERATE-PARSER=JAVA NOGOTO-DEFAULT
76 :     HACTFILE-NAME=javahdr.java HBLOCKB=/: HBLOCKE=:/ LALR=1 LIST
77 :     MAX-DISTANCE=30 MIN-DISTANCE=3 NAMES=OPTIMIZED NONT-CHECK ORMARK=|
78 :     OUTPUT-SIZE=125 PREFIX=TokenName READ-REDUCE NOSCOPES NOSHIFT-DEFAULT
79 :     SINGLE-PRODUCTIONS STACK-SIZE=128 STATES SUFFIX= TABLE=TIME TRACE=FU
80 :     VERBOSE WARNINGS XREF
81 :    
82 :    
83 :     This grammar is LALR(1).
84 :    
85 :     Number of Terminals: 105
86 :     Number of Nonterminals: 202
87 :     Number of Productions: 437
88 :     Number of Single Productions: 162
89 :     Number of Items: 1265
90 :     Number of States: 591
91 :     Number of Shift actions: 3482
92 :     Number of Goto actions: 4061
93 :     Number of Shift/Reduce actions: 369
94 :     Number of Goto/Reduce actions: 687
95 :     Number of Reduce actions: 7736
96 :     Number of Shift-Reduce conflicts: 0
97 :     Number of Reduce-Reduce conflicts: 0
98 :     Number of Reductions saved by default: 4913
99 :     Reallocating storage for TIME table, adding 3603 entries
100 :    
101 :     Length of Check table: 16836
102 :     Length of Action table: 16608
103 :     Number of entries in Action Table: 12013
104 :     Percentage of increase: 38.2%
105 :     Highest symbol in Check Table: 307
106 :     Storage Required for Tables: 66888 Bytes, 66K
107 :     Storage Required for Rules: 1308 Bytes
108 :    
109 :    
110 :     Actions in Compressed Tables:
111 :     Number of Shifts: 3482
112 :     Number of Shift/Reduces: 369
113 :     Number of Gotos: 4061
114 :     Number of Goto/Reduces: 687
115 :     Number of Reduces: 2823
116 :     Number of Defaults: 390
117 :    
118 :     Error maps storage:
119 :     Storage required for ACTION_SYMBOLS_BASE map: 1182 Bytes
120 :     Storage required for ACTION_SYMBOLS_RANGE map: 1007 Bytes
121 :     Storage required for NACTION_SYMBOLS_BASE map: 1182 Bytes
122 :     Storage required for NACTION_SYMBOLS_RANGE map: 630 Bytes
123 :     Storage required for SYMBOL_INDEX map: 616 Bytes
124 :     Storage required for STRING_BUFFER map: 4652 Bytes
125 :    
126 :     ***Warning: Base Check vector contains value > 127. 16-bit words used.
127 :     ***Warning: Terminal symbol > 127. 16-bit words used.
128 :     Escaped symbol $eof is an invalid C variable.
129 :    
130 :     Escaped symbol $error is an invalid C variable.
131 :     </PRE>
132 :     It can be quite different if the output changed since the version 2.30 of lpg. The important part is:<br>
133 :     <blockquote><b>This grammar is LALR(1).</b></blockquote>
134 :     This creates in the current directory some java source files and information files.
135 :     <blockquote><table BORDER=1 CELLSPACING=2 CELLPADDING=10>
136 :     <tr>
137 :     <th VALIGN=top align=left>java.l</th>
138 :     <td VALIGN=top>Information generated by lpg/jikespg. Enumarate all the states created for the automaton, etc.</td>
139 :     </tr>
140 :     <tr>
141 :     <th VALIGN=top align=left>JavaAction.java</th>
142 :     <td>It contains the method consumeRule(int) of the class org.eclipse.jdt.internal.compiler.parser.Parser that handles all semantic actions dispatches.</td>
143 :     </tr>
144 :     <tr>
145 :     <th VALIGN=top align=left>javahdr.java</th>
146 :     <td>You don't need this file. It is actually empty.</td>
147 :     </tr>
148 :     <tr>
149 :     <th VALIGN=top align=left>javadcl.java</th>
150 :     <td>This files is used to generate the resources files.</td>
151 :     </tr>
152 :     <tr>
153 :     <th VALIGN=top align=left>javasym.java</th>
154 :     <td>This is the contents of the class org.eclipse.jdt.core.compiler.ITerminalSymbols. You need to replace:<br>
155 :     <ul>
156 :     <li>TokenName$eof with TokenNameEOF</li>
157 :     <li>TokenName$error with TokenNameERROR</li>
158 :     </ul></td>
159 :     </tr>
160 :     <tr>
161 :     <th VALIGN=top align=left>javadef.java</th>
162 :     <td>This is the contents of the class org.eclipse.jdt.internal.compiler.parser.ParserBasicInformation.</td>
163 :     </tr>
164 :     <tr>
165 :     <th VALIGN=top align=left>javaprs.java</th>
166 :     <td>You don't need this file. Its contents is already inlined in the Parser class.</td>
167 :     </tr>
168 :     </table></blockquote>
169 :     </li>
170 :     <li><blockquote>Now we need to update the different classes and resource files.
171 :     </blockquote>
172 :     <ol>
173 :     <li>Copy the contents of the <font color="#3366FF">JavaAction.java</font> file into the <font color="#3366FF">consumeRule(int)</font> method of the org.eclipse.jdt.internal.compiler.parser.Parser class.
174 :     </li>
175 :     <li>The definition of the Parser needs to be updated with two tables from <font color="#3366FF">javadcl.java</font>. Those are <font color="#3366FF">rhs[]</font> and <font color="#3366FF">name[]</font>.
176 :     The following entries in name[] need to be replaced:
177 :     <ul>
178 :     <li><font color="#3366FF">$eof</font> with <font color="#3366FF">UNEXPECTED_EOF</font></li>
179 :     <li><font color="#3366FF">$error</font> with <font color="#3366FF">"Invalid Character"</font></li>
180 :     </ul>
181 :     The previous definition of name[] will guide you.
182 :     </li>
183 :     <li>The class <font color="#3366FF">org.eclipse.jdt.internal.compiler.parser.ParserBasicInformation</font> needs to be updated with the content of the file <font color="#3366FF">javadef.java</font>. Don't copy the
184 :     interface name. Simply copy the field declarations. The actual source of this class will guide you.</li>
185 :     <li>This is the contents of the class org.eclipse.jdt.internal.compiler.parser.TerminalSymbols. You need to replace:<br>
186 :     <ul>
187 :     <li><font color="#3366FF">TokenName$eof</font> with <font color="#3366FF">TokenNameEOF</font></li>
188 :     <li><font color="#3366FF">TokenName$error</font> with <font color="#3366FF">TokenNameERROR</font></li>
189 :     </ul>
190 :     </li>
191 :     <li>The last step is to update the resource files:<br>
192 :     Copy the jdtcore.jar file in d:\temp. Compile this <A HREF="UpdateParserFiles.java">source</A> inside d:\temp. You will have a file UpdateParserFiles.class.
193 :     Then run the following command-line:
194 :     <PRE>
195 :     D:\temp>java -classpath jdtcore.jar;. UpdateParserFiles javadcl.java
196 :     </PRE>
197 :     Once this done, you will end up with 5 new files inside d:\temp. They are called parser&lt;n&gt;.rsc, with n equals to 1..5.
198 :     All these files need to be moved to the org\eclipse\jdt\internal\compiler\parser folder. Now you are ready to execute and test
199 :     the new parser.
200 :     </li>
201 :     </ol>
202 :     </li>
203 :     </ol>
204 :     <P>
205 :     <b>NOTE:</b> <blockquote>Changing the parser is a risky operation if you miss one of the steps above. The resulting parser can be completely
206 :     unpredictable. It can go from crashing to reporting invalid errors. Be sure that you followed all the steps and that all the
207 :     files are updated and recompiled before you run it. </blockquote>
208 :     </P>
209 :     </blockquote>
210 :     </td>
211 :     </tr>
212 :     </table>
213 :     </body>
214 :     </html>