jdt-core-home/howto/generate parser/generateParser.html

Parent Directory Parent Directory | Revision Log Revision Log

Revision 1.4 - (view) (download) (as text)

1 : pmulet 1.1 <!doctype html public "-//w3c//dtd html 4.0 transitional//en">
2 :     <html>
3 :     <head>
4 :     <title>How to: Generate the Parser</title>
5 :     <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
6 :     <link rel="stylesheet" href="http://dev.eclipse.org/default_style.css" type="text/css">
7 :     </head>
8 :     <body bgcolor="#FFFFFF" text="#000000">
9 :     <table border=0 cellspacing=5 cellpadding=2 width="100%" >
10 :    
11 :     <tr>
12 :     <td align=LEFT valign=TOP colspan="2" bgcolor="#0080C0"><b><font color="#FFFFFF">
13 :     JDT Core / HowTo: Generate the Parser </font></b></td>
14 :     </tr>
15 :    
16 :     <tr>
17 :     <td ALIGN=RIGHT VALIGN=TOP WIDTH="2%"><img SRC="http://dev.eclipse.org/images/Adarrow.gif" NOSAVE BORDER=0 height=16 width=16></td>
18 :     <td WIDTH="98%"><b>Where to get the parser generator</b><br>
19 :     <blockquote>The parser files and resources are automatically generated using the LPG parser generator. This tools has
20 :     been renamed Jikes Parser generator. You can find more information and latest releases at this <A HREF="http://www-124.ibm.com/developerworks/projects/jikes/">link</A>.
21 :     The latest tool is provided in source format. We don't provide any help for compiling these
22 :     source files. Refer to the link above if you have trouble to get binaries.
23 :     <br>Our grammar is generated using the version 2.30 of LPG. If newer versions fail to generate resources from our
24 :     grammar, please send request to the Jikes Parser Generator team.
25 :     </blockquote>
26 :     </p>
27 :     </td>
28 :     </tr>
29 :     <tr>
30 :     <td ALIGN=RIGHT VALIGN=TOP WIDTH="2%"><img SRC="http://dev.eclipse.org/images/Adarrow.gif" NOSAVE BORDER=0 height=16 width=16></td>
31 :     <td WIDTH="98%"><b>Where to get the grammar</b><br>
32 : daudel 1.3 <blockquote>The latest grammar is always located in the <code>java_1_4.g</code> file. Go to the org.eclipse.jdt.core plugins
33 : pmulet 1.1 directory (<font color="#3366FF">eclipse\plugins\org.eclipse.jdt.core</font> where eclipse is the root of your eclipse installation) and open the
34 : daudel 1.3 <font color="#3366FF">grammar</font> directory. Then search for the <font color="#3366FF">java_1_4.g</font> file inside the
35 :     folder <font color="#3366FF">grammar</font>. Then copy its contents from:
36 : pmulet 1.1 <PRE>--main options
37 :     %options ACTION, AN=JavaAction.java, GP=java,
38 :     ....
39 :     $end
40 :     -- need a carriage return after the $end
41 :     </PRE>
42 :     into a file called <font color="#3366FF">java.g</font>. It is important to add a carriage return at the end of the last line.
43 :     You can save this file where you want, we will assume from thereon you saved it in <code>d:\temp\</code>.
44 :     </blockquote>
45 :     </p>
46 :     </td>
47 :     </tr>
48 :     <tr>
49 :     <td ALIGN=RIGHT VALIGN=TOP WIDTH="2%"><img SRC="http://dev.eclipse.org/images/Adarrow.gif" NOSAVE BORDER=0 height=16 width=16></td>
50 :     <td WIDTH="98%"><b>What to do with these files and update the parser class...</b><br>
51 :     <blockquote>
52 :     Assuming, the LPG executable (<code>lpg.exe</code> or <code>jikespg.exe</code>) is located inside <code>d:\lpg</code>.
53 :     <ol>
54 :     <li>First in a console, run:
55 :     <PRE>
56 :     d:
57 :     cd \temp
58 :     d:\lpg\lpg.exe java.g
59 :     </PRE>
60 :     </li>
61 :     <li>You will get an output that looks like this:
62 :     <PRE>
63 :     LPG Parser Generator (V2.30) Tue Apr 02 12:49:13 2002
64 :     %OPTIONS ACTION, AN=JavaAction.java, GP=java,
65 :     %OPTIONS FILE-PREFIX=java, ESCAPE=$, PREFIX=TokenName, OUTPUT-SIZE=125 ,
66 :     %OPTIONS NOGOTO-DEFAULT, SINGLE-PRODUCTIONS, LALR=1 , TABLE=TIME ,
67 :     %OPTIONS ERROR_MAPS
68 :     %OPTIONS first follow
69 :     %OPTIONS TRACE=FULL ,
70 :     %OPTIONS VERBOSE
71 :     Options in effect:
72 :     ACTION ACTFILE-NAME=JavaAction.java BLOCKB=/. BLOCKE=./ BYTE CONFLIC
73 :     DEFAULT=5 NODEBUG DEFERRED NOEDIT ERROR-MAPS ESCAPE=$
74 :     FILE-PREFIX=java FIRST FOLLOW GENERATE-PARSER=JAVA NOGOTO-DEFAULT
75 :     HACTFILE-NAME=javahdr.java HBLOCKB=/: HBLOCKE=:/ LALR=1 LIST
76 :     MAX-DISTANCE=30 MIN-DISTANCE=3 NAMES=OPTIMIZED NONT-CHECK ORMARK=|
77 :     OUTPUT-SIZE=125 PREFIX=TokenName READ-REDUCE NOSCOPES NOSHIFT-DEFAULT
78 :     SINGLE-PRODUCTIONS STACK-SIZE=128 STATES SUFFIX= TABLE=TIME TRACE=FU
79 :     VERBOSE WARNINGS XREF
80 :    
81 :    
82 :     This grammar is LALR(1).
83 :    
84 :     Number of Terminals: 105
85 :     Number of Nonterminals: 202
86 :     Number of Productions: 437
87 :     Number of Single Productions: 162
88 :     Number of Items: 1265
89 :     Number of States: 591
90 :     Number of Shift actions: 3482
91 :     Number of Goto actions: 4061
92 :     Number of Shift/Reduce actions: 369
93 :     Number of Goto/Reduce actions: 687
94 :     Number of Reduce actions: 7736
95 :     Number of Shift-Reduce conflicts: 0
96 :     Number of Reduce-Reduce conflicts: 0
97 :     Number of Reductions saved by default: 4913
98 :     Reallocating storage for TIME table, adding 3603 entries
99 :    
100 :     Length of Check table: 16836
101 :     Length of Action table: 16608
102 :     Number of entries in Action Table: 12013
103 :     Percentage of increase: 38.2%
104 :     Highest symbol in Check Table: 307
105 :     Storage Required for Tables: 66888 Bytes, 66K
106 :     Storage Required for Rules: 1308 Bytes
107 :    
108 :    
109 :     Actions in Compressed Tables:
110 :     Number of Shifts: 3482
111 :     Number of Shift/Reduces: 369
112 :     Number of Gotos: 4061
113 :     Number of Goto/Reduces: 687
114 :     Number of Reduces: 2823
115 :     Number of Defaults: 390
116 :    
117 :     Error maps storage:
118 :     Storage required for ACTION_SYMBOLS_BASE map: 1182 Bytes
119 :     Storage required for ACTION_SYMBOLS_RANGE map: 1007 Bytes
120 :     Storage required for NACTION_SYMBOLS_BASE map: 1182 Bytes
121 :     Storage required for NACTION_SYMBOLS_RANGE map: 630 Bytes
122 :     Storage required for SYMBOL_INDEX map: 616 Bytes
123 :     Storage required for STRING_BUFFER map: 4652 Bytes
124 :    
125 :     ***Warning: Base Check vector contains value > 127. 16-bit words used.
126 :     ***Warning: Terminal symbol > 127. 16-bit words used.
127 :     Escaped symbol $eof is an invalid C variable.
128 :    
129 :     Escaped symbol $error is an invalid C variable.
130 :     </PRE>
131 :     It can be quite different if the output changed since the version 2.30 of lpg. The important part is:<br>
132 :     <blockquote><b>This grammar is LALR(1).</b></blockquote>
133 :     This creates in the current directory some java source files and information files.
134 :     <blockquote><table BORDER=1 CELLSPACING=2 CELLPADDING=10>
135 :     <tr>
136 :     <th VALIGN=top align=left>java.l</th>
137 :     <td VALIGN=top>Information generated by lpg/jikespg. Enumarate all the states created for the automaton, etc.</td>
138 :     </tr>
139 :     <tr>
140 :     <th VALIGN=top align=left>JavaAction.java</th>
141 :     <td>It contains the method consumeRule(int) of the class org.eclipse.jdt.internal.compiler.parser.Parser that handles all semantic actions dispatches.</td>
142 :     </tr>
143 :     <tr>
144 :     <th VALIGN=top align=left>javahdr.java</th>
145 : oliviert 1.4 <td>This files is used to generate the resources files.</td>
146 : pmulet 1.1 </tr>
147 :     <tr>
148 :     <th VALIGN=top align=left>javadcl.java</th>
149 :     <td>This files is used to generate the resources files.</td>
150 :     </tr>
151 :     <tr>
152 :     <th VALIGN=top align=left>javasym.java</th>
153 : daudel 1.2 <td>This is the contents of the class org.eclipse.jdt.core.compiler.TerminalTokens. You need to replace:<br>
154 : pmulet 1.1 <ul>
155 :     <li>TokenName$eof with TokenNameEOF</li>
156 :     <li>TokenName$error with TokenNameERROR</li>
157 :     </ul></td>
158 :     </tr>
159 :     <tr>
160 :     <th VALIGN=top align=left>javadef.java</th>
161 :     <td>This is the contents of the class org.eclipse.jdt.internal.compiler.parser.ParserBasicInformation.</td>
162 :     </tr>
163 :     <tr>
164 :     <th VALIGN=top align=left>javaprs.java</th>
165 :     <td>You don't need this file. Its contents is already inlined in the Parser class.</td>
166 :     </tr>
167 :     </table></blockquote>
168 :     </li>
169 :     <li><blockquote>Now we need to update the different classes and resource files.
170 :     </blockquote>
171 :     <ol>
172 :     <li>Copy the contents of the <font color="#3366FF">JavaAction.java</font> file into the <font color="#3366FF">consumeRule(int)</font> method of the org.eclipse.jdt.internal.compiler.parser.Parser class.
173 :     </li>
174 :     <li>The class <font color="#3366FF">org.eclipse.jdt.internal.compiler.parser.ParserBasicInformation</font> needs to be updated with the content of the file <font color="#3366FF">javadef.java</font>. Don't copy the
175 :     interface name. Simply copy the field declarations. The actual source of this class will guide you.</li>
176 : daudel 1.2 <li>This is the contents of the class org.eclipse.jdt.internal.compiler.parser.TerminalTokens. You need to replace:<br>
177 : pmulet 1.1 <ul>
178 :     <li><font color="#3366FF">TokenName$eof</font> with <font color="#3366FF">TokenNameEOF</font></li>
179 :     <li><font color="#3366FF">TokenName$error</font> with <font color="#3366FF">TokenNameERROR</font></li>
180 :     </ul>
181 :     </li>
182 :     <li>The last step is to update the resource files:<br>
183 :     Copy the jdtcore.jar file in d:\temp. Compile this <A HREF="UpdateParserFiles.java">source</A> inside d:\temp. You will have a file UpdateParserFiles.class.
184 :     Then run the following command-line:
185 :     <PRE>
186 : oliviert 1.4 D:\temp>java -classpath jdtcore.jar:. UpdateParserFiles javadcl.java javahdr.java
187 : pmulet 1.1 </PRE>
188 : daudel 1.3 Once this done, you will end up with 21 new files inside d:\temp. They are called parser&lt;n&gt;.rsc, with n equals to 1..20 and readableNames.properties.
189 : pmulet 1.1 All these files need to be moved to the org\eclipse\jdt\internal\compiler\parser folder. Now you are ready to execute and test
190 :     the new parser.
191 :     </li>
192 :     </ol>
193 :     </li>
194 :     </ol>
195 :     <P>
196 :     <b>NOTE:</b> <blockquote>Changing the parser is a risky operation if you miss one of the steps above. The resulting parser can be completely
197 :     unpredictable. It can go from crashing to reporting invalid errors. Be sure that you followed all the steps and that all the
198 :     files are updated and recompiled before you run it. </blockquote>
199 :     </P>
200 :     </blockquote>
201 :     </td>
202 :     </tr>
203 :     </table>
204 :     </body>
205 :     </html>