jdt-core-home/howto/generate parser/generateParser.html

Parent Directory Parent Directory | Revision Log Revision Log

Revision 1.9 - (view) (download) (as text)

1 : pmulet 1.1 <!doctype html public "-//w3c//dtd html 4.0 transitional//en">
2 :     <html>
3 :     <head>
4 :     <title>How to: Generate the Parser</title>
5 :     <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
6 :     <link rel="stylesheet" href="http://dev.eclipse.org/default_style.css" type="text/css">
7 :     </head>
8 :     <body bgcolor="#FFFFFF" text="#000000">
9 :     <table border=0 cellspacing=5 cellpadding=2 width="100%" >
10 :    
11 :     <tr>
12 :     <td align=LEFT valign=TOP colspan="2" bgcolor="#0080C0"><b><font color="#FFFFFF">
13 :     JDT Core / HowTo: Generate the Parser </font></b></td>
14 :     </tr>
15 :    
16 :     <tr>
17 :     <td ALIGN=RIGHT VALIGN=TOP WIDTH="2%"><img SRC="http://dev.eclipse.org/images/Adarrow.gif" NOSAVE BORDER=0 height=16 width=16></td>
18 :     <td WIDTH="98%"><b>Where to get the parser generator</b><br>
19 : oliviert 1.8 <blockquote>The parser files and resources are automatically generated using the Jikespg parser generator. This tools has
20 : oliviert 1.5 been renamed Jikes Parser generator. You can find more information and latest releases at this <A HREF="http://sourceforge.net/project/showfiles.php?group_id=128803&package_id=144579">link</A>.
21 : pmulet 1.1 The latest tool is provided in source format. We don't provide any help for compiling these
22 :     source files. Refer to the link above if you have trouble to get binaries.
23 : oliviert 1.8 <br>Our grammar is generated using the version 1.3 of Jikespg. If newer versions fail to generate resources from our
24 : pmulet 1.1 grammar, please send request to the Jikes Parser Generator team.
25 :     </blockquote>
26 :     </td>
27 :     </tr>
28 :     <tr>
29 :     <td ALIGN=RIGHT VALIGN=TOP WIDTH="2%"><img SRC="http://dev.eclipse.org/images/Adarrow.gif" NOSAVE BORDER=0 height=16 width=16></td>
30 :     <td WIDTH="98%"><b>Where to get the grammar</b><br>
31 : oliviert 1.8 <blockquote>The latest grammar is always located in the <code>java_1_5.g</code> file.
32 :     (N.B.: Older versions of the grammar, such as java_1_4.g are not supported and are no longer functional.). Go to the org.eclipse.jdt.core plugins
33 : pmulet 1.1 directory (<font color="#3366FF">eclipse\plugins\org.eclipse.jdt.core</font> where eclipse is the root of your eclipse installation) and open the
34 : daudel 1.6 <font color="#3366FF">grammar</font> directory. Then search for the <font color="#3366FF">java_1_5.g</font> file inside the
35 : daudel 1.3 folder <font color="#3366FF">grammar</font>. Then copy its contents from:
36 : pmulet 1.1 <PRE>--main options
37 :     %options ACTION, AN=JavaAction.java, GP=java,
38 :     ....
39 :     $end
40 :     -- need a carriage return after the $end
41 :     </PRE>
42 :     into a file called <font color="#3366FF">java.g</font>. It is important to add a carriage return at the end of the last line.
43 : oliviert 1.8 You can save this file where you want, we will assume from thereon you saved it in <code>d:\temp\</code>.
44 : oliviert 1.9 If the end-of-line of the environment of the C compiler used to build jikespg is different from the one used when running the executable, you might end up with an exception at runtime.
45 :     Be sure to used consistent end-of-line characters.
46 : pmulet 1.1 </blockquote>
47 :     </td>
48 :     </tr>
49 :     <tr>
50 :     <td ALIGN=RIGHT VALIGN=TOP WIDTH="2%"><img SRC="http://dev.eclipse.org/images/Adarrow.gif" NOSAVE BORDER=0 height=16 width=16></td>
51 :     <td WIDTH="98%"><b>What to do with these files and update the parser class...</b><br>
52 :     <blockquote>
53 : oliviert 1.8 Assuming the Jikespg executable (<code>jikespg.exe</code>) is located in <code>d:\jikespg</code>.
54 : pmulet 1.1 <ol>
55 :     <li>First in a console, run:
56 :     <PRE>
57 :     d:
58 :     cd \temp
59 : oliviert 1.8 d:\jikespg\jikespg.exe java.g
60 : pmulet 1.1 </PRE>
61 :     </li>
62 :     <li>You will get an output that looks like this:
63 :     <PRE>
64 : oliviert 1.8 IBM Research Jikes Parser Generator Fri Oct 14 09:22:32 2005
65 : pmulet 1.1 %OPTIONS ACTION, AN=JavaAction.java, GP=java,
66 :     %OPTIONS FILE-PREFIX=java, ESCAPE=$, PREFIX=TokenName, OUTPUT-SIZE=125 ,
67 : oliviert 1.8 %OPTIONS NOGOTO-DEFAULT, SINGLE-PRODUCTIONS, LALR=1 , TABLE,
68 : pmulet 1.1 %OPTIONS ERROR_MAPS
69 :     %OPTIONS first follow
70 :     %OPTIONS TRACE=FULL ,
71 :     %OPTIONS VERBOSE
72 : oliviert 1.8 %OPTIONS DEFERRED
73 :     %OPTIONS NAMES=MAX
74 :     %OPTIONS SCOPES
75 : pmulet 1.1 Options in effect:
76 : oliviert 1.8 ACTION ACTFILE-NAME=JavaAction.java BLOCKB=/. BLOCKE=./ BYTE CONFLI
77 : pmulet 1.1 DEFAULT=5 NODEBUG DEFERRED NOEDIT ERROR-MAPS ESCAPE=$
78 :     FILE-PREFIX=java FIRST FOLLOW GENERATE-PARSER=JAVA NOGOTO-DEFAULT
79 :     HACTFILE-NAME=javahdr.java HBLOCKB=/: HBLOCKE=:/ LALR=1 LIST
80 : oliviert 1.8 MAX-DISTANCE=30 MIN-DISTANCE=3 NAMES=MAXIMUM NONT-CHECK ORMARK=|
81 :     OUTPUT-SIZE=125 PREFIX=TokenName READ-REDUCE SCOPES NOSHIFT-DEFAULT
82 :     SINGLE-PRODUCTIONS STACK-SIZE=128 STATES SUFFIX= TABLE=SPACE
83 :     TRACE=FULL VERBOSE WARNINGS XREF
84 : pmulet 1.1
85 : oliviert 1.8 *** The following Terminals are useless:
86 :     const goto
87 : pmulet 1.1
88 :     This grammar is LALR(1).
89 :    
90 : oliviert 1.8 Number of Terminals: 110
91 :     Number of Nonterminals: 306
92 :     Number of Productions: 693
93 :     Number of Single Productions: 232
94 :     Number of Items: 2165
95 :     Number of Scopes: 132
96 :     Number of States: 954
97 :     Number of Shift actions: 6057
98 :     Number of Goto actions: 7493
99 :     Number of Shift/Reduce actions: 593
100 :     Number of Goto/Reduce actions: 1041
101 :     Number of Reduce actions: 13922
102 : pmulet 1.1 Number of Shift-Reduce conflicts: 0
103 :     Number of Reduce-Reduce conflicts: 0
104 : oliviert 1.8 Reallocating storage for SPACE table, adding 2820 entries
105 : pmulet 1.1
106 : oliviert 1.8 Length of base Action Table: 12268
107 :     Number of entries in base Action Table: 9488
108 :     Percentage of increase: 29.3%
109 :     Storage required for base Tables: 24536 Bytes, 24K
110 :     Storage required for Rules: 2076 Bytes
111 :     Number of unique terminal states: 727
112 :     Number of Shift actions saved by merging: 4113
113 :     Number of Reduce actions saved by merging: 967
114 :     Number of Reduce saved by default: 8997
115 :    
116 :     Length of Terminal Check Table: 7913
117 :     Length of Terminal Action Table: 7902
118 :     Number of entries in Terminal Action Table: 7222
119 :     Percentage of increase: 9.4%
120 :     Storage required for Terminal Tables: 23717 Bytes, 24K
121 :     Total storage required for Tables: 48253 Bytes, 48K
122 : pmulet 1.1
123 :    
124 :     Actions in Compressed Tables:
125 : oliviert 1.8 Number of Shifts: 2211
126 :     Number of Shift/Reduces: 326
127 :     Number of Gotos: 7493
128 :     Number of Goto/Reduces: 1041
129 :     Number of Reduces: 3958
130 :     Number of Defaults: 475
131 : pmulet 1.1
132 :     Error maps storage:
133 : oliviert 1.8 Storage required for ACTION_SYMBOLS_BASE map: 1908 Bytes
134 :     Storage required for ACTION_SYMBOLS_RANGE map: 1709 Bytes
135 :     Storage required for NACTION_SYMBOLS_BASE map: 1908 Bytes
136 :     Storage required for NACTION_SYMBOLS_RANGE map: 960 Bytes
137 :     Storage required for TERMINAL_INDEX map: 220 Bytes
138 :     Storage required for NON_TERMINAL_INDEX map: 614 Bytes
139 :     Storage required for STRING_BUFFER map: 12004 Bytes
140 : pmulet 1.1
141 :     ***Warning: Base Check vector contains value > 127. 16-bit words used.
142 :     Escaped symbol $eof is an invalid C variable.
143 :    
144 :     Escaped symbol $error is an invalid C variable.
145 :     </PRE>
146 : oliviert 1.8 It can be quite different if the output changed since version 1.3 of jikespg. The important part is:<br>
147 : pmulet 1.1 <blockquote><b>This grammar is LALR(1).</b></blockquote>
148 : oliviert 1.7 This creates some java source files and information files in the current directory.
149 : pmulet 1.1 <blockquote><table BORDER=1 CELLSPACING=2 CELLPADDING=10>
150 :     <tr>
151 :     <th VALIGN=top align=left>java.l</th>
152 : oliviert 1.8 <td VALIGN=top>Information generated by jikespg. Enumarate all the states created for the automaton, etc.</td>
153 : pmulet 1.1 </tr>
154 :     <tr>
155 :     <th VALIGN=top align=left>JavaAction.java</th>
156 : oliviert 1.7 <td>Contains the method consumeRule(int) of class org.eclipse.jdt.internal.compiler.parser.Parser, which handles all semantic action dispatches.</td>
157 : pmulet 1.1 </tr>
158 :     <tr>
159 :     <th VALIGN=top align=left>javahdr.java</th>
160 : oliviert 1.7 <td>Used to generate the resources files.</td>
161 : pmulet 1.1 </tr>
162 :     <tr>
163 :     <th VALIGN=top align=left>javadcl.java</th>
164 : oliviert 1.7 <td>Used to generate the resources files.</td>
165 : pmulet 1.1 </tr>
166 :     <tr>
167 :     <th VALIGN=top align=left>javasym.java</th>
168 : oliviert 1.8 <td>Contents of the class org.eclipse.jdt.core.compiler.TerminalTokens.</td>
169 : pmulet 1.1 </tr>
170 :     <tr>
171 :     <th VALIGN=top align=left>javadef.java</th>
172 : oliviert 1.8 <td>Contents of the class org.eclipse.jdt.internal.compiler.parser.ParserBasicInformation.</td>
173 : pmulet 1.1 </tr>
174 :     <tr>
175 :     <th VALIGN=top align=left>javaprs.java</th>
176 :     <td>You don't need this file. Its contents is already inlined in the Parser class.</td>
177 :     </tr>
178 :     </table></blockquote>
179 :     </li>
180 :     <li><blockquote>Now we need to update the different classes and resource files.
181 :     </blockquote>
182 :     <ol>
183 :     <li>Copy the contents of the <font color="#3366FF">JavaAction.java</font> file into the <font color="#3366FF">consumeRule(int)</font> method of the org.eclipse.jdt.internal.compiler.parser.Parser class.
184 :     </li>
185 : oliviert 1.8 <li>The class <font color="#3366FF">org.eclipse.jdt.internal.compiler.parser.ParserBasicInformation</font> needs to be updated with the content of the file <font color="#3366FF">javadef.java</font>. Don't copy the
186 : pmulet 1.1 interface name. Simply copy the field declarations. The actual source of this class will guide you.</li>
187 : oliviert 1.8 <li>Similarly, update the class <font color="#3366FF">org.eclipse.jdt.internal.compiler.parser.TerminalTokens</font> with the contents of the file <font color="#3366FF">javasym.java</font>.</li>
188 : pmulet 1.1 <li>The last step is to update the resource files:<br>
189 :     Copy the jdtcore.jar file in d:\temp. Compile this <A HREF="UpdateParserFiles.java">source</A> inside d:\temp. You will have a file UpdateParserFiles.class.
190 :     Then run the following command-line:
191 :     <PRE>
192 : oliviert 1.4 D:\temp>java -classpath jdtcore.jar:. UpdateParserFiles javadcl.java javahdr.java
193 : pmulet 1.1 </PRE>
194 : daudel 1.6 Once this done, you will end up with 25 new files inside d:\temp. They are called parser&lt;n&gt;.rsc, with n equals to 1..24 and readableNames.properties.
195 : pmulet 1.1 All these files need to be moved to the org\eclipse\jdt\internal\compiler\parser folder. Now you are ready to execute and test
196 :     the new parser.
197 :     </li>
198 :     </ol>
199 :     </li>
200 :     </ol>
201 : oliviert 1.8 <p><b>Warning:</b> <blockquote>If the number of terminals is over 128, then the parser term_check field needs to be converted
202 :     to a <code>char[]</code>. Also the generation of the parser resources files needs to be changed to support
203 :     an array of <code>char[]</code>.
204 :     </blockquote></p>
205 : pmulet 1.1 <P>
206 :     <b>NOTE:</b> <blockquote>Changing the parser is a risky operation if you miss one of the steps above. The resulting parser can be completely
207 :     unpredictable. It can go from crashing to reporting invalid errors. Be sure that you followed all the steps and that all the
208 :     files are updated and recompiled before you run it. </blockquote>
209 :     </P>
210 :     </blockquote>
211 :     </td>
212 :     </tr>
213 :     </table>
214 :     </body>
215 :     </html>