jdt-core-home/howto/generate parser/generateParser.html

Parent Directory Parent Directory | Revision Log Revision Log

Revision 1.8 - (view) (download) (as text)

1 : pmulet 1.1 <!doctype html public "-//w3c//dtd html 4.0 transitional//en">
2 :     <html>
3 :     <head>
4 :     <title>How to: Generate the Parser</title>
5 :     <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
6 :     <link rel="stylesheet" href="http://dev.eclipse.org/default_style.css" type="text/css">
7 :     </head>
8 :     <body bgcolor="#FFFFFF" text="#000000">
9 :     <table border=0 cellspacing=5 cellpadding=2 width="100%" >
10 :    
11 :     <tr>
12 :     <td align=LEFT valign=TOP colspan="2" bgcolor="#0080C0"><b><font color="#FFFFFF">
13 :     JDT Core / HowTo: Generate the Parser </font></b></td>
14 :     </tr>
15 :    
16 :     <tr>
17 :     <td ALIGN=RIGHT VALIGN=TOP WIDTH="2%"><img SRC="http://dev.eclipse.org/images/Adarrow.gif" NOSAVE BORDER=0 height=16 width=16></td>
18 :     <td WIDTH="98%"><b>Where to get the parser generator</b><br>
19 : oliviert 1.8 <blockquote>The parser files and resources are automatically generated using the Jikespg parser generator. This tools has
20 : oliviert 1.5 been renamed Jikes Parser generator. You can find more information and latest releases at this <A HREF="http://sourceforge.net/project/showfiles.php?group_id=128803&package_id=144579">link</A>.
21 : pmulet 1.1 The latest tool is provided in source format. We don't provide any help for compiling these
22 :     source files. Refer to the link above if you have trouble to get binaries.
23 : oliviert 1.8 <br>Our grammar is generated using the version 1.3 of Jikespg. If newer versions fail to generate resources from our
24 : pmulet 1.1 grammar, please send request to the Jikes Parser Generator team.
25 :     </blockquote>
26 :     </td>
27 :     </tr>
28 :     <tr>
29 :     <td ALIGN=RIGHT VALIGN=TOP WIDTH="2%"><img SRC="http://dev.eclipse.org/images/Adarrow.gif" NOSAVE BORDER=0 height=16 width=16></td>
30 :     <td WIDTH="98%"><b>Where to get the grammar</b><br>
31 : oliviert 1.8 <blockquote>The latest grammar is always located in the <code>java_1_5.g</code> file.
32 :     (N.B.: Older versions of the grammar, such as java_1_4.g are not supported and are no longer functional.). Go to the org.eclipse.jdt.core plugins
33 : pmulet 1.1 directory (<font color="#3366FF">eclipse\plugins\org.eclipse.jdt.core</font> where eclipse is the root of your eclipse installation) and open the
34 : daudel 1.6 <font color="#3366FF">grammar</font> directory. Then search for the <font color="#3366FF">java_1_5.g</font> file inside the
35 : daudel 1.3 folder <font color="#3366FF">grammar</font>. Then copy its contents from:
36 : pmulet 1.1 <PRE>--main options
37 :     %options ACTION, AN=JavaAction.java, GP=java,
38 :     ....
39 :     $end
40 :     -- need a carriage return after the $end
41 :     </PRE>
42 :     into a file called <font color="#3366FF">java.g</font>. It is important to add a carriage return at the end of the last line.
43 : oliviert 1.8 You can save this file where you want, we will assume from thereon you saved it in <code>d:\temp\</code>.
44 : oliviert 1.7 <!-- Also make sure that the file has Unix-style end-of-line characters (i.e., not those of MS-DOS), as the parser generator is quite picky about this. -->
45 : pmulet 1.1 </blockquote>
46 :     </td>
47 :     </tr>
48 :     <tr>
49 :     <td ALIGN=RIGHT VALIGN=TOP WIDTH="2%"><img SRC="http://dev.eclipse.org/images/Adarrow.gif" NOSAVE BORDER=0 height=16 width=16></td>
50 :     <td WIDTH="98%"><b>What to do with these files and update the parser class...</b><br>
51 :     <blockquote>
52 : oliviert 1.8 Assuming the Jikespg executable (<code>jikespg.exe</code>) is located in <code>d:\jikespg</code>.
53 : pmulet 1.1 <ol>
54 :     <li>First in a console, run:
55 :     <PRE>
56 :     d:
57 :     cd \temp
58 : oliviert 1.8 d:\jikespg\jikespg.exe java.g
59 : pmulet 1.1 </PRE>
60 :     </li>
61 :     <li>You will get an output that looks like this:
62 :     <PRE>
63 : oliviert 1.8 IBM Research Jikes Parser Generator Fri Oct 14 09:22:32 2005
64 : pmulet 1.1 %OPTIONS ACTION, AN=JavaAction.java, GP=java,
65 :     %OPTIONS FILE-PREFIX=java, ESCAPE=$, PREFIX=TokenName, OUTPUT-SIZE=125 ,
66 : oliviert 1.8 %OPTIONS NOGOTO-DEFAULT, SINGLE-PRODUCTIONS, LALR=1 , TABLE,
67 : pmulet 1.1 %OPTIONS ERROR_MAPS
68 :     %OPTIONS first follow
69 :     %OPTIONS TRACE=FULL ,
70 :     %OPTIONS VERBOSE
71 : oliviert 1.8 %OPTIONS DEFERRED
72 :     %OPTIONS NAMES=MAX
73 :     %OPTIONS SCOPES
74 : pmulet 1.1 Options in effect:
75 : oliviert 1.8 ACTION ACTFILE-NAME=JavaAction.java BLOCKB=/. BLOCKE=./ BYTE CONFLI
76 : pmulet 1.1 DEFAULT=5 NODEBUG DEFERRED NOEDIT ERROR-MAPS ESCAPE=$
77 :     FILE-PREFIX=java FIRST FOLLOW GENERATE-PARSER=JAVA NOGOTO-DEFAULT
78 :     HACTFILE-NAME=javahdr.java HBLOCKB=/: HBLOCKE=:/ LALR=1 LIST
79 : oliviert 1.8 MAX-DISTANCE=30 MIN-DISTANCE=3 NAMES=MAXIMUM NONT-CHECK ORMARK=|
80 :     OUTPUT-SIZE=125 PREFIX=TokenName READ-REDUCE SCOPES NOSHIFT-DEFAULT
81 :     SINGLE-PRODUCTIONS STACK-SIZE=128 STATES SUFFIX= TABLE=SPACE
82 :     TRACE=FULL VERBOSE WARNINGS XREF
83 : pmulet 1.1
84 : oliviert 1.8 *** The following Terminals are useless:
85 :     const goto
86 : pmulet 1.1
87 :     This grammar is LALR(1).
88 :    
89 : oliviert 1.8 Number of Terminals: 110
90 :     Number of Nonterminals: 306
91 :     Number of Productions: 693
92 :     Number of Single Productions: 232
93 :     Number of Items: 2165
94 :     Number of Scopes: 132
95 :     Number of States: 954
96 :     Number of Shift actions: 6057
97 :     Number of Goto actions: 7493
98 :     Number of Shift/Reduce actions: 593
99 :     Number of Goto/Reduce actions: 1041
100 :     Number of Reduce actions: 13922
101 : pmulet 1.1 Number of Shift-Reduce conflicts: 0
102 :     Number of Reduce-Reduce conflicts: 0
103 : oliviert 1.8 Reallocating storage for SPACE table, adding 2820 entries
104 : pmulet 1.1
105 : oliviert 1.8 Length of base Action Table: 12268
106 :     Number of entries in base Action Table: 9488
107 :     Percentage of increase: 29.3%
108 :     Storage required for base Tables: 24536 Bytes, 24K
109 :     Storage required for Rules: 2076 Bytes
110 :     Number of unique terminal states: 727
111 :     Number of Shift actions saved by merging: 4113
112 :     Number of Reduce actions saved by merging: 967
113 :     Number of Reduce saved by default: 8997
114 :    
115 :     Length of Terminal Check Table: 7913
116 :     Length of Terminal Action Table: 7902
117 :     Number of entries in Terminal Action Table: 7222
118 :     Percentage of increase: 9.4%
119 :     Storage required for Terminal Tables: 23717 Bytes, 24K
120 :     Total storage required for Tables: 48253 Bytes, 48K
121 : pmulet 1.1
122 :    
123 :     Actions in Compressed Tables:
124 : oliviert 1.8 Number of Shifts: 2211
125 :     Number of Shift/Reduces: 326
126 :     Number of Gotos: 7493
127 :     Number of Goto/Reduces: 1041
128 :     Number of Reduces: 3958
129 :     Number of Defaults: 475
130 : pmulet 1.1
131 :     Error maps storage:
132 : oliviert 1.8 Storage required for ACTION_SYMBOLS_BASE map: 1908 Bytes
133 :     Storage required for ACTION_SYMBOLS_RANGE map: 1709 Bytes
134 :     Storage required for NACTION_SYMBOLS_BASE map: 1908 Bytes
135 :     Storage required for NACTION_SYMBOLS_RANGE map: 960 Bytes
136 :     Storage required for TERMINAL_INDEX map: 220 Bytes
137 :     Storage required for NON_TERMINAL_INDEX map: 614 Bytes
138 :     Storage required for STRING_BUFFER map: 12004 Bytes
139 : pmulet 1.1
140 :     ***Warning: Base Check vector contains value > 127. 16-bit words used.
141 :     Escaped symbol $eof is an invalid C variable.
142 :    
143 :     Escaped symbol $error is an invalid C variable.
144 :     </PRE>
145 : oliviert 1.8 It can be quite different if the output changed since version 1.3 of jikespg. The important part is:<br>
146 : pmulet 1.1 <blockquote><b>This grammar is LALR(1).</b></blockquote>
147 : oliviert 1.7 This creates some java source files and information files in the current directory.
148 : pmulet 1.1 <blockquote><table BORDER=1 CELLSPACING=2 CELLPADDING=10>
149 :     <tr>
150 :     <th VALIGN=top align=left>java.l</th>
151 : oliviert 1.8 <td VALIGN=top>Information generated by jikespg. Enumarate all the states created for the automaton, etc.</td>
152 : pmulet 1.1 </tr>
153 :     <tr>
154 :     <th VALIGN=top align=left>JavaAction.java</th>
155 : oliviert 1.7 <td>Contains the method consumeRule(int) of class org.eclipse.jdt.internal.compiler.parser.Parser, which handles all semantic action dispatches.</td>
156 : pmulet 1.1 </tr>
157 :     <tr>
158 :     <th VALIGN=top align=left>javahdr.java</th>
159 : oliviert 1.7 <td>Used to generate the resources files.</td>
160 : pmulet 1.1 </tr>
161 :     <tr>
162 :     <th VALIGN=top align=left>javadcl.java</th>
163 : oliviert 1.7 <td>Used to generate the resources files.</td>
164 : pmulet 1.1 </tr>
165 :     <tr>
166 :     <th VALIGN=top align=left>javasym.java</th>
167 : oliviert 1.8 <td>Contents of the class org.eclipse.jdt.core.compiler.TerminalTokens.</td>
168 : pmulet 1.1 </tr>
169 :     <tr>
170 :     <th VALIGN=top align=left>javadef.java</th>
171 : oliviert 1.8 <td>Contents of the class org.eclipse.jdt.internal.compiler.parser.ParserBasicInformation.</td>
172 : pmulet 1.1 </tr>
173 :     <tr>
174 :     <th VALIGN=top align=left>javaprs.java</th>
175 :     <td>You don't need this file. Its contents is already inlined in the Parser class.</td>
176 :     </tr>
177 :     </table></blockquote>
178 :     </li>
179 :     <li><blockquote>Now we need to update the different classes and resource files.
180 :     </blockquote>
181 :     <ol>
182 :     <li>Copy the contents of the <font color="#3366FF">JavaAction.java</font> file into the <font color="#3366FF">consumeRule(int)</font> method of the org.eclipse.jdt.internal.compiler.parser.Parser class.
183 :     </li>
184 : oliviert 1.8 <li>The class <font color="#3366FF">org.eclipse.jdt.internal.compiler.parser.ParserBasicInformation</font> needs to be updated with the content of the file <font color="#3366FF">javadef.java</font>. Don't copy the
185 : pmulet 1.1 interface name. Simply copy the field declarations. The actual source of this class will guide you.</li>
186 : oliviert 1.8 <li>Similarly, update the class <font color="#3366FF">org.eclipse.jdt.internal.compiler.parser.TerminalTokens</font> with the contents of the file <font color="#3366FF">javasym.java</font>.</li>
187 : pmulet 1.1 <li>The last step is to update the resource files:<br>
188 :     Copy the jdtcore.jar file in d:\temp. Compile this <A HREF="UpdateParserFiles.java">source</A> inside d:\temp. You will have a file UpdateParserFiles.class.
189 :     Then run the following command-line:
190 :     <PRE>
191 : oliviert 1.4 D:\temp>java -classpath jdtcore.jar:. UpdateParserFiles javadcl.java javahdr.java
192 : pmulet 1.1 </PRE>
193 : daudel 1.6 Once this done, you will end up with 25 new files inside d:\temp. They are called parser&lt;n&gt;.rsc, with n equals to 1..24 and readableNames.properties.
194 : pmulet 1.1 All these files need to be moved to the org\eclipse\jdt\internal\compiler\parser folder. Now you are ready to execute and test
195 :     the new parser.
196 :     </li>
197 :     </ol>
198 :     </li>
199 :     </ol>
200 : oliviert 1.8 <p><b>Warning:</b> <blockquote>If the number of terminals is over 128, then the parser term_check field needs to be converted
201 :     to a <code>char[]</code>. Also the generation of the parser resources files needs to be changed to support
202 :     an array of <code>char[]</code>.
203 :     </blockquote></p>
204 : pmulet 1.1 <P>
205 :     <b>NOTE:</b> <blockquote>Changing the parser is a risky operation if you miss one of the steps above. The resulting parser can be completely
206 :     unpredictable. It can go from crashing to reporting invalid errors. Be sure that you followed all the steps and that all the
207 :     files are updated and recompiled before you run it. </blockquote>
208 :     </P>
209 :     </blockquote>
210 :     </td>
211 :     </tr>
212 :     </table>
213 :     </body>
214 :     </html>