jdt-core-home/howto/generate parser/generateParser.html

Parent Directory Parent Directory | Revision Log Revision Log

Revision 1.7 - (view) (download) (as text)

1 : pmulet 1.1 <!doctype html public "-//w3c//dtd html 4.0 transitional//en">
2 :     <html>
3 :     <head>
4 :     <title>How to: Generate the Parser</title>
5 :     <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
6 :     <link rel="stylesheet" href="http://dev.eclipse.org/default_style.css" type="text/css">
7 :     </head>
8 :     <body bgcolor="#FFFFFF" text="#000000">
9 :     <table border=0 cellspacing=5 cellpadding=2 width="100%" >
10 :    
11 :     <tr>
12 :     <td align=LEFT valign=TOP colspan="2" bgcolor="#0080C0"><b><font color="#FFFFFF">
13 :     JDT Core / HowTo: Generate the Parser </font></b></td>
14 :     </tr>
15 :    
16 :     <tr>
17 :     <td ALIGN=RIGHT VALIGN=TOP WIDTH="2%"><img SRC="http://dev.eclipse.org/images/Adarrow.gif" NOSAVE BORDER=0 height=16 width=16></td>
18 :     <td WIDTH="98%"><b>Where to get the parser generator</b><br>
19 :     <blockquote>The parser files and resources are automatically generated using the LPG parser generator. This tools has
20 : oliviert 1.5 been renamed Jikes Parser generator. You can find more information and latest releases at this <A HREF="http://sourceforge.net/project/showfiles.php?group_id=128803&package_id=144579">link</A>.
21 : pmulet 1.1 The latest tool is provided in source format. We don't provide any help for compiling these
22 :     source files. Refer to the link above if you have trouble to get binaries.
23 : oliviert 1.5 <br>Our grammar is generated using the version 1.3 of LPG. If newer versions fail to generate resources from our
24 : pmulet 1.1 grammar, please send request to the Jikes Parser Generator team.
25 :     </blockquote>
26 :     </p>
27 :     </td>
28 :     </tr>
29 :     <tr>
30 :     <td ALIGN=RIGHT VALIGN=TOP WIDTH="2%"><img SRC="http://dev.eclipse.org/images/Adarrow.gif" NOSAVE BORDER=0 height=16 width=16></td>
31 :     <td WIDTH="98%"><b>Where to get the grammar</b><br>
32 : oliviert 1.7 <blockquote>The latest grammar is always located in the <code>java_1_5.g</code> file. (N.B.: Older versions of the grammar, such as java_1_4.g are not supported and are no longer functional.) Go to the org.eclipse.jdt.core plugins
33 : pmulet 1.1 directory (<font color="#3366FF">eclipse\plugins\org.eclipse.jdt.core</font> where eclipse is the root of your eclipse installation) and open the
34 : daudel 1.6 <font color="#3366FF">grammar</font> directory. Then search for the <font color="#3366FF">java_1_5.g</font> file inside the
35 : daudel 1.3 folder <font color="#3366FF">grammar</font>. Then copy its contents from:
36 : pmulet 1.1 <PRE>--main options
37 :     %options ACTION, AN=JavaAction.java, GP=java,
38 :     ....
39 :     $end
40 :     -- need a carriage return after the $end
41 :     </PRE>
42 :     into a file called <font color="#3366FF">java.g</font>. It is important to add a carriage return at the end of the last line.
43 : oliviert 1.7 You can save this file where you want, we will assume from thereon you saved it in <code>d:\temp\</code>.
44 :     <!-- Also make sure that the file has Unix-style end-of-line characters (i.e., not those of MS-DOS), as the parser generator is quite picky about this. -->
45 : pmulet 1.1 </blockquote>
46 :     </p>
47 :     </td>
48 :     </tr>
49 :     <tr>
50 :     <td ALIGN=RIGHT VALIGN=TOP WIDTH="2%"><img SRC="http://dev.eclipse.org/images/Adarrow.gif" NOSAVE BORDER=0 height=16 width=16></td>
51 :     <td WIDTH="98%"><b>What to do with these files and update the parser class...</b><br>
52 :     <blockquote>
53 : oliviert 1.7 Assuming the LPG executable (<code>lpg.exe</code> or <code>jikespg.exe</code>) is located in <code>d:\lpg</code>.
54 : pmulet 1.1 <ol>
55 :     <li>First in a console, run:
56 :     <PRE>
57 :     d:
58 :     cd \temp
59 :     d:\lpg\lpg.exe java.g
60 :     </PRE>
61 :     </li>
62 :     <li>You will get an output that looks like this:
63 :     <PRE>
64 :     LPG Parser Generator (V2.30) Tue Apr 02 12:49:13 2002
65 :     %OPTIONS ACTION, AN=JavaAction.java, GP=java,
66 :     %OPTIONS FILE-PREFIX=java, ESCAPE=$, PREFIX=TokenName, OUTPUT-SIZE=125 ,
67 :     %OPTIONS NOGOTO-DEFAULT, SINGLE-PRODUCTIONS, LALR=1 , TABLE=TIME ,
68 :     %OPTIONS ERROR_MAPS
69 :     %OPTIONS first follow
70 :     %OPTIONS TRACE=FULL ,
71 :     %OPTIONS VERBOSE
72 :     Options in effect:
73 :     ACTION ACTFILE-NAME=JavaAction.java BLOCKB=/. BLOCKE=./ BYTE CONFLIC
74 :     DEFAULT=5 NODEBUG DEFERRED NOEDIT ERROR-MAPS ESCAPE=$
75 :     FILE-PREFIX=java FIRST FOLLOW GENERATE-PARSER=JAVA NOGOTO-DEFAULT
76 :     HACTFILE-NAME=javahdr.java HBLOCKB=/: HBLOCKE=:/ LALR=1 LIST
77 :     MAX-DISTANCE=30 MIN-DISTANCE=3 NAMES=OPTIMIZED NONT-CHECK ORMARK=|
78 :     OUTPUT-SIZE=125 PREFIX=TokenName READ-REDUCE NOSCOPES NOSHIFT-DEFAULT
79 :     SINGLE-PRODUCTIONS STACK-SIZE=128 STATES SUFFIX= TABLE=TIME TRACE=FU
80 :     VERBOSE WARNINGS XREF
81 :    
82 :    
83 :     This grammar is LALR(1).
84 :    
85 :     Number of Terminals: 105
86 :     Number of Nonterminals: 202
87 :     Number of Productions: 437
88 :     Number of Single Productions: 162
89 :     Number of Items: 1265
90 :     Number of States: 591
91 :     Number of Shift actions: 3482
92 :     Number of Goto actions: 4061
93 :     Number of Shift/Reduce actions: 369
94 :     Number of Goto/Reduce actions: 687
95 :     Number of Reduce actions: 7736
96 :     Number of Shift-Reduce conflicts: 0
97 :     Number of Reduce-Reduce conflicts: 0
98 :     Number of Reductions saved by default: 4913
99 :     Reallocating storage for TIME table, adding 3603 entries
100 :    
101 :     Length of Check table: 16836
102 :     Length of Action table: 16608
103 :     Number of entries in Action Table: 12013
104 :     Percentage of increase: 38.2%
105 :     Highest symbol in Check Table: 307
106 :     Storage Required for Tables: 66888 Bytes, 66K
107 :     Storage Required for Rules: 1308 Bytes
108 :    
109 :    
110 :     Actions in Compressed Tables:
111 :     Number of Shifts: 3482
112 :     Number of Shift/Reduces: 369
113 :     Number of Gotos: 4061
114 :     Number of Goto/Reduces: 687
115 :     Number of Reduces: 2823
116 :     Number of Defaults: 390
117 :    
118 :     Error maps storage:
119 :     Storage required for ACTION_SYMBOLS_BASE map: 1182 Bytes
120 :     Storage required for ACTION_SYMBOLS_RANGE map: 1007 Bytes
121 :     Storage required for NACTION_SYMBOLS_BASE map: 1182 Bytes
122 :     Storage required for NACTION_SYMBOLS_RANGE map: 630 Bytes
123 :     Storage required for SYMBOL_INDEX map: 616 Bytes
124 :     Storage required for STRING_BUFFER map: 4652 Bytes
125 :    
126 :     ***Warning: Base Check vector contains value > 127. 16-bit words used.
127 :     ***Warning: Terminal symbol > 127. 16-bit words used.
128 :     Escaped symbol $eof is an invalid C variable.
129 :    
130 :     Escaped symbol $error is an invalid C variable.
131 :     </PRE>
132 : oliviert 1.7 It can be quite different if the output changed since version 2.30 of LPG. The important part is:<br>
133 : pmulet 1.1 <blockquote><b>This grammar is LALR(1).</b></blockquote>
134 : oliviert 1.7 This creates some java source files and information files in the current directory.
135 : pmulet 1.1 <blockquote><table BORDER=1 CELLSPACING=2 CELLPADDING=10>
136 :     <tr>
137 :     <th VALIGN=top align=left>java.l</th>
138 : oliviert 1.7 <td VALIGN=top>Log information generated by lpg/jikespg. Enumarates all the states created for the automaton, etc.</td>
139 : pmulet 1.1 </tr>
140 :     <tr>
141 :     <th VALIGN=top align=left>JavaAction.java</th>
142 : oliviert 1.7 <td>Contains the method consumeRule(int) of class org.eclipse.jdt.internal.compiler.parser.Parser, which handles all semantic action dispatches.</td>
143 : pmulet 1.1 </tr>
144 :     <tr>
145 :     <th VALIGN=top align=left>javahdr.java</th>
146 : oliviert 1.7 <td>Used to generate the resources files.</td>
147 : pmulet 1.1 </tr>
148 :     <tr>
149 :     <th VALIGN=top align=left>javadcl.java</th>
150 : oliviert 1.7 <td>Used to generate the resources files.</td>
151 : pmulet 1.1 </tr>
152 :     <tr>
153 :     <th VALIGN=top align=left>javasym.java</th>
154 : oliviert 1.5 <td>This is the contents of the class org.eclipse.jdt.core.compiler.TerminalTokens.</td>
155 : pmulet 1.1 </tr>
156 :     <tr>
157 :     <th VALIGN=top align=left>javadef.java</th>
158 :     <td>This is the contents of the class org.eclipse.jdt.internal.compiler.parser.ParserBasicInformation.</td>
159 :     </tr>
160 :     <tr>
161 :     <th VALIGN=top align=left>javaprs.java</th>
162 :     <td>You don't need this file. Its contents is already inlined in the Parser class.</td>
163 :     </tr>
164 :     </table></blockquote>
165 :     </li>
166 :     <li><blockquote>Now we need to update the different classes and resource files.
167 :     </blockquote>
168 :     <ol>
169 :     <li>Copy the contents of the <font color="#3366FF">JavaAction.java</font> file into the <font color="#3366FF">consumeRule(int)</font> method of the org.eclipse.jdt.internal.compiler.parser.Parser class.
170 :     </li>
171 : oliviert 1.7 <li>The class <font color="#3366FF">org.eclipse.jdt.internal.compiler.parser.ParserBasicInformation</font> needs to be updated with the contents of the file <font color="#3366FF">javadef.java</font>. Don't copy the
172 : pmulet 1.1 interface name. Simply copy the field declarations. The actual source of this class will guide you.</li>
173 : oliviert 1.7 <li>Similarly, update the class <font color="#3366FF">org.eclipse.jdt.internal.compiler.parser.TerminalTokens</font> with the contents of the file <font color="#3366FF">javasym.java</font>.
174 : pmulet 1.1 <li>The last step is to update the resource files:<br>
175 :     Copy the jdtcore.jar file in d:\temp. Compile this <A HREF="UpdateParserFiles.java">source</A> inside d:\temp. You will have a file UpdateParserFiles.class.
176 :     Then run the following command-line:
177 :     <PRE>
178 : oliviert 1.4 D:\temp>java -classpath jdtcore.jar:. UpdateParserFiles javadcl.java javahdr.java
179 : pmulet 1.1 </PRE>
180 : daudel 1.6 Once this done, you will end up with 25 new files inside d:\temp. They are called parser&lt;n&gt;.rsc, with n equals to 1..24 and readableNames.properties.
181 : pmulet 1.1 All these files need to be moved to the org\eclipse\jdt\internal\compiler\parser folder. Now you are ready to execute and test
182 :     the new parser.
183 :     </li>
184 :     </ol>
185 :     </li>
186 :     </ol>
187 :     <P>
188 :     <b>NOTE:</b> <blockquote>Changing the parser is a risky operation if you miss one of the steps above. The resulting parser can be completely
189 :     unpredictable. It can go from crashing to reporting invalid errors. Be sure that you followed all the steps and that all the
190 :     files are updated and recompiled before you run it. </blockquote>
191 :     </P>
192 :     </blockquote>
193 :     </td>
194 :     </tr>
195 :     </table>
196 :     </body>
197 :     </html>