platform-core-home/documents/update.html

Parent Directory Parent Directory | Revision Log Revision Log

Revision 1.1 - (view) (download) (as text)

1 : jeff 1.1 <html>
2 :     <head>
3 :     <title>Eclipse Update Site: Eating our own dog food</title>
4 :     <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
5 :     <link rel="stylesheet" href="http://eclipse.org/default_style.css" type="text/css">
6 :     </head>
7 :    
8 :     <body bgcolor="#FFFFFF" text="#000000">
9 :    
10 :     <h1>Eclipse Update Site Proposal</h1>
11 :     <blockquote>
12 :     <p><b>Summary</b> <br>
13 :     The current packaging of the Eclipse SDK as a set of large zips (one per platform)
14 :     is a convenient one stop shop for getting up and running with the SDK. Unfortunately,
15 :     this approach does not scale well. As more and more Eclipse projects (e.g.,
16 :     EMF, GEF, ...) come into common use, there is increasing interest in including
17 :     these in pre-packaged zips. Similarly, as Eclipse is used in more varied scenarios,
18 :     so increases the demand for different packagings (smaller and larger). It
19 :     is not feasible for eclipse.org to create, manage and distribute pre-packaged
20 :     configurations to meet all the demands. This proposal details a structure
21 :     which enables Eclipse component providers to contribute their features and
22 :     plugins and for Eclipse consumers to efficiently select, download and install
23 :     their components of interest.<br>
24 :     Last Modified: 1100 February 17, 2004</p>
25 :     </blockquote>
26 :    
27 :     <h2>Problem Definition</h2>
28 :     <p>The base Eclipse function is currently distributed as roughly the following
29 :     zip files</p>
30 :     <ul>
31 :     <li>binary Platform (one per target OS/WS) (7 x ~24MB)</li>
32 :     <li>binary JDT (one OS/WS independent) (~12MB)</li>
33 :     <li>SDK zips includind source and all doc (one per target OS/WS) (7 x ~80MB)</li>
34 :     <li>assorted other zips (e.g., examples, webdav, ...)</li>
35 :     </ul>
36 :     <p>Eclipse tools and technology project downloads are available separately and
37 :     come in a variety of flavours, sizes and organizations. People wanting the base
38 :     SDK are quite comfortable with the current situation. They need only download
39 :     the single zip and go. If they need to run on two different platforms they download
40 :     the zip for that platform and ignore the fact that 95% of the second download
41 :     is the same as the first.</p>
42 :     <p>Users looking to combine the base Eclipse with additional plugins/features
43 :     are somewhat less happy. They have to search over several web sites, identifying
44 :     downloads (zips) of interest and correlating versions manually. In general,
45 :     the suppliers of these additional zips do not make it easy for users to understand
46 :     their offerings.</p>
47 :     <p>The main issues with the current setup are:</p>
48 :     <ul>
49 :     <li>Does not promote the easy provision, acquisition and use of plugins from
50 :     a variety of souces. The basic premise of Eclipse is that plugin consumers
51 :     can gather a set of interesting plugins, put them together in an Eclipse platform
52 :     configuration and enjoy an integrated seemless environment. Currently plugin
53 :     producers provide their plugins in various forms and the lack of any managed
54 :     acquisition mechanism forces potential users to grovel around for plugins
55 :     and then hope that they got the right thing.</li>
56 :     <li>Does not scale as the scope of Eclipse increases. The current approach addresses
57 :     the provision/acquisition problem by packaging collections of interesting
58 :     plugins together in a single zip. This works for Eclipse since it is at the
59 :     bottom of the plugin stack. However, others providers are not so lucky and
60 :     are challenged with questions like &quot;how many of my prerequisites should
61 :     I include in my download?&quot;. Further, as more and more plugins are added
62 :     to the Eclipse world, the size of the complete zip increases. A zip of a reasonable
63 :     set of SDK drops from projects on eclipse.org could easily reach 150MB (i.e.,
64 :     double the Eclipse SDK size). Following the current model, there would be
65 :     one of these uber-zips per target OS/WS.</li>
66 :     <li>Does not scale as the Eclipse use scenarios increase. As the range of people
67 :     using Eclipse broadens, there will be increasing demand for alternative collections
68 :     of plugins. These will contain either more or less than the Eclipse SDK. While
69 :     we can take a stab and produce some common packagings, we will not</li>
70 :     <li>Makes poor use of existing disk and bandwidth resources. As mentioned earlier,
71 :     something like 95% of each OS/WS specific SDK zip is identical content. Further,
72 :     from build to build, not all plugins change. The user doc for example tends
73 :     to change slowly. Near the end of a milestone or release cycle we frequently
74 :     rebuild to fix isolated problems. Rather than rebuilding just the plugins
75 :     in question and providing these for download, we rebuild the whole stack and
76 :     repackage an entire set of full zips. Subsequently, interested developers
77 :     are forced to redownload the bulk of what they already have. This wastes diskspace
78 :     and bandwidth for both eclipse.org and our users.</li>
79 :     <li>Finally, it is somewhat ironic that the update/install mechanism has been
80 :     designed in part to address exactly these kinds of concerns yet it is one
81 :     of the few components that does not see regular use by the Eclipse team itself.
82 :     As a result, the update/install team misses out on the regular feedback enjoyed
83 :     by the other Eclipse teams. It is seemingly hard to justify putting update/install
84 :     forward as a useful technology but not using it ourselves. </li>
85 :     </ul>
86 :     <h2>Scenarios</h2>
87 :     <h3>Scenario 1: minimal install</h3>
88 :     <p>User starts with a minimal Eclipse base install. This contains just the runtime,
89 :     some UI elements and an Update manager UI. Based on this they want to build
90 :     an install containing additional elements from various sources.</p>
91 :     <ol>
92 :     <li>download and unzip the minimal install form eclipse.org or a mirror</li>
93 :     <li>start the new install </li>
94 :     <li>the user is presented with an update manager UI </li>
95 :     <li>user discovers or is presented with update sites </li>
96 :     <li>user selects features to install (the candidates presented are appropriate
97 :     to the the current environment e.g., OS/WS/etc as well as already installed
98 :     components)</li>
99 :     <li>the selected features are downloaded</li>
100 :     <li>the features are installed into the current configuration</li>
101 :     <li>the appropriate product/application (one that was just downloaded) is started
102 :     and the update manager GUI goes away</li>
103 :     </ol>
104 :     <h4>Variations</h4>
105 :     <p>After following the above scenario, the user can manage their configuration
106 :     either using the steps outlined in Scenario 2 or by restarting their configuration
107 :     with an indication that the update manager UI should be started. This puts them
108 :     back at step 3 but with all their previously installed plugins still installed.</p>
109 :     <h3>Scenario 2: building on a product install</h3>
110 :     <p>In this scenario, the user has an Eclipse product installed (possibly just
111 :     the Eclipse SDK from a downloaded zip file or the configuration from the steps
112 :     in scenario 1) and wants to add more function. This pretty much follows the
113 :     normal update/install scenarios.</p>
114 :     <ol>
115 :     <li>user installs an eclipse based product (e.g., Eclipse SDK)</li>
116 :     <li>at some point, while running the product, the user invokes the update manager
117 :     UI </li>
118 :     <li>user discovers or is presented with update sites </li>
119 :     <li>user selects features to install (the candidates presented are appropriate
120 :     to the the current environment e.g., OS/WS/etc as well as already installed
121 :     components)</li>
122 :     <li>the selected features are downloaded</li>
123 :     <li>the features are installed into the current configuration</li>
124 :     <li>the installed function is now available to the user</li>
125 :     </ol>
126 :     <h3>Scenario 3: Web install</h3>
127 :     <p>Note: This is an advanced scenario that may not be supported in the near term.</p>
128 :     <p>This scenario is very much like scenario 1. The main difference is in how the
129 :     initial install is obtained. Rather than manually downloading and unzipping
130 :     the minimal install, JNLP (Webstart) is used to fetch and run the minimal install.
131 :     </p>
132 :     <ol>
133 :     <li>user browses some website and clicks on a JNLP link</li>
134 :     <li>the appropriate jars are downloaded and &quot;installed&quot; using the
135 :     JNLP mechanism</li>
136 :     <li>the minimal Eclipse install is started </li>
137 :     <li>go to step 3 of scenario 1</li>
138 :     </ol>
139 :     <h2>Problem Solution</h2>
140 :     We propose to use the Eclipse update technology to address the described problems
141 :     and implement the listed scenarios. Note that this new structure is a complement
142 :     to the existing large zip packagings of Eclipse. We are not proposing to stop
143 :     creating the familiar SDK etc zips.
144 :     <h3>Update site</h3>
145 :     <p>eclipse.org (and others) will provide update sites sporting all the relevant
146 :     features and plugins. Collections of projects can cooperate and share an update
147 :     site or each supply/support their own update site. Managing your own eliminates
148 :     the need for some sort of shared site update mechanism. However, the user must
149 :     be able to navigate these individual sites seamlessly (see the section on Update
150 :     manager). </p>
151 :     <h3>Versioning</h3>
152 :     <p>Update sites are not possible unless we use the Eclipse version numbering scheme
153 :     effectively. The main challenge is for plugins to have version numbers that
154 :     indicate their content. Updating versions so they correctly reflect semantic
155 :     change would be good but not essential. What is required however is that if
156 :     two plugins have the same id and version, their content is identical. The easiest
157 :     way to do this is to use the version qualifier (i.e., the forth version element).
158 :     The qualifier should be the version tag from the map file used for the build
159 :     which generated them (e.g., 3.0.0.v20040212). The map file defines the content
160 :     going into the build. If that does not change, then we can assume that the content
161 :     of the output plugin will not change. In this way, identical plugins will not
162 :     be duplicated on the update site and consumers will not have to download them
163 :     if they already have them.</p>
164 :     <p>Features on the other hand are packagings of plugins. They are also relatively
165 :     small. Since the content of a feature is indirectly defined by the content of
166 :     the plugins included in the feature, it is hard to detect change. The safe bet
167 :     here is to use the build stamp as the qualifier for feature version (e.g., 3.0.0.I200402131200).
168 :     This has the benificial effect of clearly identifying the features for a particular
169 :     build. Further, since the entire feature set is regenerated, the collection
170 :     of features with the same version number completely defines the entire build
171 :     output.</p>
172 :     <p>Nightly builds would not go into the update site. There is no easy way to correlate
173 :     the content in a nightly build (out of HEAD) with that of an integration build
174 :     (using versioned resources). As a result, all plugins would be new and the economies
175 :     of the proposed solution would not be realized. Further, nightly builds are
176 :     primarily there for the individual teams to ensure they have not adversely affected
177 :     other components etc. That is, they do not figure prominently in the scenarios
178 :     driving this proposal.</p>
179 :     <p>Plugin directories currently reflect the id and version of the plugin. With
180 :     the addition of the map stamp, the directory names will grow by typically ~10
181 :     chars). It is unlikely but this may push some path length limits. The use of
182 :     the map stamp in a file/dir name may also place additional constraints the set
183 :     of chars that can be used in a map stamp (i.e, CVS tag).</p>
184 :     <p>Sidenote on version qualifiers: The qualifier (fourth segment) is
185 :     not interpreted. Qualifiers are compared using lexographical sorting. As such,
186 :     care should be
187 :     taken to ensure that the qualifier for subsequent builds monotonically increase.
188 :     For example, rather than prepending the build type identifier (e.g., I20040217...
189 :     or M7), the build type should be appended and the build date used as the main
190 :     body (e.g., 20040217-I and 20040213-M7).</p>
191 :     <p></p>
192 :     <p></p>
193 :     <h3>Update manager</h3>
194 :     <p>The main challenges for the update manager are in making all of this scale
195 :     and usable. </p>
196 :     <ul>
197 :     <li>the UI must support a structured (hierarchical?) view of the available features.
198 :     This view (and in fact the underlying feature structure) must be able to span
199 :     update sites/servers. </li>
200 :     <li>users must be able to select some set of root features and versions and
201 :     then click &quot;all required features&quot;. Again, this should span update
202 :     sites/servers</li>
203 :     <li>using these capabilities, user can then create their own uber-features/sites
204 :     which simply reference features/sites. This becomes in effect a favourites
205 :     list. They select their favourite root, click &quot;all required&quot; and
206 :     get everything they need.</li>
207 :     <li>the update manager mechanism may cache downloaded elements
208 :     (features and plugins) for use in future install operations. That is, if
209 :     you already have the plugin/feature, you don't need to download it again.
210 :     The
211 :     cache location should be (optionally) set by the user.</li>
212 :     <li>ideally the update/install scenarios described will be possible when running
213 :     offline providing that all required elements have been previously cached.
214 :     That is,
215 :    
216 :     if the user has already installed the features for a build, she should be
217 :     able to create a new install without touching a server.</li>
218 :     <li>as an advanced (i.e., longer term) item, update manager needs a way of
219 :     managing its cache/store -- users need a way of purging old elements. Given
220 :     a set
221 :     of 1000
222 :     plugin jars
223 :     of various
224 :     versions,
225 :     determining which to keep and which to delete is not possible without working
226 :     from higher level feature groupings. In the near term, the user can delete
227 :     the whole cache.</li>
228 :     </ul>
229 :     <p>Interesting side note: using the runtime's new ability to run directly out
230 :     of jars, it is possible to have many different eclipse configurations sharing
231 :     the same actual plugin files from the update manager's cache/store. This will
232 :     break some assumptions about plugins being together in a plugins directory etc
233 :     (the update reconciler may get upset) and it introduces some challenges for
234 :     cache clean up but it is an interesting possibility.<br>
235 :     </p>
236 :     <h3>PDE Build</h3>
237 :     <p>PDE Build already updates feature.xml files with the version numbers of the
238 :     plugins involved in the build. To support the version numbering scheme outlined
239 :     above, PDE Build will be updated to </p>
240 :     <ul>
241 :     <li>modify the plugin version number in each plugin.xml (i.e., add the map stamp).
242 :     The version number of a plugin.xml is updated only if it contains &lt;_map
243 :     stamp_&gt; as the version qualifier. This allows plugins to opt out of this
244 :     update mechanism.</li>
245 :     <li>modify the version number in the feature.xmls (i.e., add the build stamp).
246 :     The version number of a feature.xml is updated only if it contains &lt;_map
247 :     stamp_&gt; as the version qualifier. This allows features to opt out of this
248 :     update mechanism. </li>
249 :     <li>(advanced) modify the version number on &lt;import&gt; statements. Some
250 :     plugins and features may specify a perfect match rule in when listing their
251 :     dependencies. While this is not recommended, it is certainly possible. Unfortunately,
252 :     there is a conflict if the intention was that plugin A depend perfectly on
253 :     the version of plugin B with which it was built. During the build, B's version
254 :     number will be updated according to its map stamp. Plugin A's &lt;import&gt;
255 :     statement for B needs to be updated to reflect this new version. While this
256 :     modification is technically possible, it requires more complex analysis of
257 :     the plugin.xml file. Updating the version of the plugin itself can be done
258 :     using simple string replacement. In this case the builder has to
259 :     <ul>
260 :     <li>look at each &lt;import&gt; using perfect match and wanting build-time
261 :     map stamp substitutoin, </li>
262 :     <li>identify the pluign being imported, </li>
263 :     <li>discover the new version number</li>
264 :     <li>replace the &quot;substitutution desired&quot; flag in the version number
265 :     with the correct version number without loosing any other formatting,
266 :     comments etc.</li>
267 :     </ul>
268 :     </li>
269 :     </ul>
270 :     <p>Note: the plugin.xml updating outlined above must be carried out on any manifest.mf
271 :     files present in the build.</p>
272 :     <p></p>
273 :     <p>PDE Build will also be updated to output directly to update site format. This
274 :     is already largely supported but there maybe some modest work required.</p>
275 :     <p>A possible longer term advance is to change PDE build to only build the plugins
276 :     which actually changed. Since we know the map stamp of the plugins we want in
277 :     the output, we can first look for those in the update site (or some cache).
278 :     If present, there is no need to rebuild them. This will have a huge (positive)
279 :     impact on rebuilds during release cycles.</p>
280 :     <h2>Summary</h2>
281 :     <p>The problem outlined above is making it hard to scale Eclipse. It is hard for
282 :     people to use large numbers of Eclipse components (e.g., EMF, GEF, ...) and
283 :     we are failing to put forward a good model of how others would build/stock component
284 :     systems. The ability to do this easily is key to the overall Eclipse component
285 :     model. The steps proposed here are a good first try at solving the problem.
286 :     The result will not be perfect but will allow us to understand what is wrong
287 :     and fix it. Fortunately, the bulk of the technology required exists in Eclipse
288 :     today. Also, since we are proposing a complementary approach to the current
289 :     download strategy, we can stage the implementation of this new mechanism both
290 :     as we have time and as we learn more.</p>
291 :     <p>The key steps that should be taken for 3.0 are </p>
292 :     <ul>
293 :     <li>create/populate an update site</li>
294 :     <li>update PDE build to automate the stamping of plugins and features</li>
295 :     <li>a modest amount of work on update manager to enable reasonable workflows
296 :     (i.e., set a stake in the ground)</li>
297 :     </ul>
298 :     </body>
299 :     </html>