platform-core-home/documents/update.html
Parent Directory
| 
Revision Log
Revision 1.3 - (view) (download) (as text)
| 1 : | jeff | 1.3 | <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN2"> |
| 2 : | jeff | 1.1 | <html> |
| 3 : | <head> | ||
| 4 : | jeff | 1.3 | <title>Update Site Proposal</title> |
| 5 : | <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> | ||
| 6 : | <meta http-equiv="Content-Style-Type" content="text/css"> | ||
| 7 : | <link href="http://dev.eclipse.org/default_style.css" rel="stylesheet" type="text/css"> | ||
| 8 : | jeff | 1.1 | </head> |
| 9 : | jeff | 1.3 | <body bgcolor="#ffffff"> |
| 10 : | jeff | 1.1 | |
| 11 : | jeff | 1.3 | <table width="100%"> |
| 12 : | <tr><td style="background:#0080C0"><b><span style="color:white">Proposal</span></b></td></tr> | ||
| 13 : | </table> | ||
| 14 : | jeff | 1.1 | <h1>Eclipse Update Site Proposal</h1> |
| 15 : | <blockquote> | ||
| 16 : | <p><b>Summary</b> <br> | ||
| 17 : | The current packaging of the Eclipse SDK as a set of large zips (one per platform) | ||
| 18 : | is a convenient one stop shop for getting up and running with the SDK. Unfortunately, | ||
| 19 : | this approach does not scale well. As more and more Eclipse projects (e.g., | ||
| 20 : | EMF, GEF, ...) come into common use, there is increasing interest in including | ||
| 21 : | these in pre-packaged zips. Similarly, as Eclipse is used in more varied scenarios, | ||
| 22 : | so increases the demand for different packagings (smaller and larger). It | ||
| 23 : | is not feasible for eclipse.org to create, manage and distribute pre-packaged | ||
| 24 : | configurations to meet all the demands. This proposal details a structure | ||
| 25 : | which enables Eclipse component providers to contribute their features and | ||
| 26 : | dj | 1.2 | plug-ins and for Eclipse consumers to efficiently select, download and install |
| 27 : | jeff | 1.3 | their components of interest.</p> |
| 28 : | <p> Last Modified: November 22, 2004</p> | ||
| 29 : | jeff | 1.1 | </blockquote> |
| 30 : | |||
| 31 : | <h2>Problem Definition</h2> | ||
| 32 : | jeff | 1.3 | <p>The base Eclipse function is currently distributed as roughly the following |
| 33 : | zip files (as of 3.1M3)</p> | ||
| 34 : | jeff | 1.1 | <ul> |
| 35 : | jeff | 1.3 | <li>RCP Binary (one per target OS/WS) (8 x ~5MB)</li> |
| 36 : | <li>RCP SDK (one per target OS/WS) (8 x ~18MB)</li> | ||
| 37 : | <li>Platform Binary (one per target OS/WS) (8 x ~25MB)</li> | ||
| 38 : | <li>Platform SDK (one per target OS/WS) (8 x ~56MB)</li> | ||
| 39 : | <li>JDT Binary (OS/WS independent) (~15MB)</li> | ||
| 40 : | <li>JDT SDK (OS/WS independent) (~27MB)</li> | ||
| 41 : | <li>PDE Binary (OS/WS independent) (~4MB)</li> | ||
| 42 : | <li>PDE SDK (OS/WS independent) (~5MB)</li> | ||
| 43 : | <li>Full Eclipse SDK zips including source and all doc (one per target OS/WS) | ||
| 44 : | (8 x ~88MB)</li> | ||
| 45 : | jeff | 1.1 | <li>assorted other zips (e.g., examples, webdav, ...)</li> |
| 46 : | </ul> | ||
| 47 : | <p>Eclipse tools and technology project downloads are available separately and | ||
| 48 : | come in a variety of flavours, sizes and organizations. People wanting the base | ||
| 49 : | SDK are quite comfortable with the current situation. They need only download | ||
| 50 : | the single zip and go. If they need to run on two different platforms they download | ||
| 51 : | the zip for that platform and ignore the fact that 95% of the second download | ||
| 52 : | is the same as the first.</p> | ||
| 53 : | dj | 1.2 | <p>Users looking to combine the base Eclipse with additional plug-ins/features |
| 54 : | jeff | 1.1 | are somewhat less happy. They have to search over several web sites, identifying |
| 55 : | downloads (zips) of interest and correlating versions manually. In general, | ||
| 56 : | the suppliers of these additional zips do not make it easy for users to understand | ||
| 57 : | their offerings.</p> | ||
| 58 : | <p>The main issues with the current setup are:</p> | ||
| 59 : | <ul> | ||
| 60 : | jeff | 1.3 | <li>Does not promote the easy provision, acquisition and use of plug-ins from |
| 61 : | a variety of sources. The basic premise of Eclipse is that plug-in consumers | ||
| 62 : | |||
| 63 : | can gather a set of interesting plug-ins, put them together in an Eclipse | ||
| 64 : | platform configuration and enjoy an integrated seemless environment. Currently | ||
| 65 : | plug-in | ||
| 66 : | producers provide their plug-ins in various forms and the lack of any managed | ||
| 67 : | acquisition mechanism forces potential users to grovel around for plug-ins | ||
| 68 : | |||
| 69 : | jeff | 1.1 | and then hope that they got the right thing.</li> |
| 70 : | <li>Does not scale as the scope of Eclipse increases. The current approach addresses | ||
| 71 : | the provision/acquisition problem by packaging collections of interesting | ||
| 72 : | dj | 1.2 | plug-ins together in a single zip. This works for Eclipse since it is at the |
| 73 : | bottom of the plug-in stack. However, others providers are not so lucky and | ||
| 74 : | jeff | 1.1 | are challenged with questions like "how many of my prerequisites should |
| 75 : | dj | 1.2 | I include in my download?". Further, as more and more plug-ins are added |
| 76 : | jeff | 1.1 | to the Eclipse world, the size of the complete zip increases. A zip of a reasonable |
| 77 : | set of SDK drops from projects on eclipse.org could easily reach 150MB (i.e., | ||
| 78 : | double the Eclipse SDK size). Following the current model, there would be | ||
| 79 : | one of these uber-zips per target OS/WS.</li> | ||
| 80 : | jeff | 1.3 | <li>Does not scale as the Eclipse use scenarios increase. As the range of people |
| 81 : | using Eclipse broadens, there will be increasing demand for alternative | ||
| 82 : | collections | ||
| 83 : | of plug-ins. These will contain either more or less than the Eclipse SDK. | ||
| 84 : | While we can take a stab and produce some common packagings, we are are | ||
| 85 : | not likely to satisfy all/most people.</li> | ||
| 86 : | jeff | 1.1 | <li>Makes poor use of existing disk and bandwidth resources. As mentioned earlier, |
| 87 : | something like 95% of each OS/WS specific SDK zip is identical content. Further, | ||
| 88 : | dj | 1.2 | from build to build, not all plug-ins change. The user doc for example tends |
| 89 : | jeff | 1.1 | to change slowly. Near the end of a milestone or release cycle we frequently |
| 90 : | dj | 1.2 | rebuild to fix isolated problems. Rather than rebuilding just the plug-ins |
| 91 : | jeff | 1.1 | in question and providing these for download, we rebuild the whole stack and |
| 92 : | repackage an entire set of full zips. Subsequently, interested developers | ||
| 93 : | are forced to redownload the bulk of what they already have. This wastes diskspace | ||
| 94 : | and bandwidth for both eclipse.org and our users.</li> | ||
| 95 : | jeff | 1.3 | <li>Finally, it is somewhat ironic that the update/install mechanism has been |
| 96 : | designed in part to address exactly these kinds of concerns yet it is one | ||
| 97 : | |||
| 98 : | of the few components that does not see regular use by the Eclipse team itself. | ||
| 99 : | As a result, the update/install team misses out on the regular feedback | ||
| 100 : | enjoyed | ||
| 101 : | by the other Eclipse teams. It is seemingly hard to justify putting update/install | ||
| 102 : | forward as a useful technology while not using it ourselves. </li> | ||
| 103 : | jeff | 1.1 | </ul> |
| 104 : | <h2>Scenarios</h2> | ||
| 105 : | <h3>Scenario 1: minimal install</h3> | ||
| 106 : | <p>User starts with a minimal Eclipse base install. This contains just the runtime, | ||
| 107 : | some UI elements and an Update manager UI. Based on this they want to build | ||
| 108 : | an install containing additional elements from various sources.</p> | ||
| 109 : | <ol> | ||
| 110 : | <li>download and unzip the minimal install form eclipse.org or a mirror</li> | ||
| 111 : | <li>start the new install </li> | ||
| 112 : | <li>the user is presented with an update manager UI </li> | ||
| 113 : | <li>user discovers or is presented with update sites </li> | ||
| 114 : | <li>user selects features to install (the candidates presented are appropriate | ||
| 115 : | to the the current environment e.g., OS/WS/etc as well as already installed | ||
| 116 : | components)</li> | ||
| 117 : | <li>the selected features are downloaded</li> | ||
| 118 : | <li>the features are installed into the current configuration</li> | ||
| 119 : | <li>the appropriate product/application (one that was just downloaded) is started | ||
| 120 : | and the update manager GUI goes away</li> | ||
| 121 : | </ol> | ||
| 122 : | <h4>Variations</h4> | ||
| 123 : | <p>After following the above scenario, the user can manage their configuration | ||
| 124 : | either using the steps outlined in Scenario 2 or by restarting their configuration | ||
| 125 : | with an indication that the update manager UI should be started. This puts them | ||
| 126 : | dj | 1.2 | back at step 3 but with all their previously installed plug-ins still installed.</p> |
| 127 : | jeff | 1.1 | <h3>Scenario 2: building on a product install</h3> |
| 128 : | <p>In this scenario, the user has an Eclipse product installed (possibly just | ||
| 129 : | the Eclipse SDK from a downloaded zip file or the configuration from the steps | ||
| 130 : | in scenario 1) and wants to add more function. This pretty much follows the | ||
| 131 : | normal update/install scenarios.</p> | ||
| 132 : | <ol> | ||
| 133 : | <li>user installs an eclipse based product (e.g., Eclipse SDK)</li> | ||
| 134 : | <li>at some point, while running the product, the user invokes the update manager | ||
| 135 : | UI </li> | ||
| 136 : | <li>user discovers or is presented with update sites </li> | ||
| 137 : | <li>user selects features to install (the candidates presented are appropriate | ||
| 138 : | to the the current environment e.g., OS/WS/etc as well as already installed | ||
| 139 : | components)</li> | ||
| 140 : | <li>the selected features are downloaded</li> | ||
| 141 : | <li>the features are installed into the current configuration</li> | ||
| 142 : | <li>the installed function is now available to the user</li> | ||
| 143 : | </ol> | ||
| 144 : | <h3>Scenario 3: Web install</h3> | ||
| 145 : | <p>Note: This is an advanced scenario that may not be supported in the near term.</p> | ||
| 146 : | <p>This scenario is very much like scenario 1. The main difference is in how the | ||
| 147 : | initial install is obtained. Rather than manually downloading and unzipping | ||
| 148 : | the minimal install, JNLP (Webstart) is used to fetch and run the minimal install. | ||
| 149 : | </p> | ||
| 150 : | <ol> | ||
| 151 : | <li>user browses some website and clicks on a JNLP link</li> | ||
| 152 : | <li>the appropriate jars are downloaded and "installed" using the | ||
| 153 : | JNLP mechanism</li> | ||
| 154 : | <li>the minimal Eclipse install is started </li> | ||
| 155 : | <li>go to step 3 of scenario 1</li> | ||
| 156 : | </ol> | ||
| 157 : | <h2>Problem Solution</h2> | ||
| 158 : | jeff | 1.3 | <p>We propose to use the Eclipse update technology to address the described problems |
| 159 : | and implement the listed scenarios. Note that this new structure is a complement | ||
| 160 : | |||
| 161 : | to the existing large zip packagings of Eclipse. We are not proposing to stop | ||
| 162 : | |||
| 163 : | creating the familiar SDK etc zips. (we leave it to someone more brave to propose | ||
| 164 : | that!)</p> | ||
| 165 : | jeff | 1.1 | <h3>Update site</h3> |
| 166 : | <p>eclipse.org (and others) will provide update sites sporting all the relevant | ||
| 167 : | dj | 1.2 | features and plug-ins. Collections of projects can cooperate and share an update |
| 168 : | jeff | 1.1 | site or each supply/support their own update site. Managing your own eliminates |
| 169 : | the need for some sort of shared site update mechanism. However, the user must | ||
| 170 : | be able to navigate these individual sites seamlessly (see the section on Update | ||
| 171 : | manager). </p> | ||
| 172 : | jeff | 1.3 | <p>There will be one update site per <em>release family</em>. A release family |
| 173 : | is a set of releases which form a substantial and separable effort. For example, | ||
| 174 : | Eclipse 2.0, 2.1, 3.0 and 3.1 are all different release families. The life | ||
| 175 : | of release family continues after its initial release (e.g., 3.0) to include | ||
| 176 : | maintenance releases (e.g., 3.0.1, 3.0.2, ...). Isolating release families | ||
| 177 : | to their own update sites helps maintain a separation between development going | ||
| 178 : | on for the next release family and maintenance work on the previous family.</p> | ||
| 179 : | jeff | 1.1 | <h3>Versioning</h3> |
| 180 : | jeff | 1.3 | <p>Update sites are not possible unless we use the Eclipse version numbering |
| 181 : | scheme effectively. The main challenge is for plug-ins to have version numbers | ||
| 182 : | that | ||
| 183 : | accurately relate to their content. The essential element here is to ensure | ||
| 184 : | that if two plug-ins have the same id and version, their content is identical. | ||
| 185 : | The | ||
| 186 : | easiest | ||
| 187 : | way to do this is to use the version qualifier (i.e., the forth version element). | ||
| 188 : | For normal (not nightly) builds, the qualifier should be the version tag from | ||
| 189 : | the map file used for the build | ||
| 190 : | which generated them (e.g., 3.1.0.v20041122). The map file defines the content | ||
| 191 : | going into the build. If that does not change, then we can assume that the | ||
| 192 : | content | ||
| 193 : | of the output plug-in will not change. In this way, identical plug-ins will | ||
| 194 : | not be duplicated on the update site and consumers will not have to download | ||
| 195 : | them | ||
| 196 : | if they already have them. For more detail on plugin versioning, see the <a href="http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/platform-core-home/documents/plugin-versioning.html">Plug-in | ||
| 197 : | Versioning Proposal</a>.</p> | ||
| 198 : | <p>Features on the other hand are collections of plug-ins. They are also relatively | ||
| 199 : | small. Since the content of a feature is indirectly defined by the content | ||
| 200 : | of the plug-ins included in the feature, it is hard to detect change. The | ||
| 201 : | safe bet | ||
| 202 : | here is to use the build stamp as the qualifier for feature version (e.g., | ||
| 203 : | 3.0.0.200402131200). This has the benificial effect of clearly identifying | ||
| 204 : | the features for a particular | ||
| 205 : | build. Further, since the entire feature set is regenerated, the collection | ||
| 206 : | of features with the same version number completely defines the entire build | ||
| 207 : | |||
| 208 : | output.</p> | ||
| 209 : | <p>Nightly builds do not go into an update site. There is no easy way to correlate | ||
| 210 : | the content in a nightly build (out of HEAD) with that of an integration build | ||
| 211 : | |||
| 212 : | (using versioned resources). As a result, all plug-ins would be new and the | ||
| 213 : | economies of the proposed solution would not be realized. Further, nightly | ||
| 214 : | builds are | ||
| 215 : | primarily there for the individual teams to ensure they have not adversely | ||
| 216 : | affected other components etc. That is, they do not figure prominently in | ||
| 217 : | the scenarios | ||
| 218 : | jeff | 1.1 | driving this proposal.</p> |
| 219 : | dj | 1.2 | <p>Plug-in directories currently reflect the id and version of the plug-in. With |
| 220 : | jeff | 1.1 | the addition of the map stamp, the directory names will grow by typically ~10 |
| 221 : | chars). It is unlikely but this may push some path length limits. The use of | ||
| 222 : | the map stamp in a file/dir name may also place additional constraints the set | ||
| 223 : | of chars that can be used in a map stamp (i.e, CVS tag).</p> | ||
| 224 : | <p>Sidenote on version qualifiers: The qualifier (fourth segment) is | ||
| 225 : | not interpreted. Qualifiers are compared using lexographical sorting. As such, | ||
| 226 : | care should be | ||
| 227 : | taken to ensure that the qualifier for subsequent builds monotonically increase. | ||
| 228 : | jeff | 1.3 | For example, if a build type identifier is to be included in the qualifier, |
| 229 : | it should be appended and | ||
| 230 : | the build date used | ||
| 231 : | as the main | ||
| 232 : | body (e.g., 20040217-I and 20040213-M7). See the complementary proposal on | ||
| 233 : | version numbering for more information.</p> | ||
| 234 : | jeff | 1.1 | <h3>Update manager</h3> |
| 235 : | <p>The main challenges for the update manager are in making all of this scale | ||
| 236 : | and usable. </p> | ||
| 237 : | <ul> | ||
| 238 : | <li>the UI must support a structured (hierarchical?) view of the available features. | ||
| 239 : | This view (and in fact the underlying feature structure) must be able to span | ||
| 240 : | update sites/servers. </li> | ||
| 241 : | <li>users must be able to select some set of root features and versions and | ||
| 242 : | then click "all required features". Again, this should span update | ||
| 243 : | sites/servers</li> | ||
| 244 : | <li>using these capabilities, user can then create their own uber-features/sites | ||
| 245 : | which simply reference features/sites. This becomes in effect a favourites | ||
| 246 : | list. They select their favourite root, click "all required" and | ||
| 247 : | get everything they need.</li> | ||
| 248 : | <li>the update manager mechanism may cache downloaded elements | ||
| 249 : | dj | 1.2 | (features and plug-ins) for use in future install operations. That is, if |
| 250 : | you already have the plug-in/feature, you don't need to download it again. | ||
| 251 : | jeff | 1.1 | The |
| 252 : | cache location should be (optionally) set by the user.</li> | ||
| 253 : | <li>ideally the update/install scenarios described will be possible when running | ||
| 254 : | offline providing that all required elements have been previously cached. | ||
| 255 : | That is, | ||
| 256 : | |||
| 257 : | if the user has already installed the features for a build, she should be | ||
| 258 : | able to create a new install without touching a server.</li> | ||
| 259 : | <li>as an advanced (i.e., longer term) item, update manager needs a way of | ||
| 260 : | managing its cache/store -- users need a way of purging old elements. Given | ||
| 261 : | a set | ||
| 262 : | of 1000 | ||
| 263 : | plugin jars | ||
| 264 : | of various | ||
| 265 : | versions, | ||
| 266 : | determining which to keep and which to delete is not possible without working | ||
| 267 : | from higher level feature groupings. In the near term, the user can delete | ||
| 268 : | the whole cache.</li> | ||
| 269 : | </ul> | ||
| 270 : | <p>Interesting side note: using the runtime's new ability to run directly out | ||
| 271 : | of jars, it is possible to have many different eclipse configurations sharing | ||
| 272 : | dj | 1.2 | the same actual plug-in files from the update manager's cache/store. This will |
| 273 : | break some assumptions about plug-ins being together in a plug-ins directory etc | ||
| 274 : | jeff | 1.1 | (the update reconciler may get upset) and it introduces some challenges for |
| 275 : | cache clean up but it is an interesting possibility.<br> | ||
| 276 : | </p> | ||
| 277 : | <h3>PDE Build</h3> | ||
| 278 : | <p>PDE Build already updates feature.xml files with the version numbers of the | ||
| 279 : | dj | 1.2 | plug-ins involved in the build. To support the version numbering scheme outlined |
| 280 : | jeff | 1.1 | above, PDE Build will be updated to </p> |
| 281 : | <ul> | ||
| 282 : | dj | 1.2 | <li>modify the plug-in version number in each plugin.xml (i.e., add the map stamp). |
| 283 : | jeff | 1.1 | The version number of a plugin.xml is updated only if it contains <_map |
| 284 : | dj | 1.2 | stamp_> as the version qualifier. This allows plug-ins to opt out of this |
| 285 : | jeff | 1.1 | update mechanism.</li> |
| 286 : | <li>modify the version number in the feature.xmls (i.e., add the build stamp). | ||
| 287 : | The version number of a feature.xml is updated only if it contains <_map | ||
| 288 : | stamp_> as the version qualifier. This allows features to opt out of this | ||
| 289 : | update mechanism. </li> | ||
| 290 : | <li>(advanced) modify the version number on <import> statements. Some | ||
| 291 : | dj | 1.2 | plug-ins and features may specify a perfect match rule in when listing their |
| 292 : | jeff | 1.1 | dependencies. While this is not recommended, it is certainly possible. Unfortunately, |
| 293 : | dj | 1.2 | there is a conflict if the intention was that plug-in A depend perfectly on |
| 294 : | the version of plug-in B with which it was built. During the build, B's version | ||
| 295 : | number will be updated according to its map stamp. Plug-in A's <import> | ||
| 296 : | jeff | 1.1 | statement for B needs to be updated to reflect this new version. While this |
| 297 : | modification is technically possible, it requires more complex analysis of | ||
| 298 : | dj | 1.2 | the plugin.xml file. Updating the version of the plug-in itself can be done |
| 299 : | jeff | 1.1 | using simple string replacement. In this case the builder has to |
| 300 : | <ul> | ||
| 301 : | <li>look at each <import> using perfect match and wanting build-time | ||
| 302 : | map stamp substitutoin, </li> | ||
| 303 : | <li>identify the pluign being imported, </li> | ||
| 304 : | <li>discover the new version number</li> | ||
| 305 : | <li>replace the "substitutution desired" flag in the version number | ||
| 306 : | with the correct version number without loosing any other formatting, | ||
| 307 : | comments etc.</li> | ||
| 308 : | </ul> | ||
| 309 : | </li> | ||
| 310 : | </ul> | ||
| 311 : | <p>Note: the plugin.xml updating outlined above must be carried out on any manifest.mf | ||
| 312 : | files present in the build.</p> | ||
| 313 : | <p>PDE Build will also be updated to output directly to update site format. This | ||
| 314 : | is already largely supported but there maybe some modest work required.</p> | ||
| 315 : | dj | 1.2 | <p>A possible longer term advance is to change PDE build to only build the plug-ins |
| 316 : | which actually changed. Since we know the map stamp of the plug-ins we want in | ||
| 317 : | jeff | 1.1 | the output, we can first look for those in the update site (or some cache). |
| 318 : | If present, there is no need to rebuild them. This will have a huge (positive) | ||
| 319 : | impact on rebuilds during release cycles.</p> | ||
| 320 : | <h2>Summary</h2> | ||
| 321 : | <p>The problem outlined above is making it hard to scale Eclipse. It is hard for | ||
| 322 : | people to use large numbers of Eclipse components (e.g., EMF, GEF, ...) and | ||
| 323 : | we are failing to put forward a good model of how others would build/stock component | ||
| 324 : | systems. The ability to do this easily is key to the overall Eclipse component | ||
| 325 : | model. The steps proposed here are a good first try at solving the problem. | ||
| 326 : | The result will not be perfect but will allow us to understand what is wrong | ||
| 327 : | and fix it. Fortunately, the bulk of the technology required exists in Eclipse | ||
| 328 : | today. Also, since we are proposing a complementary approach to the current | ||
| 329 : | download strategy, we can stage the implementation of this new mechanism both | ||
| 330 : | as we have time and as we learn more.</p> | ||
| 331 : | <p>The key steps that should be taken for 3.0 are </p> | ||
| 332 : | <ul> | ||
| 333 : | <li>create/populate an update site</li> | ||
| 334 : | dj | 1.2 | <li>update PDE build to automate the stamping of plug-ins and features</li> |
| 335 : | jeff | 1.1 | <li>a modest amount of work on update manager to enable reasonable workflows |
| 336 : | (i.e., set a stake in the ground)</li> | ||
| 337 : | </ul> | ||
| 338 : | </body> | ||
| 339 : | </html> |
| help@eclipse.org | ViewVC Help |
| Powered by ViewVC 1.0.3 |