Community
Participate
Working Groups
The org.eclipse.releng.basebuilder cvs project is a subset of Eclipse SDK bundles that are used to build Eclipse. However, it is not efficient to check in binary jars in CVS. Basebuilder could be changed to a product build that could provision itself from a milestone repository of stable bundles.
This approach would also help the community in setting up their releng builds.
*** Bug 327536 has been marked as a duplicate of this bug. ***
I did a bit of hacking in this direction but am not ready to share yet :-)
Would we just provide a p2 director? Then the workflow could be: 1) cvs -d :pserver:anonymous@dev.eclipse.org:/cvsroot/eclipse ex -r $basebuilderTag org.eclipse.releng.director 2) org.eclipse.releng.director/createBaseBuilder.sh \ http://download.eclipse.org/eclipse/updates/3.7milestones \ org.eclipse.sdk.ide/3.7.0.I20100921-1024 \ basebuilder_M2 Then for something like my e4 builds I would be able to use: org.eclipse.releng.director/createBaseBuilder.sh \ http://download.eclipse.org/eclipse/updates/3.7milestones,http://download.eclipse.org/egit/updates-nightly \ org.eclipse.sdk.ide/3.7.0.I20100921-1024,org.eclipse.egit.fetchfactory \ basebuilder_M2git PW
Yes, that is similar. However, I was going to create a new product id so it would include the other bundles for performance tests etc.
Created attachment 181031 [details] feature projects Here is a very rough first cut at a productized and feature-ized base builder just to show direction. Note that it is based on some new features in both PDE and Equinox (I'll add the bug dependencies). I'd likely need to work some with Kim to better understand what the basebuilder is doing and what its expectations are. Check it out and see what you think. As for how this would work, my goal here is to make it easy and "normal" for people to get the infrastructure they need to build. Getting a builder.zip from download.eclipse.org sounds real good. It seems that in our local case you could a) build the basebuilder product and check the results into cvs and carry on much as you do today b) get a director and use that to install the basebuilder from some repo c) use wget to download a built basebuilder.zip, unzip and run a) is interesting as it allows for the builder archiving that we have today. Other than that it seems like an abuse of CVS :-) b) is appealling in that its a cool use of p2 and is dynamic. I do worry about the archiving and management of the repo that is supplying the basebuilder artifacts c) frankly seems like the easiest/simplest. If we build the basebuilder in build N then for build N+M we can point to N as the builder to get. As long as build N is archived then we will have the builder needed to redo N+M. Note that due to attachement size limits I had to delete the bundles out of the plugins dir that are used for the .target. The are largely ecj, javax.mail and javax.activation from Orbit and the apitools EE fragments.
I'm recently discovering basebuilder and it has some nice little hacks, about which I'm not sure if they'll survive migration to product and they can break many people's builds. E.g. there's plugins/org.eclipse.equinox.launcher.jar and exactly the same plugins/org.eclipse.equinox.launcher_1.1.0.v200123456.jar for sure it helps in ant scripts to use fixed path without qualifier, rather than play with wildcards.
(In reply to comment #7) > for sure it helps in ant scripts to use fixed path without qualifier, rather > than play with wildcards. That could be true for the releng director (i.e. plugins/org.eclipse.equinox.launcher.jar) but for the product based basebuilder, you could either go: <target name="initLauncher" unless="base.builder.launcher"> <path id="launcher.path"> <last count="1"><sort> <fileset dir="${eclipse.home}" includes="**/org.eclipse.equinox.launcher_*.jar" /> </sort></last> </path> <property name="base.builder.launcher" refid="launcher.path" /> </target> to get the correct launcher jar or since this would actually be built for your platform, one could use ${basebuilder_home}/eclipse in an exec since you would have the correct native launcher. Of course if the setup call was a script, we could always provide a copy to plugins/org.eclipse.equinox.launcher.jar after the director is finished :-) PW
This is a pretty old bug, and I've just skim read comments and have not looked at attachments ... but ... if it was me ... and, it was me for web tools and orbit :) ... we moved off the base builder, by simply installing and SDK (unzipping it) and then using p2Directory to install the few other tools we need. We actually build the tools often, but when we really need "changes in tools" we have a separate build, just for those few features, and have a separate /webtools/releng just for those tools. Which we update only occasionally (when adding new function, or fixing bugs). I like that approach, since a) seems simple :) and b) is modular ... true the whole "SDK" is more than is needed ... but ... we used a cached copy, etc. In our cases, "the tools" are not that tied to any particular version of the SDK ... I could see how it you did have tools that used lots of internal methods you might need tighter coupling. But would be nice to have loose coupling and encourage "mix and match" tools as needed. I hope to be looking at this approach in the months ahead, so thought I'd comment, in case other saw a fundamental flaw for the platform's needs.
Just to cross reference, a related bug is 379745. So far, the only clearly required "build tool" we need is the project org.eclipse.build.tools But, haven't studied in complete detail.
Changed title to better reflect what I think is right solution/direction this is headed. Installing SDK, then using p2 director to install the few additional tools needed for our build (I happen to know WTP and Orbit do it this way, each with their own slightly slightly different set of tools). A corollary effort would involve some changes to testing framework or procedures (i.e. two things in tests depends on current "base builder" one of those can be moved to eclipseBuilder bundle scripts ... the other (re-publish index pages) might depend on install same/similar in tools in "tests sdk" as we do in "builder sdk". Also, I always refer to this as "Eclipse SDK", but in reality, might end up being the Platform, plus jdt, plus pde (plus our specific tools). This is just like SDK, minus source, but allows some flexibility to "make smaller" in future, since many UI bundles not needed, and similar).
I've done a careful comparison of the "SDK" to current basebuilder to see what is in the base builder that is not in an SDK. Below are my notes. Bugs are open on the few issues (and a couple have been for along time!). Not a trivial effort, but seems feasible. I should probably be explicit, that even if we do move to a new builder as some people think we will, most of this effort would have to be done anyway, since some of the "tools" has to do with how we produce packages, how we create download pages, and would better separate "testing" from "building". = = = = = org.eclipse.build.tools ** need to build, put in some repo, some where * has tools used in build, like update pack properties? (generates exclude list from map file) https://bugs.eclipse.org/bugs/show_bug.cgi?id=252875 org.eclipse.jdt.core.contrib.converter_1.1.2.200703120901.jar This is not contained in any repo on download.eclipse.org. Could not find in any source repo. Is used for and "eclipse.convert" task that converts compile logs into our summary form. Found an long existing bug: https://bugs.eclipse.org/bugs/show_bug.cgi?id=321028 org.eclipse.core.resources.compatibility_3.4.0.v20090505.jar repo: /home/data/httpd/download.eclipse.org/eclipse/updates/3.5/R-3.5-200906111540/ org.eclipse.core.resources.compatibility * is this really needed? Not released since 3.5 I opened bug: https://bugs.eclipse.org/bugs/show_bug.cgi?id=384994 ========================== I doubt these performance bundles are required to be in base builder, per se, during the _build_ They are used in testing framework and already exist there. In trying to make a "new" base builder, consisting of only SDK plus a few extra tools, would probably just try leaving these out and see what happens. There are some things in base builder used during tests, such as to "publish results" but that's not related to these bundles. org.eclipse.test.performance.ui ** need to build, put in some repo, some where? * needed for base builder? Or, just performance testing org.eclipse.test.performance_3.6.0.v20100914.jar repo: .../eclipse/updates/ ./R-4.2-201206081400/plugins/org.eclipse.test.performance_3.8.0.v20120528-1648.jar ./R-4.2-201206081400/plugins/org.eclipse.test.performance.win32_3.1.100.jar * needed for base builder? Or, only tests? * note, the one in latest builder was "old" compared to what's in current repo ============================ The following are well understood on where/how to get (though a few cases might not literally be reqquired). org.eclipse.egit.fetchfactory_0.12.0.201108111757.jar repo: /home/data/httpd/download.eclipse.org/egit/pde/updates-nightly org.eclipse.egit.fetchfactory.feature.group TODO: eventually needs to be in (more) persistent repo org.eclipse.equinox.launcher.jar renamed version of "normal" launcher, to avoid qualifier [would require some changes to some scripts, to use SDK version (with full version/qualifier) but that is a well known pattern. org.eclipse.pde.api.tools.ee.cdcfoundation10_1.0.0.200901281111 org.eclipse.pde.api.tools.ee.cdcfoundation11_1.0.0.200901281111 org.eclipse.pde.api.tools.ee.j2se12_1.0.0.200901281111 org.eclipse.pde.api.tools.ee.j2se13_1.0.0.200901281111 org.eclipse.pde.api.tools.ee.j2se14_1.0.0.200901281111 org.eclipse.pde.api.tools.ee.j2se15_1.0.0.200901281111 org.eclipse.pde.api.tools.ee.javase16_1.0.0.200901281111 org.eclipse.pde.api.tools.ee.jre11_1.0.0.200901281111 org.eclipse.pde.api.tools.ee.osgiminimum10_1.0.0.200901281111 org.eclipse.pde.api.tools.ee.osgiminimum11_1.0.0.200901281111 org.eclipse.pde.api.tools.ee.osgiminimum12_1.0.1.200904031014 repo: /eclipse/updates/4.2 (note: these in latest basebuilder look very "old"?) Not sure if they are really needed for build? Or only for "tests" (API "tests" should be moved to "tests", anyway, IMHO, if they are not already) org.slf4j.api_1.6.1.v20100831-0715.jar repo: orbit or, likely, our own eclipse one? (not currently in SDK, though has been suggested to be in SR1) https://bugs.eclipse.org/bugs/show_bug.cgi?id=382980 [And, actually, from quick look, this might be in 3.8 Platform but not 4.2 platform?] Assuming the following fragments were included in base builder so it'd work on either windows or linux ... but that'd be handled automaticallly if Eclispe Platform used as the starting point). org.eclipse.equinox.launcher.win32.win32.x86_1.1.200.v20110908-1331 org.eclipse.swt.gtk.linux.x86_3.8.0.v3820a.jar org.eclipse.swt.win32.win32.x86_3.8.0.v3820a.jar
To cross reference, I have added 384995 to the depends-on list. This but does not literally depend on that bug, but there is a lot of related issues that will likely be solved "at the same time" so wanted to have them cross-referenced.
(In reply to comment #12) > org.slf4j.api_1.6.1.v20100831-0715.jar > repo: > orbit or, likely, our own eclipse one? > (not currently in SDK, though has been suggested to be in SR1) > https://bugs.eclipse.org/bugs/show_bug.cgi?id=382980 > [And, actually, from quick look, this might be in 3.8 > Platform but not 4.2 platform?] No. False alarm. I double checked and it is not in either. (I just confused myself with what I had installed).
I think I will create one more basebuilder in cvs. This will be, basically, 4.2 SDK plus tools ... all created "manually" and stuffed into cvs HEAD. This is my plan for a couple of reasons. 1) quick proof of concept. 2) we really do need ant 1.8.3 for unit tests "properties". 3) once we confirm that's working as expected, then when we start to "build tools" and install them with p2 director, etc., we can more easily isolate where problem might be, being confident that it does work if done "manually". I plan to do for just one platform (what we are currently using on build.eclipse.org) since presumably short lived ... a milestone or two. Just wanted to document my loose "plan" in case there's comments.
I promised, I've created a base builder based on 4.2, that would "mimic" installing "the SDK" and then installing some tools or extra bundles. This comment is just to document some of the details (so I won't forget :) I first unzipped a complete platform: getPlatform42.sh Then used p2.directory to install these other items from well known repos: getslf4j.sh getPDEBuild.sh getPDEAPI.sh getJDTCore.sh getgitFetch.sh getPerformanceTest.sh I did copy the plugins/...equinox...launcher_<version>.jar to plugins/...equinox...launcher.jar just to simplify initial steps. I think copied three jars from old base builder, to new one: org.eclipse.build.tools org.eclipse.jdt.core.contrib.converter_1.1.2.200703120901.jar org.eclipse.test.performance.ui These three are currently not in git, and we currently don't build them as part of any regular build ... which is one of the things we'll have to do, but again, to take steps, will start with simple copy of what existed (and hope all the versions/apis fit in :) From the old builder, I did NOT copy builderTemplate folder. I do not think its used during build, but is more like documentation on how to create a new builder ... and not sure it is relevant any longer? I then crammed all this into HEAD of cvs basebuilder project. This effectively makes it a "resource project" instead of a "java project" with live code. The live code, once we get to that point, should come from R38M6PlusRC3 tag (R4_2_primary_branch). Before deleting the previous head, and replacing it with this "hand crafted" version, I tagged the old HEAD version with the tag oldhead20120717 ,.. in case we need to get base to it for some reason, but as far as I know, we haven't used it in a very long time (using R4_2_primary branch instead). This "new one" is twice as large as the old, even more custom crafted version .... 130 M vs. 65 M. But .. eventually, soon ... won't be stored in a SCM repo. The next substantial step will be to figure out where to put these three projects in GIT, and when and how to build them: org.eclipse.build.tools org.eclipse.jdt.core.contrib.converter_1.1.2.200703120901.jar org.eclipse.test.performance.ui I'm trying a quick test build now, and will try an regular N build with it soon if that works at all.
> > The next substantial step will be to figure out where to put these three > projects in GIT, and when and how to build them: > org.eclipse.build.tools > org.eclipse.jdt.core.contrib.converter_1.1.2.200703120901.jar > org.eclipse.test.performance.ui > > I'm trying a quick test build now, and will try an regular N build with it soon > if that works at all. Test build didn't blow up. I'll try it in a "nightly" Wednesday afternoon. I think the three "homeless" bundles, > org.eclipse.build.tools > org.eclipse.jdt.core.contrib.converter_1.1.2.200703120901.jar > org.eclipse.test.performance.ui can just go into the existing eclipse.platform.releng git repository. But, since that repo already exists, from what I've heard, it is hard to import/merge any history from cvs into it. Hence, I'd propose we just start git with the latests and go from there. And, not worry about history. I'll always be there in cvs if we need it :) org.eclipse.jdt.core.contrib.converter literally has no history anyway. I propose we name that "new" bundle org.eclipse.build.compiler.summary The other two would keep their current names/versions. Now, for feature(s). Its tempting just to add all these three to the current org.eclipse.releng.tools feature. Currently it has (only) org.eclipse.releng.tools bundle in it and done think these other three would add all that much in size (that is, to those that have been using and getting only the org.eclipse.releng.tools bundle for their IDE, the "zipped repo" would not be much larger. I think we'll start with that, and can always change (add features) later, if needed. I think org.eclipse.test.performance.ui can be used from UI to "view performance results" if/when we get that running, but not sure what that is. And, the other two are only useful in the context of "batch build". So, at some point, we might want to have a org.eclipse.releng.batch.tools feature.
Some observations, so far. 1) my assumption it only had to run on linux is true only for build, but the basebuilder is also used to run unit tests ... hence, needs to run on windows too. The whole test starts off with a call to antRummer application, so, silly oversight. That'd be fixed by a truly "dynamic" install, but ... doesn't help during transition. I might add "windows pieces" too, to transition version, if that'd help. 2) In "researching" why such a quick failure, I looked in the "configuration" directory and saw lots of differences (I thought I'd checked before, but obviously was looking at wrong thing). The old base builder had no folders in it, (such as to hold bundle info or platform xml) but only one lone config.ini file. And that config.ini had very few values in it, one of the most different from standard was osgi.bundles=org.eclipse.equinox.common@1:start,org.eclipse.equinox.ds@3:start, org.eclipse.update.configurator@2:start,org.eclipse.core.runtime@start whereas the standard platform value of this config variable is osgi.bundles=reference\:file\:org.eclipse.equinox.simpleconfigurator_1.0.300.v20110815-1744.jar@1\:start So, I've changed it for now to be more like the old base builder configuration, but some of this (hopefully) will be solved when we do a true p2 install. Still waiting to see if the mac or linux tests run at all.
The unit tests did run (start to run) on Linux ... which is at least a good sign, but not too surprising since it "ran the build" on linux and was created, basically, from Linux. So, I'll revert now, go back to using R38M6PlusRC3. But, to will try a R38M6PlusRC3B that has ant 1.8.3 added in place of 1.8.2.
(In reply to comment #19) > The unit tests did run (start to run) on Linux ... which is at least a good > sign, but not too surprising since it "ran the build" on linux and was created, > basically, from Linux. > > So, I'll revert now, go back to using R38M6PlusRC3. > > But, to will try a R38M6PlusRC3B that has ant 1.8.3 added in place of 1.8.2. One odd finding, so far, is that the tests for Mac and Linux ran (are running, not finished yet) but the test for windows would fail to start up, with a message in log saying: !ENTRY org.eclipse.update.configurator 4 0 2012-07-25 08:11:01.988 !MESSAGE Unable to access file "C:\hb\workspace\JUnit-win2\org.eclipse.releng.basebuilder\plugins\org.apache.ant_1.8.3.v20120321-1730\META-INF\MANIFEST.MF". (and then later failures about not finding required ant bundle). I noticed that the EOLs were different between the 1.8.2 version of the MANIFEST.MF and the 1.8.3 (the former having LF only, the later having with windows type CRLF). While both should be "legal" manifest EOL characters, on a whim I tried committing a version (to base builder only) that had the manifests changed to LF only), and that version did "startup" for the windows tests. It's been tagged as R38M6PlusRC3C ... we'll see ...
FYI, I think I'd like to put the three "orphan" bundles described in comment 17 into their own repository, and their own feature. This might give me some practice or experience using maven and tycho, having its own build on hudson? I just make this comment in case there's discussion that should happen ... not anything that's going to happen in a few days or anything. Just to document it, there is a slight relationship of this solution to bug 358773. Not exactly the same thing ... but ... related.
(In reply to comment #21) > This might give me some practice or experience using maven and tycho, having > its own build on hudson? I made a start converting smaller bundles/repos to maven/tycho if you're looking for examples: http://git.eclipse.org/c/e4/org.eclipse.e4.releng.git/tree/org.eclipse.e4.builder/scripts/buildCBI.sh is a control script, and it builds with 4 repos and a signing plugin on build.eclipse.org. PW
Quick question: today are we still pulling the basebuilder from CVS?
(In reply to comment #23) > Quick question: today are we still pulling the basebuilder from CVS? Yes.
(In reply to comment #24) > (In reply to comment #23) > > Quick question: today are we still pulling the basebuilder from CVS? > > Yes. Currently we use org.eclipse.releng.basebuilder tagged as 'R38M6PlusRC3D'
I've opened bug 395776 to document the conversion of the two "tools projects" that have always been part of basebuilder (presumably which we'll continue to need at least parts of). I've also opened bug 395777 to discuss/document the idea of converting some subset of basebuilder itself, partially for future "security" of making sure at least last few releases were usable but also for the immediate "safety" of taking small steps, since we are nearing M4.
Just to give an update of status. Of the two jars that moved "out of" basebuilder into Git, org.eclipse.build.tools org.eclipse.test.performance.ui See http://git.eclipse.org/c/platform/eclipse.platform.releng.buildtools.git/tree/bundles I've done some work on org.eclipse.build.tools I turned it into a true OSGi bundle, removed stuff related to "Mail" and "RSS Feeds" ... we currently don't use, and doubt we ever will again (but, if we need it, it's there in the history!) As it is, there is not actually much in there we do still use. other than the "generate index pages", and that may be something we will want to not use in the future ... in its current form, anyway. I think that started when the end-goal was to generate pure HTML, and now that we create PHP pages, we could have much better separation of "data" versus "presentation" ... but, as an incremental step forward, I plan to build "org.eclipse.build.tools" pretty much as it is, and use in our current builds, just to get off of "base builder". In fact, a plain "Platform" will do. This bundle does have one dependency on "OSGi" (mostly on "discouraged access" method :) ... and its currently unclear to me exactly what that "tracking" code does or if it needs that level of OSGi use. If we could "get rid" of that ... then it could be a "pure Ant" bundle. But, again, that's a little longer term. I have not done any work with org.eclipse.test.performance.ui and likely won't this release. What I would like to do this release, is add a feature to build this bundle (plus the org.eclipse.releng.build.tools.convert bundle) ... along with POMs, etc. build directly on Hudson in the simplest way possible, add the feature to its own p2 repo for our "internal tools" use, and then in our build, use a milestone Platform, install these few tools into it ... and the rest should just work as it is, except we'd no longer need "the base builder". Plus, more important, this would be the first, enabling step, in making improvements to the index pages and summaries.
Before I forget, there is one "script" left in basebuilder we use (just for historical reasons, I believe, it is in base builder) and that is the "publish.xml" file. There's a few others there that I think have already been moved, but 'publish.xml' ... or, the part of it we use ... should be move "up" to "tychoeclipsebuilder", or somewhere, as we've done with similar things.
Status: We currently no longer use basebuilder during the build. Just binary platform distribution, with tools installed via p2 director. We still use it during unit tests. To fix for that, we need to make the script that gets the "new base builder" (platform and tools) just a little smarter to be sensitive to which platform and architecture we need ... on the build machine we just always get linux, gtk, 64.
With bug 492580 the last remaining use of "basebuilder" was removed. We have been using "platform + tools" for the build for several years, but still used the "basebuilder" for some cases of running the JUnit tests. Each platform was, apparently, fixed as problems occurred, and now it is "all gone". I have changed the description of the git repository as to say "obsolete". http://git.eclipse.org/c/platform/eclipse.platform.releng.basebuilder.git/ Plus, have updated the old wiki page at https://wiki.eclipse.org/Platform-releng-basebuilder to include an "Obsolete" section and re-categorized it to be under the [[Category:Eclipse_Platform_Releng_Obsolete| ]] category. = = = = = = = = And as far as I'm concerned, by getting rid of this thing, I can now "die in peace". :)
