Community
Participate
Working Groups
Having read this blog http://eclipsewebmaster.blogspot.com/2008/10/build-workshop-3-common-builder-in-its.html which led me to this script http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/org.eclipse.dash.commonbuilder.releng/tools/scripts/start.sh?root=Technology_Project&view=markup, I have to say that the direction seemingly being taken toward a common build infrastructure is disheartening. I'd expect the script used to start a build to simply invoke Ant, with as much functionality as possible being implemented in Ant throughout. With that, I'd expect as much of the build scripts invoked to be generated by an abstraction, namely a model or set of models. I've started such an approach in Amalgam, which will form the basis of the Galileo build that is product file based.
We're heading in that direction, but for now, it's something that runs on a linux server (one day, a cluster), and thus having shell start it is completely reasonable. In future, yes, we could move more of it to Ant. But really, it's just a headless PDE build, started like this: $javaHome/bin/java -enableassertions -cp $cpAndMain -application org.eclipse.ant.core.antRunner -f $relengCommonBuilderDir/buildAll.xml $antTarget -DmapVersionTag=$buildTag -DbuildType=$buildType -DbuildID=$buildID -Dtimestamp=$buildTimestamp -DbuildDirectory=$buildDirEclipse -DbuildAlias=$buildAlias -DdownloadsDir=$downloadsDir -DJAVA_HOME=$JAVA_HOME You can do the same thing under Windows or Mac, as long as you define all the required properties the build will need in a build.cfg file which its buildAll.xml will then source. You might want to consider renaming this bug so that it can be evaluated to be complete at some future point. "Maximize" implies asymptotic approach to a goal, without necessary ever reaching it.
Perhaps we should be more clear what our goals are. We weren't planning on doing .product builds for now. At least not in my understanding. And while your amalgam builds sound cool, Rich (I know we've never met but I've only ever heard you referred to as Rich ... sorry if it's presumptuous of me to call you that :), Nick's system works now and is being used by many modeling projects. I guess what I'm trying to say is that it would be nice to build up from something that works rather than to continue discussing potential technologies. Or at least have something projects can use while we work on a better system. Without something to poke fun at, we could speak forever about the best way to do it. Also, it's probably just me, but I find it _really_ restricting to work in plain ant :) .
(In reply to comment #2) > Nick's system works now and is being used by many modeling projects. Yes, but we're not all happy with the way things work in modeling builds. IMHO, a common build system needs to leverage PDE build as much as possible, feeding improvements into PDE; have a minimal setup cost (i.e. no databases, no php, minimal shell scripts); be easily installed on a local machine; consumable by adopters; easy to run under cruise control; abstract into a simple model the configuration properties and generate as much as possible; etc. IOW, give us a small, simple base to start, leaving the bells and whistles optional. BTW, product-based builds are really not much different than feature-based builds. The main differences are in some added variables and a .product file, while the same customTargets.xml is used to build what's not simply fetched for inclusion. Launchers, packaging, and p2-ification are what's additional.
(In reply to comment #3) > (In reply to comment #2) > > Nick's system works now and is being used by many modeling projects. > Yes, but we're not all happy with the way things work in modeling builds. Me neither, though it's pretty stable, if increasingly bloated and hard to add add new components. Enter CBI, Modeling Amalgam, and EPP. Individual Components, Aggregate Components, Aggregate Projects. > IMHO, a common build system needs to leverage PDE build as much as possible, > feeding improvements into PDE; have a minimal setup cost (i.e. no databases, no > php, minimal shell scripts); Yes, we're getting there. One of the goals of CBI was to rip the PHP crap out and refactor so it's one shell script which calls Ant. For crontab or web UI, that same script is called. I'm currently redoing some of the php-calls-bash / bash-calls-php duplication so that everything is a properties file which can be leveraged by shell, php, or Ant. > be easily installed on a local machine; The CBI GEF build runs locally with minimal setup. In fact, there's a setup script you can use to extract everything into a given folder prior to running start.sh. > easy to run under cruise control CBI runs a shell script which runs Ant. CC runs a crontab which runs a shell script which runs Ant. Any questions? Actually, I think Hudson might be a better solution. It supports scheduling and VCS monitoring, things I had to write by hand. And it handles clusters of servers, too, which buildX.eclipse.org will become one day. > give us a small, simple base to start, leaving the bells and whistles optional. What do you consider a bell or whistle? Being able to choose localhost or remote server build? CVS, SVN or local (workspace) sources? build from map or build from HEAD? With the Modeling build, you had one choice: on a server, from CVS, build from map. CBI will offer much more... yet somehow it's still not what you want. > BTW, product-based builds are really not much different than feature-based Yes, but for now, CBI is for _projects_. You did say you wanted no bells & whistles, right? :P
(In reply to comment #4) > Actually, I think Hudson might be a better solution. It supports scheduling and > VCS monitoring, things I had to write by hand. And it handles clusters of > servers, too, which buildX.eclipse.org will become one day. Perhaps, I've heard good things about Hudson as well. But as CruiseControl is already in use at Eclipse, and I suspect more are familiar with it, why change? > What do you consider a bell or whistle? Database-based bugzilla queries per build and cumbersome web-based build invocation, for example. Who really finds these useful? CruiseControl provides all the reporting and web interface most projects could need, imho. > CBI will offer much more... yet somehow it's still not what you want. To be seen, I guess. As mentioned above, the first artifact I saw was a script that was clearly not what I expected. > > BTW, product-based builds are really not much different than feature-based > > Yes, but for now, CBI is for _projects_. You did say you wanted no bells & > whistles, right? :P I never indicated I wanted product-based builds, only that the approach I've taken to generating these scripts from models seems like a reasonable approach. Anyway, I look forward to seeing more as CBI progresses.
> that was clearly not what I expected. I believe the team is working on a way for users to launch builds *easily*. Typing 'build abc' at the command line doesn't get any easier than that, and it abstracts pretty much all the underlying infrastructure, so I don't understand your concerns. The CBI has other constraints to consider. For instance, it needs to scale across a cluster of servers transparently (to the user). Running 20 builds on one server simultaneously is not going to work. How will we manage this using only PDE and Ant? If you have concrete suggestions on how to make this happen (ie, code, a working demo, etc.) you have my undivided attention. > Yes, but we're not all happy with the way things work in modeling builds. > IMHO, a common build system needs to leverage PDE build as much as possible, Point taken. Nick is the only one who has stepped forward and provided _actual running code_ that is capable of building something today. Patches to improve on his work are more than welcome. > only that the approach I've > taken to generating these scripts from models seems like a reasonable approach. It absolutely does. But you know the saying -- a line of code is worth a thousand words. Any givers?
(In reply to comment #6) > I believe the team is working on a way for users to launch builds *easily*. > Typing 'build abc' at the command line doesn't get any easier than that, and it > abstracts pretty much all the underlying infrastructure, so I don't understand > your concerns. My concern is that while the launch command may be simple, the properties that are being set need to be defined somewhere, preferably not always as -D options. Generating a build.properties file and picking that up within an Ant script is what I'd expect. The other concern is that the way it's being done now, I'll need to execute this shell script, which aren't as portable as vanilla-Ant. > > The CBI has other constraints to consider. For instance, it needs to scale > across a cluster of servers transparently (to the user). Running 20 builds on > one server simultaneously is not going to work. How will we manage this using > only PDE and Ant? If you have concrete suggestions on how to make this happen > (ie, code, a working demo, etc.) you have my undivided attention. Each build executes under a process, so I'm not sure how there's any difference between what I'm suggesting vs. what's being done now. I'm sure you'd agree that being able to simply invoke a build on any machine with minimal script changes is the best solution. I can't imagine a requirement for the CBI is to execute only on the Foundation's server environment. > It absolutely does. But you know the saying -- a line of code is worth a > thousand words. Any givers? I'm going to continue the model-driven approach within Amalgam, so was hoping that CBI would maximize a .properties and Ant build file approach so that I could templify these and generate much like I do today with PDE's templates. IOW, if CBI focuses on making a general set of build templates that abstract as many of the properties as possible, it would be straightforward to retarget the config model I have started to these templates in the future.
(In reply to comment #7) > I'm sure you'd agree > that being able to simply invoke a build on any machine with minimal script > changes is the best solution. Yup, we've agreed on that long ago, see http://wiki.eclipse.org/Build_Workshop_3:_Build_Hard_With_A_Purpose/Requirements ...or bug 252028 and bug 252030 > I can't imagine a requirement for the CBI is to > execute only on the Foundation's server environment. Heck no; however, as per above, our first objective is to set up a CBI on Foundation hardware for the immediate benefit of the Eclipse projects. Perhaps it was premature of me to advertise the availability of code on my blog.
(In reply to comment #8) > Perhaps it was premature of me to advertise the availability of code on my > blog. Perhaps, as it certainly raised an eyebrow (and not just mine, I suppose). I'll close this as INVALID for now. At least I know you're planning to head in the right direction now, thanks. I'd still like to circle back at perhaps the next workshop and compare notes.
To me a common build system is divided into three parts: (1) The single, tiny, simple configuration files a project has to specify in order to participate. (2) A totally-trivial-to-use build that runs on any machine. Totally-trivial means "check out all the files; type one command". (3) A build automation system that runs the totally-trivial build for every project automatically/on changes/regularly, keeps the logs in web pages, puts the downloads in the download directory, provides push (email) and pull (RSS) build status, etc. I admit to being dense, but I'm not seeing the current direction of the CBI satisfying these requirements. I imagined something like: (1) A single file stored in the {root}/releng directory of each Eclipse project. File exists: project is built by (3); file does not exist: project does their own builds. (1) The file consists of the absolutely minimum that is needed. Given that Eclipse projects already contain huge amounts of meta-data, I'm convinced that the file doesn't need much at all. (2) "ant" or "ant -f releng/build.xml" - it can't get simpler than that and it should be that simple. Maybe "releng/build.sh", I don't really care, but there shouldn't be any command line arguments for the common basic build. (3) CruiseControl or Hudson that fetches (1), then uses the contents of (1) to fetch the correct tag (or HEAD) and then runs (2) and logs all of that. This will allow us to tell: NEW PROJECTS: all you have to do is check in file (1) and your project will be built and put in the download directories automatically. CONSUMERS and TESTERS: want to build project X from scratch? Check out all the code from CVS and then type "ant".
Created attachment 116629 [details] initial (rough) build configuration model In support of Bjorn's suggested 3 elements, the attached model image is what I came up with (so far) for Amalgam/Galileo product-based builds. I'd modify his first requirement to be a single file (model) that contains all the information a project would need to define. Step 2 would involve the generation of all required build scripts, etc. from this model as the first step in running a build (not the current approach of having to maintain a half-dozen files located in various repositories). Step 3 can be largely (entirely?) done using CruiseControl, for starters. That's it from me... I'll shut up and get back to work.
(In reply to comment #10) > To me a common build system is divided into three parts: > (1) The single, tiny, simple configuration files a project has to specify in > order to participate. We're pared a whole .releng project down to: BUILD: build.properties buildExtra.xml maps/*.map TEST: testManifest.xml PROMOTE: promote.properties http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/org.eclipse.gef.releng/?root=Technology_Project > (2) A totally-trivial-to-use build that runs on any machine. Totally-trivial > means "check out all the files; type one command". This is still in progress. Commandline flags will ultimately move into the build.properties, so that you need only run: ./setup.sh ./start.sh However, if your config doesn't use standard CVS paths, you still need to tell the system where to go for sources. > (3) A build automation system that runs the totally-trivial build for every > project automatically/on changes/regularly, keeps the logs in web pages, puts > the downloads in the download directory, provides push (email) and pull (RSS) > build status, etc. Powered by promote.properties, this will be a shell script (or one day, Ant?) which handles all that. It can be run at the end of a build, or as a second, user-dependent step. > (1) A single file stored in the {root}/releng directory of each Eclipse > project. File exists: project is built by (3); file does not exist: project > does their own builds. This has never been stated before now. > (1) The file consists of the absolutely minimum that is needed. Given that > Eclipse projects already contain huge amounts of meta-data, I'm convinced that > the file doesn't need much at all. Yes, but you still need maps, JVM version & path, test manifest, email contact, newsgroup name, etc. etc. Build workflow metadata is NOT in the plugins/features. > (2) "ant" or "ant -f releng/build.xml" - it can't get simpler than that and it > should be that simple. Maybe "releng/build.sh", I don't really care, but there > shouldn't be any command line arguments for the common basic build. Well, sure, but that doesn't guarantee reproducibility. One user's system may have ANT_HOME and JAVA_HOME set differently, and you'll end up using Ant 1.7 and JDK 6; on another system, it could be Ant 1.6.5 and JDK 5. So either the build script or its sourced properties file needs to define these, and document them. Or we need another tool to inspect a user's system for these variables and set them "correctly" based on assumptions set in the features & plugins. > (3) CruiseControl or Hudson that fetches (1), then uses the contents of (1) to > fetch the correct tag (or HEAD) and then runs (2) and logs all of that. Yep, we're decoupling my web UI from the build system so that we can bolt whatever management interface we want onto it. > This will allow us to tell: > NEW PROJECTS: all you have to do is check in file (1) and your project will be > built and put in the download directories automatically. I doubt it'll be that simple, unless we have some extra linkage. 1. project commits their .releng project 2. project updates portal metadata to point some field at their .releng project 3. process on build.eclipse scans portal data for .releng projects and populates something in CC or Hudson 4. new builder appears but needs to be scheduled -- or will that information ALSO be stated in the project's .releng or portal? > CONSUMERS and TESTERS: want to build project X from scratch? Check out all the > code from CVS and then type "ant". Oh, and make sure you have java and ant properly installed. If you want email notification, make sure you have sendmail/mail/(other smtp) installed. If you want scheduling control, make sure you have a crontab or other scheduler configured. If you want to publish the build after you're done building it, make sure you have write perms on dev.eclipse.org & full shell access, and your ssh keys set up. If you want to sign your jars, make sure you're in the signers group and have shell access to build.eclipse.org. If you want newsgroup notification, set that up. You all want this to be a trivial exercise, but there's a LOT of creepy things lurking in the woods along the way to Grandmother's House, ready to pounce on an unsuspecting newbie consumer/tester/project lead.
(In reply to comment #12) > You all want this to be a trivial exercise, but there's a LOT of creepy things > lurking in the woods along the way to Grandmother's House, ready to pounce on > an unsuspecting newbie consumer/tester/project lead. Yes, I do. And having built similar systems before for previous places I've worked, I'm convinced that we can provide that simple user-interface/great level of service to our customers (the committers). However I believe this discussion highlights a design disagreement: I believe that the common build should the "convention over configuration" and that it should solve the build problem for 90% of the projects, but obviously if a project has interesting complex build issues, they will have to do their own builds. For example, TPTP builds native C code on some strange hardware: the common build is never going to support that. By "convention", I mean that there is a standard way to check projects out into your local workspace (no need for configuration); there is a standard layout of the projects in the CVS/SVN (no need for configuration); there is only one place to get the CVS code from (no need for configuration); etc. If you believe in configuration over convention, then we'll never see eye-to-eye. > We're pared a whole .releng project down to: Pretty good. > ./setup.sh > ./start.sh Why do we even need the setup.sh? Can't the start.sh discover that it hasn't been setup and do the setup? > However, if your config doesn't use standard CVS paths, you still need to tell > the system where to go for sources. My answer: don't support that. > This has never been stated before now. Well, I'm pretty sure that I promoted that idea at BW1 way back when, but I apologize for not repeating it. > Yes, but you still need maps, JVM version & path, test manifest, email contact, > newsgroup name, etc. etc. Build workflow metadata is NOT in the > plugins/features. Sure: that's in the one file. > Well, sure, but that doesn't guarantee reproducibility. One user's system may > have ANT_HOME and JAVA_HOME set differently, Or simply have the build script check the version numbers and report a warning or build failure if they are the wrong versions. That's the way I've run these things in the past. > 1. project commits their .releng project I don't see why it needs to be a separate project - why not just a subdirectory? > 2. project updates portal metadata to point some field at their .releng project Using "convention" there is no need to do this. > 3. process on build.eclipse scans portal data for .releng projects and > populates something in CC or Hudson Yep. > 4. new builder appears but needs to be scheduled -- or will that information > ALSO be stated in the project's .releng or portal? It will just run continuously on changes to the source repository. No extra scheduling step is needed. > Oh, and make sure ... Of course. But the build script can easily check for those things and report failures (and even installation instructions) when it doesn't find them. The point is that "ant" will produce a build *and* a set of warnings/reports that tell you how your configuration differs from the master configuration. So getting started is really really easy: check it out and run "ant".
I'm in the process of distilling actionable items from this discussion and will open bugs for each.
Okay, I just opened a bunch of bugs based on a distillation of the comments here. Some comments that I didn't open bugs for were things that are sort of driving principles that I didn't know how to codify in a bug: - decouple build system from web UI - leverage PDE Build - push improvements to PDE Build - prefer ant over shell (essentially the title of this bug) Input is required on a few bugs (namely deciding on shapes and conventions) so please peruse the list of open bugs and vote or add comments where appropriate. Thanks.
If there are specific improvements that are required to PDE build, I suggest that you open these bugs as soon as possible given that we are now just starting M4 and there are already many items on the PDE build plan. http://wiki.eclipse.org/PDE/Build/Galileo
