Web-Based PDE Builds, Part 2: Q&A
I received some questions from the guys in the ECF project about using my infra for doing web-based PDE builds, so I thought I’d digest them and post them here:
Q: When running a build from the web, why queue the command? Why not just exec() it directly?
A: Well, it depends on your infrastructure. On our internal IBM server or on emft.eclipse, running an older, simpler version of this code, we just have php assemble the command and exec() it. If you compare that with the newer, fancier version used for MDT or M2T and you’ll see it creates a lockfile so that apache doesn’t have to have its php doing exec() calls (which are not allowed on build.eclipse.org).
Q: Why use .htaccess instead of httpd.conf?
A: I don’t have access to that file on build.eclipse.org, because I’m not root there. Sure, I could have used httpd.conf on the other servers where I have root access, but this way it’s consistent and I can set up multiple servers the same way.
Q: What directories are you protecting?
A: .htaccess is used to protect the web UI that runs the build script (foo/build/index.php) and that promotes builds (foo/build/promo.php), plus anything else that non-authorized users shouldn’t be able to use.
Q: Where do you keep the script start.sh?
A: Depends on the server and how much control you have on it. On emf.toro and emft.eclipse, scripts live in /home/www-data/build/modeling/scripts/, because these scripts are run by apache (aka www-data). On build.eclipse, they go in ~/build/modeling/scripts because it’s a shared server so I can only work in my home dir.
The contents of this scripts folder is in fact the result of doing a cvs checkout, which makes it very easy to keep up to date when I make changes to the code base.
cvs -d :pserver:anonymous@dev.eclipse.org:/cvsroot/modeling -q co -d scripts releng-common/tools/scripts
Q: How does your stuff compare to what Cruise Control offers?
A: I’ve only ever seen Cruise Control once, as demo’d to me by Max Feldman, of Borland/GMF fame. They use it as a way of setting up cronjobs and sending email alerts, and for continuous builds. I maintain my own crontab manually (it’s pretty simple). start.sh and promoteToEclipse.sh both have hooks into a sendEmail.php script I use to fire emails notifying when builds are done or promoted, if people request such notification (it’s not global, it’s on demand).
As to continuous builds, it’s never been a requirement on us as we’re a much smaller (and more geographically constrained) team; in fact, it would be a detriment because emf.toro and emft.eclipse are used by MANY components’ builds so if there were builds running constantly it would tax the hell out of the servers and put undue strain on CVS. As it is, I can’t run more than one build for a given cvs repo at a time because of locks put in place at eclipse.org to prevent pserver abuse.
Instead, I built web UI that the developers could use to fire their own builds when they deemed it necessary, with daily scheduled Nightlies in my crontab as another assurance of quality. Plus, we never promote a build that’s not green, so we don’t have to worry about automatically spinning fixes. What has yet to be created is a log watcher to scan crontab-fired build logs and alert people via email if a problem is ever found in a Nightly. The current process is to spin a build at 3 or 4pm every day (if anything changed in CVS), so people who’ve just committed new code that day) can check about an hour later if they broke anything.
Q: How current is http://wiki.eclipse.org/EMFT_Build_Server_Setup ?
A: As current as possible, but only for emft.eclipse.org. There is no wiki doc yet for build.eclipse.org setup.
Q: Why do I need to add the ‘www’ group?
A: This is the group used for shared access to both website files (/var/www/html/modeling/…) and build files created by apache (/home/www-data/build/modeling/…). This allows me to edit website files and also to sudo to apache to edit files created by that user. (See below.)
Q: On Suse there already is a group www. It is the group that apache runs under. On Suse, apache runs as user/group = wwwrun/www. The corresponding group on Ubuntu is www-data. On Ubuntu, apache runs as user/group=www-data/www-data.
A: See, it depends on your linux distro. If it’s already there, great. If not, I’m recording what *I* did so that what we had on emf.toro (a very old Debian 2.4 kernel box set up back when we had to support JDK 1.3) could be mirrored to emft.eclipse (a Xen/FC4 2.6 kernel vserver).
Q: What is the apache user used for? It’s not the user apache runs under.
A: On Xen/FC, it is. I believe xampp tries to run as user apache too, or maybe I tweaked its startup script to allow that.
Q: I’m not familiar with the !! in the passwd field in /etc/passwd. Does the !! in the passwd field of /etc/passwd have something to do with disabling logins?
A: Yes. This is so that you can’t log in as apache, but you can sudo to that user if you’re a user in the “www” group.
Q: What does this mean in my /etc/sudoers file?
%www ALL = (apache) NOPASSWD: ALL, (root) /usr/bin/su apache
A: See the man pages for sudo or visudo for more on this. Essentially, the command says:
“Allow all users in group “www” to passwordlessly sudo to user apache and do whatever as that user.”
Granted, that’s a bit of security hole, but it’s only for users that are already authorized to ssh to the server to run builds. This lets them do cleanup (deleting old/bad builds) since everything is created via the web and thus owned by apache or www-data.
The second part of the sudo command says:
“Allow all users in group “www” to sudo as root to restart apache (with password)”
This is there for me to be able to restart the webserver without having to su first. It authorizes me to run ONLY that single two-word command.
Q: Why do you want to arrange to ssh to eclipse.org using key authentication and no password/passphrase?
A: This is so that scripts can access dev.eclipse.org or download1.eclipse.org in order to check out/in files and publish bits from build server to production. By omitting the passphrase, these ssh calls can be done unattendedly via a script, kicked via a webpage. As this is a potential risk, so it’s mitigated with .htaccess (see above).
Posted July 19th, 2007 by in category: pde, scripting, server config, web
You can skip to the end and leave a response. Pinging is currently not allowed.
One Response to “Web-Based PDE Builds, Part 2: Q&A”
Leave a Reply
You must be logged in using your Eclipse Bugzilla account to post a comment.
Jeroen Rodland Says:
August 18th, 2007 at 03:30 pm
DR. PAUL’s Child Health and Wellness Info Site…
Useful, thank you!…