Community
Participate
Working Groups
Per some conversations with Denis, we should craft some mechanism to allow projects to upgrade their HIPP instance. We don't want to support too many versions of Hudson though so to quote Denis 'I think a simple "Version X.y.z is available. Upgrade now." will suffice.'. HIPP uses symlinks to point to a version of the hudson.war file. I think this button would update the symlink to the latest version supported by the Foundation and then the project can restart their HIPP instance themselves. One additional step after that should be done is to update any plugins if applicable. We still need a way for the projects to do this themselves. One option could be to give this access to the Project Lead or similar.
(In reply to Thanh Ha from comment #0) > We don't want to support too many versions of Hudson though so to quote > Denis 'I think a simple "Version X.y.z is available. Upgrade now." will > suffice.'. The HIPP service database table will see the addition of two integer field: current_version, old_version 2.18 = 218, 3.1.2 = 312 10.5 = 1050 > HIPP uses symlinks to point to a version of the hudson.war file. I think > this button would update the symlink to the latest version supported by the > Foundation and then the project can restart their HIPP instance themselves. Thanh, can you slowly start thinking of crafting a script (or several) that, when run as a HIPP user, will: a) make a backup of everything needed to revert? I'm guessing the jobs directory and the hudson directory. Make sure the current_version value is part of the tarball name(s) b) change the symlink and perform any upgrade tasks. I will not specify which version I want to upgrade to -- there should only be one available c) make a restore script that will discard everything and replace it with the backup made in a) In similar to the Start/Stop/restart functionality, when a new version is created and available for upgrade, the Web UI will show that. Command to Upgrade and Revert will be handled by the same handlers as the Start/Stop/restart (ie, the HIPP user will do it, not root) > One additional step after that should be done is to update any plugins if > applicable. We still need a way for the projects to do this themselves. One > option could be to give this access to the Project Lead or similar. +1. Could the HIPP user (easily) do this itself from RPC or a web-api call?
> Thanh, can you slowly start thinking of crafting a script (or several) that Obviously, catching and returning error status will be important to advise committers if the upgrade failed and should be reverted (and why?)
> The HIPP service database table will see the addition of two integer field: > current_version, old_version 2.18 = 218, 3.1.2 = 312 10.5 = 1050 I'll probably also add a current_version_since datetime field. If you've been running current_version for X days, then a revert no longer becomes possible (to avoid accidental data loss).
(In reply to Denis Roy from comment #1) > (In reply to Thanh Ha from comment #0) > > We don't want to support too many versions of Hudson though so to quote > > Denis 'I think a simple "Version X.y.z is available. Upgrade now." will > > suffice.'. > > The HIPP service database table will see the addition of two integer field: > current_version, old_version 2.18 = 218, 3.1.2 = 312 10.5 = 1050 > I've been burned by this before so I think this logic is not good. As soon as any digits after the first become double digits this logic starts to fail. For example: 2.1 = 21 2.10 = 210 3.1 = 31 210 > 31 > 21 but in reality 3.1 > 2.10 > 2.1 It's a little worse since many eclipse projects use symantic versioning so we see 3 digit numbers like 1.2.3.
(In reply to Thanh Ha from comment #4) > (In reply to Denis Roy from comment #1) > > (In reply to Thanh Ha from comment #0) > > > We don't want to support too many versions of Hudson though so to quote > > > Denis 'I think a simple "Version X.y.z is available. Upgrade now." will > > > suffice.'. > > > > The HIPP service database table will see the addition of two integer field: > > current_version, old_version 2.18 = 218, 3.1.2 = 312 10.5 = 1050 > > > > I've been burned by this before so I think this logic is not good. As soon > as any digits after the first become double digits this logic starts to fail. > I suppose if we're not using the numbers to do any greater than or less than comparisons it should be ok.
Created attachment 239670 [details] patch I pushed into admintools the attached patch. This patch provides 3 new tools: hippbackup: Usage: hippbackup [userid] userid is a optional field that is only required when run as root. to specify which HIPP user to backup the instance for. It is recommended to run this script as the regular user though. At the moment backups are backed up to $HOME/backup/$USERNAME-$VERSION and creates 2 files hudson-home-$VERSION.zip and hudson-jobs-$VERSION.zip hippupgrade: Usage: hippupgrade [userid] userid is a optional field that is only required when run as root. to specify which HIPP user to backup the instance for. It is recommended to run this script as the regular user though. This script basically just deletes the existing hudson.war symlink in the HIPP user's $HOME and replaces it with a new hudson.war symlink pointing to the newer Hudson version we support. hipprestore: Usage: hipprestore <version> [userid] version is a required parameter to specify which version to restore the backup from. userid is a optional field that is only required when run as root. to specify which HIPP user to backup the instance for. It is recommended to run this script as the regular user though. This script deletes the $HOME/.hudson and /jobs/$USERNAME/* directories and unzips the tarballs to revert back to the old version.
Denis, Do you have any opinion where the backup zips should be saved to? Currently the backups go to each HIPP's $HOME directory which I don't think should be where we save them. I think common directory would be better. I view these backups as temporary backups specifically for the upgrade process since our regular backup scripts already backup the jobs. Maybe they can all go to a common directory that has a cleanup script that auto removes the backups after a few weeks or whatever we agree as long enough to disable reverting.
(In reply to Thanh Ha from comment #7) > Denis, > > Do you have any opinion where the backup zips should be saved to? > > Currently the backups go to each HIPP's $HOME directory which I don't think > should be where we save them. I think common directory would be better. > > I view these backups as temporary backups specifically for the upgrade > process since our regular backup scripts already backup the jobs. Maybe they > can all go to a common directory that has a cleanup script that auto removes > the backups after a few weeks or whatever we agree as long enough to disable > reverting. Looks like /opt/public/hipp/ is mounted rw on the HIPP servers. I propose we create a sub-directory there that's writeable by the HIPP instances and send the backups there that way there's a common directory webmaster can scan and delete old backups.
> Looks like /opt/public/hipp/ is mounted rw on the HIPP servers. I propose we > create a sub-directory there that's writeable by the HIPP instances and send > the backups there that way there's a common directory webmaster can scan and > delete old backups. +1
(In reply to Denis Roy from comment #9) > > Looks like /opt/public/hipp/ is mounted rw on the HIPP servers. I propose we > > create a sub-directory there that's writeable by the HIPP instances and send > > the backups there that way there's a common directory webmaster can scan and > > delete old backups. > > > +1 Looks like the patches were already merged into master long ago. I updated the backup & restore scripts to use /opt/public/hipp/upgrade_backups as the backup&restore directory.
(In reply to Thanh Ha from comment #10) > Looks like the patches were already merged into master long ago. I updated > the backup & restore scripts to use /opt/public/hipp/upgrade_backups as the > backup&restore directory. If we're going to allow individual HIPP instances to backup their own Hudson than the upgrade_backups directory will need to be group or world writeable. I'm thinking the best solution here is to create a "hipp" group and adding all the genie.* users to it and then making the directory writeable by the hipp group. Thoughts?
(In reply to Thanh Ha from comment #11) > (In reply to Thanh Ha from comment #10) > > Looks like the patches were already merged into master long ago. I updated > > the backup & restore scripts to use /opt/public/hipp/upgrade_backups as the > > backup&restore directory. > > If we're going to allow individual HIPP instances to backup their own Hudson > than the upgrade_backups directory will need to be group or world writeable. The end goal here is to provide a restore point should an upgrade go wrong, or prove to be unstable. Can we instead leverage a directory in the hipp user's home directory?
(In reply to Denis Roy from comment #12) > (In reply to Thanh Ha from comment #11) > > (In reply to Thanh Ha from comment #10) > > > Looks like the patches were already merged into master long ago. I updated > > > the backup & restore scripts to use /opt/public/hipp/upgrade_backups as the > > > backup&restore directory. > > > > If we're going to allow individual HIPP instances to backup their own Hudson > > than the upgrade_backups directory will need to be group or world writeable. > > > The end goal here is to provide a restore point should an upgrade go wrong, > or prove to be unstable. Can we instead leverage a directory in the hipp > user's home directory? We can and that's what the scripts used to be configured to do (comment 7). The downside though of using their home directory though is it's a little harder for us to monitor and clear up space should we need to. If I recall we only wanted to support the restore point for a short time so I imagined a cron job would run that would remove the backup after a week or 2. This would be easier if it was backed up to a shared location.
(In reply to Thanh Ha from comment #13) > I imagined a > cron job would run that would remove the backup after a week or 2. This > would be easier if it was backed up to a shared location. I think the presence of a "restore point" for a HIPP needs to be known to the web UI that will control the action (ie, upgrading an instance will add the restore point details to the database table). From that perspective, we can extend the ldap_community_creator to automatically remove unused restore points after 2 weeks (both on file and from the DB). Make sense?
(In reply to Denis Roy from comment #14) > (In reply to Thanh Ha from comment #13) > > I imagined a > > cron job would run that would remove the backup after a week or 2. This > > would be easier if it was backed up to a shared location. > > > I think the presence of a "restore point" for a HIPP needs to be known to > the web UI that will control the action (ie, upgrading an instance will add > the restore point details to the database table). From that perspective, we > can extend the ldap_community_creator to automatically remove unused restore > points after 2 weeks (both on file and from the DB). Make sense? Sounds good, I'll change the scripts to revert back to using the HIPP user's $HOME as the backup directory.
I started a wiki page to document issues I have run into and eventually steps a project can take to upgrade their HIPP instance. See: https://wiki.eclipse.org/Services/HIPP
Not really depends on but we should be aware of bug 431819. If promoted-builds-plugin is installed it will cause upgrade issues. Considering not many projects use this plugin at the moment, might be worth involving Webmaster to perform the upgrade if a project has this plugin installed.
(In reply to Thanh Ha from comment #15) > Sounds good, I'll change the scripts to revert back to using the HIPP user's > $HOME as the backup directory. Sorry, I forgot to make this change. It's been reverted now to backup to HIPP's home directory again.
Hudson version. We're currently on Hudson 3.0.1, there is a few projects that are testing out Hudson version 3.1.2 for us but there is a problem where it reports to Gerrit an invalid URL. This is fixed in Hudson 3.2.0 which I think should be the next upgrade path for Hudson. Papyrus 3.1.2: https://bugs.eclipse.org/bugs/show_bug.cgi?id=433012 HIPPs on 3.1.2: - modeling.mdt.papyrus - rt.virgo - technology.scout
We have to address this soon. I assume our current 3.0.1 installs are referencing older plugin versions that may or may not work. See bug 440860
Thanh had started work on this, and Mikael will take it over. Providing a new HIPP image (based on the latest 3.2.x release) is one of our top priorities at this time. This bug does depend on bug 431819 -- so we'll need to carefully test Hudson 3.2 before offering it as an option.
To do the upgrade, only changing the symlink to the new war is not enough. The .hudson/war has to be removed so that all cached resources are re-extracted from the standalone jar.
Also the config.xml file contains the Hudson version in a property. I don't how it can affect the Hudson behavior if we don't upgrade it.
Mikael's work on the backend infra is complete. Over to me to integrate into HIPP Control at https://dev.eclipse.org/site_login/myaccount.php
Created attachment 251152 [details] Screenshot Here's a screenshot of the finished product. I have 3 HIPP instances: Babel running 3.0.1, CBI running the current 3.2.2 and Dash also running 3.0.1. The righthand pane shows the view of the README file that Mikael can produce to give more information about the upgrade. I'll be submitting a change soon. That will complete the work here, and will enable projects to begin upgrading.
New Gerrit change created: https://git.eclipse.org/r/42836
What does "Erasing private configs" mean?
> What does "Erasing private configs" mean? It's my version of Lorem Ipsum.
Gerrit change https://git.eclipse.org/r/42836 was merged to [master]. Commit: http://git.eclipse.org/c/websites/dev.eclipse.org.git/commit/?id=cadbdec7ca7e2d4744af84ef5cc8d91d991ad397
The functionality to upgrade is now there. The upgrade output text is not well formatted for such a small space. We need to work on that.
(In reply to Denis Roy from comment #30) > The functionality to upgrade is now there. Does this mean the change is live? For the Code Recommenders HIPP at least, this doesn't quite work yet. Apparently, it can't figure out the Hudson version (FWIW, 3.0.1-b2) our HIPP is running or so the empty parentheses indicate [1]: Hudson HIPP Control Hudson Instances Code Recommenders: Running () <stop> <restart> [1] <https://dev.eclipse.org/site_login/myaccount.php>
Andreas, I've just finished updating the version info for all the HIPPs. Please reload the page and give it a whirl.
(In reply to Denis Roy from comment #32) > I've just finished updating the version info for all the HIPPs. Please > reload the page and give it a whirl. Tempting, but upgrading 5 minutes before the weekend starts...? I'll give it a try on Monday. But, yes, the page now shows the correct version and the upgrade link.
That is very prudent. Mikael and I have updated the output format to something easier to read.
(In reply to Denis Roy from comment #34) > That is very prudent. Speaking of prudence, are there any backups of (at least) the job configs being made? In the past, we had Jenkins (not Hudson) forget about some of its jobs during an update. As our jobs are rather complex (build matrix + 5 promotion jobs), I'd like to avoid having to recreate them from memory (or a "screenshot backup").
(In reply to Andreas Sewe from comment #35) > Speaking of prudence, are there any backups of (at least) the job configs > being made? In the past, we had Jenkins (not Hudson) forget about some of > its jobs during an update. As our jobs are rather complex (build matrix + 5 > promotion jobs), I'd like to avoid having to recreate them from memory (or a > "screenshot backup"). Yes, backup of jobs' config and Hudson's config are made before upgrading. If something goes wrong, just open a bug, we can restore the previous version and configuration. Please note that the workspaces are not saved. This is the only thing you could loose if anything goes wrong.
I think we can consider this one as closed.
(In reply to Andreas Sewe from comment #35) > (In reply to Denis Roy from comment #34) > > That is very prudent. > > Speaking of prudence, are there any backups of (at least) the job configs > being made? In the past, we had Jenkins (not Hudson) forget about some of > its jobs during an update. As our jobs are rather complex (build matrix + 5 > promotion jobs), I'd like to avoid having to recreate them from memory (or a > "screenshot backup"). Note that it is very easy to backup these yourself. For Sirius I use a script like this to regularly backup all our config.xml into a dedicated Git repo: #!/bin/sh # # Requires: # - curl # - jq (http://stedolan.github.io/jq/) # - git (assumes we are launched from inside a Git repo) set -e fetch_jobs() { local readonly HIPP_URL="$1" for j in $(curl -s "$HIPP_URL/api/json" | jq '.jobs | .[] | .name' | sed -s 's/"//g'); do echo "Backuping job $j..." curl -s "$HIPP_URL/job/$j/config.xml" > "$j.xml" done } cd $(dirname $0) rm -f *.xml fetch_jobs https://hudson.eclipse.org/sirius git add -A . git commit -m "Automatic backup."
