Community
Participate
Working Groups
As noted in bug 487044 we have moved all our releng cron jobs to the Platform's Releng HIPP instance. It used to be that the "e4Build" id would start up this jobs, but is one step forward to do it from a HIPP instance instead. BUT, one remaining piece of that work is that the "genie.releng" user needs the ability (permission) to "tag" all of the repositories we build, all those in "Platform" and all those in "Equinox". This is exactly like the ability that the "e4Build" id had. I believe that was implemetned with ACLs, but am not sure it will be the same for "genie.releng". But, what ever the mechanism is, we need that ability before we start up I or M builds. Currently planned for roughly June 25, but "the sooner the better" since we it would be good to test it before we start doing "real" builds. Let me know if you need any other information. Thanks,
Adding Dani and Tom as Project or PMC leads so you'll know I am just not making this up on my own. :) = = = = = FYI. Assuming this all works out, we could give up the e4Build Id in about a month.
Added Wayne and Janet for approval, at the webmasters request, since this is a small departure from the norm. That is, it is a bit of giving access "across projects". But, as mentioned, it is the way we have been doing business ever since moving our builds to eclipse.org -- only the "user id" is changing, from 'e4Build' to "genie.releng". And, if you are wondering, we are quite religous about the fact that the only thing the id can do is "tag" ... never commit any code changes! Not even minor ones. (Also, the id already has permission to upload things to Equinox download area as well as the Eclipse Platform).
We should generalise this. When a project is given a HIPP instance, we should just create a "Genie" user and add it to the project group. Project teams can opt to share a Genie user. The Genie user would have the privileges required to write to the download directory and into Git repositories. We'll have to take care to avoid a situation where the Genie user is contributing real IP to the project (tags and merge commits aren't IP). I believe that these users are set up in such a way that nobody can log in on their behalf, and only team members can configure the HIPP instances that control them. Is this accurate? How secure is the public key of such a user?
(In reply to David Williams from comment #2) > And, if you are wondering, we are quite religous about the fact that the > only thing the id can do is "tag" ... never commit any code changes! Not > even minor ones. The IP Log generator would register commits by this user as contributions. The IP Log generator doesn't know anything about UNIX groups, it works from the committer list in the database (which would not include this pseudo user).
(In reply to Wayne Beaton from comment #4) > (In reply to David Williams from comment #2) > > And, if you are wondering, we are quite religous about the fact that the > > only thing the id can do is "tag" ... never commit any code changes! Not > > even minor ones. > > The IP Log generator would register commits by this user as contributions. > The IP Log generator doesn't know anything about UNIX groups, it works from > the committer list in the database (which would not include this pseudo > user). Sorry... the point being that if this user were to commit anything, we'd notice. If it's a priority, I can configure the IP log generator to show a warning if the Genie user shows up as a contributor.
(In reply to Wayne Beaton from comment #3) > We should generalize this. When a project is given a HIPP instance, we > should just create a "Genie" user and add it to the project group. Project > teams can opt to share a Genie user. > > The Genie user would have the privileges required to write to the download > directory and into Git repositories. > > We'll have to take care to avoid a situation where the Genie user is > contributing real IP to the project (tags and merge commits aren't IP). > > I believe that these users are set up in such a way that nobody can log in > on their behalf, and only team members can configure the HIPP instances that > control them. Is this accurate? Pretty accurate. No one can login as a "genie" user directly. They "login" to Hudson using their own committer ID and password, and then "submit a job" which the genie user runs on their behalf. Hence, there is some degree of traceability -- I only say "some" since I am not sure how long "records" of the transaction are kept. From the project's point of view, they can set how long to "save logs", but not sure if there are other logs kept that we do not know about (such as LDAP?). > How secure is the public key of such a user? To use Hudson and, in turn, a 'genie user" users must use their committer id and password. No shell access to the Hudson or the genie user. There are some points that that one project can "kick off" a build on another project's HIPP instance if the two have an agreed upon "secret key" to use in https requests, typically via Curl. (For an example, see bug 494159#c9) I would say that the "secret key" in such cases is moderately secure. Care must be taken not to let that part of the "configuration" be viewed by "anonymous" users or even other committers that do not need to know. However, that "kick off another project's job" I do not think would work for "tagging a build" at least in any sane way. The simple reason is that the builder has the repositories they just checked out to build, and then they want to "tag 'head'" of those repositories. This would be overly complicated to do by several Hudson jobs (if not impossible).
(In reply to Wayne Beaton from comment #5) > Sorry... the point being that if this user were to commit anything, we'd > notice. If it's a priority, I can configure the IP log generator to show a > warning if the Genie user shows up as a contributor. This wouldn't hurt. Especially if generalized. But I don't think the Platform or Equinox need that -- for the simple reason that I know from experience that Dani keeps a close eye on committers :) And, again, *we* need this ability relatively soon so perhaps the general case can be handled via another bug?
(In reply to Wayne Beaton from comment #3) > Project teams can opt to share a Genie user. As long as we're clear this is only an option for projects like platform(which has several individual projects, but a single 'build') but not for unrelated projects(ie: technology.ease and tools.cdt). > I believe that these users are set up in such a way that nobody can log in > on their behalf, and only team members can configure the HIPP instances that > control them. Is this accurate? Yes. > How secure is the public key of such a user? It's pretty well secured, but since these systems are available on the internet, we simply cannot guaranty that it's impossible to break things. -M.
(In reply to David Williams from comment #7) > And, again, *we* need this ability relatively soon so perhaps the general > case can be handled via another bug? Webmaster, I am in favour of pushing forward with this for the Eclipse project and using that experience to sort out issues in the general case.
(In reply to David Williams from comment #2) > Added Wayne and Janet for approval, at the webmasters request, since this is > a small departure from the norm. That is, it is a bit of giving access > "across projects". But, as mentioned, it is the way we have been doing > business ever since moving our builds to eclipse.org -- only the "user id" > is changing, from 'e4Build' to "genie.releng". > > And, if you are wondering, we are quite religous about the fact that the > only thing the id can do is "tag" ... never commit any code changes! Not > even minor ones. > > (Also, the id already has permission to upload things to Equinox download > area as well as the Eclipse Platform). +1 Janet
Ok, I've added the genie.releng user to all the platform groups as well as rt.equinox. -M.
(In reply to Eclipse Webmaster from comment #11) > Ok, I've added the genie.releng user to all the platform groups as well as > rt.equinox. > > -M. Ok, thanks. And, just to be sure (since "platform" is ambiguous) I assume you included "pde" and "jdt" repositories too, right? All as part of the "Eclipse project" or "Eclipse Platform Project" as it is sometimes called?
(In reply to David Williams from comment #12) > Ok, thanks. And, just to be sure (since "platform" is ambiguous) I assume > you included "pde" and "jdt" repositories too, right? All as part of the > "Eclipse project" or "Eclipse Platform Project" as it is sometimes called? No, I didn't. I started just with the groups prefixed by 'eclipse.platform'. I've added the pde and jdt groups. -M.
(In reply to Eclipse Webmaster from comment #13) > (In reply to David Williams from comment #12) > > > Ok, thanks. And, just to be sure (since "platform" is ambiguous) I assume > > you included "pde" and "jdt" repositories too, right? All as part of the > > "Eclipse project" or "Eclipse Platform Project" as it is sometimes called? > > No, I didn't. I started just with the groups prefixed by 'eclipse.platform'. > I've added the pde and jdt groups. > > -M. Whew, glad I said something. :) And I do not see genie.releng if I do a "getfacl" on the repo directories, so assume this is handle in LDAP magic. Is there anyway to command line way to check write access before hand? Or, just test with "tagging" (which will first occur on Thursday, I believe). Thanks again for your timely help! (and education :)
(In reply to David Williams from comment #14) > Whew, glad I said something. :) So am I. > so assume this is handle in LDAP magic. Yes the releng HIPP user has been added to the project groups as if it was a committer. > Is there anyway to command line way to check write access before hand? Or, just > test with "tagging" (which will first occur on Thursday, I believe). Not really. -M.
(In reply to David Williams from comment #14) > Thanks again for your timely help! (and education :) One other bit of education needed. Previously we configured our "build repositories" with B_GIT_EMAIL=e4Build@eclipse.org B_GIT_NAME="E4 Build" COMMITTER_ID=e4Build So that the "tags" would be associated to the right "user". I assume now that should be B_GIT_EMAIL=genie.releng@eclipse.org B_GIT_NAME="Platform Releng HIPP" COMMITTER_ID=genie.releng Or, does the 'genie.releng' user have a "hudson.org" email domain? Thanks as always.
This is main fix, changing e4Build to genie.releng http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/commit/?id=102f341a44c460d716c17478da14b54875ec20f0 Still need to change the "name" from "E4 Build" to "Releng HIPP". And, that name "Releng HIPP" is intentional, instead of "Platform Releng HIPP" since on the "page of HIPP servers" at https://hudson.eclipse.org/ is it called simple "Releng" not "Platform Releng". = = = = = = But, I have noticed we do not seem to explicitly configure on clone, as I though we did. So, I will do a test build now (after deleting previous "N-build" gitCache, to see if it "ends up" with the right values. I noticed on "e4Build" id that we did have these global git configuration variables set: e4Build@build:~> git config --global --list core.autocrlf=false branch.autosetuprebase=always user.email=e4Build@eclipse.org user.name=e4Build push.default=matching My guess is we would be better off explicitly setting the values on clone. But, we'll see what we end up with. (Plus, I think core.autocrlf=input would be a better value that false. We have had cases in the past where someone checked in something with Windows CRLF (such as from OSGi) and then on the next "pull" we get an error that says "source has changed". This would be avoided by using "input" instead of "false". [Though, that might mess up our little "stashing heuristic"].
(In reply to David Williams from comment #17) Sorry, this comment was meant for bug 496490, not this one.
Note, I opened bug 496517 for the "general case" if that is ever pursued. I put it in "Servers" but might belong in "Process". It's sole contribution so far is how to change the HIPP set up process, and -- if anyone would be willing to write it -- provide a Git hook that disallowed commits from this "Hipp User" for anything other than tags.
Finally got to try a test build, and it didn't go too well. There was one repository tagged correctly, but the rest failed. Output pasted below. Technically, only one failed, but that was enough for the script to stop trying the others. The first mentioned in the output as "failing" just to happens to be the "first in the list" (under aggregator) and it was "aggregator" that was tagged OK. It seems to me the main error is "Can't locate Net/LDAP.pm in @INC". It implies something else needs to be installed ... on HIPP instance? Or, whereever "jdt" is? Perhaps because we use the "file://" protocol? Or could it be that it hits this error due to "lack of permission"? The last line, though, is where the push was successful. Pushing to file:///gitroot/platform/eclipse.platform.releng.aggregator.git Perhaps the other "platform" ones would have worked, except the script "bails out" after the first failure. That "bail out" is beyond our control (as currently written) since we use git submodule foreach [doTheTaggingForEach] And I have seen in other contexts "foreach" bails out after first failure. Is this enough information for you to help? = = = = = = = = = Creating gitLog.txt Entering 'eclipse.jdt' Pushing to file:///gitroot/jdt/eclipse.jdt remote: Can't locate Net/LDAP.pm in @INC (you may need to install the Net::LDAP module) (@INC contains: /usr/lib/perl5/site_perl/5.18.2/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.18.2 /usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.18.2 /usr/lib/perl5/5.18.2/x86_64-linux-thread-multi /usr/lib/perl5/5.18.2 /usr/lib/perl5/site_perl .) at /home/data/common/eclipse.org-common-ldap.pl line 22. remote: BEGIN failed--compilation aborted at /home/data/common/eclipse.org-common-ldap.pl line 22. remote: Compilation failed in require at hooks/update line 5. remote: error: hook declined to update refs/tags/Y20160623-1645 To file:///gitroot/jdt/eclipse.jdt ! [remote rejected] Y20160623-1645 -> Y20160623-1645 (hook declined) error: failed to push some refs to 'file:///gitroot/jdt/eclipse.jdt' Stopping at 'eclipse.jdt'; script returned non-zero status. fatal: ambiguous argument 'Y20160623-1100..Y20160623-1645': unknown revision or path not in the working tree. Use '--' to separate paths from revisions, like this: 'git <command> [<revision>...] -- [<file>...]' Stopping at 'eclipse.jdt.core'; script returned non-zero status. Pushing to file:///gitroot/platform/eclipse.platform.releng.aggregator.git
(In reply to David Williams from comment #20) > ... except the script > "bails out" after the first failure. That "bail out" is beyond our control > (as currently written) since we use > git submodule foreach [doTheTaggingForEach] > And I have seen in other contexts "foreach" bails out after first failure. I have found there is a way to avoid this "bail out" on first failure, as described in http://stackoverflow.com/questions/19728933/continue-looping-over-submodules-with-the-git-submodule-foreach-command-after. I may try that, to get more data for this test case, if nothing else.
Created attachment 262658 [details] results when attempting to tag all repositories After the fix in previous comment (using || :) then the mystery just gets deeper. If you look through the log, there are two repos that get tagged and pushed ok: eclipse.platform.releng.aggregator eclipse.platform.common (search for [new tag] to find them) All the others have that error about remote: Can't locate Net/LDAP.pm in @INC (you may need to install the Net::LDAP module) I do not know anything that would be special about "platform common" (except lots of groups have permission to write to it -- which is not try of "aggregator).
Changing to "major" since we are about to start "I" and "M" builds where this function is important, and would represent a "loss of function" compared to using cron jobs from "e4Build". Let me know if there is more information I can (try) and provide. Thanks,
Webmaster: The problem is that the "releng" HIPP misses the Net::LDAP perl module. Please install that. The error message from comment 20 appears for all repos that have a hooks/update file installed. Exceptions are the eclipse.platform.common.git and eclipse.platform.releng.aggregator.git repos, which don't have any update hook installed. The ...common.git can't use the update hook due to bug 376166. The aggregator repo is just missing the hook.
(In reply to Markus Keller from comment #24) > Webmaster: The problem is that the "releng" HIPP misses the Net::LDAP perl > module. Please install that. Mikael, could you fix this? Tomorrow's I-build should not miss the tags again.
releng's HIPP runs on a new SLES version and it seems that the Net:LDAP perl module is missing on that version and I can't change that. Webmasters will be back tomorrow and may be able to provide a fix. In the meantime what I can do is to remove access to the repo with non-gerrit access (ie no access with url others than ssh://user@git.eclipse.org:29418/ppp/ppp.git and http(s)://git.eclipse.org/r/ppp/ppp). This way I can deactivate the git hooks that checks for credentials and which requires the LDAP module (it is safe as you will only go through gerrit and it will do the authentication role itself). WDYT?
(In reply to Mikaël Barbero from comment #26) > releng's HIPP runs on a new SLES version and it seems that the Net:LDAP perl > module is missing on that version and I can't change that. Webmasters will > be back tomorrow and may be able to provide a fix. > > In the meantime what I can do is to remove access to the repo with > non-gerrit access (ie no access with url others than > ssh://user@git.eclipse.org:29418/ppp/ppp.git and > http(s)://git.eclipse.org/r/ppp/ppp). This way I can deactivate the git > hooks that checks for credentials and which requires the LDAP module (it is > safe as you will only go through gerrit and it will do the authentication > role itself). WDYT? I don't like this Gerrit restriction solution, since our production build does not use Gerrit, and if we started to require that, then it might make it difficult to reproduce the production build on "local" machines for testing. It is rather buried above, but when building on "eclipse.org" we use direct file access: file:///gitroot/platform/eclipse.platform.releng.aggregator.git To change that, at this point, seems overly complicated and error prone.
(In reply to Mikaël Barbero from comment #26) Thanks Mikaël, then we better wait for the webmaster. Globally killing direct ssh access to these repos would be worse than missing another set of I-build tags. (In reply to David Williams from comment #27) It's unfortunate that the production build seems to be tied to run on eclipse.org infrastructure where file:///gitroot/... is accessible. But I agree this is not the right time to try to change that.
(In reply to Markus Keller from comment #28) > (In reply to Mikaël Barbero from comment #26) > Thanks Mikaël, then we better wait for the webmaster. Globally killing > direct ssh access to these repos would be worse than missing another set of > I-build tags. > > (In reply to David Williams from comment #27) > It's unfortunate that the production build seems to be tied to run on > eclipse.org infrastructure where file:///gitroot/... is accessible. But I > agree this is not the right time to try to change that. It is not that tied to eclipse.org infrastructure. It is the reverse I am worried about, if we change it so that the build has to push through Gerrit, then I am not sure it would run anywhere except eclipse.org infrastructure. If you'd like to experiment, search the aggregator ("production" directory) for REPO_AND_ACCESS. That is where it is defined as either then export REPO_AND_ACCESS=file:///gitroot else export REPO_AND_ACCESS=git://git.eclipse.org/gitroot I have no doubt it could be made to work, just, as you say, not the right time to complicate our code. Plus, not sure how to make sure it gets pushed to "refs/heads/$BRANCH" instead of "refs/for/$BRANCH"? That might need more changes elsewhere? All doable, but since install the ldap libraries is a simple matter of root running perl -MCPAN -e shell Then at the cpan> prompt run install Net::LDAP And since that it apparently how we have been running on "build.eclipse.org" I'd see no reason it should not be part of Hudson's HIPP install.
(In reply to David Williams from comment #29) You can't push to git: URLs, so the "else" case already has to fail today if you really try to push the tags. Mikael suggested to push to e.g. ssh://user@git.eclipse.org:29418/platform/eclipse.platform.releng.aggregator.git That the server behind this URL is Gerrit is irrelevant. Gerrit behaves like a normal file: or ssh: Git server, accepts normal "git push ..." command line syntax, and therefore allows pushes to refs/heads/$BRANCH (no "for" etc. required, since we wouldn't create a Gerrit change at all). The only problems are: - performance (pull would ideally still use file: or at least git:) - need to set up the SSH keys that are necessary for the Gerrit server
(In reply to Markus Keller from comment #30) > (In reply to David Williams from comment #29) > You can't push to git: URLs, so the "else" case already has to fail today if > you really try to push the tags. Yes, valid point. We would not have to push when we build locally. But, I know I use a bare mirror of all our repositories when I do local builds, so in theory I (or someone) might want to push tags some day (only to local mirror, would not effect original host code) ... but, I currently do not. > Mikael suggested to push to e.g. > ssh://user@git.eclipse.org:29418/platform/eclipse.platform.releng.aggregator. > git > That the server behind this URL is Gerrit is irrelevant. Gerrit behaves like > a normal file: or ssh: Git server, accepts normal "git push ..." command > line syntax, and therefore allows pushes to refs/heads/$BRANCH (no "for" > etc. required, since we wouldn't create a Gerrit change at all). So "change id" and "signature" not are not required if using the above form? I'm always learning new things from you, Markus. :) Thanks! > The only problems are: > - performance (pull would ideally still use file: or at least git:) > - need to set up the SSH keys that are necessary for the Gerrit server Guess the key could be set up by a unique Gerrit job that was to be ran just once. I do not know if that 'genie.releng' user currently has a key under ~/,ssh or not. Never looked. But I think this effort is only worth it if there is a substantial reason that Net::LDAP can not be installed on HIPP instances.
I've added the net::ldap module to all of our HIPP configs so it should appear in the next 30 minutes. -M.
Created attachment 262931 [details] Latest attempt to "tag and push" Things are better. It seems now there are only "permission" issues left. We are getting 7 of the repositories "tagged" and "pushed" ok. Equinox seems fine. JDT and PDE do not seem fine ... and a number of others in "platform". I hope the attached output is enough for you to fix all the permissions. If not, please let me know and I'll make the list for you (but you just need to search for "permission denied").
Created attachment 262970 [details] script to assist setting ACLs Webmasters, I've checked and seems genie.releng belongs to all the right groups. I am assuming that was not a recent change (since the last day or two) or else you would have commented here. I spot checked the 'refs/tags' directories and all seem to have the correct group permissions (including 's' for "group"). I am wondering if the ACL permissions are required? Could it be related to the "maximum number of groups" a user can belong to? genie.releng belongs to 12 or 13. Plus, I have noticed both "e4Build" (our previous build id) and "gerrit" are listed in the ACLs for our repositories. So not sure why genie.releng would not have to be. If you think that would help, or at least no harm in trying :) I have written the attached script to add genie.releng to the ACL of each repository we build and need to 'tag' automatically. I can not test the script completely so please proofread or simply use as an example for your own favorite method. Are there any new rules or restrictions against "file://" access? Long ago, I believe that was recommended to us by you webmasters (well, Denni, I think) as the most efficient way. Let me know if you have other suggestions or need more data from us. The natives are getting restless. :) We could always move back to cronjobs on build machine running under 'e4Build' if you think that is better than us trying to use Hudson? Though I personally would rather try the ACL approach before giving up. Thanks for any help.
Changing to 'blocker' since the committers consider it a "blocker" in bug 497047.
I suspect this is actually a classic case of 'permissions updated, but you need to restart the process to use them'. I couldn't replicate this as the releng user via the command line, so I threw a quick job together and it also failed, so I then restarted the releng HIPP and my test job passed. -M.
(In reply to Eclipse Webmaster from comment #36) > I suspect this is actually a classic case of 'permissions updated, but you > need to restart the process to use them'. I couldn't replicate this as the > releng user via the command line, so I threw a quick job together and it > also failed, so I then restarted the releng HIPP and my test job passed. > > -M. Thanks, Matt. We will give it another try then ... right now!
Created attachment 262994 [details] log showing 25 repositories tagged, as expected Yep. All fixed! The log shows 25 repositories tagged, as expected. For the record, We did try to restart the HIPP instance a few times a few days ago, but it never worked. In fact, from my account is said the instance was "stopped" though obviously running. Oh, maybe that was Platform HIPP. At any rate, thank you thank you thank you!
Fixed, as stated in the previous comment: 25 repositories got "new tag" and no error messages. All due to servers needing a restart. I thought we were running on Linux. :)
