Bug 496316 - Investigate if "database" cron jobs can be performed on Releng Hipp
Summary: Investigate if "database" cron jobs can be performed on Releng Hipp
Status: RESOLVED FIXED
Alias: None
Product: Platform
Classification: Eclipse Project
Component: Releng (show other bugs)
Version: 4.6   Edit
Hardware: PC Linux
: P3 enhancement (vote)
Target Milestone: 4.6.1   Edit
Assignee: David Williams CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: 487044
  Show dependency tree
 
Reported: 2016-06-17 13:55 EDT by David Williams CLA
Modified: 2016-06-22 15:59 EDT (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description David Williams CLA 2016-06-17 13:55:58 EDT
As mentioned in bug 487044 some of the few remaining cronjobs I have that run under my ID are related to the database server we use for performance tests results. 

The derby instance itself is under 
/shared/eclipse/derby

It has a script file to simplify commands such as 'start', 'stop', and 'ping' (and others). 

The database that is written to during performance tests and read from for analyssis is under 
/shared/eclipse/database
and is named perfDB

There is also a derby.properties file there that defines "how to run". 

= = = = = = = =

The two cronjobs I have are to "check if derby is running". It does this with a simply "ping" command. Which will fail, if it is not running. If running, it returns something like 
Fri Jun 17 13:51:18 EDT 2016 : Connection obtained for host: [IPaddress], port number [portNumber]

(I've not listed the ip address and port number just to be not too public about them.). 

I think this "check if running" job could be moved to Releng HIPP. 

= = = = = = = = 

The other cronjob I have is that upon @reboot of the build machine, the derby data base it started, with a simple 'start' command. 

While we can not implemnt a Releng HIPP job that works off of @reboot, I think we can achieve the same function be providing a job that sends "start' to the server (and stop, and ping, and maybe a few others? if anyone knows what they are for :) 

 Usage:  ./derby/derby.sh <command>
                Where <command> is one of start, stop, ij, runtimeinfo, ping, sysinfo, help, serverhelp, dblookhelp
Comment 1 David Williams CLA 2016-06-17 13:56:54 EDT
I will take this one, and see what I can do, since that will "finish" all the cronjobs I have (for Eclipse).
Comment 2 David Williams CLA 2016-06-17 14:56:50 EDT
Not much luck, so far. I get a "connection refused" error. 
Probably related to "proxies"? 

Oddly enough, I could "start" it from Releng HIPP, but then could not access it from anywhere else! Even the HIPP instance. I suspect  due to mis-match of IP addresses. 

So, for now, I had to "kill" the version started with HIPP, and restarted under my user id. 

I'll read up on "proxies" or something to see if there is more that can be done (easily :)
Comment 3 David Williams CLA 2016-06-17 17:24:13 EDT
I have managed to do the routine "ping" to see if the database was still running or not. (Have not literally tested it yet ... will wait for a time when I know no performance tests will be running, and then turn off database for an afternoon). 

But, things like "start", "stop", or "runtimeinfo" do not work. 

When I tried to "stop" for example, got an error message that said: 

Host 172.25.102.108 is not local to the server running on build.eclipse.org, so cannot be used for NetworkServerControl commands.

Kind of makes sense, for security reasons. There are ways to "adminster a server remotely" but it takes more than a little bash job. It would require a "servlet" or similar. Way more work than we need. 

So, I will count this as "fixed" and we can leave it up to the webmaster to start and stop it :) .... or release engineers can get a shell script. 

I will send a note to webmaster.
Comment 4 David Williams CLA 2016-06-18 09:41:52 EDT
(In reply to David Williams from comment #3)
> I have managed to do the routine "ping" to see if the database was still
> running or not. (Have not literally tested it yet ... will wait for a time
> when I know no performance tests will be running, and then turn off database
> for an afternoon). 

I did test the "check if running" function and it did fail when I shutdown database Saturday morning. So, that little sanity check can be said to be complete. Now just need to figure out best way to start it up automatically. 

I think options are: 

A. Status quo: some "release engineer" will need a terminal to build machine and put the starup in their own crontab. 

B. Webmasters add it to the build machine's "startup" mechanism. 

C. In theory could run it on the HIPP instance itself, but I think that will be a LOT of extra work chaning IP addresses in several places, as well as react any time our Hipp Instance is moved to a different Hipp machine. (It is currently on "Hipp8" but believe these machines are "reconfigured" from time to time, depending on load, demand, etc. -- which would mean we'd have to go through all the reconfiguration each time that heppened. So, I think 'A' or 'B' are only viable options.
Comment 5 Eclipse Webmaster CLA 2016-06-20 14:32:24 EDT
For option C, why not just use 'localhost' as the destination, rather than by ip?  That way even if we move the HIPP instance the jobs wont actually have to be updated.

-M.
Comment 6 David Williams CLA 2016-06-20 16:00:27 EDT
(In reply to Eclipse Webmaster from comment #5)
> For option C, why not just use 'localhost' as the destination, rather than
> by ip?  That way even if we move the HIPP instance the jobs wont actually
> have to be updated.
> 
> -M.

because the performance jobs are running on a completely different host. A dedicated performance machine.

https://hudson.eclipse.org/perftests/view/Eclipse%20and%20Equinox/

So it needs to know where to "send" it's data. 

And, before you ask, the reason we don't run the database on "perftest" is that then the database operations themselves *might* effect the performance tests. The perftest machinage itself should be as "quiet" as possible, except for running performance tests. 

Hope that's what you were asking. Let me know if I am misunderstandng.
Comment 7 Eclipse Webmaster CLA 2016-06-22 13:56:07 EDT
Ok, well that makes it clearer.

For option A at some point there will be no more shell access so at that point we'll have to use something like option B, or move the db service to the platform HIPP.

But option B removes the projects ability to stop/restart the service as desired, and basically hands support over to Webmaster.

So for now we should probably go for A with the idea that the perf tests will need to be updated so that the location of the test db server can be changed with relative ease.  

-M.
Comment 8 David Williams CLA 2016-06-22 15:59:38 EDT
Ok, let's call this one "fixed" then, since I did fix the cron job that checks every now and then that the database is still running (incidentally, the fix was by more "hardcoding" the IP address :) and then I have opened bug 496587 to improve the way we define that IP address so there could (in theory) be just one place we would have to change it, and everything continue working. 

Thanks,