Community
Participate
Working Groups
We need a back end storage mechanism to persist performance test results, and allow comparing to baseline builds, and analysis of performance across builds. In the IBM performance test lab we had a Derby database for this, so that is one option. Some documentation is found here: http://dev.eclipse.org/viewcvs/viewvc.cgi/org.eclipse.test.performance/doc/Performance%20Tests%20HowTo.html?view=co We should keep in mind that other eclipse.org projects might want to make use of the same infrastructure.
Updated documentation link: http://git.eclipse.org/c/platform/eclipse.platform.releng.git/plain/bundles/org.eclipse.test.performance/doc/Performance%20Tests%20HowTo.html
John, was the decision made that we continue with the same storage for performance tests? I'm asking because setting up an embedded Derby is simple as it comes as an OSGi plugin. I'd be keen to help if desired.
Documentation has been migrated to https://wiki.eclipse.org/Performance/Automated_Tests in https://bugs.eclipse.org/bugs/show_bug.cgi?id=390820
I don't think Derby "comes as an OSGi plugin" any longer ... but sounds easy to "make one" as we need ... BUT ... I think we should still set it up as a "server" on eclipse.org infrastructure, since as an "embedded server" it would reside on the Hudson machine where ever the tests happen to be running ... and we will have limited "direct" access to that. [And, please correct me if/when/whereever I'm wrong]. Webmaster, I still have a lot of research to do, and will "test out the concepts" on my local test machine, but my initialing thinking is to get a "derby server" running on build.eclipse.org, with the files and database somewhere such as in /shared/eclipse/perfdata I believe we can then access this db (to "put in" data) using something like net:1527//build.eclipse.org/perfdata No matter which hudson instance the performance tests are running on. I assume similar to "read data" from it? My thinking is it'd be good, initially, to have such a database, that we could have "direct access to" and at same time ability to "throw it away" after a month or two (once all is working as needed). At that point, (after a month or two) would be the right time to discuss if there is a better way to set up the data base. (such as more secure, part of the "backed up" part of eclipse, etc.). But just to get started, I'd like to make things easy and flexible, since I am "learning as I go". As always, advise/suggestions welcome. And feel free to tell me if this is a really bad idea for what ever reason. Thanks,
You are free to run any reasonable type of service on build.eclipse.org (ie, provided it doesn't hog CPU, RAM and disk resources). You can use just about any port above 1024 and access it from any of the hudson/hipp machines. You could not access that port from the outside world; however, if you do have a shell on build.eclipse.org you could access your service using a tunneled port (ie, ssh -L localport:localhost:remoteport build.eclipse.org)
(In reply to Denis Roy from comment #5) > You are free to run any reasonable type of service on build.eclipse.org (ie, > provided it doesn't hog CPU, RAM and disk resources). You can use just > about any port above 1024 and access it from any of the hudson/hipp machines. > > You could not access that port from the outside world; however, if you do > have a shell on build.eclipse.org you could access your service using a > tunneled port (ie, ssh -L localport:localhost:remoteport build.eclipse.org) Ok, thanks Dennis. As I've now thought about it, don't believe there is much need for "public read access" ... if any ... Our tools can run on build.eclipse.org, can run on build.eclipse.org (or, on Hudson itself) to produce the graphics and comparison charts needed. For those following this bug, our "performance documentation" currently mentions Derby 10.4.2 (or "Cloudscape"). I plan to remove mention of Cloudscape (not sure that's even still available, except of very old archived versions?) and Derby's most recent release is 10.10.2.0 so my current thinking is to try and use that most recent release. [Which again, I mostly mention in case anyone "knows better" than I do]. [I also think a CQ needs to be entered, even for "build time dependency", just to provide a record of it, so I'll do that soon].
*** Bug 448000 has been marked as a duplicate of this bug. ***
I'm about ready to start experimenting (and learning) about this side of performance tests, so thought I'd update status, and ask questions here. I've opened CQ https://dev.eclipse.org/ipzilla/show_bug.cgi?id=8834 to cover "build/test-time only" dependency on Derby (which is just a "works with" dependency). And, I picked 10.11.1.1 simply because it is the most recent and assume the API's and ideas will all be the same.
(In reply to David Williams from comment #8) > I'm about ready to start experimenting (and learning) about this side of > performance tests, so thought I'd update status, and ask questions here. > The CQ has been approved, and while the literal "back end" is easy enough to install and "get running" I am not sure, so far, how to get the o.e.test.performance bundle to "talk" to it. Investigation will continue. There are a few changes I am going to make to o.e.test.performance bundle. See also bug 390820 and bug 441888 for similar comments or follow-on action once these changes are "confirmed". Currently the bundle has 3 optional bundles: Cloudscape;resolution:=optional, org.apache.derby;resolution:=optional, org.eclipse.test.performance.derby;bundle-version="10.4.2";resolution:=optional And the document at http://wiki.eclipse.org/Performance/Automated_Tests#Getting_and_installing_Derby describes one "test-time only" bundle to create. Turns out that org.eclipse.test.performance.derby is very similar to what the document says to make (Markus found it on an old machine ... it is not in our Git repo). And I think "Cloudscape" should be removed ... it is pretty old. Hence, I'd like to end up with one optional bundle, which I'll call org.apache.derby.core since the Apache documentation calls it that, and use it in o.e.test.performance. (and eventually in o.e.test.performance.ui). Plus, we need to change it to use "manifest form", not the "Eclipse 2.0" plugin form. I've created that bundle derby.core bundle, along with a derby.core.feature, and put in the (composite) repo at http://build.eclipse.org/eclipse/buildtools/ The test.xml framework used to "copy" the old bundle into "dropins" but I see no reason we should not now install it, with p2.
Status update: - We are now getting data into the database, on eclipse.org. - Still only running "short set" of 12 test suites -- adding the "save data to database function" does add some time to running the tests. Ironically, about 15 minutes to the 2 hours "short set" on build.eclipse.org, but an hour on "local test machine" ... but, have not been too scientific about it. - I did "reverse order" so baseline is ran first, and then "current build", so any tests that use "assert against baseline" should fail now ... if I understand how that works? - I have tried to use the "performance UI" tool to create the "pretty pictures" but need to fix a few things first. -- The tool is sensitive to "bad data" in database, and there is bad data there, as I got things automated, occasionally ${buildId} would be entered as the buildId, instead of, say I20141029-0800. -- Hence, I plan to delete current database, and start again. (Will try re-running I-build performance tests ... once I confirm "it works" on local machine. -- Not sure if it is "out of date" documentation ... or, if I am doing something wrong ... but not sure it is prepared for a "baseline" such as "4.4.1". At points, seems it might be always expecting the "dated" form? -- I am currently using "config" and "assert" values such as eclipse.perf.config=build\=I20141104-0800;buildId\=I20141104-0800;testedPlatformConfig\=linux.gtk.x86_64;jvm\=8.0 eclipse.perf.assertAgainst=build\=4.4.1;buildId\=I20141104-0800;testedPlatformConfig\=linux.gtk.x86_64;jvm\=8.0 -- Might have to change those, as I learn more of what the "performance.ui" tool is expecting. = = = = = = = BUT, the biggest problem I might have caused myself (or, that the original authors overlooked :) depending on your perspective, is what exactly does the "baseline" mean. To be concrete, I have been assuming it meant to use 4.4.1 to "run" our current 4.5.0 performance test suite. But, if that is the case, there are already a large number of 4.5 tests that do not run at all on 4.4.1, some due to changes in pre-reqs, some do to "real" code changes that make them in compatible. Some perhaps "fixable", but some I doubt are. Was the original design that the "baseline" meant to simply run 4.4.1 target and tests, and compare those with 4.5.0 target and tests? If so, I need to change a number of things ... plus ... in my view, that is not the ideal design ... it would be hard to interpret many of those comparisons, since "the tests have changed" (sort of "apples to oranges") ... not to mention, seems very difficult to add or remove tests? You'd have to add them now, just to have them for ready for Mars SR1? Or, for example, if you could improve a test so it was "faster", then that invalidates any comparison to running old test where it runs slower. In either case, there's a large problem with this aspect of it, and not sure if it is my large conceptual problem, or the hard technical problem of making sure performance tests run on N and N-1? I'll bet there's some "old-timers" :) who remember how it used to be done?
(In reply to David Williams from comment #10) > BUT, the biggest problem I might have caused myself (or, that the original > authors overlooked :) depending on your perspective, is what exactly does > the "baseline" mean. > > To be concrete, I have been assuming it meant to use 4.4.1 to "run" our > current 4.5.0 performance test suite. > > But, if that is the case, there are already a large number of 4.5 tests that > do not run at all on 4.4.1, some due to changes in pre-reqs, some do to > "real" code changes that make them in compatible. Some perhaps "fixable", > but some I doubt are. > > Was the original design that the "baseline" meant to simply run 4.4.1 target > and tests, and compare those with 4.5.0 target and tests? No. Ideally they would run the 4.5 tests but that does not work as you already figured out yourself. We achieved that by backporting new and changed performance tests to a separate branch (e.g. perf_37x) that got branched off the June release. Also, we never changed the baseline to an SRx. We always used the June release to create the baseline, so that we had the same reference point during the development of the next release. Looking at http://archive.eclipse.org/eclipse/downloads/drops/R-3.7-201106131736/performance/performance.php might provide you with some more details.
(In reply to Dani Megert from comment #11) > > > > Was the original design that the "baseline" meant to simply run 4.4.1 target > > and tests, and compare those with 4.5.0 target and tests? > > No. Ideally they would run the 4.5 tests but that does not work as you > already figured out yourself. We achieved that by backporting new and > changed performance tests to a separate branch (e.g. perf_37x) that got > branched off the June release. Ok, I think that means I have the right idea ... but, didn't realize there would be "new builds" to do, when tests are changed or backported (i.e. rebuilding the tests against baseline). For now, I'll continue "getting things to work", and we'll worry about "new builds" later. > Also, we never changed the baseline to an SRx. We always used the June > release to create the baseline, so that we had the same reference point > during the development of the next release. Not sure what the advantage of that is ... seems it would miss items that got "slower" from SR, as long as still equal to initial release? But, I will "backup" and use 4.4 instead of 4.4.1 in the spirit of "doing things like they used to be done". > Looking at > http://archive.eclipse.org/eclipse/downloads/drops/R-3.7-201106131736/ > performance/performance.php might provide you with some more details. Not much so far, but does imply the "full name", such as R-3.6-201006080911 is used instead of a short name, such as "3.6". But thanks for the reminder, I might find more on the file system (than, in the report).
I think we can close this bug as fixed, as the database is installed, working, and collecting data during our performance tests. There is still work to do to get all the data displayed, but you can see the preliminary work at URLs such as http://build.eclipse.org/eclipse/builds/4I/siteDir/eclipse/downloads/drops4/I20141111-0830/performance/org.eclipse.ui.php Plus, I opened bug 451072 for potential *future* improvements and centralization of "the database" ... but, I think for *this* bug, we have completed the essentials. There is one matter of "X-Server" ports, on build.eclipse.org that I'll communicate to webmaster via email, so as to not publish here. (Technically, unrelated to "databases", but is related to "running our performance test analysis on eclipse.org infrastructure". Thanks to all who helped along the way.
(In reply to David Williams from comment #9) > Hence, I'd like to end up with one optional bundle, which I'll call > org.apache.derby.core since the Apache documentation calls it that, and use > it in o.e.test.performance. (and eventually in o.e.test.performance.ui). > I've created that bundle derby.core bundle, along with a derby.core.feature, > and put in the (composite) repo at > http://build.eclipse.org/eclipse/buildtools/ Hi David. We use the test.performance framework along with derby. We used to get apache.derby from Orbit so we had to change our targets to use apache.derby.core and point to the new update site http://build.eclipse.org/eclipse/buildtools/ (not a big problem) However, the update site page says "It may not be long lived.". Would it be possible to make it long lived? :) Or add apache.derby.core to Orbit? It would help us make our builds more future proof. Thanks!
(In reply to Marc-Andre Laperle from comment #14) > However, the update site page says "It may not be long lived.". Would it be > possible to make it long lived? :) Or add apache.derby.core to Orbit? It > would help us make our builds more future proof. Thanks! Opened bug 483198
