Community
Participate
Working Groups
Right now, when running performance tests and keeping data in the derby database, only the average, standard deviation and sample size are kept for each test. It may be interesting to also keep the minimal and maximum values obtained by the run. The minimal value is informative, as it gives an idea of the lower bound of our test (algorithm), which we don't get from average and standard deviation alone. And while saving the minimal value, why not the maximum as well. It might be interesting too. For background, we use the performance test framework in our project and we export the results in our own front-end to show the performance trends (see http://istmffastyet.dorsal.polymtl.ca/ for the front-end). For some benchmarks, we might want to use the minimal value instead of average. A patch for this has been proposed here https://git.eclipse.org/r/#/c/31476/
Seems like a good idea. To be honest, and show my ignorance of current framework, I am surprised the data itself is not saved (and then any statistic you want could be computed) ... but ... I'll assume that's "phase 2" ... and in the mean time will look at his closer. (I'll need to be sure doesn't "interfere" with our current way of doing analysis, but as you can imagine, I have not looked at that too closely either). You obviously have a "working system" ... so, maybe you can help answer some questions I've just posted to bug 441888.
This bug does not literally "block" bug 374441, but since highly related, wanted to mark it as such to be able to track "whole performance" effort a little easier ... and make sure I didn't lose this one :)
> > You obviously have a "working system" ... so, maybe you can help answer some > questions I've just posted to bug 441888. Actually, we are using only the core of the performance test framework. I tried the performance.ui and really I couldn't make it work with satisfaction, many things were hard-coded, it expects the different build names to have a certain format (which is not very well documented). And our need was different, we use the data to populate graphs on our web site and compare performance day to day: http://istmffastyet.dorsal.polymtl.ca/ We convert the results from the database to json (see file http://git.eclipse.org/c/tracecompass/org.eclipse.tracecompass.git/tree/org.eclipse.tracecompass.alltests/src/org/eclipse/tracecompass/alltests/perf/PerfResultsToJSon.java) and more generally the org.eclipse.tracecompass.alltests project in our source tree: http://git.eclipse.org/c/tracecompass/org.eclipse.tracecompass.git
(In reply to Genevieve Bastien from comment #3) > > > > You obviously have a "working system" ... so, maybe you can help answer some > > questions I've just posted to bug 441888. > > Actually, we are using only the core of the performance test framework. Ok, thanks. You might "subscribe" to bug 448102 since it is my believe that raw data should be stored, not "statistics" ... and if that's done, could impact users such as yourself. (But, at same time, no need to panic, I'd estimate that is months and months away, if done at all.). There _might_ be ways to do both, to maintain compatibility ... but if that takes a lot of extra work, probably would not.
This still seems like a fine idea to me, but I want to get our performance tests working, before making changes, so will postpone this. Thanks for the contribution, though.
(In reply to Genevieve Bastien from comment #3) > > Actually, we are using only the core of the performance test framework. Does this mean you use the bundle named org.eclipse.test.performance? I ask, since I just changed it's "optional pre-reqs" ... and wanted to be sure I didn't mess you up. It's documented (mostly) in bug 390821. Just let me now (probably in that bug) if there is a negative impact ... else, will assume not. Thanks,
I am un-targeting, as there has been no response on Gerrit review comments. (And, even if that happened in next day or two, I'd still prefer to do this after M4, since we still have problems with "performance database" that need to be understood.).
Doing a mass "reset to default assignee" of 52 bugs to help make clear it will (very likely) not be me working on things I had previously planned to work on. I hope this will help prevent the bugs from "getting lost" in other people's queries. Feel free to "take" a bug if appropriate.
Derby no longer used.
