Bug 148652 - Interim Report of performance tests of service registry
Summary: Interim Report of performance tests of service registry
Status: RESOLVED FIXED
Alias: None
Product: Equinox
Classification: Eclipse Project
Component: Framework (show other bugs)
Version: 3.2   Edit
Hardware: PC Windows XP
: P3 enhancement (vote)
Target Milestone: 3.3 M5   Edit
Assignee: equinox.framework-inbox CLA
QA Contact:
URL:
Whiteboard:
Keywords: performance
Depends on:
Blocks:
 
Reported: 2006-06-26 10:06 EDT by Ikuo Yamasaki CLA
Modified: 2007-03-29 18:22 EDT (History)
2 users (show)

See Also:


Attachments
Service Registry Performance Test Results(060626) (213.10 KB, application/octet-stream)
2006-06-26 10:09 EDT, Ikuo Yamasaki CLA
no flags Details
Excel file of performance test result comparison between R3.2 and R3.3M5 (51.00 KB, application/octet-stream)
2007-03-13 17:00 EDT, Ikuo Yamasaki CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Ikuo Yamasaki CLA 2006-06-26 10:06:25 EDT
I'll upload zip file, which contains results of my performance tests of service registry ( and service tracker), in the next post. Please read "ReadMe.htm" first, which tells you what to read and my recommendation of orders to read. (You will need Excel to read the result).
If you have any questions or comments, please feel free to access me or post!!
Comment 1 Ikuo Yamasaki CLA 2006-06-26 10:09:35 EDT
Created attachment 45299 [details]
Service Registry Performance Test Results(060626)

This zip file includes Excel files and ReadMe.htm
Comment 2 BJ Hargrave CLA 2006-06-26 10:27:25 EDT
Lots of numbers... :-)

What was the hypothesis you were testing? I know you have identified several areas where we have made some improvements, but what were you original goals with this analysis?

Also, while these are marked as interim results, do you have an interim conclusions to share? What else needs to be done/analyzed? 

Thanks for all the work on this!
Comment 3 Ikuo Yamasaki CLA 2006-06-26 11:43:31 EDT
Thank you for your comment and sorry for lack of easiness to read, BJ.

(In reply to comment #2)
> Lots of numbers... :-)
> What was the hypothesis you were testing? I know you have identified several
> areas where we have made some improvements, but what were you original goals
> with this analysis?
> Also, while these are marked as interim results, do you have an interim
> conclusions to share? What else needs to be done/analyzed? 

Good point. These are questions I have in my mind (by myself) and would
like to ask all of you.

First of all, there was nobody who knows the results of regarding Equinox's
service registry performance (and there was no tests). In preparation
for increasing use of service registry in future, we would like to grasp
the current implementation performance results and improve it if
possible. What's why I just started this work.

# Current conclusion I have is the file in the zip "SummaryGraphs.xls", which # shows several aspects of current implementation's performance.

But I'm not sure what the goal SHOULD be.
In other words, what is good performance "enough" 
(especially in the aspect of scalablity) ?  
- CPU and memory spec, JVM
- There is no general model which is exact same as actual usage of equinox...
   (including how many services ...)

# Of course, the faster, the better. the less memory, the better. But
# there might be a tradeoff between speed and memory consumption.

In addition, there might be other tests which we should grasp.
E.g. I did test on ServiceTrackers specified by not filter but class
name. or, ServicePerformance test for Declarative Services might be
tested similarly.

Last thing I should mention is that some of these tests will be included
in Automated tests ( of releng ?). Additional improvements will be
achieved by not only me but also others, if it is found.

If you have any opinion or requests, please let's me know.

P.S. I'll have other tests' results regarding https://bugs.eclipse.org/bugs/show_bug.cgi?id=141144,
	LazyServiceTracker, SimpleServiceTracker, ...
Comment 4 Ikuo Yamasaki CLA 2007-03-13 17:00:54 EDT
Created attachment 60731 [details]
Excel file of performance test result comparison between R3.2 and R3.3M5

I've run the performance test against R3.2 and R3.3M5 by applying Bug#150872.
The result shows the improvement between two.

I'll take several results for explanation of improvement.(highlighted in the excel sheet)
------------------
[Precondition1] 20 service objects with different service properties under the same service interface are registered in a service registry for each 10,000 service interfaces. (That means, totally, 20*10,000=200,000 service objects are registered).

1) For R3.3M5, getServiceReference(className) for each service object takes 76.5 micro sec, which is 87.7% of R3.2.

2) For R3.3M5, getServiceReferences(className, filter) for each service object takes 342 micro sec, which is 90.3% of R3.2.
--------------
[Precondition2] In addition to precondition1, ServiceTracker for the service interface is opened in advance. (A tracker keep trackes 20 service objects).

3) For R3.3M5, each ServiceTracker#getService() and call a method takes 177.5 micro sec, which is 10.9% of R3.2.
----------------
[No precondition]
4) For R3.3M5, each BundleContext#registerService(,,) with 3 service properties takes 188 micro sec on average for registering 10,000 service objects, which is 1.5% of R3.2.
--------------
Alghouth I don't know about which bug fix contributed to which improvement,
I can list some of them:

Bug#151645 for all(?)
Bug#142385 for 4.
Bug#144540 for 3.
Bug#144055 for 4.
and so on...
Comment 5 Thomas Watson CLA 2007-03-29 18:22:13 EDT
Several fixes have been released in 3.3.  Nothing else is planned for service registry performance at this time.