Bug 185452 - [search] for all packages seems hung
Summary: [search] for all packages seems hung
Status: VERIFIED FIXED
Alias: None
Product: JDT
Classification: Eclipse Project
Component: UI (show other bugs)
Version: 3.3   Edit
Hardware: PC Windows XP
: P3 major (vote)
Target Milestone: 3.3 RC1   Edit
Assignee: Martin Aeschlimann CLA
QA Contact:
URL:
Whiteboard:
Keywords: performance
Depends on:
Blocks:
 
Reported: 2007-05-03 17:37 EDT by Kevin McGuire CLA
Modified: 2007-05-16 11:49 EDT (History)
5 users (show)

See Also:
benno.baumgartner: review+


Attachments
Proposed patch (4.87 KB, patch)
2007-05-09 13:41 EDT, Frederic Fusier CLA
no flags Details | Diff
patch (6.78 KB, patch)
2007-05-11 06:24 EDT, Martin Aeschlimann CLA
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Kevin McGuire CLA 2007-05-03 17:37:45 EDT
I0501

If you pick Navigate->Go To->Package you get a progress dialog up but the progress is already at 100%.  It says,
Searching...
[bar||||]
Locating Java matches...

It stays that way for a VERY LONG TIME - in a workspace with only 6 projects it was stuck like that for about 3 minutes on a 3Ghz P4.  I thought actually it was hung before finally I got a chooser up with the list of packages.

Issues:
#1: There is no progress indication, it appears hung
#2: The cancel button in the progress bar is unresponsive
#3: What the heck is it doing with all that time?
#4: The resulting list appears wrong. For example, with org.eclipse.ui and org.eclipse.ui.workbench as source projects I get 6 entries for org.eclipse.ui.about which believe I only have one of.

I have to assume nobody ever uses this menu, its pretty unusable as is.
Comment 1 Dani Megert CLA 2007-05-04 02:37:16 EDT
Can reproduce in my large dev workspace.
Comment 2 Martin Aeschlimann CLA 2007-05-04 05:18:32 EDT
This happens in org.eclipse.jdt.internal.ui.dialogs.PackageSelectionDialog.open()

The progress monitor is passed directly to
SearchPattern pattern= SearchPattern.createPattern("*", 
	IJavaSearchConstants.PACKAGE, IJavaSearchConstants.DECLARATIONS,
	SearchPattern.R_PATTERN_MATCH | SearchPattern.R_CASE_SENSITIVE);

new SearchEngine().search(pattern, ...., monitor);

So it seems the search engine isn't showing good progress here

Comment 3 Frederic Fusier CLA 2007-05-04 06:01:42 EDT
I cannot reproduce the hang on the progress dialog. I only can get this dialog when indexes are not ready and then it works properly: the bar progresses and text indicates the remaining files to index. When indexing is finished, "Locating Java Matches..." appears while 1s and then just after the 'Go to Package' dialog opens.

However, I also can observe that packages are repeated several times in the list. So, I'll see what happen here.

For the hang issue, I'd like to have more clues to reproduce as I wasn't able on a Eclipse full source workspace....
Comment 4 Frederic Fusier CLA 2007-05-04 09:38:26 EDT
Eric helps me to reproduce this problem on an 'Europa' workspace...
Comment 5 Dani Megert CLA 2007-05-04 09:50:13 EDT
In my Eclipse SDK workpsace it takes >10s when I do this (i.e. unusable).
Comment 6 Frederic Fusier CLA 2007-05-04 10:51:36 EDT
(In reply to comment #5)
> In my Eclipse SDK workpsace it takes >10s when I do this (i.e. unusable).
> 
Is it the time spent in the Progress dialog only?
If so, we definitely do not have the same numbers, on my box (T60p Intel Core 2 CPU - T7600 @ 2,33GHz), when I open package dialog on a Eclipse full source workspace (ie. 83 projects), it's instantaneous: Progress dialog opens and closes without having the time to see Locating Java matches...
Comment 7 Kevin McGuire CLA 2007-05-04 11:07:02 EDT
I had about 5 source projects (org.eclipse.ui, org.eclipse.ui.workbench, org.eclipse.jface,...ie. not a terribly big workspace) and I had used the PDE trick of adding all plugins to the java search path.  I was able to reproduce the behaviour consistently so it doesn't appear an issue of indexes being built up.
Comment 8 Kevin McGuire CLA 2007-05-04 11:45:38 EDT
(In reply to comment #6)
> Is it the time spent in the Progress dialog only?
> If so, we definitely do not have the same numbers, on my box (T60p Intel Core 2
> CPU - T7600 @ 2,33GHz), when I open package dialog on a Eclipse full source
> workspace (ie. 83 projects), it's instantaneous: Progress dialog opens and
> closes without having the time to see Locating Java matches...

Adding plugins to the java search path makes it easy to reproduce.

In a new workspace on the I20070503 build I pulled in org.eclipse.ui and the Navigate operation was quite fast. I then opened the Plug-ins View and added all to the Java Search Path.  What I observed was the progress dialog informing first of indexing (very fast) then at 100% and doing nothing for some time.
Comment 9 Dani Megert CLA 2007-05-07 04:07:15 EDT
>> In my Eclipse SDK workpsace it takes >10s when I do this (i.e. unusable).
>> 
>Is it the time spent in the Progress dialog only?
It spends quite some time in the dialog, then the dialog goes away and there's the busy cursor only. I have a T43p.
Comment 10 Frederic Fusier CLA 2007-05-09 13:20:10 EDT
Sounds not to be a problem with JDT/Core search but with the requestor used while performing the search request.

Since bug 183062 was fixed, our performance tests covering search package declarations request runs at least 20 times faster. But this test used as scope on a simple project, so I wrote a new one searching all package declarations using the JavaWorkspaceScope on a full source workspace and here are the numbers:

3.2:
FullSourceWorkspaceSearchTests#testSearchPackageDeclarationsWorkspace...
  - 26,737 package declarations in workspace.
  - CPU Time: n=10, sum=6140, av=614.0, dev=7.6, err=2.4
  - Elapsed Process: n=10, sum=6141, av=614.1, dev=7.5, err=2.4

3.3 M7:
FullSourceWorkspaceSearchTests#testSearchPackageDeclarationsWorkspace:
  - 48,104 package declarations in workspace.
  - CPU Time: n=10, sum=5282, av=528.2, dev=39.6, err=12.5
  - Elapsed Process: n=10, sum=5187, av=518.7, dev=10, err=3.1

So, Elapsed Process time goes down from 614.1ms to 518.7ms which means a gain about 15%. It's not 20 times faster but it's still faster...

The important thing is the number of reported matches: 26,737 for 3.2 but 48,104 for 3.3 M7. This is the key point as looking at the requestor used for the request in PackageSelectionDialog.open() method, I found the offending code:
if (fHideEmptyInner) {
    IPackageFragment pkg= (IPackageFragment) enclosingElement;
    if (pkg.getCompilationUnits().length == 0 && pkg.getClassFiles().length == 0) {
        return;
    }
}

The fHideEmptyInner field is true for the dialog, so, each reported package fragment is opened to get the compilation units and the class files lengthes! As this is done while accepting the match in the requestor, user gets the dialog stuck during the JDT/Core search...

So, put back to JDT/UI to see if opening packages may be avoided during the request.
Comment 11 Frederic Fusier CLA 2007-05-09 13:28:56 EDT
Finally put back in JDT/Core as I have a patch to fix reported duplicate declarations. I still think that package should not be opened by the requestor while accepting reported match but I'll open a separated bug for this issue...
Comment 12 Frederic Fusier CLA 2007-05-09 13:41:59 EDT
Created attachment 66522 [details]
Proposed patch
Comment 13 Frederic Fusier CLA 2007-05-10 13:46:37 EDT
After having talked with Jerome, we think that user may still have the hung during this action as long as the requestor opens the reported package fragments.

So, move this bug to JDT/UI to fix this issue. I've opened bug 186415 for the problem of reporting duplicate matches during the search request.
Comment 14 Frederic Fusier CLA 2007-05-10 13:47:37 EDT
Comment on attachment 66522 [details]
Proposed patch

see bug 186415 for the patch of duplicate matches
Comment 15 Martin Aeschlimann CLA 2007-05-11 06:22:45 EDT
We don't want to show empty inner packages and I don't see an other way to find that out, except testing the package children.

I think the problem is that the search engine progress monitor is at 100% when it reports all matches to the result requestor. It should save some ticks for the reporting, and should assume that the requestor code can do some work. Cancel should be tested after each call back.
But I see that this is probably a bigger change in jdt.core. So for now I moved the our testing code from the callback to after the search, using my own progress monitor.
I filed bug 185452 for the search engine issue.

JDT.UI patch to come. Benno, please review.
Comment 16 Martin Aeschlimann CLA 2007-05-11 06:24:34 EDT
Created attachment 66833 [details]
patch
Comment 17 Frederic Fusier CLA 2007-05-11 06:34:43 EDT
(In reply to comment #15)
> We don't want to show empty inner packages and I don't see an other way to find
> that out, except testing the package children.
> 
> I think the problem is that the search engine progress monitor is at 100% when
> it reports all matches to the result requestor. It should save some ticks for
> the reporting, and should assume that the requestor code can do some work.
> Cancel should be tested after each call back.
> But I see that this is probably a bigger change in jdt.core. So for now I moved
> the our testing code from the callback to after the search, using my own
> progress monitor.
> I filed bug 185452 for the search engine issue.
> 
I guess you wanted to say bug 186540...

Note that we can also think about a new SearchPattern flavor to let PackageDeclarationPattern know that it should not report the empty packages.
It will not necessary to filter them in the dialog...
Comment 18 Benno Baumgartner CLA 2007-05-11 07:12:01 EDT
Patch is good. However, you may want to try a HashSet for packageList:
"This class offers constant time performance for the basic operations (add, remove, contains and size), assuming the hash function disperses the elements properly among the buckets"
This would also "fix" the duplicate entry issue...
Comment 19 Martin Aeschlimann CLA 2007-05-11 08:14:13 EDT
The duplicated are based on the package name, not the package element. Duplicate testing is also not always required. A set might make the 'remove' less expensive, but is more memory consuming and 'add' is also more expensive.
So I'd rather stick with the lists.
Comment 20 Martin Aeschlimann CLA 2007-05-11 08:15:47 EDT
patch released > 20070511
Comment 21 Benno Baumgartner CLA 2007-05-16 11:49:59 EDT
verified in I20070516-0010