Bug 289057 - Java Content Assist taking too long
Summary: Java Content Assist taking too long
Status: VERIFIED FIXED
Alias: None
Product: JDT
Classification: Eclipse Project
Component: Core (show other bugs)
Version: 3.6   Edit
Hardware: Macintosh Mac OS X - Carbon (unsup.)
: P3 major (vote)
Target Milestone: 3.6 M6   Edit
Assignee: Satyam Kandula CLA
QA Contact:
URL:
Whiteboard:
Keywords: performance
Depends on:
Blocks: 305116
  Show dependency tree
 
Reported: 2009-09-10 06:36 EDT by Damian Harvey CLA
Modified: 2010-03-10 03:10 EST (History)
8 users (show)

See Also:


Attachments
Thread dump using Java VisualVM (12.24 KB, text/plain)
2009-09-14 12:44 EDT, Damian Harvey CLA
no flags Details
Adaptj thread dump from Windows Vista (52.31 KB, text/plain)
2009-09-15 13:28 EDT, Damian Harvey CLA
no flags Details
JAR file that causes problem (17.13 MB, application/octet-stream)
2009-09-16 05:19 EDT, Damian Harvey CLA
no flags Details
Patch demonstrating the changes (13.35 KB, patch)
2010-01-20 10:18 EST, Satyam Kandula CLA
no flags Details | Diff
Proposed patch (14.42 KB, patch)
2010-02-02 01:38 EST, Satyam Kandula CLA
no flags Details | Diff
Proposed patch (10.85 KB, patch)
2010-02-03 06:04 EST, Satyam Kandula CLA
no flags Details | Diff
Proposed patch (14.01 KB, patch)
2010-03-05 13:59 EST, Satyam Kandula CLA
no flags Details | Diff
Proposed patch (16.69 KB, patch)
2010-03-09 00:03 EST, Satyam Kandula CLA
frederic_fusier: iplog+
frederic_fusier: review+
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Damian Harvey CLA 2009-09-10 06:36:31 EDT
User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2
Build Identifier: 20090621-0832

I have a large project (although not excessively so) and Java Content Assist is so slow that it is unusable. Eclipse varies between:
- It taking 5 seconds to display the list
- Getting the "Problems During Content Assist" error modal
- Getting the red message in the status bar at the bottom - sometimes in conjunction with the error modal.

On a fresh vanilla (ie. no additional plugins) install of 3.5 20090621-0832, I then import my JSF Web project into the workspace (347MB in total). Content Assist on any Java code regardless of how many characters typed will timeout or take 5 seconds.

I have tried various things such as increasing the Xmx to 1535M etc, however it is getting to the point where I am better off typing everything.

Additionally Copy/Paste and Organize Imports are both very slow. This feels related to whatever is causing the Content Assist issue. If I create a new workspace with a new project then everything is typically fast again.

When starting Eclipse with -consolelog I get the following error in the console:

!SESSION 2009-09-10 11:23:56.988 -----------------------------------------------
eclipse.buildId=I20090611-1540
java.version=1.6.0_15
java.vendor=Apple Inc.
BootLoader constants: OS=macosx, ARCH=x86, WS=carbon, NL=en_US
Framework arguments:  -product org.eclipse.epp.package.java.product
Command-line arguments:  -os macosx -ws carbon -arch x86 -product org.eclipse.epp.package.java.product -consolelog

!ENTRY org.eclipse.core.net 1 0 2009-09-10 11:24:04.615
!MESSAGE System property http.nonProxyHosts has been set to local|*.local|169.254/16|*.169.254/16 by an external source. This value will be overwritten using the values from the preferences

!ENTRY org.eclipse.jdt.ui 2 0 2009-09-10 11:24:36.745
!MESSAGE The 'org.eclipse.jdt.ui.JavaAllCompletionProposalComputer' proposal computer from the 'org.eclipse.jdt.ui' plug-in did not complete normally. The extension took too long to return from the 'computeCompletionProposals()' operation.

!ENTRY org.eclipse.jdt.ui 2 0 2009-09-10 11:24:51.170
!MESSAGE The 'org.eclipse.jdt.ui.JavaAllCompletionProposalComputer' proposal computer from the 'org.eclipse.jdt.ui' plug-in did not complete normally. The extension took too long to return from the 'computeCompletionProposals()' operation.



Reproducible: Always

Steps to Reproduce:
1. Download Eclipse
2. Import Project
3. Hit CRTL-Space for Content Assist



I am running on a brand new MacbookPro with 4GB of RAM with OSX 10.6 (Snow Leopard). This behaviour was originally encountered on an older Macbook with 2GB RAM (the new one purchased specifically to try to address this issue)
Comment 1 Olivier Thomann CLA 2009-09-14 12:13:31 EDT
It would be nice if you could provide thread dumps while the code assist is slow. I don't see a way to "fix" this issue without any idea on what is causing the slowness.
I have a pretty big workspace and the code assist is pretty responsive. I don't use a Mac though and I don't have access to a Mac for long term testing.

This could be Mac OS specific.
Comment 2 Damian Harvey CLA 2009-09-14 12:44:04 EDT
Created attachment 147120 [details]
Thread dump using Java VisualVM
Comment 3 Damian Harvey CLA 2009-09-14 12:45:05 EDT
Added thread dump from VisualVM. Not sure if this is the correct format or approach (followed instructions here http://wiki.eclipse.org/index.php/How_to_report_a_deadlock)

Please let me know if you require anything else.
Comment 4 Olivier Thomann CLA 2009-09-14 13:43:10 EDT
I don't see anything obvious.
Could you please try the upcoming 3.6M2 and tell us if this is improving what you see ?
Do you have "Insert parameter names" checked in Preferences>Java>Editor>Content Assist>Insertion?
Comment 5 Damian Harvey CLA 2009-09-14 14:00:20 EDT
I'll download 3.6 and try it.

I had "Insert Best Guessed Arguments" selected. Changing this to "Insert Parameter Names" did not help.
Comment 6 Olivier Thomann CLA 2009-09-14 14:15:13 EDT
What about unchecking "Fill method arguments and show guessed arguments" ?
Your problem might be related to http requests to get the javadoc of the type taking too long.
Did you change the value for the time out in Preferences>Java>Editor>Content Assist>Advanced?
Comment 7 Damian Harvey CLA 2009-09-14 14:23:55 EDT
I tried unchecking that and also increasing & reducing the Javadoc timeout to 50ms and 1ms respectively.

No change to the performance of Content Assist unfortunately.
Comment 8 Damian Harvey CLA 2009-09-14 14:36:29 EDT
Would it be of value for me to take a dump using the Adaptj StackTrace app or was there enough info in the VisualVM dump?

On OSX Adaptj StackTrace has a dependency on GDB that I'll have to configure (hence why I used VisualVM to start with)
Comment 9 Olivier Thomann CLA 2009-09-14 14:42:17 EDT
You should first wait for 3.6M2. If the performance issue is no longer there, then everything is fine.
If you can reproduce with 3.6M2, we will try to see what we can do to investigate this issue.
Thanks.
Comment 10 Damian Harvey CLA 2009-09-15 08:06:05 EDT
Tried with Version: 3.6.0 Build id: N20090911-2000

Same problem.

Initially after importing the project the Content Assist seemed to work ok (although camel-case searching only worked after finding the first word). However after a few minutes Content Assist was unusable again.
Comment 11 Damian Harvey CLA 2009-09-15 08:16:01 EDT
I have taken a JING screencast showing the issue with the new Eclipse 3.6 build. It's 6MB unfortunately but does show the issue well.

http://locuslive.com/webdrive/EclipseIssue.swf

I wasn't sure if that build was 3.6M2. Is it on the nightly builds download page?
Comment 12 Damian Harvey CLA 2009-09-15 13:27:12 EDT
I've just installed Eclipse 3.5 Build id: 20090621-0832 on a Windows Vista Laptop with 2GB of RAM. Fresh install & import project into a new workspace.

Same result. So it's not Mac specific, it's Me specific.
Comment 13 Damian Harvey CLA 2009-09-15 13:28:46 EDT
Created attachment 147231 [details]
Adaptj thread dump from Windows Vista

Dump taken from Windows Vista laptop using Adaptj
Comment 14 Olivier Thomann CLA 2009-09-15 13:33:15 EDT
In this case, could it be possible to get your project ?
I don't experience anything like you describe. It looks like this is specific to your project.
Comment 15 Damian Harvey CLA 2009-09-16 05:19:42 EDT
Created attachment 147285 [details]
JAR file that causes problem

The attached JAR file is the culprit. When added to the build path of a project it will reliably cause Content Assist to fail.

The classes in it were generated by Altova Mapforce 2009 to handle EDIFACT transformations.
Comment 16 David Audel CLA 2009-09-16 10:33:11 EDT
With a simple project which reference this jar i reproduce some slowness.

When i complete a type reference i often reach the timeout but not always.
I also noticed that open type (ctrl+shilt+T) is also slow.

So this jar could cause some slowness during searchAllTypeNames() which is used by open type and code assist
Comment 17 Damian Harvey CLA 2009-09-16 10:43:34 EDT
David, I would be interested to know if you also experience any slow down in copy/paste, specifically on types in the Java editor.
Comment 18 David Audel CLA 2009-09-16 11:11:02 EDT
I do not see slow down in copy paste or organize import.
Comment 19 Frederic Allard CLA 2009-09-18 11:47:56 EDT
Hi David,

I had the same problem has you. Our project uses a lot of libs. One of them is 60mo in size.

So the content assist is real pain in the ass with all the new features. Often, eclipse froze for like 1 minute when I try to filter the liste by adding new character.

I found out that if you change the default settings of the content assist to disable the new features, it become fast as it was in 3.4 and lesser.

Here's the settings :

preferences / java / editor / content assist / advanded.

Only check in first and second list the option:
- Java Proposals
- Template Proposals

Change the value of the timeout to 1ms.

Hopes it will help.

But it's only a work around because the new features are unusable on large projects.
Comment 20 Damian Harvey CLA 2009-09-18 11:53:29 EDT
@Frederic: Thanks, but I've tried these settings and they don't help in this case.
Comment 21 Gabriele CLA 2010-01-09 05:49:44 EST
Hi, I also experience same problem on windows vista OS and Eclispe 3.3.2, the affected project become unusable. I consider thi bug very serious and hope for a rapid resolution.

Gabriele
Comment 22 Srikanth Sankaran CLA 2010-01-12 00:38:35 EST
Satyam, can you please investigate ? Thanks!
Comment 23 Keith Morgan CLA 2010-01-13 12:54:21 EST
Our whole team has been experiencing the same problem with both Ganymede and Galileo.  Our projects have a large # of jar files (around 300) and 200 to over 1k java source files per project.  Eclipse freezes for up to 30 sec every time we mistype a variable name, etc.  Turning off suggestions has been the only solution we've found but that's rather disheartening since with so many classes and a mix of US and UK spellings we had come to rely heavily on suggestions in Europa.  Hardware is a mix of pretty fast Windows XP systems, 1-2GB of memory.
Comment 24 Satyam Kandula CLA 2010-01-18 04:56:52 EST
They are 23312 classes (actually many of them are inner classes) in this attached jar, which is causing the searchAllTypeNames() to be slow. I am currently trying to see if we can improve this.
Comment 25 Satyam Kandula CLA 2010-01-18 04:58:06 EST
(In reply to comment #21)
@Gabriele, Did you try out the suggestions Frederic has mentioned in comment #20? How big are your projects or dependent jars?
Comment 26 Satyam Kandula CLA 2010-01-18 05:03:03 EST
(In reply to comment #23)
@Keith Morgan, Did you try out the suggestions Frederic has mentioned in comment#20? To make sure the problems are similar, Could you provide couple of thread dumps? You could get some instructions to get the thread dump at http://wiki.eclipse.org/index.php/How_to_report_a_deadlock.
Comment 27 Satyam Kandula CLA 2010-01-20 10:18:24 EST
Created attachment 156655 [details]
Patch demonstrating the changes 

For the attached jar, put(s) into the internal hashtable is taking lot of time. There seems to be lot of conflicts which is going for the kill. 
At this time, the hashtable is getting created from an existing persisted index file, which was stored again from an hashtable. 

I tried out re-creating the hashtable exactly similar to the one that would have got persisted. Basically I am not doing a put but almost copying the contents in order of their appearance and leaving off the empty slots. 
This optimization gave around 30% performance improvement for searchAllTypeNames for this particular jar. Content Assist returned with values with this change. However, I didn't see considerable improvement for the other projects. 
Side effect with this approaches are -- 1. The size of the hashtable needs to be written on to the index file and hence the version needs to be updated. 2. If we ever change the algorithm of the hashCode computation, we need to update the version again. 

Then, I tried one more optimization. After this hashtable was created, we were unnecessary doing a get, when we could directly access the values. After removing this get(s), I got an additional 30% improvement for this particular jar. 

The attach patch demonstrates the changes.
Comment 28 Satyam Kandula CLA 2010-02-02 01:38:44 EST
Created attachment 157871 [details]
Proposed patch

Here is another patch! 
There is one difference compared to the earlier patch while the hash table gets restored from the index file. 
The code in the earlier patch was doing an obscure way of calculating the empty slots and positioning the contents. In this new patch, the empty slots are being stored in the index file and hence restoring the hash table becomes cleaner.
Comment 29 Srikanth Sankaran CLA 2010-02-02 07:12:06 EST
(In reply to comment #28)
> Created an attachment (id=157871) [details]
> Proposed patch
> 
> Here is another patch! 
> There is one difference compared to the earlier patch while the hash table gets
> restored from the index file. 
> The code in the earlier patch was doing an obscure way of calculating the empty
> slots and positioning the contents. In this new patch, the empty slots are
> being stored in the index file and hence restoring the hash table becomes
> cleaner.

This patch looks much simpler and cleaner than the previous one.
I didn't find anything amiss, but I am new to this area of code.
I presume that all JDT/Core tests and JDT/UI tests pass.

Could you post some performance numbers with this patch on the
large jar files you have been working with ? 

Also a measure of index file sizes would help -- Thanks!
Comment 30 Satyam Kandula CLA 2010-02-03 06:04:10 EST
Created attachment 158022 [details]
Proposed patch

Here is a new patch which does not need an update of the index file. The current Hashtable's put is doing a name comparison before doing the put. However, as we know that this is really unique, these name comparisons are useless. Hence, created a new version of put which will just find out the next empty slot after the index. 
I used this new version of put in couple of other places, which is giving me 3X performance with testSearchNewAllTypeNames() for this jar. The earlier patches gave 2X performance. 
They are some more places where this can be used for better performance -- I am looking at it.
Comment 31 Frederic Fusier CLA 2010-02-05 11:40:12 EST
We currently looking at ways to improve the hashCode used in HashtableOfObject...
Comment 32 Olivier Thomann CLA 2010-02-26 14:43:33 EST
Tagging as 3.6. Not sure this can be completed for M6.
Comment 33 Olivier Thomann CLA 2010-03-05 11:58:08 EST
Targetting M6 for the fix that improves the hashcode only.
Comment 34 Satyam Kandula CLA 2010-03-05 13:59:23 EST
Created attachment 161165 [details]
Proposed patch

This new patch does two things. 
1. It uses 16 characters for hash computation instead of 8. Only Hashtable for the indexing has been modified. 
2. It also uses the new variant of put which doesn't do a name comparison before doing the put, it just finds out if the slot is empty. This new variant is called to put back the hashtable that was stored in the file. 

This patch gives almost 10X performance for this offending jar and around 4% benefit for testNewSearchAllTypeNames

Had to also change the test testBug232880i() because the order of the types that the search is returning now is little different.
Comment 35 Satyam Kandula CLA 2010-03-09 00:03:14 EST
Created attachment 161407 [details]
Proposed patch

This patch reduces the scope of the changes compared to the previous changes. However it still does the both. 
1. It uses 16 characters for hash computation instead of 8. Only Hashtable for
the indexing has been modified. 
2. It also uses the new variant of put which doesn't do a name comparison
before doing the put, it just finds out if the slot is empty. This new variant
is called to put back the hashtable that was stored in the file. 

This patch gives almost 3X performance for this offending jar.
Comment 36 Jay Arthanareeswaran CLA 2010-03-09 01:01:48 EST
Released in HEAD for 3.6M6.
Comment 37 Frederic Fusier CLA 2010-03-09 03:48:06 EST
Comment on attachment 161407 [details]
Proposed patch

Patch looks good to me
Comment 38 Jay Arthanareeswaran CLA 2010-03-10 03:10:15 EST
Verified for 3.6M6 using build I20100309-0809