Bug 268365 - [index] Indexer runs out of memory
Summary: [index] Indexer runs out of memory
Status: VERIFIED NOT_ECLIPSE
Alias: None
Product: JDT
Classification: Eclipse Project
Component: Core (show other bugs)
Version: 3.5   Edit
Hardware: PC Windows XP
: P3 normal (vote)
Target Milestone: 3.5 M7   Edit
Assignee: Frederic Fusier CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-03-12 11:08 EDT by Kent Beck CLA
Modified: 2017-06-21 09:13 EDT (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Kent Beck CLA 2009-03-12 11:08:17 EDT
Build ID: I20090202-1535

Steps To Reproduce:
I tried to write a test case to reproduce the problem and failed. However, when JUnit Max is installed, I get the error below even without invoking Max. This is Max code that works fine with 3.4.x and 3.5M4. I saw an entry in the M5 release notes that seemed related:

"Fix for bug 6930 required the index version to be incremented. Indexes will be automatically regenerated upon subsequent search queries (accounting for indexing notification in search progress dialogs) and the size of the new indexes will be bigger."


More information:
eclipse.buildId=I20090202-1535
java.version=1.6.0_06
java.vendor=Sun Microsystems Inc.
BootLoader constants: OS=win32, ARCH=x86, WS=win32, NL=en_US
Framework arguments:  -product org.eclipse.sdk.ide
Command-line arguments:  -product org.eclipse.sdk.ide -data D:\pd workspace/../runtime-New_configuration(1) -dev file:D:/pd workspace/.metadata/.plugins/org.eclipse.pde.core/Eclipse/dev.properties -os win32 -ws win32 -arch x86


Error
Thu Mar 12 07:47:58 PDT 2009
Background Indexer Crash Recovery

java.lang.OutOfMemoryError: Java heap space
at org.eclipse.jdt.internal.compiler.util.HashtableOfObject.<init>(HashtableOfObject.java:39)
at org.eclipse.jdt.internal.compiler.util.HashtableOfObject.rehash(HashtableOfObject.java:139)
at org.eclipse.jdt.internal.compiler.util.HashtableOfObject.put(HashtableOfObject.java:112)
at org.eclipse.jdt.internal.core.index.DiskIndex.copyQueryResults(DiskIndex.java:349)
at org.eclipse.jdt.internal.core.index.DiskIndex.mergeWith(DiskIndex.java:522)
at org.eclipse.jdt.internal.core.index.Index.save(Index.java:191)
at org.eclipse.jdt.internal.core.search.indexing.IndexManager.saveIndex(IndexManager.java:659)
at org.eclipse.jdt.internal.core.search.indexing.AddJarFileToIndex.execute(AddJarFileToIndex.java:204)
at org.eclipse.jdt.internal.core.search.processing.JobManager.run(JobManager.java:394)
at java.lang.Thread.run(Thread.java:619)
Comment 1 Olivier Thomann CLA 2009-03-12 11:12:36 EDT
Moving to JDT/Core
Comment 2 Frederic Fusier CLA 2009-03-12 12:17:26 EDT
What is JUnit Max? What was your -Xmx value when you run it?
Comment 3 Kent Beck CLA 2009-03-12 13:22:48 EDT
(In reply to comment #2)
> What is JUnit Max? What was your -Xmx value when you run it?

JUnit Max is a plugin that runs a project's tests riskiest-first when you save a Java file in that project and then reports the results as markers. Max is being activated at startup but it doesn't do anything significant that I could see (obviously not or the indexer wouldn't choke). The indexer runs out of memory before any file has been edited, so Max isn't running when the problem occurs.
Comment 4 Frederic Fusier CLA 2009-03-12 13:48:58 EDT
So, I guess you use the default eclipse default heap size (256M)?
Comment 5 Kent Beck CLA 2009-03-12 14:56:41 EDT
(In reply to comment #4)
> So, I guess you use the default eclipse default heap size (256M)? 

Sorry about that. Correct, default heap size.
Comment 6 Ian Bull CLA 2009-03-13 18:47:47 EDT
I am getting this with I20090313-0100 simply by starting a run-time workbench (I get it in the run-time workbench).  All I have installed in mylyn and a custom cheat sheet project.

I'm using the Eclipse defaults.
Comment 7 Frederic Fusier CLA 2009-03-14 12:57:43 EDT
(In reply to comment #6)
> I am getting this with I20090313-0100 simply by starting a run-time workbench
> (I get it in the run-time workbench).  All I have installed in mylyn and a
> custom cheat sheet project.
> 
> I'm using the Eclipse defaults.
> 
This one is a duplicate of bug 166223... You just need to set the heap size at least to 256M (default while starting a work time workbench is the default VM, i.e. 64M...)
Comment 8 Ian Bull CLA 2009-03-14 15:25:55 EDT
 
> This one is a duplicate of bug 166223... You just need to set the heap size at
> least to 256M (default while starting a work time workbench is the default VM,
> i.e. 64M...)
> 
Thanks for pointing me at this bug Fredric.  I wonder if more is going on here though. In my target workspace I only have simple mail app, and I've not added anything to my java search path.  Do you think the indexer would fail with this setup with 65M of heap space?
Comment 9 Frederic Fusier CLA 2009-03-14 16:27:01 EDT
(In reply to comment #8)
> Thanks for pointing me at this bug Fredric.  I wonder if more is going on here
> though. In my target workspace I only have simple mail app, and I've not added
> anything to my java search path.  Do you think the indexer would fail with this
> setup with 65M of heap space?
> 
I guess your app at least uses a JDK 6.0, doesn't it? This would imply the indexing of approx. 37,000 class files, hence may explain that the max memory was reached (note that starting with a heap of 64M, does not mean that the indexing can use all of it...) Could you make a try just with 128M and see if it works (I guess it should...)?
Comment 10 Ian Bull CLA 2009-03-16 00:37:57 EDT
(In reply to comment #9)
> I guess your app at least uses a JDK 6.0, doesn't it? This would imply the
> indexing of approx. 37,000 class files, hence may explain that the max memory
> was reached (note that starting with a heap of 64M, does not mean that the
> indexing can use all of it...) Could you make a try just with 128M and see if
> it works (I guess it should...)?
> 
Good call on the Java 1.6 thing.  However, I set:
-Xms40m -Xmx256m

and I'm still getting the heap space error. I'm not sure that PDE is actually using these args (when I look at the configuration in the about dialog the heap space arguments are not there).


Comment 11 Ian Bull CLA 2009-03-16 01:11:12 EDT
I opened bug 268711 for the PDE issue.  Thanks for your help Frederic.
Comment 12 Frederic Fusier CLA 2009-03-16 05:39:56 EDT
(In reply to comment #11)
> I opened bug 268711 for the PDE issue.  Thanks for your help Frederic.
> 
You're welcome, but I wonder whether there's a real issue with PDE or not. I often change -Xmx values in my launch config and it's well taken into account...

Note also that I did make a test in a brand new workspace with an empty project using JDK 6.0 and I was able to index all the classes with 64M. Does your mail app contain a lot of source files or have a lot of plugins/jars dependencies? Was there a compilation running while the OOME occurred?
Comment 13 Ian Bull CLA 2009-03-16 11:25:03 EDT
> You're welcome, but I wonder whether there's a real issue with PDE or not. I
> often change -Xmx values in my launch config and it's well taken into
> account...
Yep, my misunderstanding of how the launcher arguments are applied.

> 
> Note also that I did make a test in a brand new workspace with an empty project
> using JDK 6.0 and I was able to index all the classes with 64M. Does your mail
> app contain a lot of source files or have a lot of plugins/jars dependencies?
> Was there a compilation running while the OOME occurred?
> 
The mail app was the simple one that PDE generates (new plug-in project > create an RCP app, select "Simple RCP Mail" from the list of templates.  

There were no other custom settings, so if a compilation was occurring, it would have just been the automatic build.  

Comment 14 Frederic Fusier CLA 2009-04-20 07:41:06 EDT
(In reply to comment #13)
> > 
> > Note also that I did make a test in a brand new workspace with an empty project
> > using JDK 6.0 and I was able to index all the classes with 64M. Does your mail
> > app contain a lot of source files or have a lot of plugins/jars dependencies?
> > Was there a compilation running while the OOME occurred?
> > 
> The mail app was the simple one that PDE generates (new plug-in project >
> create an RCP app, select "Simple RCP Mail" from the list of templates.  
> 
> There were no other custom settings, so if a compilation was occurring, it
> would have just been the automatic build.  
> 
I did the same test and got only a max of 105M for the heap size after the indexing finished!? So, I retry with a -Xmx128M and never got any OOME with the same test case. Note that obviosuly starting with less memory (e.g. 64M) makes the indexer crashing...

As with a "pure" eclipse install, we're definitely far from the described OOME in both cases (Kent and you), my assumption is that the additional plug-ins (JUnitMax, MyLyn ) could be responsible of an extra memory consumption which may not leave enough room for the indexer to finish its work properly.

So, currently close this bug as NOT_ECLIPSE...

FYI, see bug 215376 to track work around the OOME raised by the indexer when there's definitely not enough room to finish the jar indexing...
Comment 15 Olivier Thomann CLA 2009-04-28 08:50:29 EDT
Verified for 3.5M7.