Bug 26820 - Out of Memory indexing new plugins
Summary: Out of Memory indexing new plugins
Status: VERIFIED FIXED
Alias: None
Product: JDT
Classification: Eclipse Project
Component: Core (show other bugs)
Version: 2.1   Edit
Hardware: PC Windows 2000
: P3 critical (vote)
Target Milestone: 2.1 M5   Edit
Assignee: Kent Johnson CLA
QA Contact:
URL:
Whiteboard:
Keywords: performance
Depends on:
Blocks:
 
Reported: 2002-11-20 18:07 EST by Jed Anderson CLA
Modified: 2006-11-29 12:31 EST (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jed Anderson CLA 2002-11-20 18:07:41 EST
Build: M3

I get out of memory exceptions when I do the following:

1. Start M3 on a fresh workspace
2. Import all external plugins
3. Switch to CVS Repository Exploring Perspective
4. Get the following:

!SESSION Nov 20, 2002 17:10:57.63 ----------------------------------------------
java.version=1.4.1-beta
java.vendor=Sun Microsystems Inc.
BootLoader constants: OS=win32, ARCH=x86, WS=win32, NL=en_US
Command-line arguments: -os win32 -ws win32 -arch x86 -data
c:\home\workspaces\autorefreshM3 -install
file:C:/home/eclipse/2.1/eclipseM3/eclipse/
!ENTRY org.eclipse.ui 4 4 Nov 20, 2002 17:10:57.63
!MESSAGE Unhandled exception caught in event loop.
!ENTRY org.eclipse.ui 4 0 Nov 20, 2002 17:10:57.63
!MESSAGE java.lang.OutOfMemoryError
!STACK 0
java.lang.OutOfMemoryError


This has happened to me three or four times now (so I'm starting to think there
is a problem here) :)
Comment 1 Jared Burns CLA 2002-11-20 18:12:28 EST
Not sure if it's related, but I've been getting OutOfMemoryErrors frequently
when launching a selfhosting instance of Eclipse. My case doesn't involve VCM
in any way that I know of. Also, people on the newsgroup have complained about
the same problem with recent builds. My hypothesis is that Eclipse's memory
usage recently surpassed some memory usage mark recently and people are hitting
this now in cases where they're not passing the -Xmx arg to increase the VMs
memory allocation.
Comment 2 Jed Anderson CLA 2002-11-20 18:14:28 EST
I should be more clear on this:

I can switch to the CVS perspective, but I run out of memory browsing the CVS
repository.
Comment 3 Jed Anderson CLA 2002-11-20 18:32:31 EST
This is a Java indexing bug, moving.

Simple test to reproduce this bug:

1. New workspace
2. Import all plugins
3. Wait...

I was running the following VM:
java version "1.3.1_03"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.1_03-b03)
Java HotSpot(TM) Client VM (build 1.3.1_03-b03, mixed mode)

I did a ctrl-break before and a ctrl-break after the out of memory exception,
and the java indexing thread had died.

Full thread dump Java HotSpot(TM) Client VM (1.4.1-beta-b14 mixed mode):

"Snapshot" prio=5 tid=0x0B071B10 nid=0x4e8 in Object.wait() [c71f000..c71fd88]
        at java.lang.Object.wait(Native Method)
        - waiting on <03BC3E58> (a org.eclipse.core.internal.resources.DelayedSn
apshotRunnable)
        at org.eclipse.core.internal.resources.DelayedSnapshotRunnable.run(Delay
edSnapshotRunnable.java:38)
        - locked <03BC3E58> (a org.eclipse.core.internal.resources.DelayedSnapsh
otRunnable)
        at java.lang.Thread.run(Thread.java:536)

"Java indexing" daemon prio=4 tid=0x0AED9370 nid=0x2a0 runnable [bb9f000..bb9fd8
8]
        at org.eclipse.jdt.internal.compiler.classfmt.ClassFileStruct.utf8At(Cla
ssFileStruct.java:73)
        at org.eclipse.jdt.internal.compiler.classfmt.MethodInfo.getMethodDescri
ptor(MethodInfo.java:78)
        at org.eclipse.jdt.internal.core.search.indexing.BinaryIndexer.indexClas
sFile(BinaryIndexer.java:493)
        at org.eclipse.jdt.internal.core.search.indexing.BinaryIndexer.indexFile
(BinaryIndexer.java:529)
        at org.eclipse.jdt.internal.core.search.indexing.AbstractIndexer.index(A
bstractIndexer.java:558)
        at org.eclipse.jdt.internal.core.index.impl.Index.add(Index.java:101)
        at org.eclipse.jdt.internal.core.search.indexing.AddJarFileToIndex.execu
te(AddJarFileToIndex.java:185)
        at org.eclipse.jdt.internal.core.search.processing.JobManager.run(JobMan
ager.java:333)
        at java.lang.Thread.run(Thread.java:536)

"Debug async queue" prio=5 tid=0x0AE81C60 nid=0x57c in Object.wait() [b9cf000..b
9cfd88]
        at java.lang.Object.wait(Native Method)
        - waiting on <031A3850> (a org.eclipse.debug.core.DebugPlugin$AsynchRunn
er)
        at java.lang.Object.wait(Object.java:426)
        at org.eclipse.debug.core.DebugPlugin$AsynchRunner.run(DebugPlugin.java:
692)
        - locked <031A3850> (a org.eclipse.debug.core.DebugPlugin$AsynchRunner)
        at java.lang.Thread.run(Thread.java:536)

"Signal Dispatcher" daemon prio=10 tid=0x00878CA8 nid=0x604 runnable [0..0]

"Finalizer" daemon prio=9 tid=0x00877150 nid=0x608 in Object.wait() [aacf000..aa
cfd88]
        at java.lang.Object.wait(Native Method)
        - waiting on <02E65210> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
        - locked <02E65210> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x00875CC8 nid=0x468 in Object.wait() [aa
8f000..aa8fd88]
        at java.lang.Object.wait(Native Method)
        - waiting on <02E65278> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:426)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:113)
        - locked <02E65278> (a java.lang.ref.Reference$Lock)

"main" prio=5 tid=0x00235090 nid=0x5f8 runnable [6f000..6fc3c]
        at org.eclipse.swt.internal.win32.OS.WaitMessage(Native Method)
        at org.eclipse.swt.widgets.Display.sleep(Display.java:1960)
        at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.java:1436)
        at org.eclipse.ui.internal.Workbench.run(Workbench.java:1418)
        at org.eclipse.core.internal.boot.InternalBootLoader.run(InternalBootLoa
der.java:831)
        at org.eclipse.core.boot.BootLoader.run(BootLoader.java:462)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:324)
        at org.eclipse.core.launcher.Main.basicRun(Main.java:247)
        at org.eclipse.core.launcher.Main.run(Main.java:703)
        at org.eclipse.core.launcher.Main.main(Main.java:539)

"VM Thread" prio=5 tid=0x00874A80 nid=0x60c runnable

"VM Periodic Task Thread" prio=10 tid=0x0023F7B8 nid=0x614 waiting on condition

"Suspend Checker Thread" prio=10 tid=0x0023FE98 nid=0x5a4 runnable
java.lang.OutOfMemoryError
Full thread dump Java HotSpot(TM) Client VM (1.4.1-beta-b14 mixed mode):

"Snapshot" prio=5 tid=0x0B071B10 nid=0x4e8 in Object.wait() [c71f000..c71fd88]
        at java.lang.Object.wait(Native Method)
        - waiting on <03BC3E58> (a org.eclipse.core.internal.resources.DelayedSn
apshotRunnable)
        at org.eclipse.core.internal.resources.DelayedSnapshotRunnable.run(Delay
edSnapshotRunnable.java:38)
        - locked <03BC3E58> (a org.eclipse.core.internal.resources.DelayedSnapsh
otRunnable)
        at java.lang.Thread.run(Thread.java:536)

"Debug async queue" prio=5 tid=0x0AE81C60 nid=0x57c in Object.wait() [b9cf000..b
9cfd88]
        at java.lang.Object.wait(Native Method)
        - waiting on <031A3850> (a org.eclipse.debug.core.DebugPlugin$AsynchRunn
er)
        at java.lang.Object.wait(Object.java:426)
        at org.eclipse.debug.core.DebugPlugin$AsynchRunner.run(DebugPlugin.java:
692)
        - locked <031A3850> (a org.eclipse.debug.core.DebugPlugin$AsynchRunner)
        at java.lang.Thread.run(Thread.java:536)

"Signal Dispatcher" daemon prio=10 tid=0x00878CA8 nid=0x604 waiting on condition
 [0..0]

"Finalizer" daemon prio=9 tid=0x00877150 nid=0x608 in Object.wait() [aacf000..aa
cfd88]
        at java.lang.Object.wait(Native Method)
        - waiting on <02E65210> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:111)
        - locked <02E65210> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:127)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x00875CC8 nid=0x468 in Object.wait() [aa
8f000..aa8fd88]
        at java.lang.Object.wait(Native Method)
        - waiting on <02E65278> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:426)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:113)
        - locked <02E65278> (a java.lang.ref.Reference$Lock)

"main" prio=5 tid=0x00235090 nid=0x5f8 runnable [6f000..6fc3c]
        at org.eclipse.swt.internal.win32.OS.WaitMessage(Native Method)
        at org.eclipse.swt.widgets.Display.sleep(Display.java:1960)
        at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.java:1436)
        at org.eclipse.ui.internal.Workbench.run(Workbench.java:1418)
        at org.eclipse.core.internal.boot.InternalBootLoader.run(InternalBootLoa
der.java:831)
        at org.eclipse.core.boot.BootLoader.run(BootLoader.java:462)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:324)
        at org.eclipse.core.launcher.Main.basicRun(Main.java:247)
        at org.eclipse.core.launcher.Main.run(Main.java:703)
        at org.eclipse.core.launcher.Main.main(Main.java:539)

"VM Thread" prio=5 tid=0x00874A80 nid=0x60c runnable

"VM Periodic Task Thread" prio=10 tid=0x0023F7B8 nid=0x614 waiting on condition

"Suspend Checker Thread" prio=10 tid=0x0023FE98 nid=0x5a4 runnable
Comment 4 Jed Anderson CLA 2002-11-20 19:48:12 EST
I've tried to track this down, but I am unable to reproduce the behaviour in a
target workspace (happens only in my host).
Comment 5 Olivier Thomann CLA 2002-11-21 09:21:39 EST
You might want to use -vmargs -Xmx312M to increase the heap size.
Comment 6 Jed Anderson CLA 2002-11-21 15:33:59 EST
Shouldn't the java indexing be able to run in a "standard" heap size, no matter
what size the workspace is?
Comment 7 Philipe Mulet CLA 2002-11-25 04:58:19 EST
Kent - pls investigate
Comment 8 Kent Johnson CLA 2002-11-25 11:57:11 EST
Nothing has changed in the indexer since the spring.

Have you changed your VM or the args used to initialize it in the last few 
weels? What VM + args are you using?

My cut feel is this is JIT related issue with the 'new' 1.4.1 Sun VMs...
Comment 9 Jed Anderson CLA 2002-11-25 15:42:29 EST
This was happening on Sun 1.3.1.

How's this for a theory.

The VM allocates MAX amount of memory for it to use.  The indexer needs n bytes
of memory to run.  Eclispe self-hosting has grown to use (MAX - n + m) where m > 0.
When the indexer starts running, it doesn't have enough memory to finish the job.

How much memory does the indexer need?  Can it run in less than n bytes?
Comment 10 Kent Johnson CLA 2002-11-25 16:05:24 EST
My understanding of the indexer is that it has its own watermark (I think its 
10Meg) & it merges the in-memory index with the disk representation whenever 
the in-memory size exceeds the limit.

Its definitely possible that with your current VM memory settings, we could be 
pushing the overall limit... but its not like the indexer uses 50Meg (or at 
least its not supposed to...).
Comment 11 Kent Johnson CLA 2003-01-06 18:00:28 EST
Removed all index consistency checks from JDT Core startup.

Released changes that add save index jobs immediately after rebuildAll jobs to 
free up space. Previously, each index could keep upto 10Mb in memory before 
flushing the information to disk. Index files were only saved when the 
IndexManager was idle.

Now with SaveIndex jobs, rebuildAll jobs which fork 10-100 small update file 
jobs, are immediately followed by save jobs to free all the space.
Comment 12 David Audel CLA 2003-02-11 09:28:36 EST
Verified.
Comment 13 Michael Giroux CLA 2006-11-29 10:00:18 EST
(In reply to comment #11)
What release is this fix targeted for?  This problem is killing my JUnit test environment in 3.2.1.

Short of a fix in the indexer, is there some way for me to detect that the indexer is running and wait for it to complete?  For example, if I use Platform.getJobManager() can I find the indexer and suspend it or even kill it? After all, this is a launched workspace and I really do not need the indexing.

Alternately, is there a way to configure the launched workspace so that it never runs the indexing thread?
Comment 14 Kent Johnson CLA 2006-11-29 10:38:14 EST
This problem was fixed and released 4 years ago.

Please open a new bug with repeatable steps and thread dump.
Comment 15 Michael Giroux CLA 2006-11-29 12:31:15 EST
See https://bugs.eclipse.org/bugs/show_bug.cgi?id=166223.

See also http://www.adam-bien.com/roller/page/abien/20060801 for another user having the same problem.

I'll try to reduce my project to something I can post for your convenience.