Community
Participate
Working Groups
Build: 20020409 JRE: IBM 1.3.1 I launched a new target from a selfhosting workspace, brought up the Java perspective, and created a simple "Bonjour, le monde!" program. <possibly irrelevant>I then switched my default JRE setting a few times from JDK 1.4 to IBM 1.3 and back.</possible irrelevant> While the target Eclipse was just sitting doing nothing, I noticed that my CPU was at 100%. By process of elimination, I determined that the "Java indexing" thread was the culprit. I suspended the thread a few times while it was still hammering the CPU to see what it was doing. The following are snapshots of the stack: Thread [Java indexing] (Suspended) org.eclipse.jdt.internal.core.index.impl.CodeByteStream.writeBits(int, int) line: 238 org.eclipse.jdt.internal.core.index.impl.CodeByteStream.writeByte(int) line: 248 org.eclipse.jdt.internal.core.index.impl.CodeByteStream.writeSmallUTF(char[], int, int) line: 289 org.eclipse.jdt.internal.core.index.impl.GammaCompressedIndexBlock.encodeEntry(org.eclipse.jdt.internal.core.index.impl.WordEntry, char[], org.eclipse.jdt.internal.core.index.impl.CodeByteStream) line: 41 org.eclipse.jdt.internal.core.index.impl.GammaCompressedIndexBlock.addEntry(org.eclipse.jdt.internal.core.index.impl.WordEntry) line: 27 org.eclipse.jdt.internal.core.index.impl.BlocksIndexOutput.addWord(org.eclipse.jdt.internal.core.index.impl.WordEntry) line: 73 org.eclipse.jdt.internal.core.index.impl.MergeFactory.mergeReferences() line: 169 org.eclipse.jdt.internal.core.index.impl.MergeFactory.merge() line: 79 org.eclipse.jdt.internal.core.index.impl.Index.merge() line: 231 org.eclipse.jdt.internal.core.index.impl.Index.add(org.eclipse.jdt.internal.core.index.IDocument, org.eclipse.jdt.internal.core.index.IIndexer) line: 81 org.eclipse.jdt.internal.core.search.indexing.AddJarFileToIndex.execute(org.eclipse.core.runtime.IProgressMonitor) line: 161 org.eclipse.jdt.internal.core.search.indexing.IndexManager(org.eclipse.jdt.internal.core.search.processing.JobManager).run() line: 298 java.lang.Thread.run() line: 512 Thread [Java indexing] (Suspended) org.eclipse.jdt.internal.core.index.impl.CodeByteStream.writeSmallUTF(char[], int, int) line: 289 org.eclipse.jdt.internal.core.index.impl.GammaCompressedIndexBlock.encodeEntry(org.eclipse.jdt.internal.core.index.impl.WordEntry, char[], org.eclipse.jdt.internal.core.index.impl.CodeByteStream) line: 41 org.eclipse.jdt.internal.core.index.impl.GammaCompressedIndexBlock.addEntry(org.eclipse.jdt.internal.core.index.impl.WordEntry) line: 27 org.eclipse.jdt.internal.core.index.impl.BlocksIndexOutput.addWord(org.eclipse.jdt.internal.core.index.impl.WordEntry) line: 73 org.eclipse.jdt.internal.core.index.impl.MergeFactory.mergeReferences() line: 169 org.eclipse.jdt.internal.core.index.impl.MergeFactory.merge() line: 79 org.eclipse.jdt.internal.core.index.impl.Index.merge() line: 231 org.eclipse.jdt.internal.core.index.impl.Index.save() line: 341 org.eclipse.jdt.internal.core.search.indexing.IndexManager.saveIndexes() line: 388 org.eclipse.jdt.internal.core.search.indexing.IndexManager.notifyIdle(long) line: 293 org.eclipse.jdt.internal.core.search.indexing.IndexManager(org.eclipse.jdt.internal.core.search.processing.JobManager).run() line: 286 java.lang.Thread.run() line: 512 Thread [Java indexing] (Suspended) org.eclipse.jdt.internal.core.index.impl.CodeByteStream.readBits(int) line: 71 org.eclipse.jdt.internal.core.index.impl.CodeByteStream.readGamma() line: 108 org.eclipse.jdt.internal.core.index.impl.GammaCompressedIndexBlock.nextEntry(org.eclipse.jdt.internal.core.index.impl.WordEntry) line: 90 org.eclipse.jdt.internal.core.index.impl.BlocksIndexInput.moveToNextWordEntry() line: 181 org.eclipse.jdt.internal.core.index.impl.MergeFactory.mergeReferences() line: 170 org.eclipse.jdt.internal.core.index.impl.MergeFactory.merge() line: 79 org.eclipse.jdt.internal.core.index.impl.Index.merge() line: 231 org.eclipse.jdt.internal.core.index.impl.Index.save() line: 341 org.eclipse.jdt.internal.core.search.indexing.IndexManager.saveIndexes() line: 388 org.eclipse.jdt.internal.core.search.indexing.IndexManager.notifyIdle(long) line: 293 org.eclipse.jdt.internal.core.search.indexing.IndexManager(org.eclipse.jdt.internal.core.search.processing.JobManager).run() line: 286 java.lang.Thread.run() line: 512 Thread [Java indexing] (Suspended) org.eclipse.jdt.internal.core.index.impl.GammaCompressedIndexBlock.nextEntry(org.eclipse.jdt.internal.core.index.impl.WordEntry) line: 74 org.eclipse.jdt.internal.core.index.impl.BlocksIndexInput.moveToNextWordEntry() line: 181 org.eclipse.jdt.internal.core.index.impl.MergeFactory.mergeReferences() line: 170 org.eclipse.jdt.internal.core.index.impl.MergeFactory.merge() line: 79 org.eclipse.jdt.internal.core.index.impl.Index.merge() line: 231 org.eclipse.jdt.internal.core.index.impl.Index.save() line: 341 org.eclipse.jdt.internal.core.search.indexing.IndexManager.saveIndexes() line: 388 org.eclipse.jdt.internal.core.search.indexing.IndexManager.notifyIdle(long) line: 293 org.eclipse.jdt.internal.core.search.indexing.IndexManager(org.eclipse.jdt.internal.core.search.processing.JobManager).run() line: 286 java.lang.Thread.run() line: 512 Thread [Java indexing] (Suspended) org.eclipse.jdt.internal.core.index.impl.CodeByteStream.readByte() line: 84 org.eclipse.jdt.internal.core.index.impl.CodeByteStream.readSmallUTF() line: 116 org.eclipse.jdt.internal.core.index.impl.GammaCompressedIndexBlock.nextEntry(org.eclipse.jdt.internal.core.index.impl.WordEntry) line: 76 org.eclipse.jdt.internal.core.index.impl.BlocksIndexInput.moveToNextWordEntry() line: 181 org.eclipse.jdt.internal.core.index.impl.MergeFactory.mergeReferences() line: 170 org.eclipse.jdt.internal.core.index.impl.MergeFactory.merge() line: 79 org.eclipse.jdt.internal.core.index.impl.Index.merge() line: 231 org.eclipse.jdt.internal.core.index.impl.Index.save() line: 341 org.eclipse.jdt.internal.core.search.indexing.IndexManager.saveIndexes() line: 388 org.eclipse.jdt.internal.core.search.indexing.IndexManager.notifyIdle(long) line: 293 org.eclipse.jdt.internal.core.search.indexing.IndexManager(org.eclipse.jdt.internal.core.search.processing.JobManager).run() line: 286 java.lang.Thread.run() line: 512
Easily reproduced. Launch a new runtime-workbench from a self-hosting workspace. Switch to the Java perspective & create a new project, then watch... the Indexing thread consumes 100% of the CPU for 1:15 when target is JDK 1.3.1 & 1:30 when target is J9. Deleting the project & recreating it does NOT reproduce the problem. You need to startup a new test workspace in debug mode.
This was not in 0321 but easily reproduced in 0409.
It happens when you run (instead of debug) a new runtime-workbench too (with the JIT enabled) but only lasts 10-15 seconds.
In debug mode with JDK 1.3.1 the method Index.merge() is called 6 times: merge took 4.006 seconds merge took 6.008 seconds merge took 7.892 seconds merge took 9.483 seconds merge took 11.206 seconds merge took 11.807 seconds The times are slightly worse with J9 as the target VM.
The Indexing thread is NOT started at a lower priority so it actually interupts the main thread... the build operation is not finished & yet the indexing thread has started its AddJarFileToIndex job. IJob's do not seem to suspend themselves, once they are started they run until finished... which accounts for the machine being completed max'ed out. ---------------------------------------------------------------------------- STARTING (Thread[Java indexing,5,main]) - ensuring consistency DONE (Thread[Java indexing,5,main]) - ensuring consistency -> requesting job (Thread[main,5,main]): indexing D:\aaa\jdk1.3.1_01 \jre\lib\rt.jar -> executing (Thread[Java indexing,5,main]): indexing D:\aaa\jdk1.3.1_01 \jre\lib\rt.jar 1 awaiting jobs. index name: D:\aaa\jdk1.3.1_01\jre\lib\rt.jar <----> 1596617441.index INDEX (Thread[Java indexing,5,main]): D:\aaa\jdk1.3.1_01\jre\lib\rt.jar index name: D:\aaa\jdk1.3.1_01\jre\lib\rt.jar <----> 1596617441.index merge merge -> requesting job (Thread[main,5,main]): indexing project /Test merge merge merge INDEX : D:\aaa\jdk1.3.1_01\jre\lib\rt.jar COMPLETE in 12538 ms -> executing (Thread[Java indexing,5,main]): indexing project /Test 1 awaiting jobs. index name: \Test <----> 1657330681.index -> merging index : D:\a0321 \workspace.indexing\.metadata\.plugins\org.eclipse.jdt.core\1596617441.index merge
Indexing the rt.jar will usually never take longer than 30 seconds, and it doesn't compete too badly with main thread on Win2k. Maybe the concurrency isn't very good on a Linux VM ?
With J9 as the target VM & the Indexing thread at NORM_PRIORITY, the indexer completes while the Build is still running... jclMax/classes.zip is only 2.2Mb compared to 13.3Mb for rt.jar. ---------------------------------------------------------------------------- STARTING (Thread[Java indexing,5,main]) - ensuring consistency DONE (Thread[Java indexing,5,main]) - ensuring consistency -> requesting job (Thread[main,5,main]): indexing D:\aaa\j9 \lib\jclMax\classes.zip -> executing (Thread[Java indexing,5,main]): indexing D:\aaa\j9 \lib\jclMax\classes.zip 1 awaiting jobs. index name: D:\aaa\j9\lib\jclMax\classes.zip <----> 2610005571.index INDEX (Thread[Java indexing,5,main]): D:\aaa\j9\lib\jclMax\classes.zip index name: D:\aaa\j9\lib\jclMax\classes.zip <----> 2610005571.index -> requesting job (Thread[main,5,main]): indexing project /Test INDEX : D:\aaa\j9\lib\jclMax\classes.zip COMPLETE in 5068 ms -> executing (Thread[Java indexing,5,main]): indexing project /Test 1 awaiting jobs. index name: \Test <----> 1657330681.index -> merging index : D:\a0321 \workspace.indexing\.metadata\.plugins\org.eclipse.jdt.core\2610005571.index merge
When debugging with JDK 1.3.1_01 as the target VM, the indexing thread took 66 seconds (5 merges) to complete while I was trying to change preferences for PDE. Increasing the Index.MAX_FOOTPRINT to 10Mb instead of 2.5Mb reduces the time to < 55 seconds (only 1 merge).
Fixed. Also note that indexing thread priority got reduced.
With the latest changes to the file structure in InMemoryIndex, the indexing time was reduced to 35 seconds from 66 seconds in debug mode.
