Community
Participate
Working Groups
Created attachment 286463 [details] the create event of the garbage collect thread Not sure who is directly to blame When i use my eclipse to debug against a eclipse product that runs with: openjdk version "16.0.1" 2021-04-20 OpenJDK Runtime Environment AdoptOpenJDK-16.0.1+9 (build 16.0.1+9) Eclipse OpenJ9 VM AdoptOpenJDK-16.0.1+9 (build openj9-0.26.0, JRE 16 Windows 10 amd64-64-Bit Compressed References 20210421_24 (JIT enabled, AOT enabled) OpenJ9 - b4cc246d9 OMR - 162e6f729 JCL - cea22090ecf based on jdk-16.0.1+9) then sometimes if i debug my whole eclipse locks down, To see what was happening i started another eclipse that runs my own eclipse in a debugger and from there on i do my normal work (so starting and debugging that eclipse based product) When it happens i then can see (just suspend the main thread) what is so busy And the thing is that the Debug view tree goes wild, because for some reason it gets a HUGE amount of threads that are very quickly created and terminated. i think they are all garbage collect threads, so it seems if for some reason a garbage collector is triggered, that somehow makes a huge amount of threads in that openjdk 16 openj9 vm And all those events are added to a queue a processes. for some reason those system threads are not ignored. In the first i attach 2 pictures, 1 is that you can see the threads that is created and the create event that then adds this to a queue of things to process the second one is is the queue and look how many there are in that one already (and i do force clear through the debugger to see if i can get it to a normal state, but they keep coming)
Created attachment 286464 [details] the huge queue
i tries Eclipse 4.20 but still the same problem a huge amount of DelayedDoModelChangedJob are created like this: and because of that my eclipse completely is kind of dead. Thread [Worker-643: Debug Event Dispatch] (Suspended (breakpoint at line 467 in TreeModelContentProvider$DelayedDoModelChangedJob)) owns: ArrayList<E> (id=1473) TreeModelContentProvider$DelayedDoModelChangedJob.runDelayed(Object) line: 467 TreeModelContentProvider$DelayedDoModelChangedJob.runDelayed(IModelDelta, IModelProxy) line: 455 TreeModelContentProvider.modelChanged(IModelDelta, IModelProxy) line: 514 AbstractModelProxy$1.run() line: 92 SafeRunner.run(ISafeRunnable) line: 45 JavaDebugTargetProxy(AbstractModelProxy).fireModelChanged(IModelDelta) line: 96 DebugEventHandler.fireDelta(IModelDelta) line: 169 JavaThreadEventHandler(ThreadEventHandler).fireDeltaAndClearTopFrame(IThread, int) line: 241 ThreadEventHandler.handleCreate(DebugEvent) line: 151 JavaThreadEventHandler.handleCreate(DebugEvent) line: 359 JavaDebugTargetProxy(EventHandlerModelProxy).dispatchCreate(DebugEventHandler, DebugEvent) line: 163 JavaDebugTargetProxy(EventHandlerModelProxy).handleDebugEvents(DebugEvent[]) line: 123 DebugPlugin$EventNotifier.run() line: 1214 SafeRunner.run(ISafeRunnable) line: 45 DebugPlugin$EventNotifier.dispatch(DebugEvent[]) line: 1248 DebugPlugin$EventDispatchJob.run(IProgressMonitor) line: 480 Worker.run() line: 63
Has it started happening now on moving to new release ? Can you attach a sample small application to reproduce?
not sure when it really started to be this way i have it with 4.19 and 4.20 Debugging against first i think java 15 (openj9) and now also jdk-16.0.1+9 java.vm.name=Eclipse OpenJ9 VM java.vm.specification.name=Java Virtual Machine Specification java.vm.specification.vendor=Oracle Corporation java.vm.specification.version=16 java.vm.vendor=Eclipse OpenJ9 java.vm.version=openj9-0.26.0 My java eclipse also now runs with that one. (and the eclipse i run in debug mode also uses that one) Its not directly reproducible but it happens a few times a day I think when i am just debugging our application and that applications triggers a certain garbage collection Then by java eclipse goes berserk. I can only recover now because my java eclipse is started also with another eclipse so i can set breakpoints in org.eclipse.debug.internal.ui.viewers.model.TreeModelContentProvider public IStatus runInUIThread(IProgressMonitor monitor) { List<Object> currentBatch = new ArrayList<>(); synchronized (fQueue) { if (shutdown || fQueue.isEmpty()) { return Status.OK_STATUS; } currentBatch = new ArrayList<>(fQueue); fQueue.clear(); } if (DebugUIPlugin.DEBUG_CONTENT_PROVIDER) { DebugUIPlugin.trace("Delayed batch size: " + currentBatch.size()); //$NON-NLS-1$ } for (Iterator<?> iterator = currentBatch.iterator(); iterator.hasNext();) { Object task = iterator.next(); on currentBatch = and Object task = lines and at the moment it hits there in the debugger i do clear() against currentBatch and then also the same against fQueue field. (and that numerous times long after i killed my target of our eclipse product) and then it comes back on. but that fQueue field can be filled up by thousends of items. And then after many clears i it comes back to live, if i don't debug my own java eclipse it pretty much can't get out of it and i need to kill the process.
Will it be too much to ask to try once on 4.17 build and see if you can still reproduce?
Which part? My java eclipse or the one I debug so our own eclipse RCP product? I think I never saw that before 419. But there are also other factors here of course, before that I guess I also used a lower java version for running and debugging I can try to run my java with 4.17 if my current workspaces work.
i tried it now for 2 days or so and when i run my eclipse with 4.17 it doesn't happen (as far as i know see, most of the time it would happen a few times a day)
when using 4.17 as my eclipse i don't get the big problem until now But i did get that the debug view does get a bit wild. You see the scrollbar constantly getting bigger and smaller (and yes i see in the application that i debug that garbage collection does happen quite a bit) But for some reason it doesn't get as bad as in 4.20 where i can just kill my own eclipse to get back into the game.. We really need to have a filter where those GC threads can be filtered out so they never are added to the ui (it should already be a system thread right?)
@Julian, Do you think it can be related to Bug 537399?
(In reply to Sarika Sinha from comment #9) > @Julian, > Do you think it can be related to Bug 537399? Seems like the issue of many shortlived GC threads is there even without that fix. Maybe temporarily disabling GC one some objects makes it worse? I don't know, I've never anything like that with openjdk. Johan, please try disabling "show logical structures" to see If that affects your issue (in 4.20).
"show logical structures" in the variables view? I wonder how that effect threads in the debug view.. (how is that related) ? What i find very weird is that those short lived gc threads are shown anyway or tried to be shown... because i do have "show system threads" disabled.. so why are those not system threads. if i enabled that i do see some gc threads by default 15 of them when starting up those are then called: Daemon System Thread [GC Worker] (Running) yesteday in my 4.17 i for the first time had the same thing, i really needed to kill my 4.17 eclipse because it was never getting back And yes the program i was debugging i think was gc'ing a bit (i think i see that on our heap status symbol that you can show in the status bar in an eclipse app) So at one point OpenJ9 makes a lot of gc threads or eclipse is interpreting that that it does generate a lot of those... The stupid thing is i can't repoduce this, i made a small example app that has 2 threads where i generate a lot of garbage and it never happens how long i let that one run... So that is very annoying..
(In reply to Johan Compagner from comment #11) > "show logical structures" in the variables view? > I wonder how that effect threads in the debug view.. (how is that related) ? Yes. For the logical structure calculation, GC gets temporarily disabled on the relevant objects. But if you got the same issue in 4.17 it's not related.
i switched for debugging to the hotspot variant and then it works fine But debugging OpenJ9 vm in eclipse can result in a full "dos" attack .. But still not sure how this happens and when
i read now they added an option: https://www.eclipse.org/openj9/docs/version0.27/#new-xx-adaptivegcthreading-option-added i guess that was already also there and enabled by default in the .26 release? Because that sounds a lot where i hit against if i use the openj9 variant. That that vm then creates and destroys a lot of gc threads.
hmm does really nobody have any problems with the Debug view as i have? Because i did think it was the OpenJ9 variant but now i again have the same thing that the debug view eats up a huge amount of resources and debugging is almost not doable because of that You see that it tries to constantly repaint the something in that tree, because the scrollbar is constantly moving a bit (getting smaller and bigger) and now it is just: java.version=17 java.version.date=2021-09-14 java.vm.compressedOopsMode=32-bit java.vm.info=mixed mode, sharing java.vm.name=OpenJDK 64-Bit Server VM java.vm.specification.name=Java Virtual Machine Specification java.vm.specification.vendor=Oracle Corporation java.vm.specification.version=17 java.vm.vendor=Eclipse Adoptium java.vm.version=17+35 the "standard" hotspot vm.. the main thread is constantly doing this: "main" #1 prio=6 os_prio=0 cpu=1019562.50ms elapsed=89031.99s tid=0x00000231db772330 nid=0x2d6c runnable [0x000000aa0ef2d000] java.lang.Thread.State: RUNNABLE at org.eclipse.swt.internal.win32.OS.SendMessage(Native Method) at org.eclipse.swt.widgets.Tree.getSelection(Tree.java:3466) at org.eclipse.swt.widgets.Tree.getSelection(Tree.java:3462) at org.eclipse.swt.widgets.Tree.getSelection(Tree.java:3462) at org.eclipse.swt.widgets.Tree.getSelection(Tree.java:3512) at org.eclipse.jface.viewers.TreeViewer.getSelection(TreeViewer.java:231) at org.eclipse.jface.viewers.AbstractTreeViewer.getSelection(AbstractTreeViewer.java:2986) at org.eclipse.jface.viewers.TreeViewer.remove(TreeViewer.java:817) at org.eclipse.debug.internal.ui.viewers.model.TreeModelContentProvider.handleRemove(TreeModelContentProvider.java:1560) at org.eclipse.debug.internal.ui.viewers.model.TreeModelContentProvider.updateNodes(TreeModelContentProvider.java:1279) at org.eclipse.debug.internal.ui.viewers.model.TreeModelContentProvider.updateNodes(TreeModelContentProvider.java:1312) at org.eclipse.debug.internal.ui.viewers.model.TreeModelContentProvider.updateNodes(TreeModelContentProvider.java:1312) at org.eclipse.debug.internal.ui.viewers.model.TreeModelContentProvider.updateNodes(TreeModelContentProvider.java:1312) at org.eclipse.debug.internal.ui.viewers.model.TreeModelContentProvider.updateModel(TreeModelContentProvider.java:575) at org.eclipse.debug.internal.ui.viewers.model.TreeModelContentProvider.doModelChanged(TreeModelContentProvider.java:543) at org.eclipse.debug.internal.ui.viewers.model.TreeModelContentProvider$DelayedDoModelChangedJob.runInUIThread(TreeModelContentProvider.java:449) at org.eclipse.ui.progress.UIJob.lambda$0(UIJob.java:95) at org.eclipse.ui.progress.UIJob$$Lambda$1040/0x0000000801611cf8.run(Unknown Source) at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:40) at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:185) - locked <0x00001007dd299ce8> (a org.eclipse.swt.widgets.RunnableLock) at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:4035) at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3635) at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$5.run(PartRenderingEngine.java:1154) at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:338) at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1045) at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:155) at org.eclipse.ui.internal.Workbench.lambda$3(Workbench.java:644) at org.eclipse.ui.internal.Workbench$$Lambda$294/0x0000000800e07328.run(Unknown Source) at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:338) at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:551) at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:156) at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:152) at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:203) at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:136) at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104) at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:401) at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@17/Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@17/NativeMethodAccessorImpl.java:77) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17/DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(java.base@17/Method.java:568) at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:659) at org.eclipse.equinox.launcher.Main.basicRun(Main.java:596) at org.eclipse.equinox.launcher.Main.run(Main.java:1467)
(In reply to Johan Compagner from comment #15) > hmm does really nobody have any problems with the Debug view as i have? > > Because i did think it was the OpenJ9 variant but now i again have the same > thing that the debug view eats up a huge amount of resources and debugging > is almost not doable because of that > > You see that it tries to constantly repaint the something in that tree, > because the scrollbar is constantly moving a bit (getting smaller and bigger) > > and now it is just: > > java.version=17 > java.version.date=2021-09-14 > java.vm.compressedOopsMode=32-bit > java.vm.info=mixed mode, sharing > java.vm.name=OpenJDK 64-Bit Server VM > java.vm.specification.name=Java Virtual Machine Specification > java.vm.specification.vendor=Oracle Corporation > java.vm.specification.version=17 > java.vm.vendor=Eclipse Adoptium > java.vm.version=17+35 > > the "standard" hotspot vm.. > > the main thread is constantly doing this: > > "main" #1 prio=6 os_prio=0 cpu=1019562.50ms elapsed=89031.99s > tid=0x00000231db772330 nid=0x2d6c runnable [0x000000aa0ef2d000] > java.lang.Thread.State: RUNNABLE > at org.eclipse.swt.internal.win32.OS.SendMessage(Native Method) > at org.eclipse.swt.widgets.Tree.getSelection(Tree.java:3466) > at org.eclipse.swt.widgets.Tree.getSelection(Tree.java:3462) > at org.eclipse.swt.widgets.Tree.getSelection(Tree.java:3462) > at org.eclipse.swt.widgets.Tree.getSelection(Tree.java:3512) > at > org.eclipse.jface.viewers.TreeViewer.getSelection(TreeViewer.java:231) > at > org.eclipse.jface.viewers.AbstractTreeViewer.getSelection(AbstractTreeViewer. > java:2986) > at org.eclipse.jface.viewers.TreeViewer.remove(TreeViewer.java:817) > at > org.eclipse.debug.internal.ui.viewers.model.TreeModelContentProvider. > handleRemove(TreeModelContentProvider.java:1560) > at > org.eclipse.debug.internal.ui.viewers.model.TreeModelContentProvider. > updateNodes(TreeModelContentProvider.java:1279) > at > org.eclipse.debug.internal.ui.viewers.model.TreeModelContentProvider. > updateNodes(TreeModelContentProvider.java:1312) > at > org.eclipse.debug.internal.ui.viewers.model.TreeModelContentProvider. > updateNodes(TreeModelContentProvider.java:1312) > at > org.eclipse.debug.internal.ui.viewers.model.TreeModelContentProvider. > updateNodes(TreeModelContentProvider.java:1312) > at > org.eclipse.debug.internal.ui.viewers.model.TreeModelContentProvider. > updateModel(TreeModelContentProvider.java:575) > at > org.eclipse.debug.internal.ui.viewers.model.TreeModelContentProvider. > doModelChanged(TreeModelContentProvider.java:543) > at > org.eclipse.debug.internal.ui.viewers.model. > TreeModelContentProvider$DelayedDoModelChangedJob. > runInUIThread(TreeModelContentProvider.java:449) > at org.eclipse.ui.progress.UIJob.lambda$0(UIJob.java:95) > at > org.eclipse.ui.progress.UIJob$$Lambda$1040/0x0000000801611cf8.run(Unknown > Source) > at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:40) > at > org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:185) > - locked <0x00001007dd299ce8> (a > org.eclipse.swt.widgets.RunnableLock) > at > org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:4035) > at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3635) > at > org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$5. > run(PartRenderingEngine.java:1154) > at > org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:338) > at > org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine. > run(PartRenderingEngine.java:1045) > at > org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench. > java:155) > at org.eclipse.ui.internal.Workbench.lambda$3(Workbench.java:644) > at > org.eclipse.ui.internal.Workbench$$Lambda$294/0x0000000800e07328.run(Unknown > Source) > at > org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:338) > at > org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:551) > at > org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:156) > at > org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication. > java:152) > at > org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java: > 203) > at > org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher. > runApplication(EclipseAppLauncher.java:136) > at > org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher. > start(EclipseAppLauncher.java:104) > at > org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:401) > at > org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255) > at > jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@17/Native > Method) > at > jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@17/ > NativeMethodAccessorImpl.java:77) > at > jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17/ > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(java.base@17/Method.java:568) > at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:659) > at org.eclipse.equinox.launcher.Main.basicRun(Main.java:596) > at org.eclipse.equinox.launcher.Main.run(Main.java:1467) Can you come up with a reproducible scenario?
not really, except my own eclipse and then it can be that it worked nicely for a day (like yesterday) but today it suddenly popuped up again i am running my own dev environment again through another workspace run so i can debug myself and again the same as the image above (the huge queue) i have again the same thing with a queue of DelayedDoModel stuff of 2257 ... i don't get why i am so special, i have this over multiply workspaces (the same eclipse) with our product over different branches. All the same
i think i finally found at least one cause (not sure if it is "the cause") We use chromium browser (native plugin) and use that as an editor now it seems if our code has some kind of small bug where the angular app that we run in that does some looping. Then suddenly the java part creates a lot (and i really mean a lot) of GC Threads that are also quickly terminated.. why this happens i dont know but it would be nice if eclipse is able to handles this and just filters or sees these things that are very short lived and ignores them very quickly also
Created attachment 288684 [details] a capture of the debug view for the terminated gc threads
