Community
Participate
Working Groups
Build Identifier: 3.6.2 M20110210-1200 Having the Problems, or Markers, view update itself during a build is extra processing that gives no value to the user. The update should wait until the build is done for 2 reasons: 1)the user only cares about the final number of errors/warnings, and 2)this processing can dramatically slow down build times. There is no point in trying to update the views during the build when the build will constantly be finding new markers to process until it is done. I have seen build times improve by 40% in when I close the Problems view in the Java perspective. I have created a workspace specifically to emphasize this problem. The workspace has 11000 java files in it, and contains ~760000 markers. While this may be an unrealistic number, this is only meant to easily show the problem. I ran two build tests with this workspace: 1)with the Problems view open, 2) with the Problems view closed. On eclipse 3.6.2, this workspace builds in 1)3 min 50 s 2)2 min 40 s ~40% improvement from test 1 --> test 2. As I watch the build run, I can see the Problems view trying to update itself. It updates first with 80000, then 160000, and so on. But as a user, I feel that these updates are of no value to me, especially considering the amount of time it is taking to do update the view. I have also seen these improvements in build time in our adopter product of eclipse 3.6.2 with real customer workspaces. Sometimes build/validation can cause a number of "false" markers to be produced when a clean build gets kicked off. If we postpone the processing of the markers until the end of the build, we will not only be improving build times but also be eliminating the time it takes to process these "false" markers. When I say "false" markers, I mean errors/warnings that are produced before they have been built/validated. Once the build/validation runs, they are no longer errors or warnings. There is no need to process these markers since they are not actually markers when the build is finished. Reproducible: Always Steps to Reproduce: To easily reproduce this: 1. Increase java heap size to accommodate the large workspace 2. Import my workspace in the Java perspective 3. Record the time for a clean build with the problems view open 4. Close the problems view, and record the time for another clean build
You can find the workspace here: ftp://perfdata:perfdata@wsperf3.torolab.ibm.com/mmartire/WorkspaceWithLotsOfMarkers.zip
(In reply to comment #0) > Having the Problems, or Markers, view update itself during a build is extra > processing that gives no value to the user. This is absolutely incorrect statement, I cannot disagree more. That might be of little use for your particular flow but when build takes long time user can start working on the first batch of problems appearing in Problems View instead of sitting idle around. For example it is not uncommon for big projects in CDT that build could take 20 min. or so and it is not really acceptable to lose that time. It was really a big deal when eclipse allowed to work with error/warning markers while build has not finished yet. Note that for C/C++ there is a flag not to stop on first build error and it is legitimate workflow.
(In reply to comment #2) > (In reply to comment #0) > > Having the Problems, or Markers, view update itself during a build is extra > > processing that gives no value to the user. > This is absolutely incorrect statement, I cannot disagree more. I agree - be forced to wait until all is built is not a good option. I think the suggestion in comment 0 was made because of the performance issue and not because having to wait until the end is seen as a preferred way of working. We should investigate where the CPU is burnt and improve this.
Created attachment 198340 [details] Workspace with a lot of markers
You are right, reducing the build time is what were are after here. The impact of processing all these markers during the build is the root problem. As can be shown with my workspace, this can have an affect of up to 40% in certain cases. I accept the fact that there are cases where this processing helps the user, but its not every case and thus it should not be considered to be always beneficial. We should consider adding an option to allow the user to turn off this incremental update so that the view updates at the end of the build only. Also, note that in some cases, for example, a clean builds of a java workspace can produce a number of transient errors/warnings that will eventually get cleared up during the build and the cost of the processing of these large sets of extra or "false" markers is high. This may be another issue worth its own bugzilla, but I'm going to state this here: Another case where users are affected by this processing are users who have the Problems view open but not visible, or users who have the Problems view as a fast view but not visible. In these cases, the Problems view still is updating itself during the build, but the user does not even see this updating. While it may be beneficial in the sense that once the user clicks on the view to open it, it will most likely be populated already, or soon after. But this does not coincide with eclipse logic of deferring all work until necessary. I think the view should not be updating itself unless visible to the user. If it is not visible then the update should wait for the view to become visible, then update. If it would help, I can provide you with a yourkit snapshot to further analyze the problem. Note: I have also attached our workspace to this bug. Disregard comment 1.
The default setting for problems view is to show only 100 markers group (View menu > Configure contents). What was this setting when you saw the 40% improvement as mentioned in comment 0 ? Does changing this setting affect performance ?
(In reply to comment #6) > The default setting for problems view is to show only 100 markers group (View > menu > Configure contents). What was this setting when you saw the 40% > improvement as mentioned in comment 0 ? Does changing this setting affect > performance ? I didnt change any preferences when running the test, so whatever the default is thats what I used. I dont think the actual displaying of the markers in the view is the problem, its the sorting/processing of the markers during the build that is causing the slowdown. I can run a test to see what happens if I increase/decrease that number.
(In reply to comment #7) > I didnt change any preferences when running the test, so whatever the default > is thats what I used. I dont think the actual displaying of the markers in the > view is the problem, its the sorting/processing of the markers during the build > that is causing the slowdown. I can run a test to see what happens if I > increase/decrease that number. Yeah, sorting/processing of markers could be the reason. Still it would be useful to know if displaying more markers changes anything.
I ran some more tests with my workspace playing with the marker limits in the Problems view preferences. The build times are as follows: problems view closed: 2.7 min problems view open - marker limit 100 (default): 3.9 min problems view open - marker limit 1000 (default): 4.5 min problems view open - marker limit 10000 (default): 4.3 min problems view open - marker limit 100000 (default): 3.8 min problems view open - no marker limit (default): 4 min It seems that there is some variance in the sorting/processing of the markers and the way it gets displayed. There may be some race conditions in the sense that in some cases the build has to wait for the processing of markers, but in other cases it passes that by. However, overall it doesnt seem that changing this preference would benefit the user in any way (performance wise).
(In reply to comment #9) > problems view open - marker limit 1000 (default): 4.5 min > problems view open - marker limit 10000 (default): 4.3 min > problems view open - marker limit 100000 (default): 3.8 min > problems view open - no marker limit (default): 4 min Note, those should not say (default), I forgot to remove that statement for those rows. The default limit is 100.
It is easy to let Problems view to cause *permanent* UI freezes, at least on Linux. * Workspace must have > 20000 warnings/errors (e.g. projects from Platform UI :-)) * Disable item limits for Problems view via "Configure Contents->[] Use item limits" * Make sure "Show->Show All" is selected, so that all markers are shown * Select "Group By -> Severity" * Expand "Warnings" node (which most likely will contain biggest part of the markers) * Switch to any view in the same stack as the Problems view to hide it * Switch to the Problems view back => UI freeze (~5 sec) * Write String s = null; in Java editor and save => UI freeze (warning added) (~5 sec) * Delete the line avove and save => UI freeze (warning removed) (~5 sec) * Select all markers => UI freeze (~5 sec) * With all selected markers, right click => UI freeze ("forever", may be 2-3 minutes, I just gave up) * With all selected markers (group must be not selected), hit "delete" key => UI freeze for ~1 minute. I'm trying to profile this now. The problem is most likely SWT/JFace (AbstractTreeViewer) and additionally the way how Problems view refreshes itself (UIUpdateJob *always* refreshes the viewer starting with root element). So even if a single marker is added or deleted, Problems view will do the refresh of *all* elements, practically re-setting each and every element via SWT. This works fine for smaller number of items but with items count over 10000 this is an overkill. Sure, removing all markers in the way how it is done here in bug description is overkill ^ 2. Stack Trace in most cases shows this: at org.eclipse.swt.internal.gtk.OS._gtk_tree_store_set(OS.java:-2) at org.eclipse.swt.internal.gtk.OS.gtk_tree_store_set(OS.java:13131) at org.eclipse.swt.widgets.TreeItem.setImage(TreeItem.java:1433) at org.eclipse.jface.viewers.TreeViewerRow.setImage(TreeViewerRow.java:110) at org.eclipse.jface.viewers.ViewerCell.setImage(ViewerCell.java:169) at org.eclipse.ui.internal.views.markers.MarkerProblemSeverityAndMessageField.update(MarkerProblemSeverityAndMessageField.java:88) at org.eclipse.ui.internal.views.markers.MarkerColumnLabelProvider.update(MarkerColumnLabelProvider.java:64) at org.eclipse.jface.viewers.ViewerColumn.refresh(ViewerColumn.java:154) at org.eclipse.jface.viewers.AbstractTreeViewer.doUpdateItem(AbstractTreeViewer.java:949) at org.eclipse.ui.internal.views.markers.MarkersTreeViewer.doUpdateItem(MarkersTreeViewer.java:74) at org.eclipse.jface.viewers.AbstractTreeViewer$UpdateItemSafeRunnable.run(AbstractTreeViewer.java:114) at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42) at org.eclipse.ui.internal.JFaceUtil$1.run(JFaceUtil.java:50) at org.eclipse.jface.util.SafeRunnable.run(SafeRunnable.java:173) at org.eclipse.jface.viewers.AbstractTreeViewer.doUpdateItem(AbstractTreeViewer.java:1029) at org.eclipse.jface.viewers.StructuredViewer$UpdateItemSafeRunnable.run(StructuredViewer.java:473) at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42) at org.eclipse.ui.internal.JFaceUtil$1.run(JFaceUtil.java:50) at org.eclipse.jface.util.SafeRunnable.run(SafeRunnable.java:173) at org.eclipse.jface.viewers.StructuredViewer.updateItem(StructuredViewer.java:2176) at org.eclipse.jface.viewers.AbstractTreeViewer.updateChildren(AbstractTreeViewer.java:2755) at org.eclipse.jface.viewers.AbstractTreeViewer.internalRefreshStruct(AbstractTreeViewer.java:1916) at org.eclipse.jface.viewers.TreeViewer.internalRefreshStruct(TreeViewer.java:684) at org.eclipse.jface.viewers.AbstractTreeViewer.internalRefreshStruct(AbstractTreeViewer.java:1923) at org.eclipse.jface.viewers.TreeViewer.internalRefreshStruct(TreeViewer.java:684) at org.eclipse.jface.viewers.AbstractTreeViewer.internalRefresh(AbstractTreeViewer.java:1891) at org.eclipse.jface.viewers.AbstractTreeViewer.internalRefresh(AbstractTreeViewer.java:1848) at org.eclipse.jface.viewers.StructuredViewer$8.run(StructuredViewer.java:1554) at org.eclipse.jface.viewers.StructuredViewer.preservingSelection(StructuredViewer.java:1462) at org.eclipse.jface.viewers.TreeViewer.preservingSelection(TreeViewer.java:366) at org.eclipse.jface.viewers.StructuredViewer.preservingSelection(StructuredViewer.java:1423) at org.eclipse.jface.viewers.StructuredViewer.refresh(StructuredViewer.java:1551) at org.eclipse.jface.viewers.ColumnViewer.refresh(ColumnViewer.java:534) at org.eclipse.jface.viewers.StructuredViewer.refresh(StructuredViewer.java:1507) at org.eclipse.ui.internal.views.markers.UIUpdateJob.runInUIThread(UIUpdateJob.java:112) at org.eclipse.ui.progress.UIJob$1.run(UIJob.java:97) at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:35) at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:136) at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:3793) at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3431) at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run(PartRenderingEngine.java:1151) at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:337) at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1032) at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:156) at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:648) at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:337) at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:592) at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:150) at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:138) at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196) at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134) at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104) at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:380) at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:235) at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:648) at org.eclipse.equinox.launcher.Main.basicRun(Main.java:603) at org.eclipse.equinox.launcher.Main.run(Main.java:1465) at org.eclipse.equinox.launcher.Main.main(Main.java:1438)
Found a really old bug 112922 which actually says: why do we refresh the entire thing if we only want to add or to remove a single marker to it?
*** Bug 207199 has been marked as a duplicate of this bug. ***
(In reply to Andrey Loskutov from comment #12) > Found a really old bug 112922 which actually says: why do we refresh the > entire thing if we only want to add or to remove a single marker to it? Maybe related to Bug 451084?
*** Bug 345608 has been marked as a duplicate of this bug. ***
I can't commit time to this issue for 4.7.
*** Bug 383733 has been marked as a duplicate of this bug. ***
Fresh stack from GTK 3.20 / Eclipse 4.7.3, where Eclipse was frozen forever with this stack (workspace had 200.000 markers and Problems view was set to not have any limit): "main" #1 prio=6 os_prio=0 tid=0x00007ffff0016800 nid=0xfbfc runnable [0x00007ffff59f8000] java.lang.Thread.State: RUNNABLE at org.eclipse.swt.internal.gtk.OS._gtk_tree_store_set(Native Method) at org.eclipse.swt.internal.gtk.OS.gtk_tree_store_set(OS.java:12850) at org.eclipse.swt.widgets.TreeItem.setImage(TreeItem.java:1578) at org.eclipse.jface.viewers.TreeViewerRow.setImage(TreeViewerRow.java:110) at org.eclipse.jface.viewers.ViewerCell.setImage(ViewerCell.java:169) at org.eclipse.ui.internal.views.markers.MarkerProblemSeverityAndMessageField.update(MarkerProblemSeverityAndMessageField.java:77) at org.eclipse.ui.internal.views.markers.MarkerColumnLabelProvider.update(MarkerColumnLabelProvider.java:54) at org.eclipse.jface.viewers.ViewerColumn.refresh(ViewerColumn.java:141) at org.eclipse.jface.viewers.AbstractTreeViewer.doUpdateItem(AbstractTreeViewer.java:946) at org.eclipse.ui.internal.views.markers.MarkersTreeViewer.doUpdateItem(MarkersTreeViewer.java:67) at org.eclipse.jface.viewers.AbstractTreeViewer$UpdateItemSafeRunnable.run(AbstractTreeViewer.java:117) at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42) at org.eclipse.ui.internal.JFaceUtil.lambda$0(JFaceUtil.java:44) at org.eclipse.ui.internal.JFaceUtil$$Lambda$8/486619248.run(Unknown Source) at org.eclipse.jface.util.SafeRunnable.run(SafeRunnable.java:173) at org.eclipse.jface.viewers.AbstractTreeViewer.doUpdateItem(AbstractTreeViewer.java:1025) at org.eclipse.jface.viewers.StructuredViewer$UpdateItemSafeRunnable.run(StructuredViewer.java:475) at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42) at org.eclipse.ui.internal.JFaceUtil.lambda$0(JFaceUtil.java:44) at org.eclipse.ui.internal.JFaceUtil$$Lambda$8/486619248.run(Unknown Source) at org.eclipse.jface.util.SafeRunnable.run(SafeRunnable.java:173) at org.eclipse.jface.viewers.StructuredViewer.updateItem(StructuredViewer.java:2159) at org.eclipse.jface.viewers.AbstractTreeViewer.createTreeItem(AbstractTreeViewer.java:840) at org.eclipse.jface.viewers.AbstractTreeViewer.updateChildren(AbstractTreeViewer.java:2735) at org.eclipse.jface.viewers.AbstractTreeViewer.internalRefreshStruct(AbstractTreeViewer.java:1894) at org.eclipse.jface.viewers.TreeViewer.internalRefreshStruct(TreeViewer.java:668) at org.eclipse.jface.viewers.AbstractTreeViewer.internalRefreshStruct(AbstractTreeViewer.java:1900) at org.eclipse.jface.viewers.TreeViewer.internalRefreshStruct(TreeViewer.java:668) at org.eclipse.jface.viewers.AbstractTreeViewer.internalRefresh(AbstractTreeViewer.java:1870) at org.eclipse.jface.viewers.AbstractTreeViewer.internalRefresh(AbstractTreeViewer.java:1827) at org.eclipse.jface.viewers.StructuredViewer.lambda$3(StructuredViewer.java:1531) at org.eclipse.jface.viewers.StructuredViewer$$Lambda$285/1157246451.run(Unknown Source) at org.eclipse.jface.viewers.StructuredViewer.preservingSelection(StructuredViewer.java:1447) at org.eclipse.jface.viewers.TreeViewer.preservingSelection(TreeViewer.java:354) at org.eclipse.jface.viewers.StructuredViewer.preservingSelection(StructuredViewer.java:1408) at org.eclipse.jface.viewers.StructuredViewer.refresh(StructuredViewer.java:1531) at org.eclipse.jface.viewers.ColumnViewer.refresh(ColumnViewer.java:535) at org.eclipse.jface.viewers.StructuredViewer.refresh(StructuredViewer.java:1492) at org.eclipse.ui.internal.views.markers.UIUpdateJob.runInUIThread(UIUpdateJob.java:105) at org.eclipse.ui.progress.UIJob.lambda$0(UIJob.java:95) at org.eclipse.ui.progress.UIJob$$Lambda$85/1474895109.run(Unknown Source) at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:37) at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:182)
See also bug 519164.
*** Bug 546206 has been marked as a duplicate of this bug. ***
*** Bug 558340 has been marked as a duplicate of this bug. ***
