Community
Participate
Working Groups
I am working with a clone of CDT repository (counted ~6000 files in workspace CDT subset I am dealing with) and decorators are lagging well behind every action I do. That refers to especially "current branch" decorator in Git Repositories view, and also to "dirty" decorators in Project Explored. One can hardly be sure which branch is checked out and which files haven't been committed. For example, checking out a different local branch: 1. Check out itself ~5 sec. 2. Refreshing Git Repository task ~5 sec. 3. Updating current branch decorator ~40 sec. Yes, *forty seconds*. It wrongly shows the old branch as current during all that time. I am busy engaging in doubts about the state of the repository refraining from further actions. Interesting that the context menu seems to be aware of the correct status. Can at least old decorator be removed or temporary "in progress" decorator be added to the new branch? Similar problems with Project Explorer decorators except I gave up relying on them long time ago anyway due to dirty decorators bugs. Windows 7 if it makes a difference.
Andrew: 1) Could you add the exact build id, since this may be important 2) Could you describe the layout of your projects? A screenshot of the project explorer should suffice. The information above may help finding *your* case faster. We sometimes blame Windows, but I'm very sure Redmond is not at fault. However, I think most of this this relates to https://bugs.eclipse.org/331273 and http://egit.eclipse.org/r/#change,1927
Created attachment 185816 [details] PE screenshot (In reply to comment #1) > 1) Could you add the exact build id, since this may be important The build id is egit 0.11.0.201012231154. > 2) Could you describe the layout of your projects? A screenshot of the project > explorer should suffice. Screenshot attached. My project is a clone of CDT and hosted on github https://github.com/angvoz/SD80. If you are really interested you could try to reproduce from that. I checkout a subset of the projects which you could see on the screenshot. > The information above may help finding *your* case faster. We sometimes blame > Windows, but I'm very sure Redmond is not at fault. > However, I think most of this this relates to https://bugs.eclipse.org/331273 > and http://egit.eclipse.org/r/#change,1927 I think you are probably right. My commits are somewhat slow but usable and Synchronize mode is extremely slow and unusable despite having i5 processor and SSD disk on my laptop which is fast in general and with smaller egit projects.
(In reply to comment #1) > However, I think most of this this relates to https://bugs.eclipse.org/331273 > and http://egit.eclipse.org/r/#change,1927 Actually, not exactly. You refer to bug 331273 having problem with opens and reads of almost all files. I am talking about the decorator marking the current branch. Why would you scan all files just to find out what the current branch is to update the decorator? Perhaps there is a room for improvement right there.
This is possibly related to 330917: ContainerTreeIterator.isEntryIgnored performs a lot of unnecessary work https://bugs.eclipse.org/bugs/show_bug.cgi?id=330917 which contains a detailed report regarding the slowness of Git's decorators.
Hi, just want to add that I'm seeing something similar on Mac OS X. I checked out the sd80 branch from github and I noticed the cpu going crazy for a couple of mins after the import of around 30 projects. I did a jstack and it was the decorators.
Found workaround for the current branch decorator problem. Refresh button in Repositories view updates the decorator immediately.
*** Bug 328918 has been marked as a duplicate of this bug. ***
Experiencing the same issue. Using java profiler I found that my eclipse spent about 80% of its processing time in the method org.eclipse.egit.ui.internal.decorators.GitLightweightDecorator.processDecoration when switching between branches, updating and so on. I am using egit on a very large project and because of this my eclipse is using one core of my processor at all time. Can this bug be quick-fixed by adding to the settings some possibility to turn off the decorations for the time better solution will be available. I dont really mind to use egit without decorations, if it will reduce the processor demands. Using egit 0.12.0.201102282024 on eclipse 3.7.0 Build id: I20110127-2034
A lot of improvements for decorator performance have been done since this bug was filed. See bug 325393. It should be magnitudes faster now ... but still I see that martin has problems with a quite recent build. To quick-fix your current problem it is easy to turn of git decorations. Just go to Window-Preferences-General-Appearance-Label decorations and deselect "Git". Is your project public - means: can I clone it from somewhere to see the current problem.
Sorry... project is not public and I cannot get you access to our sources. Project I am working on is very large project (repository takes about 3GB of my disk space) with lots of small text files and some large binary files. Eclipse is much more smoother after turning the decorations off and is now using the same amount of processor time like before the egit installation. Thanks a lot.
See also bug 346079 (in 0.12). My eclipse UI was locking up and a series of jstacks pulled up 14 concurrent egit decorator jobs each of which was doing I/O.
Created attachment 195909 [details] 19 concurrent git decrotor jobs running
This is a major problem for me. The CVS decorators are updated immediately, but now with Git, I have to wait for more than a minute even for a simple dirty indicator on a single file.
Presumably egit doesn't use a core.resources resource change listener and instead periodically does all this I/O...
(In reply to comment #14) > Presumably egit doesn't use a core.resources resource change listener and > instead periodically does all this I/O... That's well possible. When I enable system jobs in the Progress view, I see periods where nothing is running, and then a Git decorator job shows up from time to time. But it needs multiple such jobs until the decorators are in sync again. Polling is clearly a no-go for an Eclipse plug-in. core.resources should be the only plug-in that synchronizes with the file system.
They are also blocking other decorations to be updated in a timely manner like e.g. errors and warnings. After starting my workspace which has 11 big repositories it takes 5 minutes until I see the decorators. "Worker-6" prio=6 tid=0x3e7fd800 nid=0x12f8 runnable [0x4292f000] java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:120) at org.eclipse.jgit.treewalk.FileTreeIterator$FileEntry.openInputStream(FileTreeIterator.java:198) at org.eclipse.jgit.treewalk.WorkingTreeIterator.idBufferBlob(WorkingTreeIterator.java:257) at org.eclipse.jgit.treewalk.WorkingTreeIterator.idBuffer(WorkingTreeIterator.java:235) at org.eclipse.jgit.treewalk.AbstractTreeIterator.getEntryObjectId(AbstractTreeIterator.java:409) at org.eclipse.jgit.treewalk.TreeWalk.getObjectId(TreeWalk.java:675) at org.eclipse.egit.core.synchronize.ThreeWayDiffEntry.scan(ThreeWayDiffEntry.java:93) at org.eclipse.egit.core.synchronize.GitSyncCache.loadDataFromGit(GitSyncCache.java:87) at org.eclipse.egit.core.synchronize.GitSyncCache.getAllData(GitSyncCache.java:50) at org.eclipse.egit.core.synchronize.GitResourceVariantTreeSubscriber.init(GitResourceVariantTreeSubscriber.java:81) at org.eclipse.egit.core.internal.GitRepositoryProviderType.getSubscriber(GitRepositoryProviderType.java:48) at org.eclipse.team.internal.ui.TeamUIPlugin.getDecoratedStateProvider(TeamUIPlugin.java:444) - locked <0x18adfc78> (a org.eclipse.team.internal.ui.TeamUIPlugin) at org.eclipse.team.internal.ui.TeamAdapterFactory.getAdapter(TeamAdapterFactory.java:56) at org.eclipse.core.internal.adapter.AdapterFactoryProxy.getAdapter(AdapterFactoryProxy.java:80) at org.eclipse.core.internal.runtime.AdapterManager.getAdapter(AdapterManager.java:295) at org.eclipse.core.runtime.PlatformObject.getAdapter(PlatformObject.java:66) at org.eclipse.team.internal.ui.Utils.getAdapter(Utils.java:676) at org.eclipse.team.internal.ui.mapping.WorkspaceTeamStateProvider.getDecoratedStateProviderForId(WorkspaceTeamStateProvider.java:130) at org.eclipse.team.internal.ui.mapping.WorkspaceTeamStateProvider.listenerForStateChangesForId(WorkspaceTeamStateProvider.java:185) at org.eclipse.team.internal.ui.mapping.WorkspaceTeamStateProvider.handleProject(WorkspaceTeamStateProvider.java:175) at org.eclipse.team.internal.ui.mapping.WorkspaceTeamStateProvider.<init>(WorkspaceTeamStateProvider.java:48) at org.eclipse.team.internal.ui.TeamUIPlugin.getDecoratedStateProvider(TeamUIPlugin.java:461) - locked <0x18adfc78> (a org.eclipse.team.internal.ui.TeamUIPlugin) at org.eclipse.team.ui.mapping.SynchronizationStateTester.getTeamStateProvider(SynchronizationStateTester.java:167) at org.eclipse.team.ui.mapping.SynchronizationStateTester.getState(SynchronizationStateTester.java:154) at org.eclipse.team.internal.ccvs.ui.CVSLightweightDecorator.decorate(CVSLightweightDecorator.java:212) at org.eclipse.team.internal.ccvs.ui.CVSLightweightDecorator.decorate(CVSLightweightDecorator.java:161) at org.eclipse.ui.internal.decorators.LightweightDecoratorDefinition.decorate(LightweightDecoratorDefinition.java:263) at org.eclipse.ui.internal.decorators.LightweightDecoratorManager$LightweightRunnable.run(LightweightDecoratorManager.java:81) at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42) at org.eclipse.ui.internal.decorators.LightweightDecoratorManager.decorate(LightweightDecoratorManager.java:365) at org.eclipse.ui.internal.decorators.LightweightDecoratorManager.getDecorations(LightweightDecoratorManager.java:347) at org.eclipse.ui.internal.decorators.DecorationScheduler$1.ensureResultCached(DecorationScheduler.java:370) at org.eclipse.ui.internal.decorators.DecorationScheduler$1.run(DecorationScheduler.java:330) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)
Could you try again with the latest nightly build ? The following changes were submitted which help to improve decorator performance http://egit.eclipse.org/r/#change,4179 http://egit.eclipse.org/r/#change,4200 http://egit.eclipse.org/r/#change,4218 http://egit.eclipse.org/r/#change,4294 http://egit.eclipse.org/r/#change,4295 We have reimplemented the git decorator and it now also listens on resource change events to incrementally update its state, this improved performance heavily.
(In reply to comment #17) > We have reimplemented the git decorator and it now also listens on resource When you say "now also", is it still do it's periodic stat() of all files? Can this be disabled independently of the resource change hook? On NFS this really hurts. For Eclipse workspace resources, the only plugin which should periodically stat is core.resources. Generally Team Providers can throw away the sync state cache if the user requests a full-synchronize, but otherwise the sync state should be kept up to date without performing loads of I/O.
(In reply to comment #17) > Could you try again with the latest nightly build ? With 1.2.0.201109290051, small updates indeed feel snappier. But the initial decoration after starting up my workspace still takes several minutes. During this time, I also see whole projects (all files and folders) as untracked (as mentioned in bug 359264 comment 1). The wrong intermediate state is highly confusing.
(In reply to comment #18) > (In reply to comment #17) > > We have reimplemented the git decorator and it now also listens on resource > > When you say "now also", is it still do it's periodic stat() of all files? Can > this be disabled independently of the resource change hook? On NFS this really > hurts. The periodic refresh can be disabled in Team>Git>Refresh resourcrs when index change. That name is a minsomer since we listen for refs changes too and we read too many of those to get useful information for UI. Besides that, an index change should be handled differently depending on whether we changed it or someone else did (e.g. git command line). If *we* changed it, then the decorators do not need to be informed, but if the change is external we could possibly try to figure out what changed and only refresh those resources.
(In reply to comment #20) > if the change > is external we could possibly try to figure out what changed and only refresh > those resources. Exactly. The IResourceChangeListener gives you all this information for free. The information from the resource deltas should be enough to update the decorators. However, this only works for resources that are actually present in the workspace. For other resources (e.g. projects that have not been imported), you might still need to access the file system for certain operations that have to work on the whole Git repository.
(In reply to comment #21) > (In reply to comment #20) > > if the change > > is external we could possibly try to figure out what changed and only refresh > > those resources. > > Exactly. The IResourceChangeListener gives you all this information for free. > The information from the resource deltas should be enough to update the > decorators. If the resource changed outside Eclipse, then Eclipse does not know, so there is no event, which is the reason we do refresh when the index changes. Automatic workspace refresh consumes even more resources and yields very slow refresh. We refresh only those resources that are managed by the repository where the change occurred, as opposed to the whole workspace.
(In reply to comment #22) > We refresh only those resources that are managed by the repository > where the change occurred, as opposed to the whole workspace. Well that's good. So presumably on noticing index changes, you can refreshLocal the changed resources, ensuring the rest of the WS's listeners are notified (as well as egit itself).
(In reply to comment #19) > (In reply to comment #17) > > Could you try again with the latest nightly build ? > > With 1.2.0.201109290051, small updates indeed feel snappier. > > But the initial decoration after starting up my workspace still takes several > minutes. During this time, I also see whole projects (all files and folders) as > untracked (as mentioned in bug 359264 comment 1). The wrong intermediate state > is highly confusing. Can you provide a trace? Please create a .options file in the Eclipse directory with the following content: org.eclipse.egit.core/debug=true org.eclipse.egit.core/debug/core/indexdiffcache=true Then start Eclipse from the command line with the -debug option, e.g. eclipse -debug /Users/xxxx/Eclipse/Helios/eclipse/.options The trace will be written to the trace.log file in the .metadata folder of your workspace.
> Can you provide a trace? I'll try the tracing options later. Here's the stack trace taken after 5 minutes where some of my decorators are still not available: "Worker-16" prio=6 tid=0x3f4f7800 nid=0x1a40 runnable [0x42e9e000] java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:138) at org.eclipse.core.internal.filesystem.local.LocalFile.openInputStream(LocalFile.java:362) at org.eclipse.core.internal.localstore.FileSystemResourceManager.read(FileSystemResourceManager.java:797) at org.eclipse.core.internal.resources.File.getContents(File.java:289) at org.eclipse.egit.core.ContainerTreeIterator$ResourceEntry.openInputStream(ContainerTreeIterator.java:246) at org.eclipse.jgit.treewalk.WorkingTreeIterator.idBufferBlob(WorkingTreeIterator.java:257) at org.eclipse.jgit.treewalk.WorkingTreeIterator.idBuffer(WorkingTreeIterator.java:235) at org.eclipse.jgit.treewalk.AbstractTreeIterator.getEntryObjectId(AbstractTreeIterator.java:399) at org.eclipse.jgit.treewalk.WorkingTreeIterator.contentCheck(WorkingTreeIterator.java:699) at org.eclipse.jgit.treewalk.WorkingTreeIterator.isModified(WorkingTreeIterator.java:671) at org.eclipse.jgit.treewalk.filter.IndexDiffFilter.include(IndexDiffFilter.java:176) at org.eclipse.jgit.treewalk.filter.AndTreeFilter$List.include(AndTreeFilter.java:162) at org.eclipse.jgit.treewalk.TreeWalk.next(TreeWalk.java:560) at org.eclipse.jgit.lib.IndexDiff.diff(IndexDiff.java:284) at org.eclipse.egit.core.internal.indexdiff.IndexDiffCacheEntry.calcIndexDiff(IndexDiffCacheEntry.java:265) at org.eclipse.egit.core.internal.indexdiff.IndexDiffCacheEntry.access$2(IndexDiffCacheEntry.java:256) at org.eclipse.egit.core.internal.indexdiff.IndexDiffCacheEntry$3.run(IndexDiffCacheEntry.java:151) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54) "Worker-15" prio=6 tid=0x3f4f4000 nid=0x185c runnable [0x42e3f000] java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:138) at org.eclipse.jgit.treewalk.FileTreeIterator$FileEntry.openInputStream(FileTreeIterator.java:198) at org.eclipse.jgit.treewalk.WorkingTreeIterator.idBufferBlob(WorkingTreeIterator.java:257) at org.eclipse.jgit.treewalk.WorkingTreeIterator.idBuffer(WorkingTreeIterator.java:235) at org.eclipse.jgit.treewalk.AbstractTreeIterator.getEntryObjectId(AbstractTreeIterator.java:399) at org.eclipse.jgit.treewalk.WorkingTreeIterator.contentCheck(WorkingTreeIterator.java:699) at org.eclipse.jgit.treewalk.WorkingTreeIterator.isModified(WorkingTreeIterator.java:671) at org.eclipse.jgit.treewalk.filter.IndexDiffFilter.include(IndexDiffFilter.java:176) at org.eclipse.jgit.treewalk.filter.AndTreeFilter$List.include(AndTreeFilter.java:162) at org.eclipse.jgit.treewalk.TreeWalk.next(TreeWalk.java:560) at org.eclipse.jgit.lib.IndexDiff.diff(IndexDiff.java:284) at org.eclipse.egit.core.internal.indexdiff.IndexDiffCacheEntry.calcIndexDiff(IndexDiffCacheEntry.java:265) at org.eclipse.egit.core.internal.indexdiff.IndexDiffCacheEntry.access$2(IndexDiffCacheEntry.java:256) at org.eclipse.egit.core.internal.indexdiff.IndexDiffCacheEntry$3.run(IndexDiffCacheEntry.java:151) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)
(In reply to comment #25) I have some questions on your environment: - which OS / Hardware are you using? - Is your Git repository located on a network share? I tested the new decoration with big repositories and got the following calculation time for IndexDiffCache (calculates a git status, the data model for decoration): Linux Kernel Repository (30000 files): MacBook Pro Core i7 8GB: 1s Windows 7, Core i5, 8GB: 6s I also tested CDT repository on my MacBook Pro and got comparable times (~1s). So it is interesting to get the calculation times from the trace.
(In reply to comment #26) > (In reply to comment #25) > > I have some questions on your environment: Maybe as a start: I have 12 repos cloned from http://git.eclipse.org/c/. See attached picture if you want to try them out. > - which OS / Hardware are you using? Windows 7 on a Lenovo W510 with 8 GB RAM. > - Is your Git repository located on a network share? No, local disk (but not an SSD).
Created attachment 205185 [details] Picture showing all repos
(In reply to comment #24) > Can you provide a trace? I finally found some time to do some tracing. This is with 1.2.0.201111222122 where it no longer takes 5 minutes as before but still takes 1-2 minutes until the decorators are OK. The bad thing is that they also toggle during that time, e.g. the blue ? appears and goes away again.
Created attachment 207422 [details] Trace 1
Created attachment 207423 [details] Trace 2
Dani, could you provide a team project set so that we can mirror your workspace setup ?
Created attachment 208048 [details] Dani's team project set
(In reply to comment #33) > Created attachment 208048 [details] > Dani's team project set I tried to import it and got the following error logged: java.lang.IllegalArgumentException: Path for project must have only one segment. at org.eclipse.core.runtime.Assert.isLegal(Assert.java:63) at org.eclipse.core.internal.resources.WorkspaceRoot.getProject(WorkspaceRoot.java:184) at org.eclipse.team.internal.ccvs.core.CVSProjectSetCapability$LoadInfo.<init>(CVSProjectSetCapability.java:269) at org.eclipse.team.internal.ccvs.core.CVSProjectSetCapability.asProjects(CVSProjectSetCapability.java:202) at org.eclipse.team.internal.ccvs.core.CVSProjectSetCapability.addToWorkspace(CVSProjectSetCapability.java:119) at org.eclipse.team.internal.ui.ProjectSetImporter.importProjectSet(ProjectSetImporter.java:109) at org.eclipse.team.internal.ui.ProjectSetImporter.importProjectSet(ProjectSetImporter.java:67) at org.eclipse.team.internal.ui.wizards.ImportProjectSetOperation.runForFile(ImportProjectSetOperation.java:84) at org.eclipse.team.internal.ui.wizards.ImportProjectSetOperation.run(ImportProjectSetOperation.java:99) at org.eclipse.team.internal.ui.actions.ProgressDialogRunnableContext$3.run(ProgressDialogRunnableContext.java:100) at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2344) at org.eclipse.team.internal.ui.actions.ProgressDialogRunnableContext$2.run(ProgressDialogRunnableContext.java:97) at org.eclipse.jface.operation.ModalContext$ModalContextThread.run(ModalContext.java:121)
(In reply to comment #34) > (In reply to comment #33) > > Created attachment 208048 [details] > > Dani's team project set > > I tried to import it and got the following error logged: Me too. Looks like something went wrong during export.
(In reply to comment #35) > (In reply to comment #34) > > (In reply to comment #33) > > > Created attachment 208048 [details] > > > Dani's team project set > > > > I tried to import it and got the following error logged: > > Me too. Looks like something went wrong during export. Actually not. I've debugged this and there are two problems: 1. org.eclipse.egit.core.GitProjectSetCapability.addToWorkspace(String[], ProjectSetSerializationContext, IProgressMonitor) throws an exception: org.eclipse.jgit.errors.TransportException: ssh://dmegert@git.eclipse.org/gitroot/platform/eclipse.platform.git: UnknownHostKey: git.eclipse.org. RSA key fingerprint is .... Filed bug 367750 to track this. This exception is not surfaced due to the second issue (see below). The interesting part here is, that if I first open the 'Git Repositories' view and then start the import, all is fine. So this looks like an initialization problem in Git. Jens, can you check whether this workaround works for you too? 2. Error handling and reporting in org.eclipse.team.internal.ui.ProjectSetImporter.importProjectSet(String, Shell, IProgressMonitor) is wrong. Filed bug 367749 to track that.
Ping!
https://git.eclipse.org/r/#/c/7127/ and https://git.eclipse.org/r/#/c/7231/ makes a huge difference for me. Please try the nightly build.
(In reply to comment #38) > https://git.eclipse.org/r/#/c/7127/ and https://git.eclipse.org/r/#/c/7231/ > makes a huge difference for me. Please try the nightly build. I can see that it is a bit faster but still takes around 2 minutes until all my projects are updated. Since my last comment 0 I now have more repos (21) and more projects checked out (~130). Note that my problem is not a general slowness, but rather the updating after starting Eclipse.
(In reply to comment #39) > (In reply to comment #38) > > https://git.eclipse.org/r/#/c/7127/ and https://git.eclipse.org/r/#/c/7231/ > > makes a huge difference for me. Please try the nightly build. > > I can see that it is a bit faster but still takes around 2 minutes until all > my projects are updated. Since my last comment 0 I now have more repos (21) > and more projects checked out (~130). Note that my problem is not a general > slowness, but rather the updating after starting Eclipse. This is still an issue.
When I switch branches, it takes a minute or 2 before the current branch decorators in the package explorer and repositories view update.
Does https://git.eclipse.org/r/#/c/9952/ help?
Yes! Now it takes only a few seconds to update! Thanks!
(In reply to comment #43) > Yes! Now it takes only a few seconds to update! Thanks! https://git.eclipse.org/r/#/c/9952/ was merged
fixed
This is not fixed for me. It still takes more than one minute after a fresh start on my SSD. The decoration for each project seems to wait until its repository is re-indexed (fully). I have 29 repositories and hence, the decorations slowly drip in. I filed bug 405770 for that scenario. Also, the following commit message from https://git.eclipse.org/r/#/c/9952/ is not true for the startup scenario: " Finally reduce the total number of re-indexing jobs for all repositories to two. Two or three seems to be a general practical limit for how many jobs can be run in parallel without losing performance due to trashing. " see attached picture.
Created attachment 229752 [details] Picture showing multiple re-indexing jobs This is with 3.0.0.201304152313.
(In reply to comment #47) > Created attachment 229752 [details] > Picture showing multiple re-indexing jobs > > This is with 3.0.0.201304152313. Unless they are actually hard working it's not a problem. At most two are actually doing anything. Perhaps we could change the task name update could be used to indicate that.
