Bug 486844 - git rebase is taking ages and then runs out of memory
Summary: git rebase is taking ages and then runs out of memory
Status: NEW
Alias: None
Product: JGit
Classification: Technology
Component: JGit (show other bugs)
Version: unspecified   Edit
Hardware: PC Linux
: P3 normal (vote)
Target Milestone: ---   Edit
Assignee: Project Inbox CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-01-29 10:49 EST by Michael Woski CLA
Modified: 2016-01-31 15:06 EST (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Woski CLA 2016-01-29 10:49:07 EST
I regularly rebase my local commits onto a cdt master checkout.

Meanwhile I've given up doing this from inside eclipse, because rebasing bogs down the CPU using all cores and finally runs into an out-of-memory situation, which is when I have to kill the whole workbench process.

The same operation on the command line with the "real" git only takes like 1 or 2 seconds.

All eclipse plugins are from snapshot update sites, but the problem exists for a long time now.
Comment 1 Matthias Sohn CLA 2016-01-29 11:49:31 EST
Check object statistics for your repository.
In EGit select your repository in the Git repositories view and click "Properties > Statistics". If you have a lot of loose objects you should run the git gc to repack your repository.
On the command line you can do the same check using
$ git count-objects -vH

Then try rebase again. 

If you still see performance issues also check the configuration option
"Preferences > Team > Git > Window Cache > Window Cache Limit ?
Increase this to a larger value to increase the size of JGit's object cache buffering access to objects stored in git. If this cache is too small JGit may have to do a lot of I/O. If necessary increase the JVM max heap size in eclipse.ini and restart Eclipse.

Alternatively you can try to use virtual memory mapping (another option under
"Preferences > Team > Git > Window Cache"). It has the advantage that JGit then uses NIO file memory mapping instead of its default cache implementation based on byte arrays to read objects from pack files. Though with this option we depend on the JVMs garbage collector to reclaim this memory since the JVM doesn't provide an explicit unmmap().

Also you should update to EGit 4.2 if you are using an older version, we did a couple of performance improvements recently.

If all this doesn't help provide steps to reproduce with some public repository.
Comment 2 Michael Woski CLA 2016-01-29 13:13:13 EST
(In reply to Matthias Sohn from comment #1)

Hi Matthias,

thanks for this quick and thorough answer. I changed those settings and it looks as if rebasing works like a charm now. I will come back here and report whether it remains that way, so far it was just a quick test.

This, however, leaves me with the question where I could have found the information without opening a bug report. To be honest, Window Size, Window Cache Limit, etc. don't ring any bells with me and I'm still clueless what these values do or what they are good for. Maybe some different naming and/or description on that (quite empty) preference page would have helped - maybe.

Anyway, thanks a lot for your help.
Comment 3 Andrey Loskutov CLA 2016-01-30 14:01:50 EST
(In reply to Michael Woski from comment #2)
> (In reply to Matthias Sohn from comment #1)
> 
> Hi Matthias,
> 
> thanks for this quick and thorough answer. I changed those settings 

Which from many proposed by Matthias?

> and it
> looks as if rebasing works like a charm now. 

> This, however, leaves me with the question where I could have found the
> information without opening a bug report. 

Me too :-)

> To be honest, Window Size, Window
> Cache Limit, etc. don't ring any bells with me and I'm still clueless what
> these values do or what they are good for. Maybe some different naming
> and/or description on that (quite empty) preference page would have helped -
> maybe.

I wonder *if* the virtual memory mapping is that good, why don't we enable it by default?
Comment 4 Michael Woski CLA 2016-01-31 15:06:04 EST
(In reply to Andrey Loskutov from comment #3)

> Which from many proposed by Matthias?
basically all.

Meanwhile I've had the chance to rebase my patches again onto the latest cdt master branch, and it was a bit disillusioning. Although I was not running into heap exhaustion this time, my notion of performance is something different.

As for the first suggestion from comment #2, I ran gc just recently to no avail. The following is what count-objects currently spits out:

count: 3471
size: 14.71 MiB
in-pack: 602012
packs: 13
size-pack: 148.33 MiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes