Comment 1 Andrey Loskutov 2020-05-19 09:49:02 EDT

Noopur, I assume that this could be related to jgit memory usage, but better we check this with memory dump.

First get your Eclipse process id (jps -l).
With process_id you can then create 3 files that could help:

jmap -heap process_id > /tmp/memory_heap_summary.txt
jmap -histo process_id > /tmp/memory_histo_summary.txt
 
# this will take longer and can require lot of free disk space (depending on the config from ~2 to ~20 GB)!
jmap -dump:file=/tmp/memory_dump.hprof process_id

# Zip all and attach to this bug:
zip memory_dump.zip /tmp/memory_heap_summary.txt /tmp/memory_histo_summary.txt /tmp/memory_dump.hprof

Comment 2 Noopur Gupta 2020-05-20 05:20:09 EDT

I am using AdoptOpenJDK\jdk-11.0.4.11-openj9 as the default on my windows system and running these commands with -heap and -dump gives the message:

exactly one option must be selected
jmap: obtain heap information about a Java process
 Usage:
    jmap <option>* <vmid>
        <vmid>: Attach API VM ID as shown in jps or other Attach API-based tools
        <vmid>s are read from stdin if none are supplied as arguments
    -histo: print statistics about classes on the heap, including number of objects and aggregate size
    -histo:live : Print only live objects
    -J: supply arguments to the Java VM running jmap
NOTE: this utility may significantly affect the performance of the target VM.
At least one option must be selected.

Comment 3 Andrey Loskutov 2020-05-20 05:28:35 EDT

(In reply to Noopur Gupta from comment #2)
> I am using AdoptOpenJDK\jdk-11.0.4.11-openj9 as the default on my windows
> system and running these commands with -heap and -dump gives the message:
> 
> exactly one option must be selected
> jmap: obtain heap information about a Java process
>  Usage:
>     jmap <option>* <vmid>
>         <vmid>: Attach API VM ID as shown in jps or other Attach API-based
> tools
>         <vmid>s are read from stdin if none are supplied as arguments
>     -histo: print statistics about classes on the heap, including number of
> objects and aggregate size
>     -histo:live : Print only live objects
>     -J: supply arguments to the Java VM running jmap
> NOTE: this utility may significantly affect the performance of the target VM.
> At least one option must be selected.

But how did you run it? 

I wrote all the commands with only one argument:

jmap -heap process_id > /tmp/memory_heap_summary.txt
jmap -histo process_id > /tmp/memory_histo_summary.txt
jmap -dump:file=/tmp/memory_dump.hprof process_id

Comment 4 Andrey Loskutov 2020-05-20 05:33:33 EDT

(In reply to Andrey Loskutov from comment #3)
> But how did you run it? 
> 
> I wrote all the commands with only one argument:
> 
> jmap -heap process_id > /tmp/memory_heap_summary.txt
> jmap -histo process_id > /tmp/memory_histo_summary.txt
> jmap -dump:file=/tmp/memory_dump.hprof process_id

Of course if you have Windows, you can't use output redirection, so you must copy/paste the output manually 

jmap -heap process_id
# copy std out to memory_heap_summary.txt

jmap -histo process_id 
# copy std out to memory_histo_summary.txt

jmap -dump:file=C:\tmp\memory_dump.hprof process_id

Not sure regarding the last one, what is the supported path notation on windows JDK.

Comment 5 Noopur Gupta 2020-05-20 05:42:08 EDT

I ran the following (9880 is the process id):

jmap -heap 9880 > C:\tmp\memory_heap_summary.txt
jmap -histo 9880 > C:\tmp\memory_histo_summary.txt
jmap -dump:file=C:\tmp\memory_dump.hprof 9880

Only the one with -histo worked. The other two gave the message that I pasted. Probably -heap and -dump are not supported in AdoptOpenJDK\jdk-11.0.4.11-openj9. 

I will try with another JDK when the OOME happens.

Comment 6 Andrey Loskutov 2020-05-20 05:49:01 EDT

(In reply to Noopur Gupta from comment #5)
> I ran the following (9880 is the process id):
> 
> jmap -heap 9880 > C:\tmp\memory_heap_summary.txt
> jmap -histo 9880 > C:\tmp\memory_histo_summary.txt
> jmap -dump:file=C:\tmp\memory_dump.hprof 9880
> 
> Only the one with -histo worked. The other two gave the message that I
> pasted. Probably -heap and -dump are not supported in
> AdoptOpenJDK\jdk-11.0.4.11-openj9. 
> 
> I will try with another JDK when the OOME happens.

Try with

jcmd 9880 GC.class_histogram
jcmd 9880 GC.heap_dump C:\tmp\memory_dump.hprof

Comment 7 Noopur Gupta 2020-05-20 06:02:26 EDT

(In reply to Andrey Loskutov from comment #6)
> Try with
> 
> jcmd 9880 GC.class_histogram
> jcmd 9880 GC.heap_dump C:\tmp\memory_dump.hprof

Thanks, these are working. 

Now the OOME is not happening with the same steps, probably because there is nothing to merge after pulling as I just pulled them earlier. I will get the files next time.

Comment 8 Noopur Gupta 2020-05-27 03:58:58 EDT

Created attachment 283033 [details]
memory_histo_summary.txt

I got OOME in I20200526-1800 while building after pulling the repositories.

Attaching memory_histo_summary.txt. The other file memory_dump.hprof is 2GB and Bugzilla won't allow it as an attachment. I will share the link after uploading it to some other location.

Comment 9 Noopur Gupta 2020-05-27 04:52:20 EDT

(In reply to Noopur Gupta from comment #8)
> The other file memory_dump.hprof is 2GB
> and Bugzilla won't allow it as an attachment. I will share the link after
> uploading it to some other location.

https://www.dropbox.com/s/5lo99kwwavi9zf4/memory_dump.hprof?dl=0

Comment 10 Andrey Loskutov 2020-05-27 15:39:46 EDT

Created attachment 283037 [details]
yourkit screenshot

(In reply to Noopur Gupta from comment #8)
> Created attachment 283033 [details]
> memory_histo_summary.txt
> 
> I got OOME in I20200526-1800 while building after pulling the repositories.
> 
> Attaching memory_histo_summary.txt. The other file memory_dump.hprof is 2GB
> and Bugzilla won't allow it as an attachment. I will share the link after
> uploading it to some other location.

 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:        177218     1011760960  [I (java.base@13)
   2:       2494674      637787576  [B (java.base@13)

You have 1 GB with array of ints and 600 MB in arrays bytes, all retained by jgit pack files, see screenshot.

I would recommend to increase heap size or reduce number of projects in your workspace :-). 

I run with 16 GB and pretty much of SDK imported without issues, but 8 should be OK too.

Comment 11 Noopur Gupta 2020-05-28 02:43:12 EDT

(In reply to Andrey Loskutov from comment #10)
> Created attachment 283037 [details]
> yourkit screenshot
> 
> You have 1 GB with array of ints and 600 MB in arrays bytes, all retained by
> jgit pack files, see screenshot.
> 
> I would recommend to increase heap size or reduce number of projects in your
> workspace :-). 
> 
> I run with 16 GB and pretty much of SDK imported without issues, but 8
> should be OK too.

Thanks for looking into it. I use the default 2 GB but started seeing this with the same number of projects almost 1-2 months back. I will increase the heap size and report if it happens again.

Comment 12 Andrey Loskutov 2020-05-28 03:02:09 EDT

(In reply to Noopur Gupta from comment #11)
> Thanks for looking into it. I use the default 2 GB but started seeing this
> with the same number of projects almost 1-2 months back.

Which egit version are you using? May be we had a regression in jgit/egit area.

Comment 13 Noopur Gupta 2020-05-28 03:24:00 EDT

(In reply to Andrey Loskutov from comment #12)
> Which egit version are you using? 

Git integration for Eclipse	5.7.0.202003110725-r

Comment 14 Lars Vogel 2020-05-28 03:28:03 EDT

Should this bug moved to JGit or EGit?

Comment 15 Thomas Wolf 2020-05-29 18:27:12 EDT

@Matthias: could this be caused by https://git.eclipse.org/r/#/c/155149/ ? I'm particularly wary of the fact the the soft reference queue's enqueue() has been overridden to be a no-op. Previously when there was also only soft references and a ReferenceQueue, calling Ref.enqueue() _did_ enqueue the item, and a subequent WindowCache.gc() would find it on the queue. Newly PageRef.kill() will not actually enqueue the ref; that'll happen only later, whenever the Java GC feels up to it.