Out Java team tried to debug this issue using Java
tools. The problem we encountered is that this extra memory is not
tracked by Java at all (off heap memory). We realised this using NMT -
There was not change in NMT tracked memory with and without
annotations enabled. Please see the output of "jcmd <PID>
VM.native_memory summary scale=MB" below:
Glassfish +
annotations disabled (551 MB RES on startup):
Total: reserved=2397MB,
committed=1159MB
- Java
Heap (reserved=1024MB, committed=1024MB)
(mmap: reserved=1024MB, committed=1024MB)
-
Class (reserved=1077MB, committed=60MB)
(classes #10470)
(malloc=1MB #15133)
(mmap: reserved=1076MB, committed=59MB)
-
Thread (reserved=23MB, committed=23MB)
(thread #80)
(stack: reserved=23MB, committed=23MB)
-
Code (reserved=249MB, committed=28MB)
(malloc=5MB #8188)
(mmap: reserved=244MB, committed=23MB)
-
GC (reserved=3MB, committed=3MB)
(mmap: reserved=3MB, committed=3MB)
-
Internal (reserved=3MB, committed=3MB)
(malloc=3MB #13793)
-
Symbol (reserved=14MB, committed=14MB)
(malloc=10MB #111702)
(arena=4MB #1)
- Native Memory
Tracking (reserved=2MB, committed=2MB)
(tracking overhead=2MB)
Glassfish +
annotations (1909 MB RES on startup):
Total: reserved=2400MB,
committed=1162MB
- Java
Heap (reserved=1024MB, committed=1024MB)
(mmap: reserved=1024MB, committed=1024MB)
-
Class (reserved=1079MB, committed=61MB)
(classes #10552)
(malloc=1MB #16204)
(mmap: reserved=1078MB, committed=59MB)
-
Thread (reserved=22MB, committed=22MB)
(thread #74)
(stack: reserved=22MB, committed=22MB)
-
Code (reserved=249MB, committed=30MB)
(malloc=6MB #8349)
(mmap: reserved=244MB, committed=25MB)
-
GC (reserved=3MB, committed=3MB)
(mmap: reserved=3MB, committed=3MB)
-
Internal (reserved=3MB, committed=3MB)
(malloc=3MB #13773)
-
Symbol (reserved=14MB, committed=14MB)
(malloc=11MB #112818)
(arena=4MB #1)
- Native Memory
Tracking (reserved=3MB, committed=3MB)
(tracking overhead=2MB)
- Arena
Chunk (reserved=1MB, committed=1MB)
(malloc=1MB)
Apache + annotations (1900
MB RES on startup):
Total: reserved=2400MB, committed=1163MB
- Java
Heap (reserved=1024MB, committed=1024MB)
(mmap: reserved=1024MB, committed=1024MB)
-
Class (reserved=1079MB, committed=61MB)
(classes #10679)
(malloc=1MB #16766)
(mmap: reserved=1078MB, committed=60MB)
-
Thread (reserved=23MB, committed=23MB)
(thread #80)
(stack: reserved=23MB, committed=23MB)
-
Code (reserved=249MB, committed=30MB)
(malloc=6MB #8492)
(mmap: reserved=244MB, committed=25MB)
-
GC (reserved=3MB, committed=3MB)
(mmap: reserved=3MB, committed=3MB)
-
Internal (reserved=3MB, committed=3MB)
(malloc=3MB #14043)
-
Symbol (reserved=14MB, committed=14MB)
(malloc=11MB #113587)
(arena=4MB #1)
- Native Memory
Tracking (reserved=3MB, committed=3MB)
(tracking overhead=2MB)