Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [mat-dev] java.io.FileNotFoundException: /mat-work/nikhil/MAT1.11/mat/../../sacbay-m-big77GB.inbound.index3096.log (Too many open files)
Hi Andrew,

Thanks for quick work around.

But we have scenarios where we may not be able to change the limit as its container based solution. Once Hprof files are processed and HTML reports are generated the container will vanish and throughout we will not have access to the container.
And another scenario is hprof file size it may vary from 1gb to 100gb so fixing at OS level may not be the right solution. 

Please suggest.

Regards
Nikhil Dongre 

On Wed, Jul 7, 2021 at 5:09 PM Andrew Johnson <andrew_johnson@xxxxxxxxxx> wrote:
Memory Analyzer does create a lot of files at the re-indexing stage as it helps to build the inbound index. These are deleted once that stage is over, so you don't normally notice them.

You need to increase the limit on the number of open files for your process. Something like
ulimit -S -n 4096
would increase the soft limit for the process - if 4096 is too small for MAT with this dump you will still get the error. There is a hard limit of the number of open files:
ulimit -H -n
and an ordinary user can't increase the soft limit beyond the hard limit. If you need more then you need root access or a system administrator to increase the limits for your userid
The administrator might need to change something like the following.

/etc/security/limits.conf
* - nofile 16384

Once that is done you might need to log in again and run
ulimit -S -n 16384

Andrew Johnson




From:        "Nikhil Dongre" <nikhil.vilas@xxxxxxxxx>
To:        "Memory Analyzer Dev list" <mat-dev@xxxxxxxxxxx>
Date:        07/07/2021 06:51
Subject:        Re: [mat-dev] java.io.FileNotFoundException: /mat-work/nikhil/MAT1.11/mat/../../sacbay-m-big77GB.inbound.index3096.log (Too many open files)
Sent by:        "mat-dev" <mat-dev-bounces@xxxxxxxxxxx>

Here are some more details 
Please find the command ulimit -a output below :

[opc@ci-iad-mat-lab-01 mat]$ ulimit -a

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 239754

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) unlimited

open files                      (-n) 1024

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time               (seconds, -t) unlimited

max user processes              (-u) 4096

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

[opc@ci-iad-mat-lab-01 mat]$ 

Hprof file size is : 77GB

MAT version : 1.11

And its failing after this step  "Re-indexing outbound index"

[Task: Parsing /mat-work/nikhil/MAT1.11_Normal/mat/../../sacbay-m-big77GB.hprof

[

Subtask: Scanning /mat-work/nikhil/MAT1.11_Normal/mat/../../sacbay-m-big77GB.hprof

[....................

[INFO] Wrote threads call stacks to /mat-work/nikhil/MAT1.11_Normal/mat/../../sacbay-m-big77GB.threads

[....................

[INFO] Heap /mat-work/nikhil/MAT1.11_Normal/mat/../../sacbay-m-big77GB.hprof contains 1,168,946,725 objects

[....................

Subtask: Extracting objects from /mat-work/nikhil/MAT1.11_Normal/mat/../../sacbay-m-big77GB.hprof

[................................................................................Task: Removing unreachable objects

[

Subtask: Searching for unreachable objects

[

Subtask: Marking reachable objects

[.

Subtask: Re-indexing objects

[.

[INFO] Removed 72,901,319 unreachable objects using 6,692,864,976 bytes

[..

Subtask: Re-indexing classes

[...

Subtask: Writing /mat-work/nikhil/MAT1.11_Normal/mat/../../sacbay-m-big77GB.idx.index

[....

Subtask: Writing /mat-work/nikhil/MAT1.11_Normal/mat/../../sacbay-m-big77GB.o2c.index

[.....

Subtask: Writing /mat-work/nikhil/MAT1.11_Normal/mat/../../sacbay-m-big77GB.a2s.index

[......

Subtask: Re-indexing outbound index

[......]


eclipse.buildId=unknown

java.version=1.8.0_292

java.vendor=Red Hat, Inc.

BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US

Framework arguments:  -application org.eclipse.mat.api.parse -DhprofStrictnessWarning=true ../../sacbay-m-big77GB.hprof org.eclipse.mat.api:suspects org.eclipse.mat.api:top_components org.eclipse.mat.api:overview

Command-line arguments:  -os linux -ws gtk -arch x86_64 -consolelog -application org.eclipse.mat.api.parse -DhprofStrictnessWarning=true ../../sacbay-m-big77GB.hprof org.eclipse.mat.api:suspects org.eclipse.mat.api:top_components org.eclipse.mat.api:overview

!ENTRY org.eclipse.osgi 4 0 2021-07-07 05:32:46.312

!MESSAGE Application error

!STACK 1

org.eclipse.mat.SnapshotException: Error opening heap dump 'sacbay-m-big77GB.hprof'. Check the error log for further details.

at org.eclipse.mat.parser.internal.SnapshotFactoryImpl.parse(SnapshotFactoryImpl.java:331)

at org.eclipse.mat.parser.internal.SnapshotFactoryImpl.openSnapshot(SnapshotFactoryImpl.java:167)

at org.eclipse.mat.snapshot.SnapshotFactory.openSnapshot(SnapshotFactory.java:147)

at org.eclipse.mat.internal.apps.ParseSnapshotApp.parse(ParseSnapshotApp.java:138)

at org.eclipse.mat.internal.apps.ParseSnapshotApp.start(ParseSnapshotApp.java:110)

at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:203)

at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:137)

at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:107)

at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:401)

at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:657)

at org.eclipse.equinox.launcher.Main.basicRun(Main.java:594)

at org.eclipse.equinox.launcher.Main.run(Main.java:1447)

at org.eclipse.equinox.launcher.Main.main(Main.java:1420)

Caused by: java.io.FileNotFoundException: /mat-work/nikhil/MAT1.11_Normal/mat/../../sacbay-m-big77GB.inbound.index3096.log (Too many open files)

at java.io.FileOutputStream.open0(Native Method)

at java.io.FileOutputStream.open(FileOutputStream.java:270)

at java.io.FileOutputStream.<init>(FileOutputStream.java:213)

at java.io.FileOutputStream.<init>(FileOutputStream.java:162)

at org.eclipse.mat.parser.index.IndexWriter$InboundWriter.log(IndexWriter.java:1237)

at org.eclipse.mat.parser.internal.GarbageCleaner.clean(GarbageCleaner.java:360)

at org.eclipse.mat.parser.internal.SnapshotFactoryImpl.parse(SnapshotFactoryImpl.java:282)

... 17 more



On Wed, Jul 7, 2021 at 9:03 AM Nikhil Dongre <nikhil.vilas@xxxxxxxxx> wrote:
Hi Kevin,

Thanks for the quick response.

Please find the command output below :

[opc@ci-iad-mat-lab-01 mat]$ ulimit -a

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 239754

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) unlimited

open files                      (-n) 1024

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time               (seconds, -t) unlimited

max user processes              (-u) 4096

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

[opc@ci-iad-mat-lab-01 mat]$ 

Regards

Nikhil Dongre




On Wed, Jul 7, 2021 at 8:52 AM Kevin Grigorenko <kevin.grigorenko@xxxxxxxxxx> wrote:
The error "Too many open files" may occur because your file ulimit is too small, system-wide limits have been exceeded, or there is a bug in MAT creating too many file descriptors (in decreasing order of probability). Please start a terminal and send the output of the following command:

$ ulimit -a

--
Kevin Grigorenko
IBM App Platform SWAT




From:        "Nikhil Dongre" <nikhil.vilas@xxxxxxxxx>
To:        mat-dev@xxxxxxxxxxx
Date:        07/06/2021 06:01 PM
Subject:        [EXTERNAL] [mat-dev] java.io.FileNotFoundException: /mat-work/nikhil/MAT1.11/mat/../../sacbay-m-big77GB.inbound.index3096.log (Too many open files)
Sent by:        "mat-dev" <mat-dev-bounces@xxxxxxxxxxx>



Hi Team, When i tried to process process gz hprof file i got this exception  org.eclipse.mat.SnapshotException: Error opening heap dump 'sacbay-m-big77GB.hprof.gz'. Check the error log for further details. at org.eclipse.mat.parser.internal.SnapshotFactoryImpl.parse(SnapshotFactoryImpl.java:331)
Hi Team,

When i tried to process process gz hprof file i got this exception 

org.eclipse.mat.SnapshotException: Error opening heap dump 'sacbay-m-big77GB.hprof.gz'. Check the error log for further details.

at org.eclipse.mat.parser.internal.SnapshotFactoryImpl.parse(SnapshotFactoryImpl.java:331)

at org.eclipse.mat.parser.internal.SnapshotFactoryImpl.openSnapshot(SnapshotFactoryImpl.java:167)

at org.eclipse.mat.snapshot.SnapshotFactory.openSnapshot(SnapshotFactory.java:147)

at org.eclipse.mat.internal.apps.ParseSnapshotApp.parse(ParseSnapshotApp.java:138)

at org.eclipse.mat.internal.apps.ParseSnapshotApp.start(ParseSnapshotApp.java:110)

at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:203)

at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:137)

at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:107)

at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:401)

at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:657)

at org.eclipse.equinox.launcher.Main.basicRun(Main.java:594)

at org.eclipse.equinox.launcher.Main.run(Main.java:1447)

at org.eclipse.equinox.launcher.Main.main(Main.java:1420)

Caused by: java.io.FileNotFoundException: /mat-work/nikhil/MAT1.11/mat/../../sacbay-m-big77GB.inbound.index3096.log (Too many open files)

at java.io.FileOutputStream.open0(Native Method)

at java.io.FileOutputStream.open(FileOutputStream.java:270)

at java.io.FileOutputStream.<init>(FileOutputStream.java:213)

at java.io.FileOutputStream.<init>(FileOutputStream.java:162)

at org.eclipse.mat.parser.index.IndexWriter$InboundWriter.log(IndexWriter.java:1237)

at org.eclipse.mat.parser.internal.GarbageCleaner.clean(GarbageCleaner.java:360)

at org.eclipse.mat.parser.internal.SnapshotFactoryImpl.parse(SnapshotFactoryImpl.java:282)

... 17 more

MemoryAnalyzer:

An error has occurred. See the log file_______________________________________________
mat-dev mailing list
mat-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/mat-dev



_______________________________________________
mat-dev mailing list
mat-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/mat-dev_______________________________________________
mat-dev mailing list
mat-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/mat-dev



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
 _______________________________________________
mat-dev mailing list
mat-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/mat-dev

Back to the top