Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: AW: [smila-dev] OutOfMemoryException during Crawl

I only wanted to catch the difference between your and Dmitry's testing results.

--
Ivan

Juergen.Schumacher@xxxxxxxxxxx wrote:
Correct. But I wasn't looking for leaks in Aperture, but in SMILA and ODE. And for this it should be quite irrelevant what type of documents Aperture has to process, shouldn't it? Of course, in Dmitry's test case, having to process large binary documents could provoke the OoM even earlier.

Jürgen.

-----Original Message-----
From: smila-dev-bounces@xxxxxxxxxxx [mailto:smila-dev-
bounces@xxxxxxxxxxx] On Behalf Of Ivan Churkin
Sent: Tuesday, October 14, 2008 12:23 PM
To: Smila project developer mailing list
Subject: Re: AW: [smila-dev] OutOfMemoryException during Crawl

Hi,

 >indexing the JDK1.6 documentation
It does not contain representative collection of different document
types.
Guess it was mainly html documents and only html converter used which
is
not resource consuming.

--
Ivan



Juergen.Schumacher@xxxxxxxxxxx wrote:
Hi,

I did the test, too, with a standard SMILA build and configuration,
indexing the JDK1.6 documentation. Yes, in the end ODE seems to occupy
quite a lot of space, but it seems not to be a memory leak. It's just a
bit lazy with cleaning up. I've already opened an issue in the ODE JIRA
about making this configurable to be more "aggressive" (there was an
configuration option for this in an earlier version which has
disappeared again), and suppose I'll ask the ODE developers about
changing the behavior (have to think about it first to make a useful
proposal). But there should be a lot more instances left of this
BpelDAOConnectionImpl if they would not be released at all. The leak
suspects analysis shows only the VFS classes.
Dmitry, maybe with -Xmx64m your test just did not run long enough so
that ODE did not have a chance to clean up at all? At least in my test
the 64m heap usage was exceeded quite immediately after starting the
indexing.
Cheers.
Jürgen.


-----Original Message-----
From: smila-dev-bounces@xxxxxxxxxxx [mailto:smila-dev-
bounces@xxxxxxxxxxx] On Behalf Of Dmitry Hazin
Sent: Tuesday, October 14, 2008 11:58 AM
To: Smila project developer mailing list
Subject: Re: AW: [smila-dev] OutOfMemoryException during Crawl

Hi Daniel,

I tested several times with the same configuration as you described
(all components active and XmX=64m) and in all times it was
*org.apache.ode.bpel.memdao.BpelDAOConnectionImpl
*Could it be that aperture takes too much memory while converting my
local documents?*
*
Thanks,
Dmitry

Daniel.Stucky@xxxxxxxxxxx wrote:

Hi Dmitry,

I did some tests (with all components active) and XmX=64m. I was
not
able to reproduce the behavior of ODE you described.

The leak suspects are always located in commons VFS (see
attachment).
Bye,
Daniel



-----Ursprüngliche Nachricht-----
Von: smila-dev-bounces@xxxxxxxxxxx [mailto:smila-dev-
bounces@xxxxxxxxxxx] Im Auftrag von Dmitry Hazin
Gesendet: Montag, 13. Oktober 2008 14:03
An: Smila project developer mailing list
Betreff: Re: [smila-dev] OutOfMemoryException during Crawl

Hi,

I found the tool that can be useful in detecting the leak source:
Memory
Analyzer tool (it's in incubation but seems to be working good),
http://www.eclipse.org/mat/ It allows to analyze Java heap dumps
and
to generate different reports regarding memory usage. To get a
dump
on OOM Exception you should to add the following JVM parameter to
SMILA.ini:
-XX:+HeapDumpOnOutOfMemoryError or -XX:+HeapDumpOnCtrlBreak to get
heap dump on demand.
Also you can get a dump with the following command: jmap
-dump:format=b,file=<filename.hprof> <pid> (command format id for
java6).

So here are the first results so far (leak analyzer executed on
one
dump was taken with filesystem crawler executed on large amount of
files and another when crawling job was stopped). The most memory
is
accumulated by org.apache.ode and org.apache.commons.vfs, and
org.apache.ode is always at the first place:

Report with dump #1:

One instance of "org.apache.ode.bpel.engine.BpelServerImpl" loaded

by

"org.apache.ode" occupies 55,544,144 (40.42%) bytes. The memory is
accumulated in one instance of
"org.apache.ode.bpel.memdao.BpelDAOConnectionImpl" loaded by
"org.apache.ode".
Keywords
org.apache.ode.bpel.engine.BpelServerImpl
org.apache.ode
org.apache.ode.bpel.memdao.BpelDAOConnectionImpl

Details »
Problem Suspect 2

20,031 instances of
"org.apache.commons.vfs.provider.local.LocalFile",
loaded by "org.apache.commons.vfs" occupy 15,637,616 (11.38%)
bytes.
These instances are referenced from one instance of
"java.util.HashMap$Entry[]", loaded by "<system class loader>"

Keywords
org.apache.commons.vfs
java.util.HashMap$Entry[]
org.apache.commons.vfs.provider.local.LocalFile


==============
Report with dump #2:

Problem Suspect 1

One instance of "org.apache.ode.bpel.engine.BpelServerImpl" loaded

by

"org.apache.ode" occupies 43,411,080 (23.18%) bytes. The memory is
accumulated in one instance of
"org.apache.ode.bpel.memdao.BpelDAOConnectionImpl" loaded by
"org.apache.ode".
Keywords
org.apache.ode.bpel.engine.BpelServerImpl
org.apache.ode
org.apache.ode.bpel.memdao.BpelDAOConnectionImpl

Details »
Problem Suspect 2

The class "java.lang.ref.Finalizer", loaded by "<system class
loader>", occupies 39,970,744 (21.34%) bytes.
Keywords
java.lang.ref.Finalizer

Details »
Problem Suspect 3

33,379 instances of
"org.apache.commons.vfs.provider.local.LocalFile",
loaded by "org.apache.commons.vfs" occupy 26,058,944 (13.92%)
bytes.
These instances are referenced from one instance of
"java.util.HashMap$Entry[]", loaded by "<system class loader>"

Keywords
org.apache.commons.vfs
java.util.HashMap$Entry[]
org.apache.commons.vfs.provider.local.LocalFile


Thanks,
Dmitry

_______________________________________________
smila-dev mailing list
smila-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/smila-dev
------------------------------------------------------------------
--
-

---

_______________________________________________
smila-dev mailing list
smila-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/smila-dev

_______________________________________________
smila-dev mailing list
smila-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/smila-dev

_______________________________________________
smila-dev mailing list
smila-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/smila-dev
_______________________________________________
smila-dev mailing list
smila-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/smila-dev



Back to the top