Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
RE: AW: [smila-dev] OutOfMemoryException during Crawl

Hi,

I did the test, too, with a standard SMILA build and configuration, indexing the JDK1.6 documentation. Yes, in the end ODE seems to occupy quite a lot of space, but it seems not to be a memory leak. It's just a bit lazy with cleaning up. I've already opened an issue in the ODE JIRA about making this configurable to be more "aggressive" (there was an configuration option for this in an earlier version which has disappeared again), and suppose I'll ask the ODE developers about changing the behavior (have to think about it first to make a useful proposal). But there should be a lot more instances left of this BpelDAOConnectionImpl if they would not be released at all. The leak suspects analysis shows only the VFS classes.

Dmitry, maybe with -Xmx64m your test just did not run long enough so that ODE did not have a chance to clean up at all? At least in my test the 64m heap usage was exceeded quite immediately after starting the indexing.

Cheers.
Jürgen.

> -----Original Message-----
> From: smila-dev-bounces@xxxxxxxxxxx [mailto:smila-dev-
> bounces@xxxxxxxxxxx] On Behalf Of Dmitry Hazin
> Sent: Tuesday, October 14, 2008 11:58 AM
> To: Smila project developer mailing list
> Subject: Re: AW: [smila-dev] OutOfMemoryException during Crawl
> 
> Hi Daniel,
> 
> I tested several times with the same configuration as you described
> (all components active and XmX=64m) and in all times it was
> *org.apache.ode.bpel.memdao.BpelDAOConnectionImpl
> *Could it be that aperture takes too much memory while converting my
> local documents?*
> *
> Thanks,
> Dmitry
> 
> Daniel.Stucky@xxxxxxxxxxx wrote:
> > Hi Dmitry,
> >
> > I did some tests (with all components active) and XmX=64m. I was not
> able to reproduce the behavior of ODE you described.
> > The leak suspects are always located in commons VFS (see attachment).
> >
> > Bye,
> > Daniel
> >
> >
> >> -----Ursprüngliche Nachricht-----
> >> Von: smila-dev-bounces@xxxxxxxxxxx [mailto:smila-dev-
> >> bounces@xxxxxxxxxxx] Im Auftrag von Dmitry Hazin
> >> Gesendet: Montag, 13. Oktober 2008 14:03
> >> An: Smila project developer mailing list
> >> Betreff: Re: [smila-dev] OutOfMemoryException during Crawl
> >>
> >> Hi,
> >>
> >> I found the tool that can be useful in detecting the leak source:
> >> Memory
> >> Analyzer tool (it's in incubation but seems to be working good),
> >> http://www.eclipse.org/mat/ It allows to analyze Java heap dumps and
> >> to generate different reports regarding memory usage. To get a dump
> >> on OOM Exception you should to add the following JVM parameter to
> >> SMILA.ini:
> >> -XX:+HeapDumpOnOutOfMemoryError or -XX:+HeapDumpOnCtrlBreak to get
> >> heap dump on demand.
> >> Also you can get a dump with the following command: jmap
> >> -dump:format=b,file=<filename.hprof> <pid> (command format id for
> >> java6).
> >>
> >> So here are the first results so far (leak analyzer executed on one
> >> dump was taken with filesystem crawler executed on large amount of
> >> files and another when crawling job was stopped). The most memory is
> >> accumulated by org.apache.ode and org.apache.commons.vfs, and
> >> org.apache.ode is always at the first place:
> >>
> >> Report with dump #1:
> >>
> >> One instance of "org.apache.ode.bpel.engine.BpelServerImpl" loaded
> by
> >> "org.apache.ode" occupies 55,544,144 (40.42%) bytes. The memory is
> >> accumulated in one instance of
> >> "org.apache.ode.bpel.memdao.BpelDAOConnectionImpl" loaded by
> >> "org.apache.ode".
> >> Keywords
> >> org.apache.ode.bpel.engine.BpelServerImpl
> >> org.apache.ode
> >> org.apache.ode.bpel.memdao.BpelDAOConnectionImpl
> >>
> >> Details »
> >> Problem Suspect 2
> >>
> >> 20,031 instances of
> >> "org.apache.commons.vfs.provider.local.LocalFile",
> >> loaded by "org.apache.commons.vfs" occupy 15,637,616 (11.38%) bytes.
> >> These instances are referenced from one instance of
> >> "java.util.HashMap$Entry[]", loaded by "<system class loader>"
> >>
> >> Keywords
> >> org.apache.commons.vfs
> >> java.util.HashMap$Entry[]
> >> org.apache.commons.vfs.provider.local.LocalFile
> >>
> >>
> >> ==============
> >> Report with dump #2:
> >>
> >> Problem Suspect 1
> >>
> >> One instance of "org.apache.ode.bpel.engine.BpelServerImpl" loaded
> by
> >> "org.apache.ode" occupies 43,411,080 (23.18%) bytes. The memory is
> >> accumulated in one instance of
> >> "org.apache.ode.bpel.memdao.BpelDAOConnectionImpl" loaded by
> >> "org.apache.ode".
> >> Keywords
> >> org.apache.ode.bpel.engine.BpelServerImpl
> >> org.apache.ode
> >> org.apache.ode.bpel.memdao.BpelDAOConnectionImpl
> >>
> >> Details »
> >> Problem Suspect 2
> >>
> >> The class "java.lang.ref.Finalizer", loaded by "<system class
> >> loader>", occupies 39,970,744 (21.34%) bytes.
> >> Keywords
> >> java.lang.ref.Finalizer
> >>
> >> Details »
> >> Problem Suspect 3
> >>
> >> 33,379 instances of
> >> "org.apache.commons.vfs.provider.local.LocalFile",
> >> loaded by "org.apache.commons.vfs" occupy 26,058,944 (13.92%) bytes.
> >> These instances are referenced from one instance of
> >> "java.util.HashMap$Entry[]", loaded by "<system class loader>"
> >>
> >> Keywords
> >> org.apache.commons.vfs
> >> java.util.HashMap$Entry[]
> >> org.apache.commons.vfs.provider.local.LocalFile
> >>
> >>
> >> Thanks,
> >> Dmitry
> >>
> >> _______________________________________________
> >> smila-dev mailing list
> >> smila-dev@xxxxxxxxxxx
> >> https://dev.eclipse.org/mailman/listinfo/smila-dev
> >> --------------------------------------------------------------------
> -
> >> ---
> >>
> >> _______________________________________________
> >> smila-dev mailing list
> >> smila-dev@xxxxxxxxxxx
> >> https://dev.eclipse.org/mailman/listinfo/smila-dev


Back to the top