[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [eclipselink-users] memory usage
What ended up working for me was the suggestion to throw away the EntityManager after each transaction. Once I started doing that, the memory profile stayed nice and consistent. It actually seemed to improve performance a bit, too, though I do not know why that would be.

Thanks.

- DAP

On Oct 29, 2009, at 8:23 AM, Vackar Afzal wrote:

I had this problem a while back.
I was importing millions of rows using JPA, and it started out relatively fast, but after a while got very slow. Nothing I did fixed this (flushing, clearing, batch inserts etc).

The way I got around it was as follows. Split import into batches of 5000 (each with a separate transaction), and monitor the output from the console. When it got slow, I killed the process, and started it up at the next batch index.

Not perfect but it worked.

In hindsight it would probably have been better to have imported it using DBA tools instead of JPA.

Thanks,
Vackar

David Parker <dap@xxxxxxxxxxxxxxxxxxxx> 10/21/09 6:04 PM >>> 
I added a call to EntityManager.clear() after each transaction, but
these object arrays still seem to be piling up. I am doing another
test run with tracking allocations turned on, so hopefully I can pin
down better where these objects are being allocated.

Does EntityManager.clear() clear the whole cache? I don't see a
clear() method on EntityManagerFactory, which I understand wraps the
ServerSession - is there something I can try re-setting the whole thing?

Thanks.

- DAP

On Oct 20, 2009, at 11:57 AM, David Parker wrote:

Thanks, everybody, for all the responses. I appreciate your help.

Are you using JTA, or RESOURCE_LOCAL?


Local

Are you using the same EntityManager, or do you create a new one
for each
transaction?


This thing runs as a big loop sucking records out of one database
via a simple JDBC Connection, and for each record creating the JPA
objects for that record, and persisting them. As part of creating
the JPA objects there is some querying against the target database
via SQL queries.

There is one EntityManager, and a single transaction per record.

What cache setting are you using?


SoftWeak, but I am confused about what the best caching option is.
I've read the doc, and I ~think I understand the technical
description, but I'm still not sure when to use what.

I would recommend you create a new EntityManager per transaction,
otherwise
the previous objects will still be part of the persistence
context.  You
could also call clear().


I've started another run with the clear() after each commit.


You can try disabling the cache (shared=false), or using a weak
cache.

Do you run out of memory, or just notice a memory increase?  You
may wish to
try a memory profiler such as JProfiler.


Memory increases slowly but inexorably. I've had it blow the heap at
-Xmx6G -d64 -server. I am hopeful that just being more aggressive
about the cache clearing
will fix the problem. The YourKit profiler doesn't flag much it
thinks is  a leal - just a few k.

Also ensure that your application is not holding onto any or the
objects.


DavidParker wrote: 

Yeah, we are using multiple transactions, i.e.

get an EntityTransaction
do work
EntityTransation.commit()
set the local tx to null


On Oct 19, 2009, at 5:59 PM, Dave Brosius wrote:

Is the import all in one transaction? I'd be curious if you broke
up
the import (even for a test) into multiple transactions whether the
issue would go away.

-----Original Message-----
From: "David Parker" <dap@xxxxxxxxxxxxxxxxxxxx>
Sent: Monday, October 19, 2009 5:51pm
To: "EclipseLink User Discussions" <eclipselink-users@xxxxxxxxxxx>
Subject: [eclipselink-users] memory usage

I am running a long data import that uses eclipselink 1.1.2 for
persistence. I see memory increasing slowly but surely as it goes
along. The YourKit profiler does not identify anything as a link,
but
I do seem to have an increasing "retained size" for

org .eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork
[Stack Local]

I see there is a clear method on TransactionWrapperImpl, but that
does
not seem to be in the JPA EntityTransaction API.

Is there a best practice for long running data imports? Should I be
clearing something?

Thanks.


=============================================
David Parker
dap@xxxxxxxxxxxxxxxxxxxx





-----
http://wiki.eclipse.org/User:James.sutherland.oracle.com James
Sutherland
http://www.eclipse.org/eclipselink/
EclipseLink , http://www.oracle.com/technology/products/ias/ toplink/
TopLink
Wiki: http://wiki.eclipse.org/EclipseLink EclipseLink ,
http://wiki.oracle.com/page/TopLink TopLink
Forums: http://forums.oracle.com/forums/forum.jspa?forumID=48
TopLink ,
http://www.nabble.com/EclipseLink-f26430.html EclipseLink
Book: http://en.wikibooks.org/wiki/Java_Persistence Java Persistence
--
View this message in context: http://www.nabble.com/memory-usage-tp25966341p25974597.html
Sent from the EclipseLink - Users mailing list archive at Nabble.com.

_______________________________________________
eclipselink-users mailing list
eclipselink-users@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/eclipselink-users


=============================================
David Parker
dap@xxxxxxxxxxxxxxxxxxxx




_______________________________________________
eclipselink-users mailing list
eclipselink-users@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/eclipselink-users


=============================================
David Parker
dap@xxxxxxxxxxxxxxxxxxxx





The University of Dundee is a registered Scottish charity, No: SC015096
_______________________________________________
eclipselink-users mailing list
eclipselink-users@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/eclipselink-users


=============================================
David Parker
dap@xxxxxxxxxxxxxxxxxxxx