Re: [eclipselink-users] memory leak when using caches?

[Michael Pflueger <Michael.Pflueger@xxxxxx>]

Thanks for your detailed reply.

I do load them in batches or with a cursor(/which internally also batch loads), yes.
I am not keeping references to all those objects, but eclipselink seems to be doing something like it, as you suggest.
Now my question is, does eclipselink really need to keep hard references to attached objects? why aren't weak (or alternatively soft) references sufficient?
You can make the identity map of eclipselink use weak(or soft) references, wouldn't it be an option for that other place that keeps references aswell?
This could reduce the amount of memory eclipselink uses considerably, as it would not prevent objects i don't need anymore from being garbage collected.

One reason for loading them in a single transaction is that PostgreSQL requires cursors to execute within a transactional context.
Another is efficiency, it's probably better for our use case(reading and processing gigabytes to terabytes of data) to just call em.clear(), or even better, load detached entities from the database in the first place than to start new transactions and entity managers all the time.

Michael
________________________________________
From: eclipselink-users-bounces@xxxxxxxxxxx [eclipselink-users-bounces@xxxxxxxxxxx] On Behalf Of Christopher Delahunt [christopher.delahunt@xxxxxxxxxx]
Sent: Friday, September 30, 2011 9:48 PM
To: EclipseLink User Discussions
Subject: Re: [eclipselink-users] memory leak when using caches?

It depends.

If you load an infinite amount of objects in memory, you will eventually
run out of memory at a certain point if you are keeping references to
those objects.  As long as you are reference objects, they cannot be
collected by the garbage collector.   It maybe that adding memory to the
JVM; for instance if you need to bring in the entire database into the
application.

The EntityManager though is required to keep references to managed
entities. Detached is what is meant when the entityManager does not have
a reference to the entity itself (though with serialization of the
entity, it still has its own copy of that entity).  Calling clear
detaches references by removing the EntityManager's links to those
Entities - they are free to for garbage collection as long as the
application is also not referencing them.
The wikibooks topic describing clear might be useful to you:
http://en.wikibooks.org/wiki/Java_Persistence/Persisting#Clear

The dontMaintainCache() option on the ReadQuery is related to the
eclipselink.maintain-cache query hint described here:
http://wiki.eclipse.org/EclipseLink/UserGuide/JPA/Basic_JPA_Development/Query_Hints#Maintain_Cache.

It will return detached entities that you will need to merge into the
EntityManager should you need to make changes.

You might want to read the EclipseLink docs on caching found here:
http://wiki.eclipse.org/EclipseLink/Examples/JPA/Caching

Why are you loading in hundreds of thousands of objects in a single
EM/transaction? Can you read and process it in batches, and clear the EM
after each batch so the objects can be released?


Best Regards,
Chris

On 30/09/2011 2:40 PM, Michael Pflueger wrote:
> Hi,
>
> when reading lots(hundreds of thousands) of objects from a table (within a single entity manager/transaction), I seem to get memory leaks (heap usage is growing to 250MB/MAX quickly and soon thereafter I get an OutOfMemory Error).
> This happens even when I set eclipselink.cache.type.default to NONE in my persistence.xml.
>
> What does help is either using raq.dontMaintainCache(); on the query, or repeatedly calling em.clear(); on the entity manager. Can anybody explain this behavior?
> I already had this problem with openjpa and it is somewhat annoying to see outofmemory errors here again :)
>
> Regards,
> Michael
> ___________________________________________________
>
> SMA Solar Technology AG
> Aufsichtsrat: Guenther Cramer (Vorsitzender)
> Vorstand: Juergen Dolle, Roland Grebe, Uwe Hertel, Pierre-Pascal Urbon, Marko Werner
> Handelsregister: Amtsgericht Kassel HRB 3972
> Sitz der Gesellschaft: 34266 Niestetal
> USt-ID-Nr. DE 113 08 59 54
> WEEE-Reg.-Nr. DE 95881150
> ___________________________________________________
>
> _______________________________________________
> eclipselink-users mailing list
> eclipselink-users@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/eclipselink-users
>
_______________________________________________
eclipselink-users mailing list
eclipselink-users@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/eclipselink-users