Re: [eclipselink-users] Efficient loading to minimise memory spike

[James Sutherland <jamesssss@xxxxxxxxx>]

You can use pagination in JPA to limit query results.  See
Query.setFirstResult(), setMaxResults().

EclipseLink also supports Cursors, streams and scrolling.

See,
http://en.wikibooks.org/wiki/Java_Persistence/Querying#Pagination.2C_Max.2FFirst_Results

In terms of the copies of the object maintained by the persistence context
their can be 3.
1 - Is the working copy that is returned that you can edit.
2 - Is the original cache object in the shared object cache, if you disable
caching (shared=false) this object will not be built.  Alternatively, if you
are just reading an object, you could use the Query hint
"eclipselink.read-only" to only build and return this original object.
3 - The backup clone is used to track changes.  When an JVM agent is used or
weaving is enabled, no backup clone is built by default as attribute level
change tracking is used through instrumented events into your get/set
methods.  Some types of mappings however require "deferred" change tracking,
such as if your mapping for your lob used serialization to a complex object. 
This can be controlled through the @Mutable and @ChangeTracking annotations.


Brad Milne wrote:
> 
> Using Eclipselink v1.0.2 with JPA
> 
> I have a thick client that displays database records that each have 
> associated byte arrays up to around 26k. This field is lazily loaded, so 
> not all the arrays are loaded into memory at query time.
> 
> I had severe memory problems a few months ago, as Eclipselink keeps 2 or 
> more copies of each record as part of the persistence context, etc. The 
> solution was to detach all records from the context on load 
> (entityManager.clear()), then each record is merged 
> (entityManager.merge(record)) when selected, then re-detached again when 
> moving to the next. This (seemingly-ugly) solution has worked well.
> 
> A problem I'm encountering now, however, is when I have such large 
> numbers of records in a query, that even during the initial load (ie 
> before the associated byte arrays are lazily-loaded in), the memory 
> requirements are huge, exceeding reasonable limits. It seems a logical 
> approach would be to load the records in chunks, then use the detach and 
> merge as described above when needed. Is there an elegant/api way to 
> achieve this? I have heard of some mechanics for this in other 
> persistence frameworks.
> 
> Many thanks
> Brad
> 


-----
http://wiki.eclipse.org/User:James.sutherland.oracle.com James Sutherland 
http://www.eclipse.org/eclipselink/
 EclipseLink ,  http://www.oracle.com/technology/products/ias/toplink/
TopLink 
Wiki:  http://wiki.eclipse.org/EclipseLink EclipseLink , 
http://wiki.oracle.com/page/TopLink TopLink 
Forums:  http://forums.oracle.com/forums/forum.jspa?forumID=48 TopLink , 
http://www.nabble.com/EclipseLink-f26430.html EclipseLink 
Book:  http://en.wikibooks.org/wiki/Java_Persistence Java Persistence 
-- 
View this message in context: http://old.nabble.com/Efficient-loading-to-minimise-memory-spike-tp27351921p27421657.html
Sent from the EclipseLink - Users mailing list archive at Nabble.com.