[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
RE: [platform-core-dev] How much memory can a Workspace take?
|
Hi James,
IResource are just temporary handle objects, so whatever they
retain won't be relevant here. It's the ElementTree that counts.
But you are right that at the place where currently
path.lastSegment();
is called, it may help to do
new String(path.lastSegment());
instead to make sure that we really hold onto the last segment only
and don't get bitten by whatever byte[] array the internal String
representing the last segment still holds on to.
Let's continue the disucssion on the bug, as Szymon suggested.
Cheers,
--
Martin Oberhuber, Senior Member of Technical Staff, Wind River
Target Management Project Lead, DSDP PMC Member
http://www.eclipse.org/dsdp/tm
> -----Original Message-----
> From: platform-core-dev-bounces@xxxxxxxxxxx
> [mailto:platform-core-dev-bounces@xxxxxxxxxxx] On Behalf Of
> James Blackburn
> Sent: Tuesday, September 29, 2009 5:17 PM
> To: Eclipse Platform Core component developers list.
> Subject: Re: [platform-core-dev] How much memory can a Workspace take?
>
> By taking a quick pass with the Call Hierarchy view. It looks like the
> main entry point for creating the DataTreeNodes is
> ElementTree#createElement(IPath key, Obj data).
>
> This calls tree.createChild(parent, key.lastSegment(), data); which
> could be the source of the issue. IPath.lastSegment() returns the last
> segment the segments having been created by substring...
>
> As each IResource has it's fullPath as a full IPath, the result is
> that the DataTree contains yet another handle on the Resource's
> FullPath. If the Resource is re-created the DataTree still has a
> handle to the old fullpath string, for every item in the tree.
>
> Cheers,
> James
>
> 2009/9/29 Oberhuber, Martin <Martin.Oberhuber@xxxxxxxxxxxxx>:
> > Hi again,
> >
> > With some expert help from the MAT people, I think we've been
> > able to detect a suspect responsible for the problem in my case.
> >
> > Here is what we did:
> > 1. In memory analyzer, allow the default report ("wizard") to run.
> > 2. Expand details of first problem suspect.
> > 3. On first item (AbstractDataTreeNode[174] @ 0x97887040 :
> > - click > Show retained set.
> > --> 209000 objects of type char[] take 66MB
> > 4. On char[]: right-click > Java Basics > Group by Value
> > --> See 209000 Strings, all of which have the same prefix
> > -->
> > "/net/pek-wb-build1/buildarea/kcao/182581/build/linux/wrs/patches"
> >
> > At this point, it seems clear that there is room for optimization.
> > Why does the workspace retain such long Prefixes? An ElementTree
> > in the Workspace should always only hold the basename of an element
> > in each node.
> >
> > 5. Pick the first long char[].
> > Right-click > List Objects > With incoming references
> >
> > At this point, I am getting attached screenshot: It looks like the
> > long char[] array is held by a String with value
> > "fix-swapped-nibbles-in-tracer"
> > So although the value of that String is only a basename, that String
> > still retains a very long char[] array with the entire path!
> >
> > It looks like this is a peculiarity of the Java String.substring()
> > operation, which retains the entire original char[] array although
> > the String now only references the trailing part of it.
> >
> > Unfortunately the memory analyzer cannot tell us who performs the
> > substring() operation that still retains the large char[] array.
> > But it looks like it should be possible to optimize away my memory
> > problem by replacing
> >
> > fullPath.substring(lastSlashIdx);
> >
> > by
> >
> > new String(fullPath.substring(lastSlashIdx);
> >
> > by explicitly performing a new String() operation at this point, the
> > original substring which retains the char[] array will be released
> > for garbage collection, and the new String will only retain the
> > desired substring.
> >
> > Fellow committers, do you have an idea where that
> substring() operation
> > could be? I suspect somewhere in UnifiedTree from the
> refresh operation,
> > but I am not sure...
> >
> >
> > Cheers,
> > --
> > Martin Oberhuber, Senior Member of Technical Staff, Wind River
> > Target Management Project Lead, DSDP PMC Member
> > http://www.eclipse.org/dsdp/tm
> >
> >
> >
> >> -----Original Message-----
> >> From: platform-core-dev-bounces@xxxxxxxxxxx
> >> [mailto:platform-core-dev-bounces@xxxxxxxxxxx] On Behalf Of
> >> Oberhuber, Martin
> >> Sent: Tuesday, September 29, 2009 2:12 PM
> >> To: Eclipse Platform Core component developers list.
> >> Subject: RE: [platform-core-dev] How much memory can a
> Workspace take?
> >>
> >> Hi Szymon, James -
> >>
> >> my hprof.zip is at
> >> http://dl.getdropbox.com/u/1975868/java_pid11610.zip
> >>
> >> The ZIP is 68MB large (the 200MB had also included the entire
> >> .metadata with some large CDT pdom's).
> >>
> >> As mentioned this was with Eclipse-3.5.0 on Linux-GTK
> RHEL4 (32-bit).
> >> The problem happened directly after launching, during a workspace
> >> refresh that was initiated because a previous shutdown was unclean.
> >>
> >> Here are some probably interesting bits from the .log
> file. It looks
> >> like the machine was very busy at that time -- it took 6 minutes
> >> from initial Launch till refreshing the workspace, and it took 19
> >> minutes from starting the WS refresh till running into the
> >> OutOfMemoryError:
> >>
> >> !SESSION 2009-09-28 12:19:25.891
> >> -----------------------------------------------
> >> eclipse.buildId=unknown
> >> java.version=1.5.0_11
> >> java.vendor=Sun Microsystems Inc.
> >> BootLoader constants: OS=linux, ARCH=x86, WS=gtk, NL=en_US
> >> Command-line arguments: -os linux -ws gtk -arch x86 -data
> >> /folk/hxu2/WindRiver/workspace2
> >>
> >> !ENTRY org.eclipse.core.resources 2 10035 2009-09-28 12:25:34.028
> >> !MESSAGE The workspace exited with unsaved changes in the previous
> >> session; refreshing workspace to recover changes.
> >>
> >> !ENTRY org.eclipse.core.resources 4 1 2009-09-28 12:44:01.498
> >> !MESSAGE Internal Error
> >> !STACK 0
> >> java.lang.OutOfMemoryError: Java heap space
> >>
> >> Cheers,
> >> --
> >> Martin Oberhuber, Senior Member of Technical Staff, Wind River
> >> Target Management Project Lead, DSDP PMC Member
> >> http://www.eclipse.org/dsdp/tm
> >>
> >>
> >>
> >> > -----Original Message-----
> >> > From: platform-core-dev-bounces@xxxxxxxxxxx
> >> > [mailto:platform-core-dev-bounces@xxxxxxxxxxx] On Behalf Of
> >> > Szymon Brandys
> >> > Sent: Tuesday, September 29, 2009 1:28 PM
> >> > To: Eclipse Platform Core component developers list.
> >> > Subject: Re: [platform-core-dev] How much memory can a
> >> Workspace take?
> >> >
> >> > Hi Martin,
> >> > Could I get the .hprof file?
> >> > --
> >> > Szymon
> >> >
> >> >
> >> >
> >> >
> >> > "Oberhuber,
> >> >
> >> > Martin"
> >> >
> >> > <Martin.Oberhuber
> >> > To
> >> > @windriver.com> "Eclipse Platform Core
> >> > component
> >> > Sent by: developers list."
> >> >
> >> > platform-core-dev
> >> > <platform-core-dev@xxxxxxxxxxx>
> >> > -bounces@eclipse.
> >> > cc
> >> > org
> >> >
> >> >
> >> > Subject
> >> > [platform-core-dev]
> >> > How much memory
> >> > 2009-09-29 12:30 can a Workspace take?
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Please respond to
> >> >
> >> > "Eclipse Platform
> >> >
> >> > Core component
> >> >
> >> > developers list."
> >> >
> >> > <platform-core-de
> >> >
> >> > v@xxxxxxxxxxx>
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Hi fellow committers,
> >> >
> >> >
> >> > I got a report about an OutOfMemoryError generated during a
> >> Workspace
> >> > Refresh operation. When looking at the .hprof file with the
> >> > MemoryAnalyzer,
> >> > it turned out that pretty much all the memory was taken by
> >> > core/Workspace
> >> > only:
> >> > 100 MB (53%) in
> org.eclipse.core.internal.resources.Workspace
> >> > (occupied by ElementTree).
> >> > 52 MB (27%) in
> >> > org.eclipse.core.internal.dtree.DeltaDataTree (this is
> >> > a different additional DeltaDataTree)
> >> > 37 MB all the rest.
> >> >
> >> >
> >> > The workspace was arguably very large, but still I am surprised
> >> > 1. that the numbers are that large
> >> > 2. that in addition to the Workspace, there is such a
> >> > large additional
> >> > DeltaDataTree,
> >> > 3. that I see an OutOfMemoryError reported with 189MB
> >> > total where I used
> >> > -vmargs -Xmx320m ... what happened to the extra 131MB ?
> >> >
> >> >
> >> > Do my numbers sound reasonable, or could there be a bug
> >> that leads the
> >> > Workspace to take an excessive amount of memory? How much
> >> memory do we
> >> > expect a workspace to take for, say 50000 files? How much
> >> *additional*
> >> > memory may a workspace delta take that is generated during a
> >> > Refresh? Would
> >> > any additional investigation using the Coretools make sense
> >> > at this point?
> >> >
> >> >
> >> > I can make the .hprof file available on request (it's a 200MB
> >> > ZIP file).
> >> >
> >> >
> >> > Many thanks for any pointers!
> >> >
> >> >
> >> > Cheers,
> >> > --
> >> > Martin Oberhuber, Senior Member of Technical Staff, Wind River
> >> > Target Management Project Lead, DSDP PMC Member
> >> > http://www.eclipse.org/dsdp/tm
> >> >
> >> > _______________________________________________
> >> > platform-core-dev mailing list
> >> > platform-core-dev@xxxxxxxxxxx
> >> > https://dev.eclipse.org/mailman/listinfo/platform-core-dev
> >> >
> >> >
> >> > _______________________________________________
> >> > platform-core-dev mailing list
> >> > platform-core-dev@xxxxxxxxxxx
> >> > https://dev.eclipse.org/mailman/listinfo/platform-core-dev
> >> >
> >> _______________________________________________
> >> platform-core-dev mailing list
> >> platform-core-dev@xxxxxxxxxxx
> >> https://dev.eclipse.org/mailman/listinfo/platform-core-dev
> >>
> >
> > _______________________________________________
> > platform-core-dev mailing list
> > platform-core-dev@xxxxxxxxxxx
> > https://dev.eclipse.org/mailman/listinfo/platform-core-dev
> >
> >
> _______________________________________________
> platform-core-dev mailing list
> platform-core-dev@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/platform-core-dev
>
