Re: [tracecompass-dev] Question about scaling of data driven analysis

[Matthew Khouzam <matthew.khouzam@xxxxxxxxxxxx>]

Hi Robbert,

You have hit a known bug then... here is the link on bugzilla

https://bugs.eclipse.org/bugs/show_bug.cgi?id=460261


On 16-06-22 06:50 AM, Robbert Jongeling wrote:
> Hi Loïc and Matthew,
>
> Thank you for your responses.
>
> I have indeed noticed that opening the trace without the analysis
> attached is much faster.
> It takes about 50 seconds to index the 2.2 million events, however
> when the analysis is attached, it takes too long to wait for (it seems
> to index only hundreds of events per second compared to thousands or
> even tens of thousands when no analysis is attached).
> After applying the patch you both suggested, creating the view based
> on my analysis still takes too long to wait for.
> Indexing of the trace when the analysis is attached is also still a
> lot slower than when it is not attached.
>
> (By attached I mean the following:
> I currently have an analysis file in which I define a stateprovider
> and a timegraphview.
> Indexing the trace without the timegraphview opened is normal, quick,
> but indexing the trace while the timegraphview is already opened takes
> very long. 
> In both cases, the view is imported in Trace Compass.)
>
> The patch does improve the performance.
> I have profiled two runs of TraceCompass, one without the patch and
> one with the patch.
> In these runs I have first opened the trace and after indexing opened
> my custom timegraphview. 
> At that time, I have started the profiling.
> This does show a significant improvement of time spent on historytree
> related functions, 
> but not on the HTNode.readNode method, which takes up the majority of
> the time in both cases.
> Which, I think, is caused by the large amount of attributes on a level
> in the state system.
>
> When I run my analysis on a different version of the trace, in which
> the top level attribute of the state system contains not thousands of
> attributes but just one, generation of the view is almost instantanious.
> Is it possible to somehow group the attributes in order to make the
> generation faster?
>
>
> Kind regards,
>
> Robbert
>
> ------------------------------------------------------------------------
> *Van:* tracecompass-dev-bounces@xxxxxxxxxxx
> <tracecompass-dev-bounces@xxxxxxxxxxx> namens Matthew Khouzam
> <matthew.khouzam@xxxxxxxxxxxx>
> *Verzonden:* dinsdag 21 juni 2016 21:15:08
> *Aan:* tracecompass-dev@xxxxxxxxxxx
> *Onderwerp:* Re: [tracecompass-dev] Question about scaling of data
> driven analysis
>  
> Hi Robbert!
>
> Quick question, can you load the 200mb trace easily if there is no
> analysis? (remove the analysis from tc)
>
> if that is the case, the problem lies in the state system, perhaps this
> patch can fix it. https://git.eclipse.org/r/#/c/61062/
>
> Please keep us posted,
>
> Br,
>
> Matthew
>
> On 16-06-21 09:06 AM, Robbert Jongeling wrote:
> > Dear Trace Compass developers,
> >
> > I am trying to apply Trace Compass to custom, text-based traces and to
> > this end have created a parser and an xml-analysis.
> > My question relates to the scaling of the application or the analysis
> > to larger traces.
> > In case of a large trace (200MB) my analysis takes about 2 hours to
> > complete, whereas for small traces (10MB) it takes seconds.
> > What can I do to increase the performance of my xml analysis?
> >
> > I have noticed that the issue may be caused because of the size of my
> > state system structure, which is about 4 levels deep but on each level
> > there may be >1k different entries. E.g.:
> >
> > |- trace
> > || - thingid1
> > || - subthingid 1
> > || - somevalue
> > |...
> > || - thigid1000
> > || - subthingid 1
> > |...
> > || - subthingid 1000
> > || - someOtherValue
> >
> >
> > I have noticed that the getRelevantInterval method in
> >
> org.eclipse.tracecompass.internal.statesystem.core.backend.historytree.HTNode
> > takes up quite a lot of time.
> > Also, I found this bug
> > https://bugs.eclipse.org/bugs/show_bug.cgi?id=492525, which discusses
> > that CTF traces can be packetized.
> > Is there a similar way to handle custom (text-based) traces?
> >
> > I would greatly appreciate any pointers you can give me.
> >
> >
> > Kind regards,
> >
> > Robbert
> >
> >
> >
> > _______________________________________________
> > tracecompass-dev mailing list
> > tracecompass-dev@xxxxxxxxxxx
> > To change your delivery options, retrieve your password, or
> unsubscribe from this list, visit
> > https://dev.eclipse.org/mailman/listinfo/tracecompass-dev
>
> _______________________________________________
> tracecompass-dev mailing list
> tracecompass-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or
> unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/tracecompass-dev
>
>
> _______________________________________________
> tracecompass-dev mailing list
> tracecompass-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/tracecompass-dev