Bug 112967 - poor performance on large documents
Summary: poor performance on large documents
Status: CLOSED FIXED
Alias: None
Product: WTP Source Editing
Classification: WebTools
Component: wst.xsd (show other bugs)
Version: 0.7.1   Edit
Hardware: PC Windows XP
: P2 major (vote)
Target Milestone: 1.5.1 M151   Edit
Assignee: Valentin Baciu CLA
QA Contact:
URL:
Whiteboard: reviewed_1.5
Keywords: performance
Depends on:
Blocks:
 
Reported: 2005-10-18 13:33 EDT by Franklin de Graaf CLA
Modified: 2007-02-08 11:27 EST (History)
6 users (show)

See Also:


Attachments
XML Schema for (Mozilla) XUL documents (106.87 KB, text/plain)
2005-10-18 22:16 EDT, Franklin de Graaf CLA
no flags Details
Comment changed stack trace (3.50 KB, text/plain)
2006-08-04 03:27 EDT, Valentin Baciu CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Franklin de Graaf CLA 2005-10-18 13:33:18 EDT
On an XML Schema of about 2000 lines any changes slow the performance of the 
editor down to a crawl. typing of a single character may result in a response 
with a 5 second delay. cut and paste operations may result in as much as a 5 
minute delay.
Comment 1 David Williams CLA 2005-10-18 15:13:00 EDT
Thanks ... and our favorite reply ... any chance you can attach a 
a document, or link to a public one? Its seldom size, per se, 
so want to be sure we have a good test. 

thanks,
Comment 2 Franklin de Graaf CLA 2005-10-18 22:16:31 EDT
Created attachment 28420 [details]
XML Schema for (Mozilla) XUL documents

Validation of this XML Schema (xsd) file indicates it has errors. This is as
expected.
Comment 3 Franklin de Graaf CLA 2005-10-18 22:39:56 EDT
An example edit that demonstrates the poor performance: try commenting out a
block of code by placing an opening comment marker ("<!--") before a before an
opening tag and placing a closing comment marker ("-->") after the corresponding
closing tag.

This particular edit task identifies a couple of possible enhancements: (1)
select a block of code and with a single command - menu selection or key code,
bracket the code with opening and closing comment markers. I will enter this
suggestion as a separate bug.
Comment 4 David Williams CLA 2005-10-18 23:06:39 EDT
FYI ... the "toggle comment" action is already there ... see "Source" menu. 

(We'll still look at performance problem). 
Comment 5 David Williams CLA 2005-12-01 01:48:43 EST
By WTP convention, I'm marking as P4 indicating this critical defect will not be fixed for Release 1.0. I think since we have the "toggle comment" function there, this bug is not critical anyway (major, sure). 

We will investgate perrmance improvments for 1.5. 

Comment 6 Arthur Ryman CLA 2005-12-07 10:44:00 EST
Franklin, please use the Bugzilla definitions of severity:

Blocker  	Blocks development and/or testing work
Critical 	crashes, loss of data, severe memory leak
Major 	        major loss of function

This one is Major because it renders the editor fairly unusable. I am setting the severity accordingly.
Comment 7 Craig Salter CLA 2006-08-03 14:00:57 EDT
Valentin, can you take a look and see if this is a XSD issues of an SSE issue?  I suspect it's a bit of both so maybe there's something we can do on the XSD side.
Comment 8 Valentin Baciu CLA 2006-08-03 16:28:40 EDT
As a simple test to determine whether the problem is in the XSD editor or somewhere else I tried typing comments in large schema files opened in both the XML and XSD editors. The XSD editor is definitely slower - but also does much more than the XML editor. 

With a little profiling and code inspection I found the following: the SSE editor sends notifications for every change in the source. The XSD editor listens to these changes and attempts to keep the XSD EMF model in synch (we call this 'to reconcile') - which is quite expensive. This is normal and desired for things that the EMF model is concerned with - like things in the schema namespace - but not for things like comments and xsd:appinfo and xsd:documentation contents (thanks to Craig for pointing these out). It just so happens that these types of nodes are the ones where the user does most of the typing. 

For the appinfo and documentation content the fix could be quite simple: we can look at the element's namespace or check its parent and avoid reconciling the EMF model.

Comments seem to be a bit more tricky: for every keystroke typed inside a comment we get notified three times: 
- once that the comment node is removed, 
- once that the comment node is added with the new value and 
- once that the contents of the parent node (let's say we added a comment inside an xsd:schema element) has changed. 

The first two can be handled properly by the EMF model reconciler and avoid reconciling the model by looking at the feature or oldValue/newValue event properties.

The third unfortunately only says that the parent element's content has changed and does not offer additional information. It is the third notification on which we rely to reconcile the model and because it does not seem to tell us that it was a change to a comment which triggered it - we are forced to trigger the reconciliation to be on the safe side.

David, is there any way the third notification could provide more information about what the change was about? Or some other way for us to detect that the change is triggered by a comment change? Or by adding only text nodes (speeding up during format operations)?

Just before anyone suggests it - we are also considering batching changes and doing bulk reconciliation - but that in itself is fairly tricky and can lead to weird side effects.
Comment 9 David Williams CLA 2006-08-03 16:55:24 EDT
Here's a couple of suggestions/questions to get you a little further... 

What's that third notification? A "structured changed" or a "content changed"? 
Its supposed to be the latter, and that's supposed to be your signal that an EMF reconcile should not be needed -- I only say "supposed to be" because I don't remember testing or designing this for schema's specifically ... mostly XML, so, I hope it's working there too. 

If that's not working (and couldn't be fixed) then one approach might be for you to also listen to "about to change model" and "changed model" events. I'd have to double check, but, as I recall, we've gotten the design to the point where "notifiiedChange" events are always ~bracketed~ by the "about" and "changed" events, so that clients can keep track of if they are in a "change" window, and just set flags for what they may eventually want to do when the "changed" event finally comes through. 

So, let's start there ... and see what's next. 
Comment 10 Valentin Baciu CLA 2006-08-04 03:26:51 EDT
Thanks for the tips David. 

It seems that the third notification is a STRUCTURE_CHANGED and not a CONTENT_CHANGED. 

I have started to look at the XML core code to understand what the "change" event types really mean but I would appreciate it if you can point me to any existing information about the CHANGE, STRUCTURE_CHANGED and CONTENT_CHANGED event types, hopefully with some examples. For example it seems CHANGE is sent when an element's attribute changes.

I am attaching next a stack trace I got by sticking a breakpoint in XMLModelNotifierImlp. I noticed that in XMLModelParser#changeData comments are treated the same as elements...which probably explains why they trigger the structure changed notification.

...
    case Node.COMMENT_NODE :
    case Node.ELEMENT_NODE :
	// comment tag
	changeStructuredDocumentRegion(flatNode);
	return;
...

Anyway, over to you for comments for now.
Comment 11 Valentin Baciu CLA 2006-08-04 03:27:41 EDT
Created attachment 47385 [details]
Comment changed stack trace

Here's the stack trace I mentioned.
Comment 12 David Williams CLA 2006-08-11 02:45:03 EDT
No, no docs that would help you. And, no, I don't think structure changed can be changed (I think I was wrong about the content changed ... that's probably only for text nodes). 

And, actually, its one of the "special features" of our DOM Impl that comments can be treated as elements ... this is important for what we call "comment elements". (which is a currently undocumented extension point, but one we need to be API eventually). 

But ... seems like we could be smarter about the "add and remove", just for typing inside of a comment ... but ... right off, that sounds like a pretty big item :) ... probably no quick fix ... but, not impossible. 

So ... if you see a fix for THAT! let me know! 
Comment 13 Keith Chong CLA 2006-08-17 14:44:07 EDT
Valentin, attach a patch for what you said you can fix, close off this bug, and open any remaining problems that still need to be addressed.
Comment 14 Keith Chong CLA 2006-09-07 11:15:45 EDT
Patch to improve performance changing annotation content has been checked in for 1.5.1.
Comment 15 David Williams CLA 2006-09-07 13:12:52 EDT
Keith, can you, as you suggested, close this one as fixed in 151, and open a new bug for the additional work. 

Comment 16 Keith Chong CLA 2006-09-07 13:15:37 EDT
Sure.  Bug set to resolved, fixed.
Comment 17 Keith Chong CLA 2006-09-07 13:42:34 EDT
See bug 156564
Comment 18 Steven Wasleski CLA 2006-09-12 11:58:15 EDT
I am confused by comment 14, comment 15 and the target milestone.  What was fixed in 1.5.1 and what is left for 1.5.2?
Comment 19 Valentin Baciu CLA 2006-09-12 12:49:15 EDT
I provided a work-in-progress patch to Keith for review. The patch contained proposed fixes for speeding up typing inside xsd:annotation elements as well as typing inside text and comment nodes.
By looking at the CVS history for org.eclipse.wst.xsd.ui.internal.text.XSDModelReconcileAdapter and org.eclipse.wst.xsd.ui.internal.util.ModelReconcileAdapter it appears that the only changes committed are the ones for annotations and text nodes. The comment nodes fix did not make it.

In addition to avoiding DOM->EMF reconciliation as much as possible, we also worked with the XSD team to speed up the XSD EMF model reconciliation for large documents - see https://bugs.eclipse.org/bugs/show_bug.cgi?id=154290

I believe the milestone is now wrong - at some point in time we were talking about postponing this for 1.5.2. I think the target milestone for this bug should be changed back to 1.5.1.
Comment 20 Valentin Baciu CLA 2007-02-08 11:27:47 EST
Closing old bugs.