[news.eclipse.technology.dltk] Re: Semantic checking - via validators?

["Chuck Doucette" <cdoucette@xxxxxxxxxxx>]

Based on what I found below, here are my conclusions:
We should find the smallest/earliest place that we can change that is called 
from the editor reconciliation and ideally no place else.
I see two reasonable solutions:

1) Overriding m), i.e. AbstractSourceElementParser.parseSourceModule, since 
we already have a class which extends this. It would do the following:

public void parseSourceModule(char[] contents, ISourceModuleInfo astCache, 
char[] filename) {
super.parseSourceModule(contents, astCache, filename);
// do semantic checking
}

The only problem with this approach is that it is also called by 
SourceIndexer.index.

2) Override VScriptSourceViewerConfiguration.getReconciler, and specifically 
to add child class of ScriptReconcilingStrategy which implements reconcile. 
It would consist of the following:

protected void reconcile(final boolean initialReconcile) {
super.reconcile(initialReconcile);
// do semantic checking
}

This would not affect SourceIndexer.index, and it is close to the 
recommended strategy of implementing my own recognizer, but it builds off of 
the existing recognizer instead but is *very* dependent on the current 
implementation so as to change/implement as few lines as possible.

Chuck

"Chuck Doucette" <cdoucette@xxxxxxxxxxx> wrote in message 
news:g3plql$hkg$1@xxxxxxxxxxxxxxxxxxxx
> As I said in another reply, I have created a semantic checker, and I am 
> trying to determine where/how to invoke it.
>
> In addition to invoking it in a builder, we also wanted to invoke it 
> dynamically in the editor, so syntax and semantic errors show up 
> immediately (just like in JDT). Thus, I believe that we should try to 
> follow your advice to invoke it via the reconciler.
>
> In the process of trying to determine exactly where/how to do that, I 
> found out many things that I want to share with other users who may be 
> trying to do the same thing.
>
> Here's what I have found:
>
> a) My (concrete) class VScriptSourceViewerConfiguration extends the DLTK 
> abstract class ScriptSourceViewerConfiguration which defines this method:
>
> /**
> * Returns the reconciler ready to be used with the given source viewer.
> * This implementation always returns <code>null</code>.
> *
> * @param sourceViewer the source viewer to be configured by this 
> configuration
> * @return a reconciler or <code>null</code> if reconciling should not be 
> supported
> */
> public IReconciler getReconciler(ISourceViewer sourceViewer) {
>
> b) The DLTK class ScriptReconciler implements IReconciler. Its super 
> class, MonoReconciler depends on an instance of IReconcilingStrategy to do 
> the reconciliation.
>
> c) The method getReconciler above creates an instance of the DLTK class 
> ScriptCompositeReconcilingStrategy whose superclass, 
> CompositeReconcilingStrategy, implements IReconcilingStrategy. It also 
> depends on an instance of the more straight forward class 
> ScriptReconcilingStrategy.
>
> d) ScriptReconcilingStrategy depends on ISourceModule.reconcile
>
> e) SourceModule extends AbstractSourceModule (both of which implements 
> ISourceModule)
> The reconcile method calls ReconcileWorkingCopyOperation.runOperation, 
> which does this:
>
> if (isReadOnly()) {
> run(monitor);
> } else {
> // Use IWorkspace.run(...) to ensure that a build will be done
> // in autobuild mode.
> // Note that if the tree is locked, this will throw a
> // CoreException, but this is ok
> // as this operation is modifying the tree (not read-only) and a
> // CoreException will be thrown anyway.
> IWorkspace wc = ResourcesPlugin.getWorkspace();
> wc.run(this, getSchedulingRule(), IWorkspace.AVOID_UPDATE, monitor);
>
> f) IWorkspace.run calls IWorkspaceRunnable.run, and that is defined by 
> ModelOperation.run, which calls executeOperation.
>
> g) ReconcileWorkingCopyOperation.executeOperation calls makeConsistent, 
> which calls SourceModule.makeConsistent:
>
> h) SourceModule.makeConsistent does this:
>
> // Remove AST Cache element
> ISourceModuleInfoCache sourceModuleInfoCache = ModelManager
> .getModelManager().getSourceModuleInfoCache();
> sourceModuleInfoCache.remove(this);
> openWhenClosed(createElementInfo(), monitor);
>
> i) ModelElement.openWhenClosed calls Openable.generateInfos:
>
> /**
> * Generates the element infos for this element, its ancestors (if they are
> * not opened) and its children (if it is an Openable). Puts the newly
> * created element info in the given map.
> */
> protected abstract void generateInfos(Object info, HashMap newElements,
> IProgressMonitor pm) throws ModelException;
>
> j) Openable.generateInfos calls AbstractSourceModule.buildStructure:
>
> /**
> * Builds this element's structure and properties in the given info object,
> * based on this element's current contents (reuse buffer contents if this
> * element has an open buffer, or resource contents if this element does 
> not
> * have an open buffer). Children are placed in the given newElements table
> * (note, this element has already been placed in the newElements table).
> * Returns true if successful, or false if an error is encountered while
> * determining the structure of this element.
> */
> protected abstract boolean buildStructure(OpenableElementInfo info,
> IProgressMonitor pm, Map newElements, IResource underlyingResource)
> throws ModelException;
>
> k) AbstractSourceModule.buildStructure calls 
> SourceParserUtil.parseSourceModule
>
> l) SourceParserUtil.parseSourceModule calls 
> AbstractSourceElementParser.parseSourceModule
>
> m) AbstractSourceElementParser.parseSourceModule calls 
> SourceParserUtil.getModuleDeclaration:
>
> n) public static ModuleDeclaration getModuleDeclaration(char[] filename,
> char[] content, String nature, IProblemReporter reporter,
> ISourceModuleInfo mifo, int flags) {
> ISourceParser sourceParser;// = new SourceParser(this.fReporter);
> sourceParser = DLTKLanguageManager.getSourceParser(nature);
> if (sourceParser instanceof ISourceParserExtension) {
> ((ISourceParserExtension) sourceParser).setFlags(flags);
> }
> ModuleDeclaration moduleDeclaration = SourceParserUtil
> .getModuleFromCache(mifo, flags);
> if (moduleDeclaration == null) {
> if (reporter != null) {
> reporter.clearMarkers();
> }
> moduleDeclaration = sourceParser.parse(filename, content, reporter);
> SourceParserUtil.putModuleToCache(mifo, moduleDeclaration, flags);
> }
> return moduleDeclaration;
> }
>
> o) My class VScriptSourceParser extends AbstractSourceParser which 
> implements ISourceParser.
> The parse method instantiates a vscriptParser and uses it.
>
> NOTES:
>
> o) ISourceParser.parse is also called by:
>    i) AbstractASTFoldingStructureProvider.getCodeBlocks, which is 
> ultimately called     by:AbstractASTFoldingStructureProvider.initialized 
> and 
> AbstractASTFoldingStructureProvider.ElementChangedListener.elementChanged
>
> m) AbstractSourceElementParser.parseSourceModule is also called by:i) 
> SourceIndexer.indexDocument
>
> n/p) SourceParserUtil.getModuleDeclaration (different from the 
> getModuleDeclaration above):
> public static ModuleDeclaration getModuleDeclaration(ISourceModule module,
> IProblemReporter reporter) {
> is called from these places:
>
> i) VScriptCompletionEngine.locateVScriptContext(SourceModule, int)
> ii) VScriptSelectionEngine.select(ISourceModule, int, int)
> iii) MatchLocatorParser.parse(PossibleMatch)
> iv) SemanticPositionUpdater.reconcile(ISourceModule sourceModule, 
> SemanticHighlightingPresenter presenter, Highlighting[] highlightings, 
> List currentPositions)
>
> "Andrei Sobolev" <haiodo@xxxxxxxxx> wrote in message 
> news:g09mhq$9i2$1@xxxxxxxxxxxxxxxxxxxx
>> Hi Chuck,
>>
>> Nice to see, what DLTK help you.
>>
>> We don't have something special in area of semantic analysis, etc.
>>
>> Where is some ways to implement such features:
>> 1) Using of ScriptBuilder.
>> In DLTK we have feature named ScriptBuilder. This is incremental resource 
>> builder, used for building Mixin models, etc.
>> It will give you list of changed resources, and you cold check all this 
>> files and set appropriate markers.
>> All Eclipse markers will be show in DLTK Text editors. ScriptBuilder will 
>> contain information about builded resources,
>> and will be executed on each resource operations, like "resource added", 
>> "resource content changed".
>>
>> 2) You could use Validator, Validators framework uses Script Builder for 
>> incremental execution of validators. One more
>> additional point, validators framework manages validator instances and 
>> allow unified way to sore configurations, etc.
>> Also user could turn on/off and execute validators from DLTK UI.
>>
>> 3) If you also plan to make semantic checks for file edited, you need to 
>> extend Reconciler from your
>> ScriptSourceViewerConfiguration class. Reconciler will rebuild model, and 
>> do checks, after user will make some changes
>> in code. By default it is 500 milliseconds delay, so reconciler will be 
>> executed only if user not type code 500
>> milliseconds.
>>
>>
>> References:
>> 1) Extension point to implement: org.eclipse.dltk.core.builder
>> 2) We have "package require" checker implemented for Tcl. 
>> (TclCheckBuilder class).
>> 3) ScriptSourceViewerConfiguration.getReconciler()
>>
>> Best regards,
>> Andrei Sobolev.
>>
>>
>>> Thanks to ANTLR and DLTK, I now have an advanced editor which can
>>> immediately flag syntactic problems in our language source files.
>>>
>>> My question is if/when/how should I find/flag semantic errors in the 
>>> source
>>> file?
>>>
>>> 1.    Is there any part of DLTK framework which could/should be used to
>>> invoke/perform semantic checks?    I assume the answer to this is no. I 
>>> know
>>> you support external validators - but we'd like the semantic checking to 
>>> be
>>> performed implicitly.
>>>
>>> 2.    When should semantic checks be performed - whenever the parser is
>>> invoked?
>>>
>>> 3.    How should errors be reported - using the parser's error reporter?
>>>         Also, how can we determine the line number for populating 
>>> IProblem
>>> in the file if that information is:
>>>         a) not available in DLTKToken
>>>         b) not kept in ASTNode
>>>
>>> Thanks,
>>> Chuck
>>>
>>>
>
>