[news.eclipse.technology.dltk] Re: Semantic checking - via validators?

["Chuck Doucette" <cdoucette@xxxxxxxxxxx>]

As I said in another reply, I have created a semantic checker, and I am 
trying to determine where/how to invoke it.

In addition to invoking it in a builder, we also wanted to invoke it 
dynamically in the editor, so syntax and semantic errors show up immediately 
(just like in JDT). Thus, I believe that we should try to follow your advice 
to invoke it via the reconciler.

In the process of trying to determine exactly where/how to do that, I found 
out many things that I want to share with other users who may be trying to 
do the same thing.

Here's what I have found:

a) My (concrete) class VScriptSourceViewerConfiguration extends the DLTK 
abstract class ScriptSourceViewerConfiguration which defines this method:

/**
* Returns the reconciler ready to be used with the given source viewer.
* This implementation always returns <code>null</code>.
*
* @param sourceViewer the source viewer to be configured by this 
configuration
* @return a reconciler or <code>null</code> if reconciling should not be 
supported
*/
public IReconciler getReconciler(ISourceViewer sourceViewer) {

b) The DLTK class ScriptReconciler implements IReconciler. Its super class, 
MonoReconciler depends on an instance of IReconcilingStrategy to do the 
reconciliation.

c) The method getReconciler above creates an instance of the DLTK class 
ScriptCompositeReconcilingStrategy whose superclass, 
CompositeReconcilingStrategy, implements IReconcilingStrategy. It also 
depends on an instance of the more straight forward class 
ScriptReconcilingStrategy.

d) ScriptReconcilingStrategy depends on ISourceModule.reconcile

e) SourceModule extends AbstractSourceModule (both of which implements 
ISourceModule)
The reconcile method calls ReconcileWorkingCopyOperation.runOperation, which 
does this:

if (isReadOnly()) {
run(monitor);
} else {
// Use IWorkspace.run(...) to ensure that a build will be done
// in autobuild mode.
// Note that if the tree is locked, this will throw a
// CoreException, but this is ok
// as this operation is modifying the tree (not read-only) and a
// CoreException will be thrown anyway.
IWorkspace wc = ResourcesPlugin.getWorkspace();
wc.run(this, getSchedulingRule(), IWorkspace.AVOID_UPDATE, monitor);

f) IWorkspace.run calls IWorkspaceRunnable.run, and that is defined by 
ModelOperation.run, which calls executeOperation.

g) ReconcileWorkingCopyOperation.executeOperation calls makeConsistent, 
which calls SourceModule.makeConsistent:

h) SourceModule.makeConsistent does this:

// Remove AST Cache element
ISourceModuleInfoCache sourceModuleInfoCache = ModelManager
.getModelManager().getSourceModuleInfoCache();
sourceModuleInfoCache.remove(this);
openWhenClosed(createElementInfo(), monitor);

i) ModelElement.openWhenClosed calls Openable.generateInfos:

/**
* Generates the element infos for this element, its ancestors (if they are
* not opened) and its children (if it is an Openable). Puts the newly
* created element info in the given map.
*/
protected abstract void generateInfos(Object info, HashMap newElements,
IProgressMonitor pm) throws ModelException;

j) Openable.generateInfos calls AbstractSourceModule.buildStructure:

/**
* Builds this element's structure and properties in the given info object,
* based on this element's current contents (reuse buffer contents if this
* element has an open buffer, or resource contents if this element does not
* have an open buffer). Children are placed in the given newElements table
* (note, this element has already been placed in the newElements table).
* Returns true if successful, or false if an error is encountered while
* determining the structure of this element.
*/
protected abstract boolean buildStructure(OpenableElementInfo info,
IProgressMonitor pm, Map newElements, IResource underlyingResource)
throws ModelException;

k) AbstractSourceModule.buildStructure calls 
SourceParserUtil.parseSourceModule

l) SourceParserUtil.parseSourceModule calls 
AbstractSourceElementParser.parseSourceModule

m) AbstractSourceElementParser.parseSourceModule calls 
SourceParserUtil.getModuleDeclaration:

n) public static ModuleDeclaration getModuleDeclaration(char[] filename,
char[] content, String nature, IProblemReporter reporter,
ISourceModuleInfo mifo, int flags) {
ISourceParser sourceParser;// = new SourceParser(this.fReporter);
sourceParser = DLTKLanguageManager.getSourceParser(nature);
if (sourceParser instanceof ISourceParserExtension) {
((ISourceParserExtension) sourceParser).setFlags(flags);
}
ModuleDeclaration moduleDeclaration = SourceParserUtil
.getModuleFromCache(mifo, flags);
if (moduleDeclaration == null) {
if (reporter != null) {
reporter.clearMarkers();
}
moduleDeclaration = sourceParser.parse(filename, content, reporter);
SourceParserUtil.putModuleToCache(mifo, moduleDeclaration, flags);
}
return moduleDeclaration;
}

o) My class VScriptSourceParser extends AbstractSourceParser which 
implements ISourceParser.
The parse method instantiates a vscriptParser and uses it.

NOTES:

o) ISourceParser.parse is also called by:
    i) AbstractASTFoldingStructureProvider.getCodeBlocks, which is 
ultimately called     by:AbstractASTFoldingStructureProvider.initialized and 
AbstractASTFoldingStructureProvider.ElementChangedListener.elementChanged

m) AbstractSourceElementParser.parseSourceModule is also called by:i) 
SourceIndexer.indexDocument

n/p) SourceParserUtil.getModuleDeclaration (different from the 
getModuleDeclaration above):
public static ModuleDeclaration getModuleDeclaration(ISourceModule module,
IProblemReporter reporter) {
is called from these places:

i) VScriptCompletionEngine.locateVScriptContext(SourceModule, int)
ii) VScriptSelectionEngine.select(ISourceModule, int, int)
iii) MatchLocatorParser.parse(PossibleMatch)
iv) SemanticPositionUpdater.reconcile(ISourceModule sourceModule, 
SemanticHighlightingPresenter presenter, Highlighting[] highlightings, List 
currentPositions)

"Andrei Sobolev" <haiodo@xxxxxxxxx> wrote in message 
news:g09mhq$9i2$1@xxxxxxxxxxxxxxxxxxxx
> Hi Chuck,
>
> Nice to see, what DLTK help you.
>
> We don't have something special in area of semantic analysis, etc.
>
> Where is some ways to implement such features:
> 1) Using of ScriptBuilder.
> In DLTK we have feature named ScriptBuilder. This is incremental resource 
> builder, used for building Mixin models, etc.
> It will give you list of changed resources, and you cold check all this 
> files and set appropriate markers.
> All Eclipse markers will be show in DLTK Text editors. ScriptBuilder will 
> contain information about builded resources,
> and will be executed on each resource operations, like "resource added", 
> "resource content changed".
>
> 2) You could use Validator, Validators framework uses Script Builder for 
> incremental execution of validators. One more
> additional point, validators framework manages validator instances and 
> allow unified way to sore configurations, etc.
> Also user could turn on/off and execute validators from DLTK UI.
>
> 3) If you also plan to make semantic checks for file edited, you need to 
> extend Reconciler from your
> ScriptSourceViewerConfiguration class. Reconciler will rebuild model, and 
> do checks, after user will make some changes
> in code. By default it is 500 milliseconds delay, so reconciler will be 
> executed only if user not type code 500
> milliseconds.
>
>
> References:
> 1) Extension point to implement: org.eclipse.dltk.core.builder
> 2) We have "package require" checker implemented for Tcl. (TclCheckBuilder 
> class).
> 3) ScriptSourceViewerConfiguration.getReconciler()
>
> Best regards,
> Andrei Sobolev.
>
>
>> Thanks to ANTLR and DLTK, I now have an advanced editor which can
>> immediately flag syntactic problems in our language source files.
>>
>> My question is if/when/how should I find/flag semantic errors in the 
>> source
>> file?
>>
>> 1.    Is there any part of DLTK framework which could/should be used to
>> invoke/perform semantic checks?    I assume the answer to this is no. I 
>> know
>> you support external validators - but we'd like the semantic checking to 
>> be
>> performed implicitly.
>>
>> 2.    When should semantic checks be performed - whenever the parser is
>> invoked?
>>
>> 3.    How should errors be reported - using the parser's error reporter?
>>         Also, how can we determine the line number for populating 
>> IProblem
>> in the file if that information is:
>>         a) not available in DLTKToken
>>         b) not kept in ASTNode
>>
>> Thanks,
>> Chuck
>>
>>