[news.eclipse.modeling.tmf] Re: Keywords vs. tokens

[Sebastian Zarnekow <Sebastian.Zarnekow@xxxxxxxxx>]

Hi Marco,

you could use data type rules to achieve this:

From_import:
  'from' FQN 'import' '*' ...

(btw: there are no assignments in your rule for the sake of simplification?)

Operator returns ecore::EString:
  '+' | '*' | ...;

Generally speaking you should try to avoid to introduce many terminal 
rules as they are contextless and will lead to the problems that you 
described. Data type rules will be handled in a subsequent, context 
aware step and the parser will handle this gracefully.

Hope that helps,
Sebastian
--
Need professional support for Eclipse Modeling?
Go visit: http://xtext.itemis.com

Marco schrieb:
> Hi,
>
> I do not really understand how the lexer works.
>
> It looks like every string literal of my parsing rules are use to 
> identify tokens with even higher priority than terminal rules. So e.g.:
>
> From_import :
> ____'from' FQN 'import' ('*' | ((FQN ('as' FQN)?) (',' FQN ('as' 
> FQN)?)*)) ;
>
> destroys
>
> terminal ARITHMETIC_OPERATOR : '+' | '-' | '*' | '/' | '%' | '&' | '|' ;
>
> Is there an elegant way to avoid this?
>
> I understand that in principle there is first lexing and then parsing. 
> But is there a way where I can have different tokens depending of the 
> context? I.e. Having words or symbols being a keyword in one section of 
> a model and being normal text or just parts of another keyword in a 
> different section?
>
> I have for example a section where every line which does not include a 
> certain symbol is treated as title. Currently I don't know better than 
> listing all string literal and all terminal rules except the one symbol 
> as alternatives for allowed characters. This hard to maintain.
>
> Any hints or correction are very appreciated.
>
> Best regards,
> Marco