[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [Dltk-dev] heredoc scanner help
|
actually, on further thought, i want this to be in its own partition b/c then i can assign a specific 'color' scanner to it that offers me more flexibility/options.
it'd be nice if there were a way pull offsets out of the ModuleDeclaration to supplement rule based parsing...is that even possible?
On Wed, May 16, 2012 at 8:40 PM, Jae Gangemi
<jgangemi@xxxxxxxxx> wrote:
started looking into this and have a very basic implementation kind of working, but i've hit the first snag and wanted to see what others thought on how to over come it...
i've taken the route of trying to heredoc be it's own partition, so this:
<<EOF
EOF
yields this document structure:
partition type: __perl_heredoc, offset: 0, length: 5
---------------------------
<<EOF
---------------------------
partition type: __dftl_partition_content_type, offset: 5, length: 1
---------------------------
---------------------------
partition type: __perl_heredoc, offset: 6, length: 3
---------------------------
EOF
---------------------------
in order to do this, i start tracking that i've seen a heredoc token in my partition scanner and once i see that the next character is going to be a newline, the scanner will start consuming each line until it sees the terminator, at which point it ends the partition and resets the state.
the problem that i am having is this, if i add a newline to the document before the start of <<EOF, i hit this block of code starting at line 371 in the FastPartioner
// if position already exists and we have scanned at least the
// area covered by the event, we are done
if (fDocument.containsPosition(fPositionCategory, start, length)) {
if (lastScannedPosition >= e.getOffset() + newLength)
return createRegion();
++ first;
} else {
and the scanning stops and my partition scanner is left thinking that the next time 'nextToken()' is invoked, it's in heredoc mode.
i really don't want to have to make a copy of the FastPartitioner just to add some way to 'reset' my partition scanner, so what other options exist?
is creating a partition for this just the wrong way to go? eventually i'd like to be able to offer a folding option for heredoc, but i believe i could also accomplish that by having it represented in the AST.
i haven't tried going through the code scanner yet to see if it's possible that way - but i am worried that i will encounter the same 'state' problem i saw with the partition scanner - but perhaps not.
either way - if anyone has anything to contribute, i'd love to here it! :)On Wed, May 16, 2012 at 2:28 PM, Jae Gangemi
<jgangemi@xxxxxxxxx> wrote:
in the 2nd case you're going to have to write your own rule, but that still should be much simpler to handle b/c the ';' appears on the terminator line, not after the heredoc start.
your rule would have to check for q" followed by some character and if it saw a char after the ", read in all the chars up until the new line, save that as your terminator, and keep reading lines until you hit the terminator.
sub-classing the pattern/multi-line rule to do this should get you want you want.
On Wed, May 16, 2012 at 6:40 PM, Jae Gangemi
<jgangemi@xxxxxxxxx> wrote:
actually, i just went and re-read the wiki page, shouldn't this be simple for D?
you'd just need multiple multi-line rules that look something like this (not sure if there is an escape char):
new MultiLineRule("q(", ")\n", token, (char) 0, true);
new MultiLineRule("q{", "}\n", token, (char) 0, true);
no?
That should be correct to:
new MultiLineRule("q\"(", ")\"", token, (char) 0, true);
but anyways: yes and no, that would work for the heredoc with
"a delimiter character (any of () <> {} or []) " which is the first case as shown in the wiki, but it wouldn't work for the second case with a delimiter identifier string:
int main() {
string list = q"IDENT
1. Item One
2. Item Two
3. Item Three
IDENT";
writef( list );
}
which is more like the Ruby and Perl heredoc.
--
Bruno Medeiros
--
-jae
-jae
--
-jae
