On Wed, May 16, 2012 at 6:40 PM, Jae Gangemi <jgangemi@xxxxxxxxx> wrote:

  actually, i just went and re-read the wiki page, shouldn't this be simple for D?

  you'd just need multiple multi-line rules that look something like this (not sure if there is an escape char):

    new MultiLineRule("q(", ")\n", token, (char) 0, true);
    new MultiLineRule("q{", "}\n", token, (char) 0, true);

  no?

Actually, there's another bug here, and it's a tricky one, so heads up in case you might run to into in similar code on own work: If you use different rules to match the same token, the partitioner can at some times (especially for multi-line rules) try to resume scanning (look only for the end of the partition starting from the middle of it) using a rule different from the one that started the partition. So code like this:

q(
blah
}

might be scanned correctly as a partition of that token type, because one rule would start it, but another rule might end the scanning successfully. I mentioned this in more detail in this Eclipse bug:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=346723


--
Bruno Medeiros

_______________________________________________
dltk-dev mailing list
dltk-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/dltk-dev

  actually, i just went and re-read the wiki page, shouldn't this be simple for D?

  you'd just need multiple multi-line rules that look something like this (not sure if there is an escape char):

    new MultiLineRule("q(", ")\n", token, (char) 0, true);
    new MultiLineRule("q{", "}\n", token, (char) 0, true);

  no?

  ha! i think i figured out the answer :)

  the easiest thing to do will be to just build a token buffer if/when heredoc is encountered and then resume normal scanning once it's empty.

  i'll report back once i try this out.

On Wed, May 16, 2012 at 8:53 PM, Jae Gangemi <jgangemi@xxxxxxxxx> wrote:

  actually, on further thought, i want this to be in its own partition b/c then i can assign a specific 'color' scanner to it that offers me more flexibility/options.

  it'd be nice if there were a way pull offsets out of the ModuleDeclaration to supplement rule based parsing...is that even possible?

On Wed, May 16, 2012 at 8:40 PM, Jae Gangemi <jgangemi@xxxxxxxxx> wrote:

 
  started looking into this and have a very basic implementation kind of working, but i've hit the first snag and wanted to see what others thought on how to over come it...

  i've taken the route of trying to heredoc be it's own partition, so this:

  <<EOF
  EOF
 
  yields this document structure:

partition type: __perl_heredoc, offset: 0, length: 5
---------------------------
<<EOF
---------------------------
partition type: __dftl_partition_content_type, offset: 5, length: 1
---------------------------
---------------------------
partition type: __perl_heredoc, offset: 6, length: 3
---------------------------
EOF
---------------------------

  in order to do this, i start tracking that i've seen a heredoc token in my partition scanner and once i see that the next character is going to be a newline, the scanner will start consuming each line until it sees the terminator, at which point it ends the partition and resets the state.

  the problem that i am having is this, if i add a newline to the document before the start of <<EOF, i hit this block of code starting at line 371 in the FastPartioner

    // if position already exists and we have scanned at least the
    // area covered by the event, we are done
    if (fDocument.containsPosition(fPositionCategory, start, length)) {
        if (lastScannedPosition >= e.getOffset() + newLength)
            return createRegion();
            ++ first;
        } else {

  and the scanning stops and my partition scanner is left thinking that the next time 'nextToken()' is invoked, it's in heredoc mode.

  i really don't want to have to make a copy of the FastPartitioner just to add some way to 'reset' my partition scanner, so what other options exist?

  is creating a partition for this just the wrong way to go? eventually i'd like to be able to offer a folding option for heredoc, but i believe i could also accomplish that by having it represented in the AST.

  i haven't tried going through the code scanner yet to see if it's possible that way - but i am worried that i will encounter the same 'state' problem i saw with the partition scanner - but perhaps not.

  either way - if anyone has anything to contribute, i'd love to here it! :)

On Wed, May 16, 2012 at 2:28 PM, Jae Gangemi <jgangemi@xxxxxxxxx> wrote:

  in the 2nd case you're going to have to write your own rule, but that still should be much simpler to handle b/c the ';' appears on the terminator line, not after the heredoc start.

  your rule would have to check for q" followed by some character and if it saw a char after the ", read in all the chars up until the new line, save that as your terminator, and keep reading lines until you hit the terminator.

  sub-classing the pattern/multi-line rule to do this should get you want you want.

On Wed, May 16, 2012 at 2:21 PM, Bruno Medeiros <bruno.do.medeiros@xxxxxxxxx> wrote:

On Wed, May 16, 2012 at 6:40 PM, Jae Gangemi <jgangemi@xxxxxxxxx> wrote:

  actually, i just went and re-read the wiki page, shouldn't this be simple for D?

  you'd just need multiple multi-line rules that look something like this (not sure if there is an escape char):

    new MultiLineRule("q(", ")\n", token, (char) 0, true);
    new MultiLineRule("q{", "}\n", token, (char) 0, true);

  no?

That should be correct to:
  new MultiLineRule("q\"(", ")\"", token, (char) 0, true);
but anyways: yes and no, that would work for the heredoc with "a delimiter character (any of () <> {} or []) " which is the first case as shown in the wiki, but it wouldn't work for the second case with a delimiter identifier string:

int main() {
    string list = q"IDENT
1. Item One
2. Item Two
3. Item Three
IDENT";
    writef( list );
}

which is more like the Ruby and Perl heredoc.

--
Bruno Medeiros

_______________________________________________
dltk-dev mailing list
dltk-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/dltk-dev

--
-jae

--
-jae

--
-jae

--
-jae

  actually, on further thought, i want this to be in its own partition b/c then i can assign a specific 'color' scanner to it that offers me more flexibility/options.

  it'd be nice if there were a way pull offsets out of the ModuleDeclaration to supplement rule based parsing...is that even possible?

On Wed, May 16, 2012 at 8:40 PM, Jae Gangemi <jgangemi@xxxxxxxxx> wrote:

 
  started looking into this and have a very basic implementation kind of working, but i've hit the first snag and wanted to see what others thought on how to over come it...

  i've taken the route of trying to heredoc be it's own partition, so this:

  <<EOF
  EOF
 
  yields this document structure:

partition type: __perl_heredoc, offset: 0, length: 5
---------------------------
<<EOF
---------------------------
partition type: __dftl_partition_content_type, offset: 5, length: 1
---------------------------
---------------------------
partition type: __perl_heredoc, offset: 6, length: 3
---------------------------
EOF
---------------------------

  in order to do this, i start tracking that i've seen a heredoc token in my partition scanner and once i see that the next character is going to be a newline, the scanner will start consuming each line until it sees the terminator, at which point it ends the partition and resets the state.

  the problem that i am having is this, if i add a newline to the document before the start of <<EOF, i hit this block of code starting at line 371 in the FastPartioner

    // if position already exists and we have scanned at least the
    // area covered by the event, we are done
    if (fDocument.containsPosition(fPositionCategory, start, length)) {
        if (lastScannedPosition >= e.getOffset() + newLength)
            return createRegion();
            ++ first;
        } else {

  and the scanning stops and my partition scanner is left thinking that the next time 'nextToken()' is invoked, it's in heredoc mode.

  i really don't want to have to make a copy of the FastPartitioner just to add some way to 'reset' my partition scanner, so what other options exist?

  is creating a partition for this just the wrong way to go? eventually i'd like to be able to offer a folding option for heredoc, but i believe i could also accomplish that by having it represented in the AST.

  i haven't tried going through the code scanner yet to see if it's possible that way - but i am worried that i will encounter the same 'state' problem i saw with the partition scanner - but perhaps not.

  either way - if anyone has anything to contribute, i'd love to here it! :)

On Wed, May 16, 2012 at 2:28 PM, Jae Gangemi <jgangemi@xxxxxxxxx> wrote:

  in the 2nd case you're going to have to write your own rule, but that still should be much simpler to handle b/c the ';' appears on the terminator line, not after the heredoc start.

  your rule would have to check for q" followed by some character and if it saw a char after the ", read in all the chars up until the new line, save that as your terminator, and keep reading lines until you hit the terminator.

  sub-classing the pattern/multi-line rule to do this should get you want you want.

On Wed, May 16, 2012 at 2:21 PM, Bruno Medeiros <bruno.do.medeiros@xxxxxxxxx> wrote:

On Wed, May 16, 2012 at 6:40 PM, Jae Gangemi <jgangemi@xxxxxxxxx> wrote:

  actually, i just went and re-read the wiki page, shouldn't this be simple for D?

  you'd just need multiple multi-line rules that look something like this (not sure if there is an escape char):

    new MultiLineRule("q(", ")\n", token, (char) 0, true);
    new MultiLineRule("q{", "}\n", token, (char) 0, true);

  no?

That should be correct to:
  new MultiLineRule("q\"(", ")\"", token, (char) 0, true);
but anyways: yes and no, that would work for the heredoc with "a delimiter character (any of () <> {} or []) " which is the first case as shown in the wiki, but it wouldn't work for the second case with a delimiter identifier string:

int main() {
    string list = q"IDENT
1. Item One
2. Item Two
3. Item Three
IDENT";
    writef( list );
}

which is more like the Ruby and Perl heredoc.

--
Bruno Medeiros

_______________________________________________
dltk-dev mailing list
dltk-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/dltk-dev

--
-jae

--
-jae

--
-jae

 
  started looking into this and have a very basic implementation kind of working, but i've hit the first snag and wanted to see what others thought on how to over come it...

  i've taken the route of trying to heredoc be it's own partition, so this:

  <<EOF
  EOF
 
  yields this document structure:

partition type: __perl_heredoc, offset: 0, length: 5
---------------------------
<<EOF
---------------------------
partition type: __dftl_partition_content_type, offset: 5, length: 1
---------------------------
---------------------------
partition type: __perl_heredoc, offset: 6, length: 3
---------------------------
EOF
---------------------------

  in order to do this, i start tracking that i've seen a heredoc token in my partition scanner and once i see that the next character is going to be a newline, the scanner will start consuming each line until it sees the terminator, at which point it ends the partition and resets the state.

  the problem that i am having is this, if i add a newline to the document before the start of <<EOF, i hit this block of code starting at line 371 in the FastPartioner

    // if position already exists and we have scanned at least the
    // area covered by the event, we are done
    if (fDocument.containsPosition(fPositionCategory, start, length)) {
        if (lastScannedPosition >= e.getOffset() + newLength)
            return createRegion();
            ++ first;
        } else {

  and the scanning stops and my partition scanner is left thinking that the next time 'nextToken()' is invoked, it's in heredoc mode.

  i really don't want to have to make a copy of the FastPartitioner just to add some way to 'reset' my partition scanner, so what other options exist?

  is creating a partition for this just the wrong way to go? eventually i'd like to be able to offer a folding option for heredoc, but i believe i could also accomplish that by having it represented in the AST.

  i haven't tried going through the code scanner yet to see if it's possible that way - but i am worried that i will encounter the same 'state' problem i saw with the partition scanner - but perhaps not.

  either way - if anyone has anything to contribute, i'd love to here it! :)

On Wed, May 16, 2012 at 2:28 PM, Jae Gangemi <jgangemi@xxxxxxxxx> wrote:

  in the 2nd case you're going to have to write your own rule, but that still should be much simpler to handle b/c the ';' appears on the terminator line, not after the heredoc start.

  your rule would have to check for q" followed by some character and if it saw a char after the ", read in all the chars up until the new line, save that as your terminator, and keep reading lines until you hit the terminator.

  sub-classing the pattern/multi-line rule to do this should get you want you want.

On Wed, May 16, 2012 at 2:21 PM, Bruno Medeiros <bruno.do.medeiros@xxxxxxxxx> wrote:

On Wed, May 16, 2012 at 6:40 PM, Jae Gangemi <jgangemi@xxxxxxxxx> wrote:

  actually, i just went and re-read the wiki page, shouldn't this be simple for D?

  you'd just need multiple multi-line rules that look something like this (not sure if there is an escape char):

    new MultiLineRule("q(", ")\n", token, (char) 0, true);
    new MultiLineRule("q{", "}\n", token, (char) 0, true);

  no?

That should be correct to:
  new MultiLineRule("q\"(", ")\"", token, (char) 0, true);
but anyways: yes and no, that would work for the heredoc with "a delimiter character (any of () <> {} or []) " which is the first case as shown in the wiki, but it wouldn't work for the second case with a delimiter identifier string:

int main() {
    string list = q"IDENT
1. Item One
2. Item Two
3. Item Three
IDENT";
    writef( list );
}

which is more like the Ruby and Perl heredoc.

--
Bruno Medeiros

_______________________________________________
dltk-dev mailing list
dltk-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/dltk-dev

--
-jae

--
-jae

  in the 2nd case you're going to have to write your own rule, but that still should be much simpler to handle b/c the ';' appears on the terminator line, not after the heredoc start.

  your rule would have to check for q" followed by some character and if it saw a char after the ", read in all the chars up until the new line, save that as your terminator, and keep reading lines until you hit the terminator.

  sub-classing the pattern/multi-line rule to do this should get you want you want.

On Wed, May 16, 2012 at 2:21 PM, Bruno Medeiros <bruno.do.medeiros@xxxxxxxxx> wrote:

On Wed, May 16, 2012 at 6:40 PM, Jae Gangemi <jgangemi@xxxxxxxxx> wrote:

  actually, i just went and re-read the wiki page, shouldn't this be simple for D?

  you'd just need multiple multi-line rules that look something like this (not sure if there is an escape char):

    new MultiLineRule("q(", ")\n", token, (char) 0, true);
    new MultiLineRule("q{", "}\n", token, (char) 0, true);

  no?

That should be correct to:
  new MultiLineRule("q\"(", ")\"", token, (char) 0, true);
but anyways: yes and no, that would work for the heredoc with "a delimiter character (any of () <> {} or []) " which is the first case as shown in the wiki, but it wouldn't work for the second case with a delimiter identifier string:

int main() {
    string list = q"IDENT
1. Item One
2. Item Two
3. Item Three
IDENT";
    writef( list );
}

which is more like the Ruby and Perl heredoc.

--
Bruno Medeiros

_______________________________________________
dltk-dev mailing list
dltk-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/dltk-dev

--
-jae

On Wed, May 16, 2012 at 6:40 PM, Jae Gangemi <jgangemi@xxxxxxxxx> wrote:

  actually, i just went and re-read the wiki page, shouldn't this be simple for D?

  you'd just need multiple multi-line rules that look something like this (not sure if there is an escape char):

    new MultiLineRule("q(", ")\n", token, (char) 0, true);
    new MultiLineRule("q{", "}\n", token, (char) 0, true);

  no?

That should be correct to:
  new MultiLineRule("q\"(", ")\"", token, (char) 0, true);
but anyways: yes and no, that would work for the heredoc with "a delimiter character (any of () <> {} or []) " which is the first case as shown in the wiki, but it wouldn't work for the second case with a delimiter identifier string:

int main() {
    string list = q"IDENT
1. Item One
2. Item Two
3. Item Three
IDENT";
    writef( list );
}

which is more like the Ruby and Perl heredoc.

--
Bruno Medeiros

_______________________________________________
dltk-dev mailing list
dltk-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/dltk-dev

  actually, i just went and re-read the wiki page, shouldn't this be simple for D?

  you'd just need multiple multi-line rules that look something like this (not sure if there is an escape char):

    new MultiLineRule("q(", ")\n", token, (char) 0, true);
    new MultiLineRule("q{", "}\n", token, (char) 0, true);

  no?

  i think you should be able to handle that case using a pattern rule, you just need to account for the "<<" not being on the delimiter, which may require you to subclass the rule to handle that.

On Wed, May 16, 2012 at 1:06 PM, Bruno Medeiros <bruno.do.medeiros@xxxxxxxxx> wrote:

On Tue, May 1, 2012 at 8:25 PM, Jae Gangemi <jgangemi@xxxxxxxxx> wrote:

hello all -

  i was hoping someone here might have a suggestion on how to approach dealing w/ heredoc in order to create a document partition so it can be 'colored'

  i can't just have a rule that starts the partition as starting as soon as it sees the heredoc, b/c things like this are valid:

    $x = <<EOF . "blah"; 
    heredoc
    EOF

  and the
 
    . "blah";

  should be treated differently from the rest of the statement.

  i'm guessing that i'm going to have to override some portion of the RuleBasedPartitionScanner in order to trace some kind of state that indicates heredoc was seen and all text after the newline up until the terminator should be treated as such??

  if there's a better idea/approach, please let me know. i checked the ruby plugin for ideas since it supports heredoc, but it doesn't look like anyone's tackled this there either.

  thanks!

--
-jae

_______________________________________________
dltk-dev mailing list
dltk-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/dltk-dev

Hum, do you know if the simpler case of Heredoc scanner has been solved/implemented somewhere? That is, when there is nothing after the heredoc initial delimiter, like this:

$x = <<EOF
    heredoc
EOF

Such code would be useful for me as this is an open issue in the D language IDE. D supports heredoc, but only in the simpler form above (http://en.wikipedia.org/wiki/Here_document#D ), there is no way to continue a statement/_expression_ right after the initial delimiter/tag, like Perl and Ruby support.

It seems this simpler case can be implemented by just creating a new PatternRule, no need to change the RuleBasedPartitionScanner.

--
Bruno Medeiros

_______________________________________________
dltk-dev mailing list
dltk-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/dltk-dev

--
-jae