SynEdit Highlighter

From Lazarus wiki
Revision as of 11:34, 19 March 2011 by Martin (talk | contribs) (Step 2: Using Ranges)

Understanding the SynEdit Highlighter

SynEdit - Highlighter relationship

SynEdit <-> Highlighter have a n to 1 relationship.

  • 1 (instance of a) Highlighter can serve n (many) (instances of) SynEdits
  • Each SynEdit only has one Highlighter

As a result of this:

  • no Highlighter Instance has a (fixed) reference to the SynEdit.
(Highlighters however keep a list of SynEditTextBuffers to which they are attached)
  • All data for the Highlighter is (and must be) stored on the SynEdit (actually on the TextBuffer of SynEdit (referred to as "Lines").

However SynEdit ensures before each call to the Highlighter that Highlighter.CurrentLines is set to the current SynEdits Lines. This way the highlighter can access the data whenever needed. The Format of the data-storage is determined by the highlighter (TSynCustomHighlighter.AttachToLines)

Scanning and Returning Highlight attributes

The Highlighter is expected to work on a per Line base.

If any text was modified, SynEdit will call (TSynCustomHighlighter.ScanFrom / Currently called from TSynEdit.ScanFrom) with the line range. The Highlighter should know the state of the previous line.

If Highlight attributes are required SynEdit will request them per Line too. SynEdit will loop through individual tokens on a line. This currently happens from nested proc PaintLines in SynEdit.PaintTextLines. It calls TSynCustomHighlighter.StartAtLineIndex, followed by HL.GetTokenEx/HL.GetTokenAttribute for as long as HL.GetEol is false

Also the BaseClass for the Highlighter's data (see AttachToLines) is based on per line storage, and SynEdit's TextBuffer (Lines) do maintenance on this data to keep it synchronized. That is when ever lines of text are inserted or removed, so are entries inserted or removed from the highlighters data (hence it must have one entry per line).

Usually Highlighters store the end-of-line-status in this field. So if the highlighter is going to work on a line, it will continue with the state-entry from the previous line.

Folding

SynEdit's folding is handled by unit SynEditFoldedView and SynGutterCodeFolding. Highlighter that implement folding are to be based on TSynCustomFoldHighlighter

The basic information for communication between SynEditFoldedView and the HL requires 2 values stored for each line. (Of course the highlighter itself can store more information):

  • FoldLevel at the end of line
  • Minimum FoldLevel encountered anywhere on the line

The Foldlevel indicates how many (nested) folds exist. It goes up whenever a fold begins, and down when a fold ends

                           EndLvl   MinLvl
 Procedure a;               1 -      0
 Begin                      2 --     1 -
   b:= 1;                   2 --     2 --
   if c > b then begin      3 ---    2 --
     c:=b;                  3 ---    3 ---
   end else begin           3 ---    2 --
     b:=c;                  3 ---    3 ---
   end;                     2 --     2 --
 end;                       0        0  // The end closes both: begin and procedure fold

In the line

 Procedure a;               1 -      0

the MinLvl is 0, because the line started with a Level of 0 (and it never went down / no folds closed). Similar in all lines where there is only an opening fold keyword ("begin").

But the line

   end else begin           3 ---    2 --

starts with a level of 3, and also ends with it (one close, one open). But since it went down first, the minimum level encountered anywhere on the line is 2.

Without the MinLvl it would not be possible to tell, that a fold ends in this line.

There is no such thing as a MaxLvl, because folds that start and end on the same line can not be folded anyway. No need to detect them.

 if a then begin b:=1; c:=2; end; // no fold on that line


Creating a SynEdit Highlighter

This section is under construction

The Basics: Returning Tokens and Attributes

Below is a very basic highlighter for demonstration purposes.

What it does:

  • It splits each line into words and spaces
    • A space is anything between #0 and #32 (newline, tab, space, ...). (This works for Ascii and Utf8)
    • The spaces are part of the text, and must be highlighted too
  • It matches each word (case-insensitive) against "comment", and highlights any match.

How it works:

  • Creation

The Highlighter creates Attributes that it can return the Words and Spaces.

  • SetLine

Is called by SynEdit before a line gets painted (or before highlight info is needed)

  • GetTokenEx, GetTokenAttribute, Next, GetEol

Are used by SynEdit to iterate over the Line. Note that the first Token (Word or Spaces) must be ready after SetLine, without a call to Next.

Important: The tokens returned for each line, must represent the original line-text, and be returned in the correct order.

  • GetToken, GetTokenPos, GetTokenKind

SynEdit uses them e.g for finding matching brackets. If tokenKind returns different values per Attribute, then brackets only match, if they are of the same kind (e.g, if there was a string attribute, brackets outside a string would not match brackets inside a string)


Other notes:

For readability the below highlighter has no optimization, so it may be very slow on larger texts. Many of the supplied highlighers use hash functions, to find what word (or any group of chars) is.

<delphi> interface

type

 TSynDemoHl = class(TSynCustomHighlighter)
 private
   fCommentAttri: TSynHighlighterAttributes;
   fIdentifierAttri: TSynHighlighterAttributes;
   fSpaceAttri: TSynHighlighterAttributes;
   FTokenPos, FTokenEnd: Integer;
   FLineText: String;
   procedure FindTokenEnd;
 public
   procedure SetLine(const NewValue: String; LineNumber: Integer); override;
   procedure GetTokenEx(out TokenStart: PChar; out TokenLength: integer); override;
   function GetTokenAttribute: TSynHighlighterAttributes; override;
   procedure Next; override;
   function GetEol: Boolean; override;
 public
   function GetToken: String; override;
   function GetTokenPos: Integer; override;
   function GetTokenKind: integer; override;
   function GetDefaultAttribute(Index: integer): TSynHighlighterAttributes; override;
   constructor Create(AOwner: TComponent); override;
 published
   property CommentAttri: TSynHighlighterAttributes read fCommentAttri
     write fCommentAttri;
   property IdentifierAttri: TSynHighlighterAttributes read fIdentifierAttri
     write fIdentifierAttri;
   property SpaceAttri: TSynHighlighterAttributes read fSpaceAttri
     write fSpaceAttri;
 end;

implementation

procedure TSynDemoHl.FindTokenEnd; var

 l: Integer;

begin

 l := length(FLineText);
 FTokenEnd := FTokenPos;
 If FTokenPos > l then exit
 else
 if FLineText[FTokenEnd] in [#9, ' '] then
 	 while (FTokenEnd <= l) and (FLineText[FTokenEnd] in [#0..#32]) do inc (FTokenEnd)
 else
 	 while (FTokenEnd <= l) and not(FLineText[FTokenEnd] in [#9, ' ']) do inc (FTokenEnd)

end;

procedure TSynDemoHl.SetLine(const NewValue: String; LineNumber: Integer); begin

 inherited;
 FTokenPos := 1;
 FLineText := NewValue;
 FindTokenEnd;

end;

procedure TSynDemoHl.GetTokenEx(out TokenStart: PChar; out TokenLength: integer); begin

 TokenStart := @FLineText[FTokenPos];
 TokenLength := FTokenEnd - FTokenPos;

end;

function TSynDemoHl.GetTokenAttribute: TSynHighlighterAttributes; begin

 if FLineText[FTokenPos] in [#9, ' '] then
   Result := SpaceAttri
 else
 if LowerCase(copy(FLineText, FTokenPos, FTokenEnd-FTokenPos)) = 'comment' then
   Result := CommentAttri
 else
   Result := IdentifierAttri;

end;

procedure TSynDemoHl.Next; begin

 FTokenPos := FTokenEnd;
 FindTokenEnd;

end;

function TSynDemoHl.GetEol: Boolean; begin

 Result := FTokenPos > length(FLineText);

end;

function TSynDemoHl.GetDefaultAttribute(Index: integer): TSynHighlighterAttributes; begin

 case Index of
   SYN_ATTR_COMMENT: Result := fCommentAttri;
   SYN_ATTR_IDENTIFIER: Result := fIdentifierAttri;
   SYN_ATTR_WHITESPACE: Result := fSpaceAttri;
   else Result := nil;
 end;

end;

function TSynDemoHl.GetToken: String; begin

 Result := copy(FLineText, FTokenPos, FTokenEnd - FTokenPos);

end;

function TSynDemoHl.GetTokenPos: Integer; begin

 Result := FTokenPos - 1;

end;

function TSynDemoHl.GetTokenKind: integer; var

 a: TSynHighlighterAttributes;

begin

 a := GetTokenAttribute;
 Result := 0;
 if a = fSpaceAttri then Result := 1;
 if a = fCommentAttri then Result := 2;
 if a = fIdentifierAttri then Result := 3;

end;

constructor TSynDemoHl.Create(AOwner: TComponent); begin

 inherited Create(AOwner);
 fCommentAttri := TSynHighlighterAttributes.Create('comment', 'comment');
 AddAttribute(fCommentAttri);
 fIdentifierAttri := TSynHighlighterAttributes.Create('ident', 'ident');
 fIdentifierAttri.Style := [fsBold];
 AddAttribute(fIdentifierAttri);
 fSpaceAttri := TSynHighlighterAttributes.Create('space', 'space');
 AddAttribute(fSpaceAttri);

end; </delphi>

<delphi> procedure Form1.Init; var

 hl: TSynDemoHl;

begin

 hl := TSynDemoHl.Create(Self);
 SynEdit1.Highlighter := hl;
 hl.CommentAttri.Foreground := clRed;
 hl.IdentifierAttri.Foreground := clGreen;

end; </delphi>

Step 2: Using Ranges

The next example allows content of a line, influences other lines that follows. E.g if a "(*" in pascal makes all following lines a comment until a "*)" is found.

In this example all content between a free-standing " ( " and a " ) " (note the spaces around the brackets) will be highlighted differently


We add an attribute: <delphi>

 private
   fBlockAttri: TSynHighlighterAttributes;
 published
   property BlockAttri: TSynHighlighterAttributes read fBlockAttri
     write fBlockAttri;

constructor TSynDemoHl.Create(AOwner: TComponent); begin

 // ......
 fBlockAttri := TSynHighlighterAttributes.Create('block', 'block');
 AddAttribute(fBlockAttri);

end; </delphi>

And the methods to store the currrent nest level of ():

GetRange
Called after a line is completly scanned, to get the value at the end of the line. The value will be stored.
SetRange
Called before a line get's scanned. Sets the value stored from the end of the previous line.
ResetRange
Called before the 1st line is scanned (As there is no previous line).

<delphi>

 private
   FCurRange: Integer;
 public
   procedure SetRange(Value: Pointer); override;
   procedure ResetRange; override;
   function GetRange: Pointer; override;

procedure TSynDemoHl.SetRange(Value: Pointer); begin

 FCurRange := PtrInt(Value);

end;

procedure TSynDemoHl.ResetRange; begin

 FCurRange := 0;

end;

function TSynDemoHl.GetRange: Pointer; begin

 Result := Pointer(PtrInt(FCurRange));

end; </delphi>

Then we extend the scanner. The pre-scan to store the information, calls the same functions than the highlighter, and is automatically called, if anything changes. (It is called for all lines below the changed line, until a line returns the same Range-value as it already had)

  • NOTE: A scan is triggered by *every* change to a line (every keystroke). It scans the current line, and all lines below, until a line returns the same range that it already had.

<delphi> procedure TSynDemoHl.FindTokenEnd; var

 l: Integer;

begin

 l := length(FLineText);
 FTokenEnd := FTokenPos;
 If FTokenPos > l then exit
 else
 if FLineText[FTokenEnd] in [#9, ' '] then
 	 while (FTokenEnd <= l) and (FLineText[FTokenEnd] in [#0..#32]) do inc (FTokenEnd)
 else
 	 while (FTokenEnd <= l) and not(FLineText[FTokenEnd] in [#9, ' ']) do inc (FTokenEnd);
 // NEW: Check for ()
 if (FTokenEnd = FTokenPos+1) and (FLineText[FTokenPos] = '(') then
   inc(FCurRange);
 if (FTokenEnd = FTokenPos+1) and (FLineText[FTokenPos] = ')') and (FCurRange > 0) then
   dec(FCurRange);

end;

function TSynDemoHl.GetTokenAttribute: TSynHighlighterAttributes; begin

 if FCurRange > 0 then
   Result := BlockAttri
 else
 if FLineText[FTokenPos] in [#9, ' '] then
   Result := SpaceAttri
 else
 if LowerCase(copy(FLineText, FTokenPos, FTokenEnd-FTokenPos)) = 'comment' then
   Result := CommentAttri
 else
   Result := IdentifierAttri;

end; </delphi>

Step 3:

References

Threads on the forum:

http://www.lazarus.freepascal.org/index.php/topic,10260.0.html

http://www.lazarus.freepascal.org/index.php/topic,7879.0.html

http://www.lazarus.freepascal.org/index.php/topic,7338.0.html

http://www.lazarus.freepascal.org/index.php/topic,10959.msg54714

http://www.lazarus.freepascal.org/index.php/topic,11064

http://www.lazarus.freepascal.org/index.php/topic,11384.msg57160.html#msg57160 (obtaining highlight for printing)