Difference between revisions of "SynEdit Highlighter"

From Lazarus wiki
(Step 2: Using Ranges)
(Configurable Highlighters (incl. 3rd party): Added info about SynUniHighlighter with link to first old web-site)
 
(46 intermediate revisions by 9 users not shown)
Line 1: Line 1:
 +
{{LanguageBar}}
 +
 +
For more info on SynEdit go to: [[SynEdit]]
 +
<br/>Also see [[SynEdit Markup]]
 +
----
 +
 
= Understanding the SynEdit Highlighter =
 
= Understanding the SynEdit Highlighter =
  
 
== SynEdit - Highlighter relationship ==
 
== SynEdit - Highlighter relationship ==
  
SynEdit <-> Highlighter have a n to 1 relationship.
+
'''SynEdit to Highlighter''' have an '''N to 1''' relationship.
* 1 (instance of a) Highlighter can serve n (many) (instances of) SynEdits  
+
* One instance of a Highlighter can serve '''N''' (many) instances of SynEdits  
 
* Each SynEdit only has one Highlighter
 
* Each SynEdit only has one Highlighter
 +
* But: one text (text-buffer) can have many highlighters, if shared by several SynEdit (each SynEdit will have one HL, but all HL will work on the same document)
  
 
As a result of this:
 
As a result of this:
Line 12: Line 19:
 
* All data for the Highlighter is (and must be) stored on the SynEdit (actually on the TextBuffer of SynEdit (referred to as "Lines").
 
* All data for the Highlighter is (and must be) stored on the SynEdit (actually on the TextBuffer of SynEdit (referred to as "Lines").
  
However SynEdit ensures before each call to the Highlighter that Highlighter.CurrentLines is set to the current SynEdits Lines. This way the highlighter can access the data whenever needed.
+
However, before each call to the Highlighter, SynEdit ensures that Highlighter.CurrentLines is set to the current SynEdits Lines. This way the highlighter can access the data whenever needed.
The Format of the data-storage is determined by the highlighter (TSynCustomHighlighter.AttachToLines)
+
The Format of the data storage is determined by the highlighter (TSynCustomHighlighter.AttachToLines).
  
 
== Scanning and Returning Highlight attributes ==
 
== Scanning and Returning Highlight attributes ==
Line 21: Line 28:
 
If any text was modified, SynEdit will call (TSynCustomHighlighter.ScanFrom / Currently called from TSynEdit.ScanFrom) with the line range. The Highlighter should know the state of the previous line.
 
If any text was modified, SynEdit will call (TSynCustomHighlighter.ScanFrom / Currently called from TSynEdit.ScanFrom) with the line range. The Highlighter should know the state of the previous line.
  
If Highlight attributes are required SynEdit will request them per Line too. SynEdit will loop through individual tokens on a line. This currently happens from nested proc PaintLines in SynEdit.PaintTextLines. It calls TSynCustomHighlighter.StartAtLineIndex, followed by HL.GetTokenEx/HL.GetTokenAttribute for as long as HL.GetEol is false
+
If Highlight attributes are required, SynEdit will request them per Line too. SynEdit will loop through individual tokens on a line. This currently happens from nested proc PaintLines in SynEdit.PaintTextLines. It calls TSynCustomHighlighter.StartAtLineIndex, followed by HL.GetTokenEx/HL.GetTokenAttribute for as long as HL.GetEol is false.
  
Also the BaseClass for the Highlighter's data (see AttachToLines) is based on per line storage, and SynEdit's TextBuffer (Lines) do maintenance on this data to keep it synchronized. That is when ever lines of text are inserted or removed, so are entries inserted or removed from the highlighters data (hence it must have one entry per line).
+
Also the BaseClass for the Highlighter's data (see AttachToLines) is based on per line storage, and SynEdit's TextBuffer (Lines) do maintenance on this data to keep it synchronized. That is: whenever lines of text are inserted or removed, so are entries inserted or removed from the highlighters data (hence it must have one entry per line).
  
 
Usually Highlighters store the end-of-line-status in this field. So if the highlighter is going to work on a line, it will continue with the state-entry from the previous line.
 
Usually Highlighters store the end-of-line-status in this field. So if the highlighter is going to work on a line, it will continue with the state-entry from the previous line.
Line 29: Line 36:
 
== Folding ==
 
== Folding ==
  
SynEdit's folding is handled by unit SynEditFoldedView and SynGutterCodeFolding. Highlighter that implement folding are to be based on TSynCustomFoldHighlighter
+
SynEdit's folding is handled by unit SynEditFoldedView and SynGutterCodeFolding. Highlighters that implement folding are to be based on TSynCustomFoldHighlighter.
  
 
The basic information for communication between SynEditFoldedView and the HL requires 2 values stored for each line. (Of course the highlighter itself can store more information):
 
The basic information for communication between SynEditFoldedView and the HL requires 2 values stored for each line. (Of course the highlighter itself can store more information):
Line 35: Line 42:
 
* Minimum FoldLevel encountered anywhere on the line
 
* Minimum FoldLevel encountered anywhere on the line
  
The Foldlevel indicates how many (nested) folds exist. It goes up whenever a fold begins, and down when a fold ends
+
The Foldlevel indicates how many (nested) folds exist. It goes up whenever a fold begins, and down when a fold ends:
  
 
                             EndLvl  MinLvl
 
                             EndLvl  MinLvl
Line 56: Line 63:
 
starts with a level of 3, and also ends with it (one close, one open). But since it went down first, the minimum level encountered anywhere on the line is 2.
 
starts with a level of 3, and also ends with it (one close, one open). But since it went down first, the minimum level encountered anywhere on the line is 2.
  
Without the MinLvl it would not be possible to tell, that a fold ends in this line.
+
Without the MinLvl it would not be possible to tell that a fold ends in this line.
  
 
There is no such thing as a MaxLvl, because folds that start and end on the same line can not be folded anyway. No need to detect them.
 
There is no such thing as a MaxLvl, because folds that start and end on the same line can not be folded anyway. No need to detect them.
Line 64: Line 71:
 
= Creating a SynEdit Highlighter =
 
= Creating a SynEdit Highlighter =
  
This section is under construction
+
Since 0.9.31 Revision 35115 the fold-highlighter has changed. Implementing basic folding is now easier.
  
== The Basics: Returning Tokens and Attributes ==
+
All Sources can be found in the Lazarus installation directory under:
 +
  examples\SynEdit\NewHighlighterTutorial\
 +
 
 +
The project HighlighterTutorial contains 3 difference example highlighters:
 +
* SimpleHl: used in Step 1 below
 +
* ContextHl: used in Step 2 below
 +
* FoldHl: used in Step 3 below
 +
 
 +
SimpleHl and ContextHl will work with 0.9.30 too
 +
 
 +
The folding highlighter in action:
  
Below is a very basic highlighter for demonstration purposes.
+
[[image:SynEditFoldingHighlighterDemo.png]]
  
<b>What it does:</b>
+
== The Basics: Returning Tokens and Attributes ==
 +
As indicated, the '''SimpleHl''' unit demonstrates this process.
  
* It splits each line into words and spaces
+
=== What it does===
** A space is anything between #0 and #32 (newline, tab, space, ...). (This works for Ascii and Utf8)
+
* It splits each line into words and spaces (or tabs)
 
** The spaces are part of the text, and must be highlighted too
 
** The spaces are part of the text, and must be highlighted too
* It matches each word (case-insensitive) against "comment", and highlights any match.
+
* This example allows specifying different colors for
 
+
::- text (defaults to not-highlighted)
<b>How it works:</b>
+
::- spaces (defaults to silver frame)
 +
::- words, separated by spaces, that start with a,e,i,o,u (defaults to bold)
 +
::- the word "not" (defaults to red background)
  
 +
=== How it works ===
 
* Creation  
 
* Creation  
 
The Highlighter creates Attributes that it can return the Words and Spaces.
 
The Highlighter creates Attributes that it can return the Words and Spaces.
Line 89: Line 110:
 
Note that the first Token (Word or Spaces) must be ready after SetLine, without a call to Next.
 
Note that the first Token (Word or Spaces) must be ready after SetLine, without a call to Next.
  
Important: The tokens returned for each line, must represent the original line-text, and be returned in the correct order.
+
[[Important:]] The tokens returned for each line must represent the original line-text, and be returned in the correct order.
  
 
* GetToken, GetTokenPos, GetTokenKind
 
* GetToken, GetTokenPos, GetTokenKind
 
SynEdit uses them e.g for finding matching brackets.
 
SynEdit uses them e.g for finding matching brackets.
If tokenKind returns different values per Attribute, then brackets only match, if they are of the same kind (e.g, if there was a string attribute, brackets outside a string would not match brackets inside a string)
+
If tokenKind returns different values per Attribute, then brackets only match, if they are of the same kind (e.g. if there was a string attribute, brackets outside a string would not match brackets inside a string).
 +
 
 +
=== Other notes===
 +
For readability, the highlighter has no optimization, so it may be very slow on larger texts.
 +
Many of the supplied highlighters use hash functions, to find what word (or any group of chars) is.
 +
 
 +
== Step 2: Using Ranges ==
 +
As indicated, the '''ContextHl''' unit demonstrates this process
  
 +
The next example allows content of a line that influences other lines that follow. An example:  a "(*" in Pascal makes all following lines a comment until a "*)" is found.
  
<b>Other notes:</b>
+
This example extends the '''SimpleHl''' show above:
 +
The tokens -- and ++ (must be surrounded by space or line-begin/end to be a token of their own) will toggle words that start with a,e,i,o,u
  
For readability the below highlighter has no optimization, so it may be very slow on larger texts.
+
Multiple ++ and -- can be nested. Then for each -- a ++ must be given, before the words highlight again.
Many of the supplied highlighers use hash functions, to find what word (or any group of chars) is.
 
  
<delphi>
+
Then we extend the scanner. The pre-scan to store the information calls the same functions as the highlighter. It is automatically called, if anything changes. (It is called for all lines below the changed line, until a line returns the same Range-value as it already had)
interface
 
  
type
+
The current amount of "--" is counted in
 +
<syntaxhighlight lang="pascal">
 +
  FCurRange: Integer;
 +
</syntaxhighlight>
  
  TSynDemoHl = class(TSynCustomHighlighter)
+
The amount is decreased by "++"
  private
 
    fCommentAttri: TSynHighlighterAttributes;
 
    fIdentifierAttri: TSynHighlighterAttributes;
 
    fSpaceAttri: TSynHighlighterAttributes;
 
    FTokenPos, FTokenEnd: Integer;
 
    FLineText: String;
 
    procedure FindTokenEnd;
 
  public
 
    procedure SetLine(const NewValue: String; LineNumber: Integer); override;
 
    procedure GetTokenEx(out TokenStart: PChar; out TokenLength: integer); override;
 
    function GetTokenAttribute: TSynHighlighterAttributes; override;
 
    procedure Next; override;
 
    function GetEol: Boolean; override;
 
  public
 
    function GetToken: String; override;
 
    function GetTokenPos: Integer; override;
 
    function GetTokenKind: integer; override;
 
    function GetDefaultAttribute(Index: integer): TSynHighlighterAttributes; override;
 
    constructor Create(AOwner: TComponent); override;
 
  published
 
    property CommentAttri: TSynHighlighterAttributes read fCommentAttri
 
      write fCommentAttri;
 
    property IdentifierAttri: TSynHighlighterAttributes read fIdentifierAttri
 
      write fIdentifierAttri;
 
    property SpaceAttri: TSynHighlighterAttributes read fSpaceAttri
 
      write fSpaceAttri;
 
  end;
 
  
implementation
+
To store the info we use:
  
procedure TSynDemoHl.FindTokenEnd;
+
;GetRange: Called after a line is completely scanned, to get the value at the end of the line. The value will be stored.
var
+
;SetRange: Called before a line gets scanned. Sets the value stored from the end of the previous line.
  l: Integer;
+
;ResetRange: Called before the 1st line is scanned (as there is no previous line).
begin
+
 
  l := length(FLineText);
+
{{Note|A scan is triggered by *every* change to a line (every keystroke). It scans the current line, and all lines below, until a line returns the same range that it already had. See: http://forum.lazarus.freepascal.org/index.php/topic,21727.msg139420.html#msg139420 }}
   FTokenEnd := FTokenPos;
+
 
   If FTokenPos > l then exit
+
=== Important note on Ranges ===
  else
+
 
  if FLineText[FTokenEnd] in [#9, ' '] then
+
The purpose of the range is to allow the HL to start scanning in any line. The HL will not never need to look at a previous line. Any information needed to scan the current line can be derived from the ranges value.
  while (FTokenEnd <= l) and (FLineText[FTokenEnd] in [#0..#32]) do inc (FTokenEnd)
+
 
  else
+
Example:
  while (FTokenEnd <= l) and not(FLineText[FTokenEnd] in [#9, ' ']) do inc (FTokenEnd)
+
<syntaxhighlight lang="pascal">
end;
+
  writeln; (*
 +
   readln;
 +
   *)
 +
</syntaxhighlight>
 +
 
 +
when scanning the "readln" the HL knows from the range that it is in a comment, it does not need to look back at previous lines.
 +
 
 +
Therefore scanning can start at any line.
 +
 
 +
This also explains the note in the previous chapter. "until a line returns the same range that it already had". For even if the text of a line was not changed, if the value of the range at the lines start changed, then the scan result will change too.
  
procedure TSynDemoHl.SetLine(const NewValue: String; LineNumber: Integer);
+
== Step 3: Add Folding ==
begin
+
As indicated, the '''FoldHl''' unit demonstrates this process
  inherited;
 
  FTokenPos := 1;
 
  FLineText := NewValue;
 
  FindTokenEnd;
 
end;
 
  
procedure TSynDemoHl.GetTokenEx(out TokenStart: PChar; out TokenLength: integer);
+
For the example, the highlighter should fold everything between free-standing "-(-", "-)-".
begin
 
  TokenStart := @FLineText[FTokenPos];
 
  TokenLength := FTokenEnd - FTokenPos;
 
end;
 
  
function TSynDemoHl.GetTokenAttribute: TSynHighlighterAttributes;
+
Change inheritance:
begin
 
  if FLineText[FTokenPos] in [#9, ' '] then
 
    Result := SpaceAttri
 
  else
 
  if LowerCase(copy(FLineText, FTokenPos, FTokenEnd-FTokenPos)) = 'comment' then
 
    Result := CommentAttri
 
  else
 
    Result := IdentifierAttri;
 
end;
 
  
procedure TSynDemoHl.Next;
+
<syntaxhighlight lang="pascal">
begin
+
  uses SynEditHighlighterFoldBase;
   FTokenPos := FTokenEnd;
+
  ...
  FindTokenEnd;
+
   TSynDemoHl = class(TSynCustomFoldHighlighter)
end;
+
</syntaxhighlight>
  
function TSynDemoHl.GetEol: Boolean;
+
Change the way range info is stored, since the base class uses it for fold-info:
begin
 
  Result := FTokenPos > length(FLineText);
 
end;
 
  
function TSynDemoHl.GetDefaultAttribute(Index: integer): TSynHighlighterAttributes;
+
<syntaxhighlight lang="pascal">
 +
procedure TSynDemoHl.SetRange(Value: Pointer);
 
begin
 
begin
   case Index of
+
   inherited;
    SYN_ATTR_COMMENT: Result := fCommentAttri;
+
  FCurRange := PtrInt(CodeFoldRange.RangeType);
    SYN_ATTR_IDENTIFIER: Result := fIdentifierAttri;
 
    SYN_ATTR_WHITESPACE: Result := fSpaceAttri;
 
    else Result := nil;
 
  end;
 
 
end;
 
end;
  
function TSynDemoHl.GetToken: String;
+
procedure TSynDemoHl.ResetRange;
 
begin
 
begin
   Result := copy(FLineText, FTokenPos, FTokenEnd - FTokenPos);
+
   inherited;
 +
  FCurRange := 0;
 
end;
 
end;
  
function TSynDemoHl.GetTokenPos: Integer;
+
function TSynDemoHl.GetRange: Pointer;
 
begin
 
begin
   Result := FTokenPos - 1;
+
   CodeFoldRange.RangeType := Pointer(PtrInt(FCurRange));
 +
  inherited;
 
end;
 
end;
 +
</syntaxhighlight>
  
function TSynDemoHl.GetTokenKind: integer;
+
Now add code to the scanner which tells the highlighter about opening and closing folds:
var
 
  a: TSynHighlighterAttributes;
 
begin
 
  a := GetTokenAttribute;
 
  Result := 0;
 
  if a = fSpaceAttri then Result := 1;
 
  if a = fCommentAttri then Result := 2;
 
  if a = fIdentifierAttri then Result := 3;
 
end;
 
  
constructor TSynDemoHl.Create(AOwner: TComponent);
+
<syntaxhighlight lang="pascal">
 +
procedure TSynDemoHl.FindTokenEnd;
 
begin
 
begin
  inherited Create(AOwner);
+
  ...
  fCommentAttri := TSynHighlighterAttributes.Create('comment', 'comment');
 
  AddAttribute(fCommentAttri);
 
 
 
  fIdentifierAttri := TSynHighlighterAttributes.Create('ident', 'ident');
 
  fIdentifierAttri.Style := [fsBold];
 
  AddAttribute(fIdentifierAttri);
 
  
   fSpaceAttri := TSynHighlighterAttributes.Create('space', 'space');
+
   if (FTokenEnd = FTokenPos+1) and (FLineText[FTokenPos] = '[') then
  AddAttribute(fSpaceAttri);
+
    StartCodeFoldBlock(nil);
 +
  if (FTokenEnd = FTokenPos+1) and (FLineText[FTokenPos] = ']') then
 +
    EndCodeFoldBlock();
 
end;
 
end;
</delphi>
+
</syntaxhighlight>
  
<delphi>
+
* ''' For 0.9.30 '''
procedure Form1.Init;
+
Please see the history of this page, if you are using a 0.9.30 version [[http://wiki.lazarus.freepascal.org/index.php?title=SynEdit_Highlighter&oldid=76020]]
var
 
  hl: TSynDemoHl;
 
begin
 
  hl := TSynDemoHl.Create(Self);
 
  SynEdit1.Highlighter := hl;
 
  hl.CommentAttri.Foreground := clRed;
 
  hl.IdentifierAttri.Foreground := clGreen;
 
end;
 
</delphi>
 
 
 
== Step 2: Using Ranges ==
 
 
 
The next example allows content of a line, influences other lines that follows. E.g if a "(*" in pascal makes all following lines a comment until a "*)" is found.
 
 
 
In this example all content between a free-standing " ( " and a " ) " (note the spaces around the brackets) will be highlighted differently
 
  
 +
==== More info on StartCodeFoldBlock / EndCodeFoldBlock ====
  
We add an attribute:
+
<syntaxhighlight lang="pascal">
<delphi>
+
   function StartCodeFoldBlock(ABlockType: Pointer; IncreaseLevel: Boolean = true): TSynCustomCodeFoldBlock; virtual;
   private
+
</syntaxhighlight>
    fBlockAttri: TSynHighlighterAttributes;
 
  published
 
    property BlockAttri: TSynHighlighterAttributes read fBlockAttri
 
      write fBlockAttri;
 
  
constructor TSynDemoHl.Create(AOwner: TComponent);
+
;ABlockType: Can be used to specify an ID for the block.<br/>
begin
+
The field is not normally used as a pointer (though this is permitted). Normally IDs are a numeric enumeration.<br/>
  // ......
+
If you have different types of block (e.g. in Pascal: begin/end; repeat/until, ...), and you do not want a block being closed by the wrong keyword ("end" should not close "repeat"), then you can give them IDs:<br/>
 +
<b>StartCodeFoldBlock(PtrUInt(1))</b> // or other numbers<br/>
  
  fBlockAttri := TSynHighlighterAttributes.Create('block', 'block');
+
;IncreaseLevel: If set to False, then a block will be inserted that can not be folded.<br/> Blocks like that can be used for internal tracking.
  AddAttribute(fBlockAttri);
+
NOTE: All folds must be nested, they can not overlap. That is, the last opened fold must be closed first.<br/>
end;
+
This refers to the "EndLvl" as shown in "Folding"(1.3) above<br/>
</delphi>
+
Overlaps (like IFDEF and begin in the IDE) can not be done this way.
  
And the methods to store the currrent nest level of ():
+
<syntaxhighlight lang="pascal">
 +
  procedure EndCodeFoldBlock(DecreaseLevel: Boolean = True); virtual;
 +
</syntaxhighlight>
  
;GetRange: Called after a line is completly scanned, to get the value at the end of the line. The value will be stored.
+
;DecreaseLevel: This *must* match IncreaseLevel, as it was given on the StartCodeFoldBlock<br/>
;SetRange: Called before a line get's scanned. Sets the value stored from the end of the previous line.
+
True means a fold ends; False means an internal block is ended.
;ResetRange: Called before the 1st line is scanned (As there is no previous line).
+
If mismatched then folds will either continue, or end early.<br/>
 +
TopCodeFoldBlockType can be used to indicate the ID of the innermost open block. One can use different IDs for internal blocks, and use this to set the value.
  
<delphi>
+
<syntaxhighlight lang="pascal">
   private
+
   function TopCodeFoldBlockType(DownIndex: Integer = 0): Pointer;
    FCurRange: Integer;
+
</syntaxhighlight>
  public
+
Returns the ID of the innermost block.
    procedure SetRange(Value: Pointer); override;
 
    procedure ResetRange; override;
 
    function GetRange: Pointer; override;
 
  
procedure TSynDemoHl.SetRange(Value: Pointer);
+
;DownIndex: can be used to get ID for the other block.  
begin
+
DownIndex=1 means the block that is surrounding the innermost.
  FCurRange := PtrInt(Value);
 
end;
 
  
procedure TSynDemoHl.ResetRange;
+
= Configurable Highlighters (incl. 3rd party) =
begin
 
  FCurRange := 0;
 
end;
 
  
function TSynDemoHl.GetRange: Pointer;
+
== SynAnySyn ==
begin
 
  Result := Pointer(PtrInt(FCurRange));
 
end;
 
</delphi>
 
  
Then we extend the scanner. The pre-scan to store the information, calls the same functions than the highlighter, and is automatically called, if anything changes. (It is called for all lines below the changed line, until a line returns the same Range-value as it already had)
+
A real simple Highlighter. More a reference implementation, than a real life useable Highlighter.
  
* NOTE: A scan is triggered by *every* change to a line (every keystroke). It scans the current line, and all lines below, until a line returns the same range that it already had.
+
== SynUniHighlighter ==
  
<delphi>
+
Uses XML-based configurations files. [http://web.archive.org/web/20090610040212/http://www.delphist.com:80/UniHighlighter.html old official link]
procedure TSynDemoHl.FindTokenEnd;
 
var
 
  l: Integer;
 
begin
 
  l := length(FLineText);
 
  FTokenEnd := FTokenPos;
 
  If FTokenPos > l then exit
 
  else
 
  if FLineText[FTokenEnd] in [#9, ' '] then
 
  while (FTokenEnd <= l) and (FLineText[FTokenEnd] in [#0..#32]) do inc (FTokenEnd)
 
  else
 
  while (FTokenEnd <= l) and not(FLineText[FTokenEnd] in [#9, ' ']) do inc (FTokenEnd);
 
  
  // NEW: Check for ()
+
== SynFacilSyn ==
  if (FTokenEnd = FTokenPos+1) and (FLineText[FTokenPos] = '(') then
 
    inc(FCurRange);
 
  if (FTokenEnd = FTokenPos+1) and (FLineText[FTokenPos] = ')') and (FCurRange > 0) then
 
    dec(FCurRange);
 
end;
 
  
function TSynDemoHl.GetTokenAttribute: TSynHighlighterAttributes;
+
Flexible fully configurable Highlighter.
begin
+
https://github.com/t-edson/SynFacilSyn
  if FCurRange > 0 then
 
    Result := BlockAttri
 
  else
 
  if FLineText[FTokenPos] in [#9, ' '] then
 
    Result := SpaceAttri
 
  else
 
  if LowerCase(copy(FLineText, FTokenPos, FTokenEnd-FTokenPos)) = 'comment' then
 
    Result := CommentAttri
 
  else
 
    Result := IdentifierAttri;
 
end;
 
</delphi>
 
  
 
= References =
 
= References =
 
Threads on the forum:
 
Threads on the forum:
  
http://www.lazarus.freepascal.org/index.php/topic,10260.0.html
+
* "Making highligher for C syntaxis" [[http://www.lazarus.freepascal.org/index.php/topic,10260.0.html topic,10260]]
 
+
* "Creating a Syntax Highlighter" [[http://www.lazarus.freepascal.org/index.php/topic,10959.msg54714 topic,10959.msg54714]]
http://www.lazarus.freepascal.org/index.php/topic,7879.0.html
+
* obtaining highlight for printing [[http://www.lazarus.freepascal.org/index.php/topic,11384.msg57160.html#msg57160 11384.msg57160]]
 
+
* Some details on ranges, and using objects to store more info [[http://forum.lazarus.freepascal.org/index.php/topic,21727.msg139354.html#msg139354 topic,21727.msg139354]] and [[http://forum.lazarus.freepascal.org/index.php/topic,21727.msg139419.html#msg139419 topic,21727.msg139419]]
http://www.lazarus.freepascal.org/index.php/topic,7338.0.html
+
* More on ranges, working copy vs immutable, foldblocks... [[http://forum.lazarus.freepascal.org/index.php/topic,30685.msg195464.html#msg195464 http://forum.lazarus.freepascal.org/index.php/topic,30685]]
 
+
* Markup (Highlight) all occurrences of a token (or regex) [[http://forum.lazarus.freepascal.org/index.php?topic=26833 topic=26833]]
http://www.lazarus.freepascal.org/index.php/topic,10959.msg54714
+
* Markup and foldinfo: [[http://forum.lazarus.freepascal.org/index.php?topic=30122]]
 +
* How to do invalidation in TSynEditMarkup [[http://forum.lazarus.freepascal.org/index.php/topic,30122.msg207931.html#msg207931]]
  
http://www.lazarus.freepascal.org/index.php/topic,11064
+
* Folding
 +
** "SynEdit - improved highlighters to handle Code Folding?" [[http://www.lazarus.freepascal.org/index.php/topic,7879.0.html topic,7879]]
 +
** "SynEdit - Add support code folding for Java" [[http://www.lazarus.freepascal.org/index.php/topic,7338.0.html topic,7338]]
 +
** "CodeFolding" Config [[http://www.lazarus.freepascal.org/index.php/topic,11064 topic,11064]]
 +
** Fold blocks "end" keyword (end on the line before the next keyword) [[http://forum.lazarus.freepascal.org/index.php/topic,23411.msg139621.html#msg139621 topic,23411.msg139621]]
 +
** Folding selected text from code (user/application code): [[http://forum.lazarus.freepascal.org/index.php/topic,24473.msg147312.html#msg147312 24473.msg147312]]
 +
** Obtaining state of folding (save fold state with session) [[http://forum.lazarus.freepascal.org/index.php?topic=26748 topic=26748]]
  
http://www.lazarus.freepascal.org/index.php/topic,11384.msg57160.html#msg57160 (obtaining highlight for printing)
+
[[Category:SynEdit]]

Latest revision as of 10:03, 5 April 2020

English (en) русский (ru)

For more info on SynEdit go to: SynEdit
Also see SynEdit Markup


Understanding the SynEdit Highlighter

SynEdit - Highlighter relationship

SynEdit to Highlighter have an N to 1 relationship.

  • One instance of a Highlighter can serve N (many) instances of SynEdits
  • Each SynEdit only has one Highlighter
  • But: one text (text-buffer) can have many highlighters, if shared by several SynEdit (each SynEdit will have one HL, but all HL will work on the same document)

As a result of this:

  • no Highlighter Instance has a (fixed) reference to the SynEdit.
(Highlighters however keep a list of SynEditTextBuffers to which they are attached)
  • All data for the Highlighter is (and must be) stored on the SynEdit (actually on the TextBuffer of SynEdit (referred to as "Lines").

However, before each call to the Highlighter, SynEdit ensures that Highlighter.CurrentLines is set to the current SynEdits Lines. This way the highlighter can access the data whenever needed. The Format of the data storage is determined by the highlighter (TSynCustomHighlighter.AttachToLines).

Scanning and Returning Highlight attributes

The Highlighter is expected to work on a per Line base.

If any text was modified, SynEdit will call (TSynCustomHighlighter.ScanFrom / Currently called from TSynEdit.ScanFrom) with the line range. The Highlighter should know the state of the previous line.

If Highlight attributes are required, SynEdit will request them per Line too. SynEdit will loop through individual tokens on a line. This currently happens from nested proc PaintLines in SynEdit.PaintTextLines. It calls TSynCustomHighlighter.StartAtLineIndex, followed by HL.GetTokenEx/HL.GetTokenAttribute for as long as HL.GetEol is false.

Also the BaseClass for the Highlighter's data (see AttachToLines) is based on per line storage, and SynEdit's TextBuffer (Lines) do maintenance on this data to keep it synchronized. That is: whenever lines of text are inserted or removed, so are entries inserted or removed from the highlighters data (hence it must have one entry per line).

Usually Highlighters store the end-of-line-status in this field. So if the highlighter is going to work on a line, it will continue with the state-entry from the previous line.

Folding

SynEdit's folding is handled by unit SynEditFoldedView and SynGutterCodeFolding. Highlighters that implement folding are to be based on TSynCustomFoldHighlighter.

The basic information for communication between SynEditFoldedView and the HL requires 2 values stored for each line. (Of course the highlighter itself can store more information):

  • FoldLevel at the end of line
  • Minimum FoldLevel encountered anywhere on the line

The Foldlevel indicates how many (nested) folds exist. It goes up whenever a fold begins, and down when a fold ends:

                           EndLvl   MinLvl
 Procedure a;               1 -      0
 Begin                      2 --     1 -
   b:= 1;                   2 --     2 --
   if c > b then begin      3 ---    2 --
     c:=b;                  3 ---    3 ---
   end else begin           3 ---    2 --
     b:=c;                  3 ---    3 ---
   end;                     2 --     2 --
 end;                       0        0  // The end closes both: begin and procedure fold

In the line

 Procedure a;               1 -      0

the MinLvl is 0, because the line started with a Level of 0 (and it never went down / no folds closed). Similar in all lines where there is only an opening fold keyword ("begin").

But the line

   end else begin           3 ---    2 --

starts with a level of 3, and also ends with it (one close, one open). But since it went down first, the minimum level encountered anywhere on the line is 2.

Without the MinLvl it would not be possible to tell that a fold ends in this line.

There is no such thing as a MaxLvl, because folds that start and end on the same line can not be folded anyway. No need to detect them.

 if a then begin b:=1; c:=2; end; // no fold on that line


Creating a SynEdit Highlighter

Since 0.9.31 Revision 35115 the fold-highlighter has changed. Implementing basic folding is now easier.

All Sources can be found in the Lazarus installation directory under:

 examples\SynEdit\NewHighlighterTutorial\

The project HighlighterTutorial contains 3 difference example highlighters:

  • SimpleHl: used in Step 1 below
  • ContextHl: used in Step 2 below
  • FoldHl: used in Step 3 below

SimpleHl and ContextHl will work with 0.9.30 too

The folding highlighter in action:

SynEditFoldingHighlighterDemo.png

The Basics: Returning Tokens and Attributes

As indicated, the SimpleHl unit demonstrates this process.

What it does

  • It splits each line into words and spaces (or tabs)
    • The spaces are part of the text, and must be highlighted too
  • This example allows specifying different colors for
- text (defaults to not-highlighted)
- spaces (defaults to silver frame)
- words, separated by spaces, that start with a,e,i,o,u (defaults to bold)
- the word "not" (defaults to red background)

How it works

  • Creation

The Highlighter creates Attributes that it can return the Words and Spaces.

  • SetLine

Is called by SynEdit before a line gets painted (or before highlight info is needed)

  • GetTokenEx, GetTokenAttribute, Next, GetEol

Are used by SynEdit to iterate over the Line. Note that the first Token (Word or Spaces) must be ready after SetLine, without a call to Next.

Important: The tokens returned for each line must represent the original line-text, and be returned in the correct order.

  • GetToken, GetTokenPos, GetTokenKind

SynEdit uses them e.g for finding matching brackets. If tokenKind returns different values per Attribute, then brackets only match, if they are of the same kind (e.g. if there was a string attribute, brackets outside a string would not match brackets inside a string).

Other notes

For readability, the highlighter has no optimization, so it may be very slow on larger texts. Many of the supplied highlighters use hash functions, to find what word (or any group of chars) is.

Step 2: Using Ranges

As indicated, the ContextHl unit demonstrates this process

The next example allows content of a line that influences other lines that follow. An example: a "(*" in Pascal makes all following lines a comment until a "*)" is found.

This example extends the SimpleHl show above: The tokens -- and ++ (must be surrounded by space or line-begin/end to be a token of their own) will toggle words that start with a,e,i,o,u

Multiple ++ and -- can be nested. Then for each -- a ++ must be given, before the words highlight again.

Then we extend the scanner. The pre-scan to store the information calls the same functions as the highlighter. It is automatically called, if anything changes. (It is called for all lines below the changed line, until a line returns the same Range-value as it already had)

The current amount of "--" is counted in

  FCurRange: Integer;

The amount is decreased by "++"

To store the info we use:

GetRange
Called after a line is completely scanned, to get the value at the end of the line. The value will be stored.
SetRange
Called before a line gets scanned. Sets the value stored from the end of the previous line.
ResetRange
Called before the 1st line is scanned (as there is no previous line).
Note-icon.png

Note: A scan is triggered by *every* change to a line (every keystroke). It scans the current line, and all lines below, until a line returns the same range that it already had. See: http://forum.lazarus.freepascal.org/index.php/topic,21727.msg139420.html#msg139420

Important note on Ranges

The purpose of the range is to allow the HL to start scanning in any line. The HL will not never need to look at a previous line. Any information needed to scan the current line can be derived from the ranges value.

Example:

  writeln; (*
  readln;
  *)

when scanning the "readln" the HL knows from the range that it is in a comment, it does not need to look back at previous lines.

Therefore scanning can start at any line.

This also explains the note in the previous chapter. "until a line returns the same range that it already had". For even if the text of a line was not changed, if the value of the range at the lines start changed, then the scan result will change too.

Step 3: Add Folding

As indicated, the FoldHl unit demonstrates this process

For the example, the highlighter should fold everything between free-standing "-(-", "-)-".

Change inheritance:

  uses SynEditHighlighterFoldBase;
  ...
  TSynDemoHl = class(TSynCustomFoldHighlighter)

Change the way range info is stored, since the base class uses it for fold-info:

procedure TSynDemoHl.SetRange(Value: Pointer);
begin
  inherited;
  FCurRange := PtrInt(CodeFoldRange.RangeType);
end;

procedure TSynDemoHl.ResetRange;
begin
  inherited;
  FCurRange := 0;
end;

function TSynDemoHl.GetRange: Pointer;
begin
  CodeFoldRange.RangeType := Pointer(PtrInt(FCurRange));
  inherited;
end;

Now add code to the scanner which tells the highlighter about opening and closing folds:

procedure TSynDemoHl.FindTokenEnd;
begin
   ...

  if (FTokenEnd = FTokenPos+1) and (FLineText[FTokenPos] = '[') then
    StartCodeFoldBlock(nil);
  if (FTokenEnd = FTokenPos+1) and (FLineText[FTokenPos] = ']') then
    EndCodeFoldBlock();
end;
  • For 0.9.30

Please see the history of this page, if you are using a 0.9.30 version [[1]]

More info on StartCodeFoldBlock / EndCodeFoldBlock

  function StartCodeFoldBlock(ABlockType: Pointer; IncreaseLevel: Boolean = true): TSynCustomCodeFoldBlock; virtual;
ABlockType
Can be used to specify an ID for the block.

The field is not normally used as a pointer (though this is permitted). Normally IDs are a numeric enumeration.
If you have different types of block (e.g. in Pascal: begin/end; repeat/until, ...), and you do not want a block being closed by the wrong keyword ("end" should not close "repeat"), then you can give them IDs:
StartCodeFoldBlock(PtrUInt(1)) // or other numbers

IncreaseLevel
If set to False, then a block will be inserted that can not be folded.
Blocks like that can be used for internal tracking.

NOTE: All folds must be nested, they can not overlap. That is, the last opened fold must be closed first.
This refers to the "EndLvl" as shown in "Folding"(1.3) above
Overlaps (like IFDEF and begin in the IDE) can not be done this way.

  procedure EndCodeFoldBlock(DecreaseLevel: Boolean = True); virtual;
DecreaseLevel
This *must* match IncreaseLevel, as it was given on the StartCodeFoldBlock

True means a fold ends; False means an internal block is ended. If mismatched then folds will either continue, or end early.
TopCodeFoldBlockType can be used to indicate the ID of the innermost open block. One can use different IDs for internal blocks, and use this to set the value.

  function TopCodeFoldBlockType(DownIndex: Integer = 0): Pointer;

Returns the ID of the innermost block.

DownIndex
can be used to get ID for the other block.

DownIndex=1 means the block that is surrounding the innermost.

Configurable Highlighters (incl. 3rd party)

SynAnySyn

A real simple Highlighter. More a reference implementation, than a real life useable Highlighter.

SynUniHighlighter

Uses XML-based configurations files. old official link

SynFacilSyn

Flexible fully configurable Highlighter. https://github.com/t-edson/SynFacilSyn

References

Threads on the forum:

  • Folding
    • "SynEdit - improved highlighters to handle Code Folding?" [topic,7879]
    • "SynEdit - Add support code folding for Java" [topic,7338]
    • "CodeFolding" Config [topic,11064]
    • Fold blocks "end" keyword (end on the line before the next keyword) [topic,23411.msg139621]
    • Folding selected text from code (user/application code): [24473.msg147312]
    • Obtaining state of folding (save fold state with session) [topic=26748]