Difference between revisions of "Codetools/de"

From Lazarus wiki
Jump to navigationJump to search
m
m (updated; translated the first part)
Line 1: Line 1:
 
{{Codetools}}
 
{{Codetools}}
  
== Was sind die CodeTools ==
+
= Was sind die CodeTools =
  
 
Die CodeTools sind ein Lazarus-Package mit Werkzeugen zum Parsen, Untersuchen, Bearbeiten und Refaktorieren von Pascal-Quelltexten.
 
Die CodeTools sind ein Lazarus-Package mit Werkzeugen zum Parsen, Untersuchen, Bearbeiten und Refaktorieren von Pascal-Quelltexten.
Die CodeTools sind ein Modul für sich und unter der GPL lizensiert. Zahlreiche Beispiele darüber, wie Sie die CodeTools in Ihren eigenen Programmen einsetzen können, finden Sie unter 'components/codetools/examples'.
+
Die CodeTools sind ein Modul für sich und unter der GPL lizensiert. Zahlreiche Beispiele, wie Sie die CodeTools in Ihren eigenen Programmen einsetzen können, finden Sie unter 'components/codetools/examples'.
  
 
svn:
 
svn:
 
*Lazarus: http://svn.freepascal.org/svn/lazarus/trunk
 
*Lazarus: http://svn.freepascal.org/svn/lazarus/trunk
*Only CodeTools: http://svn.freepascal.org/svn/lazarus/trunk/components/codetools
+
*Nur CodeTools: http://svn.freepascal.org/svn/lazarus/trunk/components/codetools
  
== Verwendung der CodeTools ohne die IDE (gut für Testzwecke) ==
+
= Verwendung der CodeTools ohne die IDE (gut für Testzwecke) =
  
 
Sie können die CodeTools ohne die IDE verwenden. Dies kann zum Testen eines neuen Werkzeugs genutzt werden. Ein einfaches Beispiel ist
 
Sie können die CodeTools ohne die IDE verwenden. Dies kann zum Testen eines neuen Werkzeugs genutzt werden. Ein einfaches Beispiel ist
 
   <lazarusdir>/components/codetools/examples/methodjumping.lpi
 
   <lazarusdir>/components/codetools/examples/methodjumping.lpi
  
Um die ''find declaration'' zu testen, müssen die CodeTools die Quellen analysieren, besonders die RTL- und FCL-Quellen. FPC ist ein sehr komplexes Projekt mit vielen Suchpfaden, Include-Dateien und Makros. Die CodeTools müssen alle diese Pfade und Makros kennen, um diesen Dschungel zu analysieren. Um dies alles einfach einzurichten, enthalten die CodeTools vordefinierte Schablonen für FPC, Lazarus, [[Glossary#Delphi|Delphi]] und [[Glossary#Kylix|Kylix]] Quellverzeichnisse.  
+
Um die ''find declaration'' zu testen, müssen die CodeTools die Quellen analysieren, besonders die RTL- und FCL-Quellen. Die Beispiele benutzen folgende Umgebungsvariablen:
 +
 
 +
*FPCDIR: der Pfad zu den FPC-Quelltexten, die Vorgabe ist ''~/freepascal/fpc''.
 +
*PP: der Pfad zum ausführbaren Kompiler (/usr/bin/fpc oder /usr/bin/ppc386 oder C:\lazarus\ppc386.exe). Die CodeTools müssen den Kompiler nach den Einstellungen fragen. Als Vorgabe wird mittels der Variablen 'PATH' nach 'fpc' gesucht.
 +
*FPCTARGETOS: weist die CodeTools an, nach einem weiteren Betriebssystem zu suchen ("cross compiling"). Zum Beispiel: linux, freebsd, darwin, win32, win64, wince
 +
*FPCTARGETCPU: sucht nach einer anderen CPU. Zum Beispiel: i386, powerpc, x86_64, arm, sparc
 +
*LAZARUSDIR: der Pfad zu den Lazarus-Quelltexten. Wird nur benötigt, wenn Sie diese untersuchen wollen.
 +
 
 +
FPC ist ein sehr komplexes Projekt mit vielen Suchpfaden, Include-Dateien und Makros. Die CodeTools müssen alle diese Pfade und Makros kennen, um diesen Dschungel zu analysieren. Um dies alles einfach einzurichten, enthalten die CodeTools vordefinierte Schablonen für FPC-, Lazarus-, Delphi- und Kylix-Quellverzeichnisse.  
 
Betrachten Sie für ein ''find declaration'' Beispiel
 
Betrachten Sie für ein ''find declaration'' Beispiel
 
   <lazarusdir>/components/codetools/examples/finddeclaration.lpi
 
   <lazarusdir>/components/codetools/examples/finddeclaration.lpi
  
Weil die FPC Quellen mehrere Versionen einiger Units enthalten und die FPC Quellen oft geändert werden, verwenden die CodeTools keine starre Pfadtabelle. Stattdessen scannen sie zuerst die gesamte FPC Verzeichnisstruktur und versuchen zu raten, welche Quelle die richtige ist für das gegenwärtige ZielOS und ZielCPU. Dieser Scan kann eine Weile dauern. Damit wird das Ergebnis gespeichert. Wann immer sich die FPC Quellen bewegt haben oder eine Unit umbenannt wurde, muss dieser Scan wiederholt werden. Die IDE erledigt diesen Rescan, wann immer das Compiler executable geändert wurde oder der Benutzer dies erzwingt mit 'Einstellungen > FPC-Quelltextverzeichnis neu einlesen'.
+
Weil die FPC-Quellen mehrere Versionen einiger Units enthalten und die FPC-Quellen oft geändert werden, verwenden die CodeTools keine starre Pfadtabelle. Stattdessen scannen sie zuerst die gesamte FPC Verzeichnisstruktur und versuchen mittels einiger Regeln herauszufinden, welche Quelle die richtige ist für das gegenwärtige ZielOS und ZielCPU. Dieser Scan kann eine Weile dauern. Alle Beipiele speichern das Ergebnis in '''codetools.config''', damit diese Suche beim nächsten Start übersprungen wird.
 +
 
 +
Wann immer die FPC-Quellen verschoben wurden oder eine Unit umbenannt wurde, löschen Sie einfach die Datei '''codetools.config'''. Die Lazarus-IDE hat ihre eigene Konfigurationsdatei und erledigt den Rescan, wann immer das Compiler executable geändert wurde oder der Benutzer dies erzwingt mit 'Einstellungen > FPC-Quelltextverzeichnis neu einlesen'.
  
== Verwendung der CodeTools in der IDE mit IDEIntf ==
+
= Verwendung der CodeTools in der IDE mit dem IDEIntf =
  
 
Siehe das <lazarusdir>/examples/idequickfix/quickfixexample.lpk Package. Es demonstriert:
 
Siehe das <lazarusdir>/examples/idequickfix/quickfixexample.lpk Package. Es demonstriert:
* Wie man ein IDE Package schreibt.
+
* Wie man ein IDE-Package schreibt. Wenn Sie es installieren, wird es ein Quick-Fix-Element registrieren.
  Wenn Sie es installieren, wird es ein Quick Fix Element registrieren.
+
* Wie man ein Quick-Fix-Element schreibt für die Compilermeldung 'Parameter "Sender" not used'
* Wie man ein Quick Fix Element schreibt für Compilermeldungen 'Parameter "Sender" not used'
+
* Wie man die CodeTools verwendet, um
* Wie man die CodeTools verwendet um
+
** eine Unit zu analysieren
      * eine Unit zu analysieren
+
** Dateiname, Zeile, Spalte in eine CodeTools Quelltextposition umzuwandeln
      * conversion of Filename,Line,Column to codetools source position
+
** ein CodeTools-Element an der Cursorposition zu finden
      * finding a codetools node at a cursor position
+
** ein 'procedure'-Element und das 'begin..end'-Element zu finden
      * finding a procedure node and the begin..end node
+
** für eine Anweisung eine günstige Einfügeposition am Anfang eines 'begin..end'-Blocks zu ermitteln
      * creating a nice insertion position for a statement at the beginning of
+
** die Einrückung einer Zeile zu erhalten, damit sich die neue Zeile in die Prozedur einfügt
        the begin..end block
+
** Code einzufügen mit den CodeTools
      * getting the indentation of a line, so that the new line will
+
 
        work in sub procedure as well
+
=Codetools rules for FPC sources=
      * Code einzufügen mit den codetools
+
 
 +
When the codetools searches the source of a fpc ppu it uses a set of rules. You can write your own rules, but normally you will use the standard rules, which are defined in the include file components/codetools/fpcsrcrules.inc. You can test the rules with the command line utility: components/codetools/examples/testfpcsrcunitrules.lpi.
 +
 
 +
==Usage of testfpcsrcunitrules==
 +
 
 +
<pre>
 +
Usage: lazarus/components/codetools/examples/testfpcsrcunitrules -h
 +
 
 +
  -c <compiler file name>, --compiler=<compiler file name>
 +
        Default is to use environment variable PP.
 +
        If this is not set, search for fpc
 +
 
 +
  -T <target OS>, --targetos=<target OS>
 +
        Default is to use environment variable FPCTARGET.
 +
        If this is not set, use the default of the compiler.
 +
 
 +
  -P <target CPU>, --targetcpu=<target CPU>
 +
        Default is to use environment variable FPCTARGETCPU.
 +
        If this is not set, use the default of the compiler.
 +
 
 +
  -F <FPC source directory>, --fpcsrcdir=<FPC source directory>
 +
        Default is to use environment variable FPCDIR.
 +
        There is no default.
 +
 
 +
  -u <unit name>, --checkunit=<unit name>
 +
        Write a detailed report about this unit.
 +
</pre>
 +
 
 +
==Example for testfpcsrcunitrules==
 +
 
 +
Open the testfpcsrcunitrules.lpi in the IDE and compile it. Then run the utility in a terminal/console:
 +
./testfpcsrcunitrules -F ~/fpc/sources/2.5.1/fpc/
 +
 
 +
This will tell you what compiler is used, what compiler is executed, what config files were tested and parsed, it warns about duplicate units in the FPC search path and duplicate source files for the same unit.
 +
 
 +
==Duplicate source files==
 +
 
 +
You find out that the codetools opens for target wince/arm the wrong source of the unit ''mmsystem''. Run the tool with the -u parameter:
 +
 
 +
./testfpcsrcunitrules -F ~/fpc/2.5.1/fpc/ -T wince -P arm -u mmsystem
 +
 
 +
This will give you a detailed report where this unit was found and what score each source file got. For example:
 +
 
 +
Unit report for mmsystem
 +
  WARNING: mmsystem is not in PPU search path
 +
GatherUnitsInFPCSources UnitName=mmsystem File=packages/winunits-base/src/mmsystem.pp Score=11
 +
GatherUnitsInFPCSources UnitName=mmsystem File=packages/winceunits/src/mmsystem.pp Score=11 => duplicate
 +
 
 +
This means there are two source files with the same score, so the codetools took the first. The last one in winceunits is for target wince and the first one is for win32 and win64.
 +
 
 +
Now open the rules file fpcsrcrules.inc.
 +
 
 +
Rules work like this:
 +
<Delphi>
 +
Score:=10;
 +
Targets:='wince';
 +
Add('packages/winceunits');
 +
</Delphi>
 +
 
 +
The '''Add''' adds a rule for all files beginning with 'packages/winceunits' that adds a score of 10 to all these files. The '''Targets''' is a comma separated list of target operating systems and/or target processors. For example Targets='wince,linux,i386' means: apply this rules to TargetOS wince or linux and to all TargetCPU i386.
 +
 
 +
=How the codetools parses sources, difference to a compiler=
 +
 
 +
A compiler is optimized to parse code linear and load needed units and include files as soon as it parses a uses section or a directive.
 +
The codetools are optimized to parse only parts of code. For example jumping from the method declaration to the method body only needs the unit and its include files. When a codetool search a declaration it searches backwards. That means it starts searching in the local variables, then upwards the implementation. When it finds a uses section it searches the identifier in the interface section of the units. When the identifier is found it stops. The result and some middle steps are cached. Because it often only needs to parse some interface sections it finds a single identifier fast.
 +
 
 +
The codetools do not parse a source in one step like the compiler but in several steps, depending on the need of the current function:
 +
 
 +
*First a source file is loaded in a TCodeBuffer. The IDE uses this step to change the encoding to UTF8. The files are kept in memory and only reloaded if the modification date changes or a file is manually reverted. There are several tools and function which work directly on the buffer.
 +
*The next level is parsing a unit or include file. A unit must be parsed from the beginning, so the codetools try to find the main file, the first file of a unit. It does that by looking for a directive in the first line like {%MainUnit ../lclintf.pp}. If that does not exist, it searches in the includelink cache. The IDE saves this cache to disk, so the IDE learns over time.
 +
*After finding the main file TLinkScanner parses the source. It handles compiler directives, like include directives and if-else directives. The scanner can be given a range, so it can for instance only parse the interface of a unit. The scanner creates the '''clean source'''. The clean source is put together of all include files and stripped off of all skipped code in else parts. It also create a list of '''links''', that maps between the clean source and the real source files. The clean source is now pascal. Note: there are also tools to scan a single source for all directives and create a tree of directives.
 +
*After creating the clean source a TCodeTool parses it and creates a tree of TCodeTreeNode. It can also be given a range. This parser skips a few parts, for example class members, begin..end blocks and parameter lists. Many tools don't need them. These sub nodes are created on demand. A TCodeTreeNode has a range StartPos..EndPos which are clean positions, that means positions in the clean source. There are only nodes for the important parts. Creating nodes for every detail would need more memory than the source itself and is seldom needed. There are plenty of functions to find out the details. For example if a function has calling convention 'cdecl'.
 +
*When searching for an identifier the search stores the found base types and creates caches for all identifiers in the interface section.
 +
 
 +
Every level has its own caches, which need to be checked and updated before calling a function. Many high level functions accessible via the CodeToolBoss do that automatically. For others it is the responsibility of the caller.
 +
 
 +
Example for:
 +
 
 +
unti1.pas:
 +
 
 +
<Delphi>
 +
unit Unit1;
 +
{$I settings.inc}
 +
interface
 +
uses
 +
  {$IFDEF Flag}
 +
  unix,
 +
  {$ELSE}
 +
  windows,
 +
  {$ENDIF}
 +
  Classes;
 +
</Delphi>
 +
 
 +
settings.inc:
 +
 
 +
<Delphi>
 +
{%MainUnit unit1.pas}
 +
{$DEFINE Flag}
 +
</Delphi>
 +
 
 +
clean source:
 +
 
 +
<Delphi>
 +
unit Unit1;
 +
{$I settings.inc}{%MainUnit unit1.pas}
 +
{$DEFINE Flag}
 +
interface
 +
uses
 +
  {$IFDEF Flag}
 +
  unix,
 +
  {$ELSE}{$ENDIF}
 +
  Classes;
 +
</Delphi>
 +
 
 +
 
 +
Hint: To easily parse a unit and build the nodes, use '''CodeToolBoss.Explore'''.
 +
 
 +
=CleanPos and CursorPos=
 +
 
 +
There are several methods to define a position in the codetools.
 +
 
 +
Absolute position are related to a source as string and starts at 1. For example a TCodeBuffer holds the file content as one string in the property Source.
 +
Caret or cursor positions are given as X,Y, where Y is the line number starting at 1 and X is the column number starting at 1. A TCodeBuffer provides the member functions '''LineColToPosition''' and '''AbsoluteToLineCol'' to convert.
 +
When working with multiple source files, like a unit, that can consists of several include files, the clean position relates to the absolute position in the stripped code '''Src'''. Src which is a string and clean positions start at 1. Cursor positions are specified as TCodeXYPosition (Code,X,Y). A TCodeTool provides the functions '''CaretToCleanPos''', '''CleanPosToCaret''' to convert.
 +
 
 +
=Inserting, deleting, replacing - the TSourceChangeCache=
 +
 
 +
When making changes to the source code of a unit (or its include files) you should use the CodetoolBoss.SourceChangeCache, because:
 +
* Simple usage. Connect, Replace, Replace, ... Apply. See below.
 +
* You can use cleanpos as given by the node tree OR you can use direct position in a file.
 +
* You can use Replace to insert, delete, which automatically calls events, so connected editors are notified of changes.
 +
* It can automatically insert needed spaces, line breaks or empty lines in front or behind each Replace. For example you define that there should be an empty line in front. The SourceChangeCache checks what is inserted and how much space there is already and will insert needed space.
 +
* It checks if the replaced/deleted span is writable.
 +
* You can do multiple Replaces and you control when they are applied. Keep in mind that inserting code means that the parsed tree becomes invalid and needs rebuilding.
 +
* Multiple replaces are checked for intersection. For example an insert in the middle of deleted code gives an error.
 +
* Mutiple insertions at the same place are added FIFO - first at the top.
 +
* You can combine several functions altering code to one bigger function. See below.
 +
 
 +
==Usage==
 +
 
 +
The SourceChangeCache works on a unit, so you need to get a TCodeTool and scan a unit/include file. For example:
 +
 
 +
<Delphi>
 +
  // Step 1: load the file and parse it
 +
  Code:=CodeToolBoss.LoadFile(Filename,false,false);
 +
  if Code=nil then
 +
    raise Exception.Create('loading failed '+Filename);
 +
  if not CodeToolBoss.Explore(Code,Tool,false) then
 +
    ...;// parse error ...
 +
 
 +
  // Step 2: connect the SourceChangeCache
 +
  CodeToolBoss.SourceChangeCache.MainScanner:=Tool.Scanner;
 +
 
 +
  // Step 3: use Replace to insert and/or delete code
 +
  // The first two parameters are the needed spaces in front and behind the insertion
 +
  // The FromPos,ToPos defines the deleted/replaced range in CleanPos positions.
 +
  // The NewCode is the string of new code. Use '' for a delete.
 +
  if not CodeToolBoss.SourceChangeCache.Replace(gtNone,gtNone,FromPos,ToPos,NewCode) then
 +
    exit; // e.g. source read only or a former Replace has deleted the place
 +
  ...do some more Replace...
 +
 
 +
  // Step 4: Apply the changes
 +
  if not CodeToolBoss.SourceChangeCache.Apply then
 +
    exit; // apply was aborted
 +
</Delphi>
 +
 
 +
==BeginUpdate/EndUpdate==
 +
 
 +
BeginUpdate/EndUpdate delays the Apply. This is useful when combining several code changing functions. For example:
 +
 
 +
You want to scan the unit, add a unit to the interface uses section, and remove the unit from the implementation uses section. The two functions AddUnitToMainUsesSection and RemoveUnitFromUsesSection use Apply, altering the source, so the second function would rescan the unit a second time. But since the two functions are independent of each other (they change different parts of the source) you can combine them and do it with one scan:
 +
 
 +
<Delphi>
 +
  // Step 1: parse unit and connect SourceChangeCache
 +
  if not CodeToolBoss.Explore(Code,Tool,false) then
 +
    ...;// parse error ...
 +
  CodeToolBoss.SourceChangeCache.MainScanner:=Tool.Scanner;
 +
 
 +
  // Step 2: delay Apply
 +
  CodeToolBoss.SourceChangeCache.BeginUpdate;
 +
 
 +
  // Step 3: add unit to interface section
 +
  // AddUnitToMainUsesSection would apply and change the code
 +
  // Because of the BeginUpdate the change is not yet done, but stored in the SourceChangeCache
 +
  if not Tool.AddUnitToMainUsesSection('Classes','',CodeToolBoss.SourceChangeCache) then exit;
 +
 
 +
  // Step 4: remove unit from implementation section
 +
  // Without the BeginUpdate the RemoveUnitFromUsesSection would rescan the unit
 +
  if Tool.FindImplementationUsesSection<>nil then
 +
    if not Tool.RemoveUnitFromUsesSection(Tool.FindImplementationUsesSection,'Classes',CodeToolBoss.SourceChangeCache) then exit;
 +
 
 +
  // Step 5: apply all changes
 +
  if not CodeToolBoss.SourceChangeCache.EndUpdate then
 +
    exit; // apply was aborted
 +
</Delphi>
 +
 
 +
BeginUpdate/EndUpdate work with a counter, so if you call BeginUpdate twice you need to call EndUpdate twice. This means you can put the above example in a function and combine that with another function.
 +
 
 +
==Saving changes to disk==
 +
 
 +
The above changes are made to the code buffers and the buffers are marked modified. To save the changes to disk, you have to call Save for each modified buffer.
 +
*The buffers that will be modified in the next Apply/EndUpdate are in SourceChangeCache.BuffersToModify and BuffersToModifyCount.
 +
*The events SourceChangeCache.OnBeforeApplyChanges/OnAfterApplyChanges are used by the CodeToolBoss, which connects it to its own OnBeforeApplyChanges/OnAfterApplyChanges. The Lazarus IDE sets these events and automatically opens modified files in the source editor, so all changes go into the undo list of synedit.

Revision as of 18:53, 20 April 2011

Deutsch (de) English (en) français (fr) русский (ru)

Was sind die CodeTools

Die CodeTools sind ein Lazarus-Package mit Werkzeugen zum Parsen, Untersuchen, Bearbeiten und Refaktorieren von Pascal-Quelltexten. Die CodeTools sind ein Modul für sich und unter der GPL lizensiert. Zahlreiche Beispiele, wie Sie die CodeTools in Ihren eigenen Programmen einsetzen können, finden Sie unter 'components/codetools/examples'.

svn:

Verwendung der CodeTools ohne die IDE (gut für Testzwecke)

Sie können die CodeTools ohne die IDE verwenden. Dies kann zum Testen eines neuen Werkzeugs genutzt werden. Ein einfaches Beispiel ist

 <lazarusdir>/components/codetools/examples/methodjumping.lpi

Um die find declaration zu testen, müssen die CodeTools die Quellen analysieren, besonders die RTL- und FCL-Quellen. Die Beispiele benutzen folgende Umgebungsvariablen:

  • FPCDIR: der Pfad zu den FPC-Quelltexten, die Vorgabe ist ~/freepascal/fpc.
  • PP: der Pfad zum ausführbaren Kompiler (/usr/bin/fpc oder /usr/bin/ppc386 oder C:\lazarus\ppc386.exe). Die CodeTools müssen den Kompiler nach den Einstellungen fragen. Als Vorgabe wird mittels der Variablen 'PATH' nach 'fpc' gesucht.
  • FPCTARGETOS: weist die CodeTools an, nach einem weiteren Betriebssystem zu suchen ("cross compiling"). Zum Beispiel: linux, freebsd, darwin, win32, win64, wince
  • FPCTARGETCPU: sucht nach einer anderen CPU. Zum Beispiel: i386, powerpc, x86_64, arm, sparc
  • LAZARUSDIR: der Pfad zu den Lazarus-Quelltexten. Wird nur benötigt, wenn Sie diese untersuchen wollen.

FPC ist ein sehr komplexes Projekt mit vielen Suchpfaden, Include-Dateien und Makros. Die CodeTools müssen alle diese Pfade und Makros kennen, um diesen Dschungel zu analysieren. Um dies alles einfach einzurichten, enthalten die CodeTools vordefinierte Schablonen für FPC-, Lazarus-, Delphi- und Kylix-Quellverzeichnisse. Betrachten Sie für ein find declaration Beispiel

 <lazarusdir>/components/codetools/examples/finddeclaration.lpi

Weil die FPC-Quellen mehrere Versionen einiger Units enthalten und die FPC-Quellen oft geändert werden, verwenden die CodeTools keine starre Pfadtabelle. Stattdessen scannen sie zuerst die gesamte FPC Verzeichnisstruktur und versuchen mittels einiger Regeln herauszufinden, welche Quelle die richtige ist für das gegenwärtige ZielOS und ZielCPU. Dieser Scan kann eine Weile dauern. Alle Beipiele speichern das Ergebnis in codetools.config, damit diese Suche beim nächsten Start übersprungen wird.

Wann immer die FPC-Quellen verschoben wurden oder eine Unit umbenannt wurde, löschen Sie einfach die Datei codetools.config. Die Lazarus-IDE hat ihre eigene Konfigurationsdatei und erledigt den Rescan, wann immer das Compiler executable geändert wurde oder der Benutzer dies erzwingt mit 'Einstellungen > FPC-Quelltextverzeichnis neu einlesen'.

Verwendung der CodeTools in der IDE mit dem IDEIntf

Siehe das <lazarusdir>/examples/idequickfix/quickfixexample.lpk Package. Es demonstriert:

  • Wie man ein IDE-Package schreibt. Wenn Sie es installieren, wird es ein Quick-Fix-Element registrieren.
  • Wie man ein Quick-Fix-Element schreibt für die Compilermeldung 'Parameter "Sender" not used'
  • Wie man die CodeTools verwendet, um
    • eine Unit zu analysieren
    • Dateiname, Zeile, Spalte in eine CodeTools Quelltextposition umzuwandeln
    • ein CodeTools-Element an der Cursorposition zu finden
    • ein 'procedure'-Element und das 'begin..end'-Element zu finden
    • für eine Anweisung eine günstige Einfügeposition am Anfang eines 'begin..end'-Blocks zu ermitteln
    • die Einrückung einer Zeile zu erhalten, damit sich die neue Zeile in die Prozedur einfügt
    • Code einzufügen mit den CodeTools

Codetools rules for FPC sources

When the codetools searches the source of a fpc ppu it uses a set of rules. You can write your own rules, but normally you will use the standard rules, which are defined in the include file components/codetools/fpcsrcrules.inc. You can test the rules with the command line utility: components/codetools/examples/testfpcsrcunitrules.lpi.

Usage of testfpcsrcunitrules

Usage: lazarus/components/codetools/examples/testfpcsrcunitrules -h

  -c <compiler file name>, --compiler=<compiler file name>
         Default is to use environment variable PP.
         If this is not set, search for fpc

  -T <target OS>, --targetos=<target OS>
         Default is to use environment variable FPCTARGET.
         If this is not set, use the default of the compiler.

  -P <target CPU>, --targetcpu=<target CPU>
         Default is to use environment variable FPCTARGETCPU.
         If this is not set, use the default of the compiler.

  -F <FPC source directory>, --fpcsrcdir=<FPC source directory>
         Default is to use environment variable FPCDIR.
         There is no default.

  -u <unit name>, --checkunit=<unit name>
         Write a detailed report about this unit.

Example for testfpcsrcunitrules

Open the testfpcsrcunitrules.lpi in the IDE and compile it. Then run the utility in a terminal/console:

./testfpcsrcunitrules -F ~/fpc/sources/2.5.1/fpc/

This will tell you what compiler is used, what compiler is executed, what config files were tested and parsed, it warns about duplicate units in the FPC search path and duplicate source files for the same unit.

Duplicate source files

You find out that the codetools opens for target wince/arm the wrong source of the unit mmsystem. Run the tool with the -u parameter:

./testfpcsrcunitrules -F ~/fpc/2.5.1/fpc/ -T wince -P arm -u mmsystem

This will give you a detailed report where this unit was found and what score each source file got. For example:

Unit report for mmsystem
  WARNING: mmsystem is not in PPU search path
GatherUnitsInFPCSources UnitName=mmsystem File=packages/winunits-base/src/mmsystem.pp Score=11
GatherUnitsInFPCSources UnitName=mmsystem File=packages/winceunits/src/mmsystem.pp Score=11 => duplicate

This means there are two source files with the same score, so the codetools took the first. The last one in winceunits is for target wince and the first one is for win32 and win64.

Now open the rules file fpcsrcrules.inc.

Rules work like this: <Delphi> Score:=10; Targets:='wince'; Add('packages/winceunits'); </Delphi>

The Add adds a rule for all files beginning with 'packages/winceunits' that adds a score of 10 to all these files. The Targets is a comma separated list of target operating systems and/or target processors. For example Targets='wince,linux,i386' means: apply this rules to TargetOS wince or linux and to all TargetCPU i386.

How the codetools parses sources, difference to a compiler

A compiler is optimized to parse code linear and load needed units and include files as soon as it parses a uses section or a directive. The codetools are optimized to parse only parts of code. For example jumping from the method declaration to the method body only needs the unit and its include files. When a codetool search a declaration it searches backwards. That means it starts searching in the local variables, then upwards the implementation. When it finds a uses section it searches the identifier in the interface section of the units. When the identifier is found it stops. The result and some middle steps are cached. Because it often only needs to parse some interface sections it finds a single identifier fast.

The codetools do not parse a source in one step like the compiler but in several steps, depending on the need of the current function:

  • First a source file is loaded in a TCodeBuffer. The IDE uses this step to change the encoding to UTF8. The files are kept in memory and only reloaded if the modification date changes or a file is manually reverted. There are several tools and function which work directly on the buffer.
  • The next level is parsing a unit or include file. A unit must be parsed from the beginning, so the codetools try to find the main file, the first file of a unit. It does that by looking for a directive in the first line like {%MainUnit ../lclintf.pp}. If that does not exist, it searches in the includelink cache. The IDE saves this cache to disk, so the IDE learns over time.
  • After finding the main file TLinkScanner parses the source. It handles compiler directives, like include directives and if-else directives. The scanner can be given a range, so it can for instance only parse the interface of a unit. The scanner creates the clean source. The clean source is put together of all include files and stripped off of all skipped code in else parts. It also create a list of links, that maps between the clean source and the real source files. The clean source is now pascal. Note: there are also tools to scan a single source for all directives and create a tree of directives.
  • After creating the clean source a TCodeTool parses it and creates a tree of TCodeTreeNode. It can also be given a range. This parser skips a few parts, for example class members, begin..end blocks and parameter lists. Many tools don't need them. These sub nodes are created on demand. A TCodeTreeNode has a range StartPos..EndPos which are clean positions, that means positions in the clean source. There are only nodes for the important parts. Creating nodes for every detail would need more memory than the source itself and is seldom needed. There are plenty of functions to find out the details. For example if a function has calling convention 'cdecl'.
  • When searching for an identifier the search stores the found base types and creates caches for all identifiers in the interface section.

Every level has its own caches, which need to be checked and updated before calling a function. Many high level functions accessible via the CodeToolBoss do that automatically. For others it is the responsibility of the caller.

Example for:

unti1.pas:

<Delphi> unit Unit1; {$I settings.inc} interface uses

 {$IFDEF Flag}
 unix,
 {$ELSE}
 windows,
 {$ENDIF}
 Classes;

</Delphi>

settings.inc:

<Delphi> {%MainUnit unit1.pas} {$DEFINE Flag} </Delphi>

clean source:

<Delphi> unit Unit1; {$I settings.inc}{%MainUnit unit1.pas} {$DEFINE Flag} interface uses

 {$IFDEF Flag}
 unix,
 {$ELSE}{$ENDIF}
 Classes;

</Delphi>


Hint: To easily parse a unit and build the nodes, use CodeToolBoss.Explore.

CleanPos and CursorPos

There are several methods to define a position in the codetools.

Absolute position are related to a source as string and starts at 1. For example a TCodeBuffer holds the file content as one string in the property Source. Caret or cursor positions are given as X,Y, where Y is the line number starting at 1 and X is the column number starting at 1. A TCodeBuffer provides the member functions LineColToPosition' and AbsoluteToLineCol to convert. When working with multiple source files, like a unit, that can consists of several include files, the clean position relates to the absolute position in the stripped code Src. Src which is a string and clean positions start at 1. Cursor positions are specified as TCodeXYPosition (Code,X,Y). A TCodeTool provides the functions CaretToCleanPos, CleanPosToCaret to convert.

Inserting, deleting, replacing - the TSourceChangeCache

When making changes to the source code of a unit (or its include files) you should use the CodetoolBoss.SourceChangeCache, because:

  • Simple usage. Connect, Replace, Replace, ... Apply. See below.
  • You can use cleanpos as given by the node tree OR you can use direct position in a file.
  • You can use Replace to insert, delete, which automatically calls events, so connected editors are notified of changes.
  • It can automatically insert needed spaces, line breaks or empty lines in front or behind each Replace. For example you define that there should be an empty line in front. The SourceChangeCache checks what is inserted and how much space there is already and will insert needed space.
  • It checks if the replaced/deleted span is writable.
  • You can do multiple Replaces and you control when they are applied. Keep in mind that inserting code means that the parsed tree becomes invalid and needs rebuilding.
  • Multiple replaces are checked for intersection. For example an insert in the middle of deleted code gives an error.
  • Mutiple insertions at the same place are added FIFO - first at the top.
  • You can combine several functions altering code to one bigger function. See below.

Usage

The SourceChangeCache works on a unit, so you need to get a TCodeTool and scan a unit/include file. For example:

<Delphi>

 // Step 1: load the file and parse it
 Code:=CodeToolBoss.LoadFile(Filename,false,false);
 if Code=nil then
   raise Exception.Create('loading failed '+Filename);
 if not CodeToolBoss.Explore(Code,Tool,false) then
   ...;// parse error ...
 // Step 2: connect the SourceChangeCache
 CodeToolBoss.SourceChangeCache.MainScanner:=Tool.Scanner;
 // Step 3: use Replace to insert and/or delete code
 // The first two parameters are the needed spaces in front and behind the insertion
 // The FromPos,ToPos defines the deleted/replaced range in CleanPos positions.
 // The NewCode is the string of new code. Use  for a delete.
 if not CodeToolBoss.SourceChangeCache.Replace(gtNone,gtNone,FromPos,ToPos,NewCode) then
   exit; // e.g. source read only or a former Replace has deleted the place
 ...do some more Replace...
 // Step 4: Apply the changes
 if not CodeToolBoss.SourceChangeCache.Apply then
   exit; // apply was aborted

</Delphi>

BeginUpdate/EndUpdate

BeginUpdate/EndUpdate delays the Apply. This is useful when combining several code changing functions. For example:

You want to scan the unit, add a unit to the interface uses section, and remove the unit from the implementation uses section. The two functions AddUnitToMainUsesSection and RemoveUnitFromUsesSection use Apply, altering the source, so the second function would rescan the unit a second time. But since the two functions are independent of each other (they change different parts of the source) you can combine them and do it with one scan:

<Delphi>

 // Step 1: parse unit and connect SourceChangeCache
 if not CodeToolBoss.Explore(Code,Tool,false) then
   ...;// parse error ...
 CodeToolBoss.SourceChangeCache.MainScanner:=Tool.Scanner;
 // Step 2: delay Apply
 CodeToolBoss.SourceChangeCache.BeginUpdate;
 // Step 3: add unit to interface section
 // AddUnitToMainUsesSection would apply and change the code
 // Because of the BeginUpdate the change is not yet done, but stored in the SourceChangeCache
 if not Tool.AddUnitToMainUsesSection('Classes',,CodeToolBoss.SourceChangeCache) then exit;
 // Step 4: remove unit from implementation section
 // Without the BeginUpdate the RemoveUnitFromUsesSection would rescan the unit
 if Tool.FindImplementationUsesSection<>nil then
   if not Tool.RemoveUnitFromUsesSection(Tool.FindImplementationUsesSection,'Classes',CodeToolBoss.SourceChangeCache) then exit;
 // Step 5: apply all changes
 if not CodeToolBoss.SourceChangeCache.EndUpdate then
   exit; // apply was aborted

</Delphi>

BeginUpdate/EndUpdate work with a counter, so if you call BeginUpdate twice you need to call EndUpdate twice. This means you can put the above example in a function and combine that with another function.

Saving changes to disk

The above changes are made to the code buffers and the buffers are marked modified. To save the changes to disk, you have to call Save for each modified buffer.

  • The buffers that will be modified in the next Apply/EndUpdate are in SourceChangeCache.BuffersToModify and BuffersToModifyCount.
  • The events SourceChangeCache.OnBeforeApplyChanges/OnAfterApplyChanges are used by the CodeToolBoss, which connects it to its own OnBeforeApplyChanges/OnAfterApplyChanges. The Lazarus IDE sets these events and automatically opens modified files in the source editor, so all changes go into the undo list of synedit.