Codetools/ru

From Lazarus wiki
Jump to navigationJump to search

│ Deutsch (de) │ English (en) │ français (fr) │ русский (ru) │

Что такое codetools?

Codetools - это пакет Lazarus, предоставляющий инструменты для анализа, изучения, редактирования и рефакторинга исходных кодов Pascal. Codetools был упакован в своем собственном модуле и лицензирован под лицензией GPL. Многие примеры, показывающие, как использовать codetools в ваших собственных программах, можно найти в components/codetools/examples.

svn:

Использование codetools без IDE

Вы можете использовать codetools без IDE. Это может быть использовано для тестирования нового инструмента. Простой пример 

 <lazarusdir>/components/codetools/examples/methodjumping.lpi

Чтобы проверить декларацию find, codetools необходимо проанализировать исходники. Особенно исходники RTL и FCL. В примерах используются следующие переменные среды:

  • FPCDIR: путь к исходникам FPC, по умолчанию это ~/freepascal/fpc.
  • PP: pпуть к исполняемому файлу компилятора (/usr/bin/fpc or /usr/bin/ppc386 or C:\lazarus\ppc386.exe). Codetools нужно спросить компилятор о настройках. По умолчанию поиск 'fpc' осуществляется через переменную PATH.
  • FPCTARGETOS: скажите codetools сканировать другую операционную систему (кросс-компиляция). Например: linux, freebsd, darwin, win32, win64, wince
  • FPCTARGETCPU: при сканировании для другого процессора. Например: i386, powerpc, x86_64, arm, sparc
  • LAZARUSDIR: путь к исходникам Lazarus. Требуется только если вы хотите их сканировать.

FPC - очень сложный проект с множеством путей поиска, включающих файлы и макросы. Codetools должен знать все эти пути и макросы для анализа этих джунглей. Чтобы легко все это настроить, codetools содержит предопределенные шаблоны для исходных каталогов FPC, Lazarus, Delphi и Kylix.

Смотрите пример объявления поиска 

 <lazarusdir>/components/codetools/examples/finddeclaration.lpi

Поскольку исходники FPC содержат несколько версий некоторых модулей, и исходники FPC часто меняются, codetools не используют фиксированную таблицу путей, а скорее сканируют всю структуру каталогов FPC при первом использовании, применяя набор правил, определяющих, что это - правильный исходник для текущих TargetOS и TargetCPU. Это сканирование может занять некоторое время в зависимости от скорости вашего диска. Все примеры сохраняют результат в codetools.config, так что при следующем запуске Lazarus это возможно длительное сканирование будет пропущено.

Когда бы ни изменялись исходники FPC или переименовывалось устройство, просто удалите файл codetools.config. IDE Lazarus имеет свой собственный файл конфигурации и выполняет повторное сканирование всякий раз, когда изменяется исполняемый файл компилятора (или когда пользователь вызывает 'Tools > Rescan FPC source directory').

Задание путей поиска и макросов

Codetools использует [конструкцию] "define templates" (шаблоны определений) для генерации путей поиска и макросов. Шаблон определений является деревом правил.

Вот пример того, как расширить глобальный путь включения и запросить путь включения каталога: 

uses
  Classes, SysUtils, CodeToolManager, DefineTemplates, FileProcs;
  
var
  Directory: String;
  IncPathTemplate: TDefineTemplate;
begin
  Directory:=ExpandFileNameUTF8(GetCurrentDirUTF8);

  // добавление подшаблона для расширения включающего пути поиска #IncPath.
  IncPathTemplate:=TDefineTemplate.Create(
    'Add myincludes to the IncPath',  // имя шаблона, полезно для его поиска в дальнейшем (необязательно)
    'Add /tmp/myincludes to the include search path', // описание (необязательно)
    IncludePathMacroName,  // имя переменной: #IncPath
    '$('+IncludePathMacroName+');/tmp/myincludes' // новое значение: $(#IncPath);/tmp/myincludes
    ,da_DefineRecurse
    );
  // добавляем шаблон включающего пути в дерево
  CodeToolBoss.DefineTree.Add(IncPathTemplate);

  writeln('Directory="',Directory,'"',
    ' IncPath="',CodeToolBoss.GetIncludePathForDirectory(Directory),'"');
end.

Определение шаблона макроса

Макросы, заданные [при помощи конструкции] определения шаблонов, используются синтаксическим анализатором Pascal. В Codetools есть функции, позволяющие запрашивать у Free Pascal Compiler свои макросы и преобразовывать их для определения шаблонов. Это делается [с помощью] CodeToolBoss.Init(Options);, который вы можете найти во многих примерах.

Кроме того, есть несколько предопределенных макросов, которые начинаются с хеша (#). См. модуль definetemplates для [получения] полного списка. Вот несколько важных [из них]: 

  ExternalMacroStart = '#';
  // Стандартные макросы
  DefinePathMacroName = ExternalMacroStart+'DefinePath'; // текущий каталог
  UnitPathMacroName = ExternalMacroStart+'UnitPath'; // путь поиска объекта, разделенный точкой с запятой (такой же, как для FPC)
  IncludePathMacroName = ExternalMacroStart+'IncPath'; // путь поиска включаемого файла, разделенный точкой с запятой (такой же, как для FPC)
  SrcPathMacroName = ExternalMacroStart+'SrcPath'; // путь поиска исходника модуля, разделенный точкой с запятой (не указывается для FPC)

Например, IncludePathMacroName - это #IncPath и используется для определения пути поиска включаемого файла. Имейте в виду, что значения макросов зависят от каталога.

Значения определений шаблонов могут содержать макросы. Макрос должен быть заключен в [символ] доллара и скобки: $(macro).

В приведенном выше примере значение было '$('+IncludePathMacroName+');/tmp/myinclude' , которое эквивалентно '$(#IncPath);/tmp/myinclude' , и которое выполняется для старого включаемого пути поиска + ;/tmp/myinclude, что означает добавление пути. Для удобства чтения вы можете использовать константу IncludePathMacro вместо '$('+IncludePathMacroName+')' : 

  DefinePathMacro          = '$('+DefinePathMacroName+')'; // путь к определенному шаблону
  UnitPathMacro            = '$('+UnitPathMacroName+')';
  IncludePathMacro         = '$('+IncludePathMacroName+')';
  SrcPathMacro             = '$('+SrcPathMacroName+')';

Шаблон определений действий

Шаблон определений имеет имя, необязательное описание, переменную, значение и действие. Имя и описание не являются обязательными. Значение переменной и значение [шаблона определений] зависит от действия. Вы можете добавить шаблоны определения как дочерние элементы шаблонов определения - создавая дерево шаблонов определения. Вы можете увидеть много примеров для определения шаблонов в диалоговом окне Lazarus'а Tools / CodeTools Defines Editor.

  • da_Block - Used for grouping templates. When the block is executed all children are executed.
  • da_Directory - Use this to define all rules of a directory. If this is a root directory set Value to the full expanded directory path (use function CleanAndExpandDirectory). If this is a sub directory (parent template is a directory) Value is the sub path. Can contain macros. Children are only executed if directory fits.
  • da_Define - sets a macro (Variable) value (Value) of the current directory. The value can contain macros. Note that when a macro value is set to empty string it is still defined, that means {$IFDEF variable} will still result in true. Children are not executed.
  • da_Undefine - clears a macro (Variable) of the current directory. {$IFDEF macro} results in false. Children are not executed.
  • da_DefineRecurse - as da_Define, but for current directory and sub directories.
  • da_UndefineRecurse - as da_Undefine, but for current directory and sub directories.
  • da_UndefineAll - clear all macro values.
  • da_IfDef - if macro Variable is defined then execute children and skip following da_Else and da_ElseIf.
  • da_IfNDef - if macro Variable is not defined then execute children and skip following da_Else and da_ElseIf.
  • da_If,daElseIf - if the boolean expression Value executes to true then execute children and skip following da_Else and da_ElseIf. Value can contain macros.
  • da_Else - When this template is executed then execute all children.

How to extend the include path of a directory

See the example lazarus/components/codetools/examples/setincludepath.lpr. It demonstrates the use of relative paths, absolute paths, how to use the DefinePathMacro and the difference between da_Define and da_DefineRecurse.

Using the codetools in the IDE with the IDEIntf

See <lazarusdir>/examples/idequickfix/quickfixexample.lpk package. It demonstrates:

  • How to write an IDE package. When you install this package it will register a Quick Fix item in the IDE.
  • How to write a Quick Fix item for the compiler message: 'Parameter "Sender" not used'
  • How to use the codetools to:
    • parse a unit
    • convert Filename, Line, and Column data into a codetools source position
    • find a codetools node at a cursor position
    • find a procedure node, and the begin..end node
    • create a nice insertion position for a statement at the beginning of the begin..end block
    • obtain line indentation information so that a new line will work in a sub-procedure as well
    • insert code using the codetools

Codetools rules for FPC sources

When the codetools searches the source of a fpc ppu it uses a set of rules. You can write your own rules, but normally you will use the standard rules, which are defined in the include file components/codetools/fpcsrcrules.inc. You can test the rules with the command line utility: components/codetools/examples/testfpcsrcunitrules.lpi.

Usage of testfpcsrcunitrules

Usage: lazarus/components/codetools/examples/testfpcsrcunitrules -h

  -c <compiler file name>, --compiler=<compiler file name>
         Default is to use environment variable PP.
         If this is not set, search for fpc

  -T <target OS>, --targetos=<target OS>
         Default is to use environment variable FPCTARGET.
         If this is not set, use the default of the compiler.

  -P <target CPU>, --targetcpu=<target CPU>
         Default is to use environment variable FPCTARGETCPU.
         If this is not set, use the default of the compiler.

  -F <FPC source directory>, --fpcsrcdir=<FPC source directory>
         Default is to use environment variable FPCDIR.
         There is no default.

  -u <unit name>, --checkunit=<unit name>
         Write a detailed report about this unit.

Example for testfpcsrcunitrules

Open the testfpcsrcunitrules.lpi in the IDE and compile it. Then run the utility in a terminal/console: 

./testfpcsrcunitrules -F ~/fpc/sources/2.5.1/fpc/

This will tell you which compiler is used, which compiler executes, which config files were tested and parsed, and it warns you about duplicate units in the FPC search path and about duplicated unit source files. Note: This example caches results in the file codetools.config. You should delete codetools.config when you update the compiler or your fpc.cfg file.

Duplicate source files

You find out that the codetools opens for target wince/arm the wrong source of the unit mmsystem. Run the tool with the -u parameter:

./testfpcsrcunitrules -F ~/fpc/2.5.1/fpc/ -T wince -P arm -u mmsystem

This will give you a detailed report where this unit was found and what score each source file got. For example: 

Unit report for mmsystem
  WARNING: mmsystem is not in PPU search path
GatherUnitsInFPCSources UnitName=mmsystem File=packages/winunits-base/src/mmsystem.pp Score=11
GatherUnitsInFPCSources UnitName=mmsystem File=packages/winceunits/src/mmsystem.pp Score=11 => duplicate

This means there are two source files with the same score, so the codetools took the first. The last one in winceunits is for target wince and the first one is for win32 and win64.

Now open the rules file fpcsrcrules.inc.

Rules work like this: 

Score:=10;
Targets:='wince';
Add('packages/winceunits');

The Add adds a rule for all files beginning with 'packages/winceunits' that adds a score of 10 to all these files. The Targets is a comma separated list of target operating systems and/or target processors. For example Targets='wince,linux,i386' means: apply this rules to TargetOS wince or linux and to all TargetCPU i386.

How the codetools parses sources differently from the compiler

A compiler is optimized to parse code linearly and to load needed units and include files as soon as it parses a uses section or a directive. The codetools are optimized to parse only certain code sections. For example jumping from the method declaration to the method body only needs the unit and its include files. When a codetool searches a declaration it searches backwards. That means it starts searching in the local variables, then upwards from the implementation. When it finds a uses section it searches the identifiers in the interface section of the units. When the identifier is found it stops. The result and some intervening steps are cached. Because it often only needs to parse a few interface sections it finds an individual identifier very quickly.

The codetools do not parse a source in a single step (as the compiler does) but in several steps, which depend on what the current codetool needs:

  • First a source file is loaded in a TCodeBuffer. The IDE uses this step to change the encoding to UTF8. The files are kept in memory and only reloaded if the modification date changes or if a file is manually reverted. There are several tools and functions which work directly on the buffer.
  • The next step is to parse the unit (or include file). A unit must be parsed from the beginning, so the codetools tries to find the main file, the first file of a unit. It does that by looking for a directive in the first line like {%MainUnit ../lclintf.pp}. If that does not exist, it searches in the includelink cache. The IDE saves this cache to disk, so the IDE learns over time.
  • After finding the main file TLinkScanner parses the source. It handles compiler directives, such as include directives and conditional directives. The scanner can be given a range, so it might, for instance, parse only a unit's interface. The scanner creates the clean source. The clean source is put together from all include files, having been stripped of code in the else part of conditional directives, which is skipped. It also creates a list of links which map the clean source to the real source files. The clean source is now Pascal containing no else code. Note: there are also tools designed to scan a single source for all directives. These tools create a tree of directives.
  • After creating the clean source a TCodeTool parses it and creates a tree of TCodeTreeNode. It can also be given a range. This parser skips a few parts, for example class members, begin..end blocks and parameter lists. Many tools don't need them. These sub nodes are created on demand. A TCodeTreeNode has a range StartPos..EndPos which are clean positions, that means positions in the clean source. There are only nodes for the important parts. Creating nodes for every detail would need more memory than the source itself and is seldom needed. There are plenty of functions to find out the details. For example if a function has the 'cdecl' calling convention.
  • When searching for an identifier the search stores the base types it finds and creates caches for all identifiers in the interface section.

Every level has its own caches, which need to be checked and updated before calling a function. Many high level functions accessible via the CodeToolBoss do that automatically. For others it is the responsibility of the caller.

Example for:

unit1.pas: 

unit Unit1;
{$I settings.inc}
interface
uses
  {$IFDEF Flag}
  unix,
  {$ELSE}
  windows,
  {$ENDIF}
  Classes;

settings.inc: 

{%MainUnit unit1.pas}
{$DEFINE Flag}

clean source: 

unit Unit1;
{$I settings.inc}{%MainUnit unit1.pas}
{$DEFINE Flag}
interface
uses
  {$IFDEF Flag}
  unix,
  {$ELSE}{$ENDIF}
  Classes;


Hint: To easily parse a unit and build the nodes, use CodeToolBoss.Explore.

CleanPos and CursorPos

There are several methods to define a position in the codetools.

The codetools Absolute position is related to the source as a continuous string starting at character 1. For example a TCodeBuffer holds the file contents as a single string in its Source property. Caret or cursor positions are given as X,Y where X is the columne number and Y the line number. Each value (X and Y) starts at 1. A TCodeBuffer provides the member functions LineColToPosition' and AbsoluteToLineCol to convert between (X,Y) values and the Absolute codetools position. When working with multiple source files (such as a unit which may contain several include files), the clean position relates to the absolute position in the stripped code Src. Here Src is a string whose clean positions start at 1. Cursor positions are specified as TCodeXYPosition (Code,X,Y). A TCodeTool provides the functions CaretToCleanPos and CleanPosToCaret to convert between them.

Inserting, deleting, replacing - the TSourceChangeCache

When making changes to the source code of a unit (or its include files) you should use the CodetoolBoss.SourceChangeCache instead of altering the source directly.

  • Simple usage: Connect, Replace, Replace, ... Apply. See below.
  • You can use cleanpos as given by the node tree OR you can use direct position in a file.
  • You can use Replace to insert and delete, which automatically calls events, so connected editors are notified of changes.
  • It can automatically insert needed spaces, line breaks or empty lines in front or behind each Replace. For example you define that there should be an empty line in front. The SourceChangeCache checks what is inserted and how much space there is already and will insert needed space.
  • It checks if the replaced/deleted span is writable.
  • You can do multiple Replaces and you control when they are applied. Keep in mind that inserting code means that the parsed tree becomes invalid and needs rebuilding.
  • Multiple replaces are checked for intersection. For example an insert in the middle of deleted code gives an error.
  • Mutiple insertions at the same place are added FIFO - first at the top.
  • You can combine several functions altering code to one bigger function. See below.

Usage

The SourceChangeCache works on a unit, so you need to get a TCodeTool and scan a unit/include file. For example: 

  // Step 1: load the file and parse it
  Code:=CodeToolBoss.LoadFile(Filename,false,false);
  if Code=nil then
    raise Exception.Create('loading failed '+Filename);
  if not CodeToolBoss.Explore(Code,Tool,false) then
    ...;// parse error ...

  // Step 2: connect the SourceChangeCache
  CodeToolBoss.SourceChangeCache.MainScanner:=Tool.Scanner;

  // Step 3: use Replace to insert and/or delete code
  // The first two parameters are the needed spaces in front and behind the insertion
  // The FromPos,ToPos defines the deleted/replaced range in CleanPos positions.
  // The NewCode is the string of new code. Use '' for a delete.
  if not CodeToolBoss.SourceChangeCache.Replace(gtNone,gtNone,FromPos,ToPos,NewCode) then
    exit; // e.g. source read only or a former Replace has deleted the place
  ...do some more Replace...

  // Step 4: Apply the changes
  if not CodeToolBoss.SourceChangeCache.Apply then
    exit; // apply was aborted

BeginUpdate/EndUpdate

BeginUpdate/EndUpdate delays the Apply. This is useful when combining several code changing functions. For example:

You want to scan the unit, add a unit to the interface uses section, and remove the unit from the implementation uses section. The two functions AddUnitToMainUsesSection and RemoveUnitFromUsesSection use Apply, altering the source, so the second function would rescan the unit a second time. But since the two functions are independent of each other (they change different parts of the source) you can combine them and do it with one scan: 

  // Step 1: parse unit and connect SourceChangeCache
  if not CodeToolBoss.Explore(Code,Tool,false) then
    ...;// parse error ...
  CodeToolBoss.SourceChangeCache.MainScanner:=Tool.Scanner;

  // Step 2: delay Apply
  CodeToolBoss.SourceChangeCache.BeginUpdate;

  // Step 3: add unit to interface section
  // AddUnitToMainUsesSection would apply and change the code
  // Because of the BeginUpdate the change is not yet done, but stored in the SourceChangeCache
  if not Tool.AddUnitToMainUsesSection('Classes','',CodeToolBoss.SourceChangeCache) then exit;

  // Step 4: remove unit from implementation section
  // Without the BeginUpdate the RemoveUnitFromUsesSection would rescan the unit
  if Tool.FindImplementationUsesSection<>nil then
    if not Tool.RemoveUnitFromUsesSection(Tool.FindImplementationUsesSection,'Classes',CodeToolBoss.SourceChangeCache) then exit;

  // Step 5: apply all changes
  if not CodeToolBoss.SourceChangeCache.EndUpdate then
    exit; // apply was aborted

BeginUpdate/EndUpdate work with a counter, so if you call BeginUpdate twice you need to call EndUpdate twice. This means you can put the above example in a function and combine that with another function.

Saving changes to disk

The above changes are made to the code buffers and the buffers are marked modified. To save the changes to disk, you have to call Save for each modified buffer.

  • The buffers that will be modified in the next Apply/EndUpdate are in SourceChangeCache.BuffersToModify and BuffersToModifyCount.
  • The events SourceChangeCache.OnBeforeApplyChanges/OnAfterApplyChanges are used by the CodeToolBoss, which connects it to its own OnBeforeApplyChanges/OnAfterApplyChanges. The Lazarus IDE sets these events and automatically opens modified files in the source editor, so all changes go into the undo list of synedit.

Hints/Tips/Guide Lines

  • BuildTree checks the current state and will only parse if needed. If a former call has parsed the interface and you need the interface again, BuildTree will not do anything. If you need the full unit, only the implementation will be parsed. Nodes are only freed if some files have changed on disk or some settings have changed (initial macros). BuildTree does not check all files on every call. Instead it uses the directory cache of the codetools. So, if nothing has changed it will return very quickly. You should call it before any search operation.
  • Do not call BuildTree after an operation. That is a waste of CPU.
  • BuildTree raises an exception when there are missing include files or syntax errors. You should enclose your code into
try
  Tool.BuildTree(lsrEnd);
  ... search ... replace  
except
  on E: Exception do
    CodeToolBoss.HandleException(E);
end;
  • After calling SourceChangeCache.Replace (multiple times) the sources (TCodeBuffers) have not changed immediately. You must call SourceChangeCache.Apply to change the sources. This will not save the changes to file.

Links

  • Lazarus IDE Tools - A tutorial about the built-in tools of the standard IDE
  • Cody - IDE package adding advanced code tools to the IDE
  • Extending the IDE - How to write your own codetools plugins for the IDE.
Retrieved from "http://wiki.freepascal.org/index.php?title=Codetools/ru&oldid=122418"