Creating bindings for C libraries/fr

From Lazarus wiki
Revision as of 23:51, 12 August 2007 by Paulatreides (talk | contribs)
Jump to navigationJump to search

English (en) español (es) français (fr) 日本語 (ja) русский (ru)

Overview

This page describes how to create pascal bindings for C libraries. Normally Pascal can not use C libraries directly. You have to create a pascal translation for every C function, type and variable. There is the H2Pas tool, which can automatically translate many common C things. And there is a GUI for Lazarus using h2pas and other tools to automate the creation. And the GUI helps to create a rule set, that can be used to update the bindings. So the next version of the c library can be converted far more easier. A nice feature of the h2pas wizard is, that it automatically use temporary files, so the c header files will never be changed automatically.

Work flow

  • Fetch the C header files you want to translate.
  • Create a working directory and give your bindings a name
  • Create a new h2pas project with the h2pas wizard.
  • add the .h files to the project
  • setup the h2pas options
  • run the wizard
  • fix errors by adding text tools and run the wizard again
  • when h2pas runs without errors, test compilation and add optional tools to beautify the output
  • write some test programs to test your bindings
  • publish your bindings on lazarus-ccr or the Free Pascal website

Install tools

The h2pas tool comes with every normal fpc installation.

Install the h2paswizard package in the Lazarus IDE. Go to "Components -> Configure installed packages ...", select from the right list the H2PasWizard package and click 'Install selection', then 'Save and rebuild IDE'. Restart the IDE and you get a new menu entry: Tools -> h2pas

Fetch C header files

The C header files .h describes the interface of a C library. They are normally not provided with the library. You have to get the sources or the development package of the library. For example the C header files of the gtk libraries are in the package gtk+-devel.

Example: MPICH2

Download the mpich2-1.0.3.tar.gz from http://www-unix.mcs.anl.gov/mpi/mpich2/ and unpack them. The .h files are under mpich2-1.0.3/src/include.

Create a working directory and name your bindings

Create a directory with a useful name. A name should not contain special characters, like spaces, umlaute, points or commas. Copy the .h files.

Example: MPICH2

The h2p directory will be used for the pascal files. The h2p/c_sources directory will be used for the .h files.

 mkdir -p h2p/c_sources
 cp mpich2-1.0.3/src/include/*.h h2p/c_sources/

Create a new h2pas project with the h2pas wizard

Open the h2pas wizard: "Tools -> h2pas". This will open a window. You can switch between this window and the other IDE windows. Your last h2pas project will be loaded automatically. To create a new project click on "Settings -> New/Clear settings". Then click on the bottom button "Save settings" and choose a filename.

Example: MPICH2

Click on "Settings -> New/Clear settings". Then click on the bottom button "Save settings" and save it as h2p/mpi2.h2p.

Add the .h files to the project

In the "C header files" page you can add/delete the .h files. You can enable/disable .h files to convert only part of the files.

Example: MPICH2

Click on "C header files -> Add .h files ..." and select "mpi.h". It will be enabled automatically.

Setup the h2pas options

Under "h2pas Options" you can set the parameters for the h2pas program.

Example: MPICH2

  • Enable -e, -D, -p, -w and disable all others.
  • The -l library path is "mpich".
  • Output extension is ".pas"
  • Output directory is h2pas/, which is the default, so leave it empty

Run the wizard

Click on the bottom button "Run h2pas". This will copy the <example>.h file to a temporary <example>.tmp.h file and run the tools listed under "Before h2pas". Then it runs h2pas to convert the <example>.tmp.h into <example>.inc or <example>.pas or whatever output extension you setup on the h2pas page.

If h2pas finds a syntax error, the IDE will open the example.tmp.h file and jumps to the error line. h2pas often only reports 'syntax error', which is very unspecific. See below for a list of common problems and possible fixes.

Example: MPICH2

The h2pas wizard already contains all tools to convert all the specials of this header file, so h2pas runs without errors. But the created unit is not yet ready. Read further.

h2pas reports an error

Here is a list of common C structures, not recognized by h2pas and how to fix them:

h2pas problem: extern "C"

Some header files contain the C++ namespace block:

 #if defined(__cplusplus)
 extern "C" {
 #endif
 ...
 #if defined(__cplusplus)
 }
 #endif

Fix: Add the Remove C++ 'extern "C"' lines to the before h2pas tools.

h2pas problem: Empty macro

Some header files contain empty macros used for further extensions:

 #define MPI_FILE_DEFINED

Fix: Add the Remove empty C macros to the before h2pas tools.

h2pas problem: Implicit array types

C allows implicit arrays in parameter types. For example:

 int MPI_Group_range_incl(MPI_Group, int, int [][3], MPI_Group *);

The int [][3] is an implicit type, which is not allowed under Pascal. h2pas supports adding pointer types. So, it enough to replace all [] with *.

Fix: Add the Replace [] with * to the before h2pas tools.

h2pas problem: Makros for 0 pointers

Some header files contain typed 0 pointers:

 #define MPI_BOTTOM      (void *)0

Fix: Add the Replace macro values 0 pointer like (char *)0 with NULL to the before h2pas tools.

Test compilation and add tools to beautify the output

When h2pas runs without errors it has created a pascal file. The -i switch defines if it is a unit or an include file. The next step is test compilation. Setup up a test project that uses the new pascal source. Then use the 'Run h2pas and compile' button in the h2pas wizard.

Example: MPICH2

Create a new project with Project -> New Project -> Program. Save it a mpihelloworld. Change the code to

program MPIHelloWorld;

{$mode objfpc}{$H+}
{$linklib mpich}
{$linklib rt}

uses
  {$IFDEF UNIX}pthreads{$ENDIF}, MPI;

begin
  MPI_Init(@argc, @argv);
  writeln('Hello, world.');
  MPI_Finalize;
end.

Add the h2p/mpi.pas to the project. Project -> Add editor file to project.

Common compiler errors on h2pas output

Sometimes h2pas does not create valid pascal code. Here are the common problems and how to fix them.

Unit name contains the file path

h2pas sometimes add the whole path to the unitname.

Fix: Add Replace "unit filename;" with "unit name;" to the After h2pas tools.

Missing include files

h2pas converts #include directives to pascal {$i } directives. If you create a unit for wach header file, then you can remove all include directives with the tool Remove includes.

Forward type not resolved

For example: mpi.pas(26,16) Error: Forward type not resolved "PMPI_Aint" The error line is often a pointer type to a record:

 PMPI_Aint = ^MPI_Aint;

h2pas adds PMPI_Aint to give nameless C pointers like *MPI_Aint a name. It adds the pointers to the start of the file. But pascal requires, that the forward definition is in the same type section. Sometimes h2pas adds a pointer type, although it already exists.

Fix: Add the tool Remove redefined pointer types to the After h2pas tools.

Removing system types

h2pas adds some system types, like PLongint, which are nowadays part of the system unit.

Fix: Add Remove type redefinitons like PLongint to the After h2pas tools.

Empty type/var/const sections

After the above tools removed some variables, types, constants some setions become empty, which is not allowed in pascal.

Fix: Add Remove empty type/var/const sections to the After h2pas tools.

Implicit Types

C allows implicits types in parameters. For example: int [][3]. Pascal does not allow this, so you must give it a name and declare a type. But h2pas does not do this automatically and adds instead the implicit type in a quasi pascal syntax. For example:

 int some_func(int [][3]);

becomes

 function some_func(_para1:array[0..2] of Plongint):longint;cdecl;external;

Fix: Luckily there is now a tool to remove these implicit types and add explicit types, it's called Replace implicit types. Add it to the after tool list.

Array of nothing

Sometimes h2pas converts the C ellipsis parameter type '...' wrongly to 'array of )'. It should be 'array of const)'.

Fix: Add the tool Fix open arrays to the after tools.

Identifier not found

There are three cases:

The identifier is defined in the unit, but after it is used

In pascal forward defined types are only allowed in a type section. For example:

 type
   TMyRecord = record
     Next: PMyRecord; // using the forward definition
   end;
   PMyRecord = ^TMyRecord;

The below is not allowed, because the forward definition is in another type section:

 type
   TMyRecord = record
     Next: PMyRecord; // using the forward definition
   end;
 type
   PMyRecord = ^TMyRecord;

Solution: The code must be reordered. There is no tool yet, to do this automatically.

The identifier is defined in another unit (another .h file)

Solution 1: Add the unit to the uses section

If the other unit is already using this unit, then you have a circle. A circle dependency is allowed between .h files, but not between pascal units. In this case you must move code between both units or use IFDEFs like the gtk2 bindings or use the below merge function of the h2pas wizard.

Solution 2: Merge the two include files

The h2pas wizard can merge include files into one. For example first.h includes via the c include directive the file second.h. Select the second.h in the wizard and check the 'Merge file' feature. Then the source of second.h' will be appended to first.h before sending the output to h2pas. As always: This not change the header files.

Proposal

A tool to fix most of the above issues could work this way: Gather all constants of all units of the bindings and put them into one big constant section. Do the same for all types and put them all into one big type section behind the constant section.

Disadvantage: You can no longer see, which identifier belongs to which .h file. At least some comments should be added.

Example: MPICH2

The type MPI_Request is defined in mpi.h and used in mpio.h. Both c header files heavily use each other, so you can not put them into two separated units. Select mpio.h and enable the 'Merge file' function.

The identifier is not defined anywhere

Probably you are missing a header file or you are using one with the wrong version or it is private identifier.

Solution: Search the header file and convert it too. If you can not find it or you don't want to translate this file, you can comment the identifier or replace it.

Example: MPICH2

The mpi.h file defines MPI_File as pointer of ADIOI_FileD, but ADIOI_FileD is not defined public:

 typedef struct ADIOI_FileD *MPI_File;

The h2pas tool translated this to:

    MPI_File = ^ADIOI_FileD;

Because ADIOI_FileD is not defined public, it is a private structure, so you can simply replace it with a Pointer: Add to the 'After h2pas' tools a new tool of type 'Search and replace' with the following properties:

Property Value
Caption Replace ADIOI_FileD with Pointer
Name ReplaceADIOI_FileDwithPointer
SearchFor ADIOI_FileD
ReplaceWith Pointer
Options [trtMatchCase,trtWholeWord]

Illegal expression

Example: MPICH2

Compiler gives Illegal expression on the following statement

 const
    MPI_LONG_LONG = MPI_LONG_LONG_INT;

The reason is, that h2pas translated the MPI_LONG_LONG_INT constant to a function.

Solution: Fix h2pas or use a trick:

Replace the

 #define MPI_LONG_LONG   MPI_LONG_LONG_INT

with

 #define MPI_LONG_LONG      ((MPI_Datatype)0x4c000809)

by adding a Search and replace tool before h2pas.

Common other issues

The pascal file contains a lot of unneccessary ifdef

h2pas translates even the C #ifdef directives. Many of them are C specific and not needed under FPC and can even make it impossible many tools on this side to explore the code. That's why there is a tool to clean up and disable or remove many unneeded IFDEFs: Reduce compiler directives in pascal file.


The C header files contain redefinitions

This is ok for C compilers, because a type is the same if its definition is the same. But for Pascal each type is a different type.

Solution: Add the tool Remove redefinitions in pascal unit to the after h2pas tools. As the name implies, it only works on units, not on include files. And it expect at least a valid pascal syntax. That's why it is important to add this tool after the tools that fix the pascal syntax like Replace "unit filename;" with "unit name;", Remove empty type/var/const sections, Remove includes, Replace implicit types and Fix open arrays.

Example: MPICH2

The mpi.h contains several redefinitions like MPIIMPL_HAVE_MPI_TYPE_CREATE_DARRAY. They can all be fixed with this tool.

Alias macros are always translated by h2pas to constants

h2pas converts macros like

 #define A B

to

 const A = B;

which is almost always correct. But some c header contain aliases for types, variables and functions too.

Example: MPICH2

The mpi.h contains the macro

 #define MPI_LONG_LONG      MPI_LONG_LONG_INT

h2pas converts this simply to

 const
    MPI_LONG_LONG = MPI_LONG_LONG_INT;     

but MPI_LONG_LONG_INT is a function.

ToDo: Write a tool to auto fix aliases.

h2pas creates functions for constants

For example:

 function MPI_2COMPLEX : MPI_Datatype;
     begin
        MPI_2COMPLEX:=MPI_Datatype(MPI_DATATYPE_NULL);
     end;

All used identifiers are types and constants, so this function can be replaced with a simple constant. This is checked and done by the tool Replace simple functions with constants.

The C header files do not contain parameter names

For example there is an ugly function without parameter names:

 int MPI_Intercomm_create(MPI_Comm, int, MPI_Comm, int, int, MPI_Comm * );

The parameter names are defined in the corresponding .c file:

 int MPI_Intercomm_create(MPI_Comm local_comm, int local_leader, 
                          MPI_Comm peer_comm, int remote_leader, int tag, 
                          MPI_Comm *newintercomm)

Solution: There is no solution yet. Except manual editing or searching for other .h files. Often such .h files were auto generated and there are some nicer header files somewhere on the internet.

Proposal: Write a clever tool, that searches functions with missing parameter names and the corresponding functions in the .c files with parameter names and improve the .h file.

Publish your bindings on lazarus-ccr or Free Pascal

ToDo

Writing your own converter tools

Using the tool "Search and replace"

Many things like renaming a variable can be done by the Search and replace tool. Add the tool via the Add new tool button on either the 'Before h2pas' or the 'After h2pas' page. Then set the SearchFor, ReplaceWith, Options and Caption property.

Example: rename an identifier Tguint to guint

Property Value
Caption Rename Tguint to guint
SearchFor Tguint
ReplaceWith guint
Options [trtMatchCase,trtWholeWord]

Example: rename several identifiers

Rename Tguint to guint, Tgulong to gulong, Tgint to gint:

Property Value
Caption Rename Tguint to guint
SearchFor gint|gulong)
ReplaceWith $1
Options [trtMatchCase,trtWholeWord,trtRegExpr]

Improving an existing tool

You found a bug and want to fix it or you want to extend one of the above tools. Great!

Most of the above tools are defined in the h2pasconvert unit, which is part of the h2paswizard package. Basically a tool needs a classname, a description and an Execute method. To test/debug a tool outside the IDE and save a lot of compilation time, see the project components/simpleideintf/examples/testh2pastool.lpi. Compile it and start it on a console/terminal with the filename of a .h file as first command parameter. For example:

 ./testh2pastool files/h2pastest.pas


Writing a custom tool

You can write your own conversion tools and register them in the IDE easily. Start a package and a add a new unit (say unit1) for your new class. See the existing tools as example and read the prior section. When you wrote a new tool and tested it with the above simpleideintf project, then register it in the IDE: Add a register procedure to your unit (unit1), like the following (pseudo code):

uses
  Classes, ..., IDETextConverter;

type
  TYourToolClass = class(TCustomTextConverterTool)
  public
    class function ClassDescription: string; override;
    function Execute(aText: TIDETextConverter): TModalResult; override;
  end;

procedure Register;

implementation

procedure Register;
begin
  TextConverterToolClasses.RegisterClass(TYourToolClass);
end;

Do not forget to enable the register checkbox of the unit in the package editor, otherwise the Register procedure will not be called by the IDE. Then install your package in the IDE.

Future work / Missing stuff

  • a high lvl tool to gather many tools into one.
  • h2pas adds function bodies without the IFDEFs. Write a tool to add these. This must be done before the tool that cleans up the directives is run otherwise you don't know what definition correspond to which body.
  • h2pas converts macros to simple "constant A=B;" statements. A tool is needed to fix the cases where B is a type, var or function.
  • Enhance the Reduce compiler directives in pascal file tool to reduce something of the following examples:
 {$ifndef MPI_INCLUDED}
 {$define MPI_INCLUDED}
 ...
 {$endif}

The ifndef and endif can be removed.

 {$define HAVE_MPI_OFFSET}  

Defines can be moved up, just behind its last use.

 {$define HAVE_MPI_GREQUEST}  
 {$ifndef HAVE_MPI_GREQUEST}
 {$endif}

The block is never parsed and can be removed.

  • A tool to reorder a unit to fix forward definitions. This requires a tool to remove unneeded IFDEFs.
  • A tool to fix functions without parameter names by searching the .c files.
  • Find missing identifiers and let the user check which one to comment or replace with base types.
  • Create a list of macros, which were half translated.