spelling

From Lazarus wiki
Revision as of 11:32, 17 February 2019 by Raf (talk | contribs) (→‎About Hunspell)
Jump to navigationJump to search

English (en) español (es) русский (ru)



Using hunspell with Lazarus

This page is current as of August, 2018 but things change .....

This page is about using the hunspell library with Lazarus. It outlines a model that does, sort of, work. You will almost certainly need to make some changes for your specific purposes but hopefully, this page will give you a good start.

Firstly, in the forum, there are several references to some code that will work with the hunspell library. The hunspell.pas unit is based heavily on these blocks of code. Most have no license information and an assumption is being made that they are "common knowledge" and thus free of any constraint. I have added a bit that manages the problem of finding the library and dictionary files. And established a reasonable interface.

Further, user rvk from the forum has build a windows 64 bit DLL as there seemed little alternative for Windows users.


About Hunspell

Hunspell is an active open source project, distributed under the Mozilla Public License. The hunspell library is used in products like Libra Office, Open Office and Firefox. Its can be made work on Windows, Linux and Mac (and probably heaps of other platforms). See the platform Specific pages below. Dictionaries for Hunspell are readily available and may already be installed on many machines. Even if you cannot access another application's library, you can use its dictionary.

Dictionaries for Hunspell come as a *.dic and *.aff file pair. For example, the Australian dictionary consists of en_AU.dic and en_AU.aff . The 'en' indicates its english and the 'AU' says its specifically for Australia. As an English speaker, I note that the en_US dictionaries always seem to be installed and I add the Australian ones. I don't know how widespread this pattern is in non English speaking systems. You can find dictionaries here https://github.com/LibreOffice/dictionaries and also some information related to it here https://wiki.documentfoundation.org/Development/Dictionaries

Note about dictionaries

In order to display the suggested words correctly - the dictionaries must be encoded in UTF-8. When you have noticed that some words are not showed correctly - for example - a word - apple in Polish is jabłko - and you got - jab?ko - it means that dictionary pl_PL.aff and pl_PL.dic are not encoded in UTF-8 therefore you have to convert them. My solution for this is:

I don't know how to covert dictionaries into UTF-8 in Windows but I can do it in Linux (I use Debian but it should work in any Linux). The good thing is that converted dictionaries can be used in Windows. Check this thread http://forum.lazarus.freepascal.org/index.php/topic,44298.0.html

1. Start terminal as a root

  su

2. Go to /usr/share/hunspell/ where dictionaries are kept if not install hunspell (hunspell-pl is for Polish dictionary) more at https://packages.debian.org/search?keywords=hunspell

  apt-get install hunspell
  apt-get install hunspell-pl

3. Go to the directory usr/share/hunspell/ and create folder dic

  cd /usr/share/hunspell/
  mkdir dic

4. Convert for example dictionary pl_PL.aff and pl_PL.dic Use ISO-8859-2 for (Eastern Europe) https://en.wikipedia.org/wiki/ISO/IEC_8859-2 or ISO-8859-1 for (Western Europe) https://en.wikipedia.org/wiki/ISO/IEC_8859-1 or ISO-8859-5 for (Russian I am not sure)

  iconv -f ISO-8859-2 -t UTF-8 /usr/share/hunspell/pl_PL.aff | sed 's/^SET ISO8859-2$/SET UTF-8/g' > dic/pl_PL.aff
  iconv -f ISO-8859-2 -t UTF-8 /usr/share/hunspell/pl_PL.dic > dic/pl_PL.dic

5. Copy your dic folder or dictionaries from dic folder to your application

Platform Specific

Linux

Many Linux distributions will have Hunspell installed by default along with the appropriate language dictionaries. If not, its probably just a case of using the distribution's package manager. If all else fails, grab the source from the hunspell github site and build it yourself. Linux users are like that.

To see if you do in fact have a hunspell library installed, try this command - ldconfig -p | grep hunspell. Similarly, you can probably find some dictionaries with ls -l /usr/share/hunspell. If that does not work, try find /usr -name *.aff, it will take a bit longer ..

Windows

Installing the hunspell library on windows is more of an issue. There is apparently no pre compiled 'kit' available and most Windows apps that use Hunspell appear to bind it staticly thus no hunspell.dll left lying around for you to use. But just to be sure, try searching for *hunspell*.dll. The Hunspell github site lists a recipe to build one but it involves installing MSYS2 and is quite involved. The resulting DLL also needs a couple of gcc DLLs along as well.

Fortunately user rvk on the Lazarus Forum has built us a nice statically (ie stand alone) linked DLL using Microsoft Visual Studio Community 2015. As such, you can use and distribute this DLL with your programme subject to the Mozilla Public License.

You will find this DLL bundled with the 64bit (pre) release of tomboy-ng, just download the zip file, unzip and discard (how sad) the tomboy-ng binary. See https://github.com/tomboy-notes/tomboy-ng/releases

Mac

The author's Mac appears to have had the Hunspell library installed when Sierra was installed. But maybe, just maybe, it came along with Firefox. I'd like some feed back ....

To see if you aleady have a hunspell library installed, try this command - find / 2>&1 | grep "\hunspell", it will run for some time, depending on how many files on your system. It will likely find several files including some in your XCode directory. However, end users will probably not have XCode installed. The one particularly interesting file to me was /usr/lib/libhunspell-1.2.dylib. Version 1.2 is a bit older than elsewhere but it worked fine.

If you don't find a usable library, I suggest you install one using a package manager like MacPorts, Fink or brew.

Next issue issue is you will need some dictionaries. Similar command, find / 2>&1 | grep "\.aff", again, slow, it searches your whole disk. I found /Applications/Firefox.app/Contents/Resources/dictionaries/en-US.aff. And a quick 'ls' assured me that there was a matching 'en-US.dic' files so all good.

A more experienced Mac user might like to suggest better search strategies. Please !

The Hunspell Unit

Demo 1 simple command line

Here is a very simple command line demo of how to use hunspell.pas. Sadly, this particular demo is only suitable for linux as explained below. Save this block of code as testhun.pas and save hunspell.pas (below) in a directory and give this command :

fpc -Fu/usr/share/lazarus/1.8.0/components/lazutils/lib/x86_64-linux -Fu. testhun.pas

If you are using 32bit linux or your lazarus is installed "somewhere else" you will need to adjust the parameter to -Fu

program testhun;

{$mode objfpc}{$H+}

uses
    Classes, hunspell, sysutils;

var
  Spell : THunspell;
  Sts : TStringList;
  I : integer;

begin
    Spell := THunspell.Create();
    if Spell.ErrorMessage = '' then begin
      if Spell.SetDictionary('/usr/share/hunspell/en_US.dic') then begin
            writeln('speller ' + booltostr(Spell.Spell('speller'), True));
            writeln('badspeller ' + booltostr(Spell.Spell('badspeller'), True));
            Sts := TStringList.Create();
            Spell.Suggest('badspeller', Sts);
            for i := 0 to Sts.Count -1 do
                writeln('    ' + Sts.Strings[I]);
            Sts.Free;
      end else
        writeln('ERROR - Dictionary not loaded.');
    end else writeln('ERROR - Library not loaded.');
    Spell.Free;
end.

Why is this particular demo Linux only ? The hunspell unit is designed for GUI apps, it uses a Unit called Forms that don't make sense in a command line app. On Windows and Mac, a Forms method, Application.ExeName is used to determine where the binary is in case you have put the hunspell Library there (on Linux it has a predefined place to live).

Demo 2 in a Full GUI

A Lazarus GUI demo makes a lot more sense and has been tested on Linux, Mac and Windows. But it is a bit harder to copy and paste.

For this demo, you'll need a form with two TMemos, Memo1 and MemoMsg. A button, ButtonSpell, a Tlistbox, Listbox1. Make the following event handlers : FormCreate for the main form; a double click for the listbox and click on the Button.

First you must create the hunspell object and see if it found a library its own way, here is an example in the FormCreate() method ....

uses hunspell;
var
    Form1: TForm1;
    Sp: THunspell;
    DictPath : AnsiString;              

procedure TForm1.FormCreate(Sender: TObject);
begin
    SetDefaultDicPath();
    Sp := THunspell.Create();
    if Sp.ErrorMessage = '' then begin
        MemoMsg.append('Library Loaded =' + Sp.LibraryFullName);
        ButtonSpell.enabled := CheckForDict();
    end else
        MemoMsg.append(SP.ErrorMessage);
end;

In this example, we are writing some status message to MemoMsg, its an easy way to see whats happening. The ButtonSpell is NOT enabled until the dictionaries have been set. Wait for it ....

We now need two methods, one reads a nominated directory looking for any likely dictionary files, the other manages the decisions. If we find just one dictionary set, use it, if we find none, complain. But if we find several, and thats most likely, we have to ask the user which dictionary (ie language) they wish to use.

function TForm1.FindDictionary(const Dict : TStrings; const DPath : AnsiString) : boolean;
var
    Info : TSearchRec;
begin
    Dict.Clear;
    if FindFirst(AppendPathDelim(DPath) + '*.dic', faAnyFile and faDirectory, Info)=0 then begin
        repeat
            Dict.Add(Info.Name);
        until FindNext(Info) <> 0;
    end;
    FindClose(Info);
    Result := Dict.Count >= 1;
end;

function TForm1.CheckForDict() : boolean;
begin
    Result := False;
    EditDictPath.Caption := DictPathAlt;
    if not FindDictionary(ListBox1.Items, DictPath) then
        MemoMsg.Append('ERROR - no dictionaries found in ' + DictPath);
    if ListBox1.Items.Count = 1 then begin                   // Exactly one returned.
        if not Sp.SetDictionary(AppendPathDelim(DictPath) + ListBox1.Items.Strings[0]) then
            MemoMsg.Append('ERROR ' + SP.ErrorMessage)
        else
            MemoMsg.Append('Dictionary set to ' + DictPath + ListBox1.Items.Strings[0]);
    end;
    Result := SP.GoodToGo;   // only true if count was exactly one or FindDict failed and nothing changed
end;

Ah, you say, but where are we looking for the dictionaries ? Sadly, I don't have a good solution for that. Here is where I found mine -

procedure TForm1.SetDefaultDicPath();
begin
    {$ifdef LINUX}
    DictPath := '/usr/share/hunspell/';
    {$ENDIF}
    {$ifdef WINDOWS}
    DictPath := ExtractFilePath(Application.ExeName);
    //DictPath := 'C:\Program Files\LibreOffice 5\share\extensions\dict-en\';
    {$ENDIF}
    {$ifdef DARWIN}
    DictPath := '/Applications/Firefox.app/Contents/Resources/dictionaries/';
    //DictPathAlt := ExtractFilePath(Application.ExeName);
    {$endif}
end;

Maybe, if other users contribute where they found usable hunspell dictionaries, we can build a list for each platform. Or just take the easy way out and tell user to get some dictionaries and put them into the application directory on Windows and Mac. Your thoughts very welcome....

So far, if there is exactly one dictionary set in the indicated directory, all good. But what if there are several ? Our list box has a list of them, if the user double clicks one, they trigger this method -

procedure TForm1.ListBox1DblClick(Sender: TObject);
begin
    if ListBox1.ItemIndex > -1 then
        ButtonSpell.enabled := Sp.SetDictionary( AppendPathDelim(DictPath) + ListBox1.Items.Strings[ListBox1.ItemIndex]);
    if SP.ErrorMessage = '' then begin
        MemoMsg.Append('Good To Go =' + booltostr(Sp.GoodToGo, True));
        MemoMsg.Append('Dictionary set to ' + AppendPathDelim(DictPath) + ListBox1.Items.Strings[ListBox1.ItemIndex]);
    end else
        MemoMsg.append('ERROR ' + SP.ErrorMessage);
end;

Assuming we now have everything "good to go" we can press the Spell button and trigger this -

procedure TForm1.ButtonSpellClick(Sender: TObject);
begin
    if not Sp.Spell(Edit1.text) then begin
        Memo1.Lines.BeginUpdate;
        Sp.Suggest('badspeller', Memo1.lines);
        Memo1.Lines.EndUpdate;
    end else
        Memo1.Lines.Clear;
end;

Memo1 now contains some suggestions for better ways to spell badspeller !

Important ! Don't forget to free up our hunspeller object, memory leaks are evil !

procedure TForm1.FormDestroy(Sender: TObject);
begin
    Sp.free;
    Sp := nil;
end;

This module will do considerably more but its presented here in its most stripped down form for readability.

A note to people new to Lazarus, methods with "(Sender: TObject)" shown above cannot just be pasted into your source, use the Form's Object Inspector to create the events first, then paste my sample code into the method.

Actual Code

(sorry, it really is too long to paste into and out of a wiki page but that seems my only option)

{$MODE objfpc}{$H+}
unit hunspell;

{   Hunspell interface.
    Based on code that seems to appear in lots of places in the Lazarus Forum
    and elsewhere.

    With additions and corrections by dbannon to make it a little easier to use.

    As such, its assumed to be free to use by anyone for any purpose.
}

{   A Unit to connect to the hunspell library and check some spelling.
    First, create the class, it will try and find a library to load.
    Check ErrorMessage.
    Then call SetDictionary(), with a full filename of the dictionary to use.
    If GoodToGo is true, you can call Spell() and Suggests()
    otherwise, look in ErrorString for what went wrong.

    Look in FindLibrary() for default locations of Library.
}


interface
uses Classes, dynlibs;

type
  THunspell_create = function(aff_file: PChar; dict_file: PChar): Pointer; cdecl;
  THunspell_destroy = procedure(spell: Pointer); cdecl;
  THunspell_spell = function(spell: Pointer; word: PChar): Boolean; cdecl;
  THunspell_suggest = function(spell: Pointer; out slst: PPChar; word: PChar): Integer; cdecl;
  THunspell_analyze = function(spell: Pointer; var slst: PPChar; word: PChar): Integer; cdecl;
  THunspell_stem = function(spell: Pointer; var slst: PPChar; word: PChar): Integer; cdecl;
  THunspell_free_list = procedure(spell: Pointer; var slst: PPChar; n: integer); cdecl;
  THunspell_get_dic_encoding = function(spell: Pointer): PChar; cdecl;
  THunspell_add = function(spell: Pointer; word: PChar): Integer; cdecl;
  THunspell_remove = function(spell: Pointer; word: PChar): Integer; cdecl;

   { THunspell }

  THunspell = class
  private
    Speller: Pointer;
        { Loads indicated library, returns False and sets ErrorMessage if something wrong }
    function LoadHunspellLibrary(LibraryName: AnsiString): Boolean;
  public
    	    { set to True if speller is ready to accept requests }
    GoodToGo : boolean;
    	    { empty if OK, contains an error message if something goes wrong }
    ErrorMessage : ANSIString;
            { Will have a full name to library if correctly loaded at create }
    LibraryFullName : string;
            { Will have a "first guess" as to where dictionaries are, poke another name in
            and call FindDictionary() if default did not work }
    constructor Create();
    destructor Destroy; override;
            { Returns True if word spelt correctly }
    function Spell(Word: string): boolean;
            { Returns with List full of suggestions how to spell Word }
    procedure Suggest(Word: string; List: TStrings);
            { untested }
    procedure Add(Word: string);
            { untested }
    procedure Remove(Word: string);
            { returns a full library name or '' if it cannot find anything suitable }
    function FindLibrary(out FullName : AnsiString) : boolean;
            { returns true if it successfully set the indicated dictionary }
    function SetDictionary(const FullDictName: string) : boolean;
    function SetNewLibrary(const LibName : string) : boolean;
  end;

var Hunspell_create: THunspell_create;
var Hunspell_destroy: THunspell_destroy;
var Hunspell_spell: Thunspell_spell;
var Hunspell_suggest: Thunspell_suggest;
var Hunspell_analyze: Thunspell_analyze;
var Hunspell_stem: Thunspell_stem;
var Hunspell_get_dic_encoding: Thunspell_get_dic_encoding;
var Hunspell_add: THunspell_add;
var Hunspell_free_list: THunspell_free_list;
var Hunspell_remove: THunspell_remove;

var HunLibLoaded: Boolean = False;
var HunLibHandle: THandle;

implementation

uses LazUTF8, SysUtils, {$ifdef linux}Process, {$else} Forms, {$endif} LazFileUtils;
// Forms needed so we can call Application.~

{ THunspell }

function THunspell.LoadHunspellLibrary(libraryName: Ansistring): Boolean;
begin
    Result := false;
    HunLibHandle := LoadLibrary(PAnsiChar(libraryName));
    if HunLibHandle = NilHandle then
        ErrorMessage := 'Failed to load library ' + libraryName
    else begin
        Result := True;
        Hunspell_create := THunspell_create(GetProcAddress(HunLibHandle, 'Hunspell_create'));
        if not Assigned(Hunspell_create) then Result := False;
        Hunspell_destroy := Thunspell_destroy(GetProcAddress(HunLibHandle, 'Hunspell_destroy'));
        if not Assigned(Hunspell_destroy) then Result := False;
        Hunspell_spell := THunspell_spell(GetProcAddress(HunLibHandle, 'Hunspell_spell'));
        if not Assigned(Hunspell_spell) then Result := False;
        Hunspell_suggest := THunspell_suggest(GetProcAddress(HunLibHandle, 'Hunspell_suggest'));
        if not Assigned(Hunspell_suggest) then Result := False;
        Hunspell_analyze := THunspell_analyze(GetProcAddress(HunLibHandle, 'Hunspell_analyze'));  // not used here
        if not Assigned(Hunspell_analyze) then Result := False;
        Hunspell_stem := THunspell_stem(GetProcAddress(HunLibHandle, 'Hunspell_stem'));           // not used here
        if not Assigned(Hunspell_stem) then Result := False;
        Hunspell_get_dic_encoding := THunspell_get_dic_encoding(GetProcAddress(HunLibHandle, 'Hunspell_get_dic_encoding'));   // not used here
        if not Assigned(Hunspell_get_dic_encoding) then Result := False;
        Hunspell_free_list := THunspell_free_list(GetProcAddress(HunLibHandle, 'Hunspell_free_list'));
        if not Assigned(Hunspell_free_list) then Result := False;
        Hunspell_add := THunspell_add(GetProcAddress(HunLibHandle, 'Hunspell_add'));
        if not Assigned(Hunspell_add) then Result := False;
        Hunspell_remove := THunspell_remove(GetProcAddress(HunLibHandle, 'Hunspell_remove'));
        if not Assigned(Hunspell_remove) then Result := False;
        HunLibLoaded := Result;
    end;
    if ErrorMessage = '' then
        if not Result then ErrorMessage := 'Failed to find functions in ' + LibraryName;
end;

constructor THunspell.Create();
begin
    ErrorMessage := '';
    if Not FindLibrary(LibraryFullName) then begin
        ErrorMessage := 'Cannot find Hunspell library';
        exit();
    end;
    LoadHunspellLibrary(LibraryFullName);    // will flag any errors it finds
    Speller := nil;           // we are not GoodToGo yet, need a dictionary ....
end;

destructor THunspell.Destroy;
begin
    if (HunLibHandle <> 0) and HunLibLoaded then begin
        if Speller<>nil then hunspell_destroy(Speller);
        Speller:=nil;
        if HunLibHandle <> 0 then FreeLibrary(HunLibHandle);
        HunLibLoaded := false;
    end;
    inherited Destroy;
end;

function THunspell.Spell(Word: string): boolean;
begin
    Result := hunspell_spell(Speller, PChar(Word))
end;

procedure THunspell.Suggest(Word: string; List: TStrings);
var i, len: Integer;
	SugList, Words: PPChar;
begin
    List.clear;
    try
        len := hunspell_suggest(Speller, SugList, PChar(Word));
        Words := SugList;
        for i := 1 to len do begin
            List.Add(Words^);
            Inc(PtrInt(Words), sizeOf(Pointer));
        end;
    finally
        Hunspell_free_list(Speller, SugList, len);
    end;
end;

procedure THunspell.Add(Word: string);
begin
    Hunspell_add(Speller, Pchar(Word));
end;

procedure THunspell.Remove(Word: string);
begin
    Hunspell_remove(Speller, Pchar(Word));
end;

function THunspell.FindLibrary(out FullName : ANSIString):boolean;
var
    {$ifdef LINUX} I : integer = 1; {$endif}
    Info : TSearchRec;
    Mask : ANSIString;
begin
    Result := False;
    {$IFDEF LINUX}
    // Assumes ldconfig always returns same format, better than searching several dirs
    if RunCommand('/bin/bash',['-c','ldconfig -p | grep hunspell'], FullName) then begin
        while UTF8Pos(' ', FullName, I) <> 0 do inc(I);
        if I=1 then exit();
        UTF8Delete(FullName, 1, I-1);
        UTF8Delete(FullName, UTF8Pos(#10, FullName, 1), 1);
        Result := True;
    end;
    exit();
    {$ENDIF}
    {$IFDEF WINDOWS}		// Look for a dll in application home dir.
    Mask := '*hunspell*.dll';
    FullName := ExtractFilePath(Application.ExeName);
    {$endif}
    {$ifdef DARWIN}
    Mask := 'libhunspell*';
    FullName := '/usr/lib/';
    {$endif}
    if FindFirst(FullName + Mask, faAnyFile and faDirectory, Info)=0 then begin
        FullName := FullName + Info.name;
        Result := True;
    end;
    FindClose(Info);
end;

function THunspell.SetDictionary(const FullDictName: string) : boolean;
var
    FullAff : string;
begin
    FullAff := FullDictName;
    UTF8Delete(FullAff, UTF8Length(FullAff) - 2, 3);
    FullAff := FullAff + 'aff';
    if Speller <> Nil then
        hunspell_destroy(Speller);
    Speller := hunspell_create(PChar(FullAff), PChar(FullDictName));
    GoodToGo := Speller <> Nil;
    if not GoodToGo then
        ErrorMessage := 'Failed to set Dictionary ' + FullDictName;
    Result := GoodToGo;
end;

function THunspell.SetNewLibrary(const LibName: string): boolean;
begin
    LibraryFullName := LibName;
    Result := LoadHunspellLibrary(LibraryFullName);
end;

end.

Note that this Unit uses LazUTF8, LazFileUtils and Forms. If you wish to use it as a simple command line application, you might add LCL to Required Packages in the Project Inspector or drop back to the FPC versions of Pos() etc but at the expense of UTF8 compatibility. And you cannot use Forms to provide Application.ExeName.

Further Reading and Links

https://github.com/hunspell/hunspell

https://github.com/Homebrew - probably sensible way to get hunspell on your mac if its not already there.

https://github.com/tomboy-notes/tomboy-ng/releases - Contains The 64bit Windows DLL in a tomboy-ng_win64_<ver>.zip

https://github.com/cutec-chris/hunspell