LCL Unicode Support/ko

│ Deutsch (de) │ English (en) │ español (es) │ français (fr) │ 日本語 (ja) │ 한국어 (ko) │ русский (ru) │ 中文（中国大陆）‎ (zh_CN) │ 中文（台灣）‎ (zh_TW) │

소개

0.9.25 에서 라자루스는 Gtk1을 제외한 모든 플랫폼에서 완전한 유니코드를 지원하게 되었다. 이 페이지는 FPC 2.6.4를 사용하는 라자루스 1.4 까지의 유니코드에 대해 다룬다. 2.0을 시작하면서 라자루스는 FPC 3.0+의 특징을 사용하여 유니코드 지원을 개선하였다. 자세한 내용은 다음을 보시길 : 라자루스에서 개선 된 유니코드 지원

사용자를 위한 지침

라자루스가 유니코드 위젯 세트를 가지고 있다지만 모든 것이 유니코드가 아니란 것을 알아야 한다. 문자열 인코딩의 시작이 무엇인지 알고 다른 인코딩을 필요로하는 라이브러리간에 적절히 인코딩하는 것은 개발자의 책임이다.

보통 인코딩은 라이브러리 단위로 이루어진다(예를 들면, 동적 라이브러리인 DLL이나 라자루스 패키지). 각 라이브러리는 한 종류의 인코딩만을 요구하고, 이것들은 대개 Unicode(라자루스에서는 UTF-8) 또는 ANSI 이다(이것은 실제로는 시스템 인코딩으로 UTF-8 이거나 아니라는 뜻이다). 2.6 이하의 FPC의 RTL 과 FCL 은 ANSI 문자열이 필요하다.

유니코드와 ANSI는 다음 코드를 사용하여 변환할 수 있다.

• UTF8ToAnsi 와 AnsiToUTF8은 (FPC) System 유닛에서 기능한다.
• 또는 UTF8ToSys 와 SysToUTF8은 (Lazarus) FileUtil 유닛에서 기능한다.

두번째 것이 더 빠르고 산뜻하지만 더 많은 코드가 프로그램에 들어가게 된다.

FPC 는 유니코드를 알고 있지 않다

현재의 FPC버전(2.6.x 이하)에서 프리파스칼 런타임 라이브러리(RTL)과 프리파스칼 컴포넌트 라이브러리(FCL)은 ANSI라서 유니코드 라이브러리에서 가져오거나 유니코드 라이브러리로 보내는(예, LCL) 경우에는 문자열 변환이 필요하다.

String 을 포함하여 FPC 2.7.1 계열의 개발 버전에 중요한 개선점들이 있다. RawByteString 과 UTF8String 에 대해서는 FPC의 유니코드 지원 을 보세요.

ANSI 와 Unicode 간의 변환

  Note: AnsiToUTF8 와 UTF8ToAnsi 는 리눅스, BSD, Mac OS X 에서 사용할 경우 widestring 관리자가 필요하다. SysToUTF8 과 UTF8ToSys 함수를 사용할 수 있다.(unit FileUtil) 또는 프로그램이 사용하는 첫번째 uses 섹션에 cwstring을 추가하여 widestring 매니저를 추가할 수 있게된다.

예:

TEdit에서 문자열을 얻고 어떤 RTL 파일 루틴에 문자열을 보내고 싶다고 하자.

var
  MyString: string; // utf-8 로 인코드 되어 있다.
begin
  MyString := MyTEdit.Text;
  SomeRTLRoutine(UTF8ToAnsi(MyString));
end;

반대로 보낼 경우에는:

var
  MyString: string; // ANSI 로 인코드 되어 있다.
begin
  MyString := SomeRTLRoutine;
  MyTEdit.Text := AnsiToUTF8(MyString);
end;

Widestrings 과 Ansistrings

widestring 은 2-바이트 크기의 요소를 가지는 기본데이터의 문자열 타입이다. Widestring은 거의 대부분 UTF-16 로 인코딩된 데이터를 가진다. widestring 을 보세요

widestring의 배열로 억세스할 수 있는 각 데이터 포인트는 2 바이트를 가지며, UTF-16에서 문자(character)는 1 또는 2 데이터 포인트일 수 있어서, 2 또는 4 바이트를 차지할 수 있다. 이는 Widestring을 배열로 억세스하고 이러한 방법으로 UTF-16 문자들을 얻으려고 하는 것은 확실히 잘못된 것이어서 문자열에 4 바이트 문자가 들어있을 때는 실패하게 될 것이다. 또한 UTF-8 과 마찬가지로 UTF-16은 분리되는 문자를 가질 수도 있다. 예를 들어 문자 "Á" 는 단일 문자로 인코드할 수 있거나 두개의 문자로 인코드 될 수도 있다:"A" + 수정 액센트. 그러므로 Unicode에서 엑센트 문자를 가진 문장은 다양한 방법으로 인코드 될 수 있어서 라나루스나 FPC는 자동으로 이들을 다룰 수가 없다.

Ansistrings 를 Widestrings 로 전달할 때 인코딩을 변환해야 한다.

var 
  w: widestring;
begin
  w:='Über'; // 틀림, FPC 는 시스템 코드페이지를 UTF16 로 변경할 것이다.
  w:=UTF8ToUTF16('Über'); // 맞음
  Button1.Caption:=UTF16ToUTF8(w);
end;

UTF8 문자열 과 문자들

Lazarus 0.9.30 까지 UTF-8 핸들링 루틴은 LCLProc유닛 내의 LCL에 있었다. 라자루스 0.9.31+에서 LCLProc 내의 루틴들은 여전히 호환성 문제 때문에 쓸모가 있으나 UTF-8을 다루는 실제코드는 lazutf8 유닛내의 lazutils 패키지에 있다. Until Lazarus 0.9.30 the UTF-8 handling routines were in the LCL in the unit LCLProc. In Lazarus 0.9.31+ the routines in LCLProc are still available for backwards compatibility but the real code to deal with UTF-8 is located in the lazutils package in the unit lazutf8.

To execute operations on UTF-8 strings please use routines from the unit lazutf8 instead of routines from the SysUtils routine from Free Pascal, because SysUtils is not yet prepared to deal with Unicode, while lazutf8 is. Simply substitute the routines from SysUtils with their lazutf8 equivalent, which always has the same name except for an added "UTF8" prefix.

UTF-8 String Copy, Length, LowerCase, etc

Nearly all operations which one might want to execute with UTF-8 strings are covered by the routines in the unit lazutf8 (unit LCLProc for Lazarus 0.9.30 or lower). See the following list of routines taken from lazutf8.pas:

function UTF8CharacterLength(p: PChar): integer;
function UTF8Length(const s: string): PtrInt;
function UTF8Length(p: PChar; ByteCount: PtrInt): PtrInt;
function UTF8CharacterToUnicode(p: PChar; out CharLen: integer): Cardinal;
function UnicodeToUTF8(u: cardinal; Buf: PChar): integer; inline;
function UnicodeToUTF8SkipErrors(u: cardinal; Buf: PChar): integer;
function UnicodeToUTF8(u: cardinal): shortstring; inline;
function UTF8ToDoubleByteString(const s: string): string;
function UTF8ToDoubleByte(UTF8Str: PChar; Len: PtrInt; DBStr: PByte): PtrInt;
function UTF8FindNearestCharStart(UTF8Str: PChar; Len: integer;
                                  BytePos: integer): integer;
// find the n-th UTF8 character, ignoring BIDI
function UTF8CharStart(UTF8Str: PChar; Len, CharIndex: PtrInt): PChar;
// find the byte index of the n-th UTF8 character, ignoring BIDI (byte len of substr)
function UTF8CharToByteIndex(UTF8Str: PChar; Len, CharIndex: PtrInt): PtrInt;
procedure UTF8FixBroken(P: PChar);
function UTF8CharacterStrictLength(P: PChar): integer;
function UTF8CStringToUTF8String(SourceStart: PChar; SourceLen: PtrInt) : string;
function UTF8Pos(const SearchForText, SearchInText: string): PtrInt;
function UTF8Copy(const s: string; StartCharIndex, CharCount: PtrInt): string;
procedure UTF8Delete(var s: String; StartCharIndex, CharCount: PtrInt);
procedure UTF8Insert(const source: String; var s: string; StartCharIndex: PtrInt);

function UTF8LowerCase(const AInStr: string; ALanguage: string=''): string;
function UTF8UpperCase(const AInStr: string; ALanguage: string=''): string;
function FindInvalidUTF8Character(p: PChar; Count: PtrInt;
                                  StopOnNonASCII: Boolean = false): PtrInt;
function ValidUTF8String(const s: String): String;

procedure AssignUTF8ListToAnsi(UTF8List, AnsiList: TStrings);

//compare functions

function UTF8CompareStr(const S1, S2: string): Integer;
function UTF8CompareText(const S1, S2: string): Integer;

Dealing with directory and filenames

Lazarus controls and functions expect filenames and directory names in UTF-8 encoding, but the RTL uses ANSI strings for directories and filenames.

For example, consider a button which sets the Directory property of the TFileListBox to the current directory. The RTL Function GetCurrentDir is ANSI, not Unicode, so conversion is needed:

procedure TForm1.Button1Click(Sender: TObject);
begin
  FileListBox1.Directory:=SysToUTF8(GetCurrentDir);
  // or use the functions from the FileUtil unit
  FileListBox1.Directory:=GetCurrentDirUTF8;
end;

The unit FileUtil defines common file functions with UTF-8 strings:

// basic functions similar to the RTL but working with UTF-8 instead of the
// system encoding

// AnsiToUTF8 and UTF8ToAnsi need a widestring manager under Linux, BSD, Mac OS X
// but normally these OS use UTF-8 as system encoding so the widestringmanager
// is not needed.
function NeedRTLAnsi: boolean;// true if system encoding is not UTF-8
procedure SetNeedRTLAnsi(NewValue: boolean);
function UTF8ToSys(const s: string): string;// as UTF8ToAnsi but more independent of widestringmanager
function SysToUTF8(const s: string): string;// as AnsiToUTF8 but more independent of widestringmanager

// file operations
function FileExistsUTF8(const Filename: string): boolean;
function FileAgeUTF8(const FileName: string): Longint;
function DirectoryExistsUTF8(const Directory: string): Boolean;
function ExpandFileNameUTF8(const FileName: string): string;
function ExpandUNCFileNameUTF8(const FileName: string): string;
{$IFNDEF VER2_2_0}
function ExtractShortPathNameUTF8(Const FileName : String) : String;
{$ENDIF}
function FindFirstUTF8(const Path: string; Attr: Longint; out Rslt: TSearchRec): Longint;
function FindNextUTF8(var Rslt: TSearchRec): Longint;
procedure FindCloseUTF8(var F: TSearchrec);
function FileSetDateUTF8(const FileName: String; Age: Longint): Longint;
function FileGetAttrUTF8(const FileName: String): Longint;
function FileSetAttrUTF8(const Filename: String; Attr: longint): Longint;
function DeleteFileUTF8(const FileName: String): Boolean;
function RenameFileUTF8(const OldName, NewName: String): Boolean;
function FileSearchUTF8(const Name, DirList : String): String;
function FileIsReadOnlyUTF8(const FileName: String): Boolean;
function GetCurrentDirUTF8: String;
function SetCurrentDirUTF8(const NewDir: String): Boolean;
function CreateDirUTF8(const NewDir: String): Boolean;
function RemoveDirUTF8(const Dir: String): Boolean;
function ForceDirectoriesUTF8(const Dir: string): Boolean;

// environment
function ParamStrUTF8(Param: Integer): string;
function GetEnvironmentStringUTF8(Index : Integer): String;
function GetEnvironmentVariableUTF8(const EnvVar: String): String;
function GetAppConfigDirUTF8(Global: Boolean): string;

East Asian languages on Windows

The default font (Tahoma) for user interface controls under Windows XP is capable of correctly displaying several scripts/alphabets/languages, including Arabic, Russian (Cyrillic alphabet) and Western languages (Latin/Greek alphabets), but not East Asian languages, like Chinese, Japanese and Korean.

Simply by going to the Control Panel, choosing Regional Settings, clicking on the Languages Tab and installing the East Asia Language Pack, the standard user interface font will start showing those languages correctly. Obviously Windows XP versions localized for those languages will already have this language pack installed. Extended instructions here.

Later Windows versions presumably have support for these languages out of the box.

Usage of UTF-8

See details in UTF8_strings_and_characters

Free Pascal Particularities

UTF8 and source files - the missing BOM

When you create source files with Lazarus and type some non-ASCII characters the file is saved in UTF8. It does not use a BOM (Byte Order Mark). You can change the encoding via right click on source editor / File Settings / Encoding. Apart from the fact that UTF-8 files are not supposed to have BOMs, the reason for the lacking BOM is how FPC treats Ansistrings. For compatibility the LCL uses Ansistrings and for portability the LCL uses UTF8.

Note: Some MS Windows text editors might treat the files as encoded with the system codepage (OEM codepage) and show them as invalid characters. Do not add the BOM. If you add the BOM you have to change all string assignments.

For example:

Button1.Caption := 'Über';

When no BOM is given (and no codepage parameter was passed) the compiler treats the string as system encoding and copies each byte unconverted to the string. This is how the LCL expects strings.

// source file saved as UTF without BOM
if FileExists('Über.txt') then ; // wrong, because FileExists expects system encoding
if FileExistsUTF8('Über.txt') then ; // correct

Unicode essentials

The Unicode standard maps integers from 0 to 10FFFF(h) to characters. Each such mapping is called a code point. In other words, Unicode characters are in principle defined for code points from U+000000 to U+10FFFF (0 to 1 114 111).

There are three major schemes for representing Unicode code points as unique byte sequences. These schemes are called Unicode transformation formats: UTF-8, UTF-16 and UTF-32. Conversions between all of them are possible. Here are their basic properties:

                           UTF-8 UTF-16 UTF-32
Smallest code point [hex] 000000 000000 000000
Largest code point  [hex] 10FFFF 10FFFF 10FFFF
Code unit size [bits]          8     16     32
Minimal bytes/character        1      2      4
Maximal bytes/character        4      4      4

UTF-8 has several important and useful properties: It is interpreted as a sequence of bytes, so that the concept of lo- and hi-order byte does not exist. Unicode characters U+0000 to U+007F (ASCII) are encoded simply as bytes 00h to 7Fh (ASCII compatibility). This means that files and strings which contain only 7-bit ASCII characters have the same encoding under both ASCII and UTF-8. All characters >U+007F are encoded as a sequence of several bytes, each of which has the two most significant bits set. No byte sequence of one character is contained within a longer byte sequence of another character. This allows easy searching for substrings. The first byte of a multibyte sequence (representing a non-ASCII character) is always in the range C0h to FDh and it indicates how many bytes follow for this character. All further bytes in a multibyte sequence are in the range 80h to BFh. This allows easy resynchronization and robustness.

UTF-16 has the following most important properties: It uses a single 16-bit word to encode characters from U+0000 to U+d7ff, and a pair of 16-bit words to encode any of the remaining Unicode characters.

Finally, any Unicode character can be represented as a single 4 byte/32-bit unit in UTF-32.

For more, see: Unicode FAQ - Basic questions, Unicode FAQ - UTF-8, UTF-16, UTF-32 & BOM, Wikipedia: UTF-8 [1]

Implementation Details

Lazarus/LCL generally uses only UTF-8

Since the GTK1 interface was declared obsolete in Lazarus 0.9.31, all LCL interfaces are Unicode capable and the LCL uses and accepts only UTF-8 encoded strings, unless in routines explicitly marked as accepting other encodings.

Unicode-enabling the win32 interface

Overview

First, and most importantly, all Unicode patches for the Win32 interface must be enclosed by IFDEF WindowsUnicodeSupport, to avoid breaking the existing ANSI interface. After this stabilizes, all ifdefs will be removed and only the Unicode part will remain. At his moment all existing programs that use ANSI characters will need migration to Unicode.

No Unicode support on Win9x

Windows platforms <=Win9x are based on ISO code page standards and only partially support Unicode. Windows platforms starting with Windows NT (e.g. Windows 2000, XP, Vista, 7, 8) and Windows CE fully support Unicode.

Win 9x and NT offer two parallel sets of API functions: the old ANSI enabled *A and the new, Unicode enabled *W. *W functions accept wide strings - UTF-16 encoded strings - as parameters.

Windows 9x has all *W functions but they mostly have empty implementations, so they do nothing. Only some some *W functions are fully implemented in 9x; these are listed below in the section "Wide functions present on Windows 9x". This property is relevant as it allows to have one single application for both Win9x and WinNT and detect at runtime which set of APIs to use.

Windows CE only uses Wide API functions.

Wide functions present on Windows 9x

Some Wide API functions are present on Windows 9x. Here is a list of such functions: http://support.microsoft.com/kb/210341

Conversion example:

GetTextExtentPoint32(hdcNewBitmap, LPSTR(ButtonCaption),
Length(ButtonCaption), TextSize);

Becomes:

{$ifdef WindowsUnicodeSupport}
  GetTextExtentPoint32W(hdcNewBitmap, PWideChar(Utf8Decode(ButtonCaption)), Length(WideCaption), TextSize);
{$else}
  GetTextExtentPoint32(hdcNewBitmap, LPSTR(ButtonCaption), Length(ButtonCaption), TextSize);
{$endif}

Functions that need Ansi and Wide versions

First Conversion example:

function TGDIWindow.GetTitle: String;
var
 l: Integer;
begin
   l := Windows.GetWindowTextLength(Handle);
   SetLength(Result, l);
   Windows.GetWindowText(Handle, @Result[1], l);
end;

Becomes:

function TGDIWindow.GetTitle: String;
var
  l: Integer;
  AnsiBuffer: string;
  WideBuffer: WideString;
begin

{$ifdef WindowsUnicodeSupport}

if UnicodeEnabledOS then
begin
  l := Windows.GetWindowTextLengthW(Handle);
  SetLength(WideBuffer, l);
  l := Windows.GetWindowTextW(Handle, @WideBuffer[1], l);
  SetLength(WideBuffer, l);
  Result := Utf8Encode(WideBuffer);
end
else
begin
  l := Windows.GetWindowTextLength(Handle);
  SetLength(AnsiBuffer, l);
  l := Windows.GetWindowText(Handle, @AnsiBuffer[1], l);
  SetLength(AnsiBuffer, l);
  Result := AnsiToUtf8(AnsiBuffer);
end;

{$else}

   l := Windows.GetWindowTextLength(Handle);
   SetLength(Result, l);
   Windows.GetWindowText(Handle, @Result[1], l);

{$endif}

end;

Screenshots

FPC codepages

The compiler (FPC) supports specifying the code page in which the source code has been written via the command option -Fc (e.g. -Fcutf8) and the equivalent codepage directive (e.g. {$codepage utf8}). In this case, rather than literally copying the bytes that represent the string constants in your program, the compiler will interpret all character data according to that codepage. There are two things to watch out for though:

• on Unix platforms, make sure you include a widestring manager by adding the cwstring unit to your uses-clause. Without it, the program will not be able to convert all character data correctly when running. It's not included by default because this unit makes your program dependent on libc, which makes cross-compilation harder.
• The compiler converts all string constants that contain non-ASCII characters to widestring constants. These are automatically converted back to ansistring (either at compile time or at run time), but this can cause one caveat if you try to mix both characters and ordinal values in a single string constant:

For example:

program project1;
{$codepage utf8}
{$mode objfpc}{$H+}
{$ifdef unix}
uses cwstring;
{$endif}
var
  a,b,c: string;
begin
  a:='ä';
  b:='='#$C3#$A4; // #$C3#$A4 is UTF-8 for ä
  c:='ä='#$C3#$A4;
  writeln(a,b); // writes ä=ä
  writeln(c);   // writes ä=ä
end.

When compiled and executed, this will write:

ä=ä
ä=ä

The reason is once the ä is encountered, as mentioned above the rest of the constant string assigned to 'c' will be parsed as a widestring. As a result the #$C3 and #$A4 are interpreted as widechar(#$C3) and widechar(#$A4), rather than as ansichars.

See Also

• Character and string types
• UTF-8 - Description of UTF-8 strings
• FPC Unicode support
• for-in_loop#Traversing_UTF-8_strings - Code snippet that shows how to traverse UTF8 strings
• unicode use cases