Difference between revisions of "UTF8 Tools"

From Lazarus wiki
Jump to navigationJump to search
m (moved Theodp to UTF8 Tools: name UTF8tools is from download name of file. old name strange.)
 
(4 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 +
{{LanguageBar|UTF8 Tools}}
 +
 +
__TOC__
 +
 
== About ==
 
== About ==
  
Line 12: Line 16:
 
Also it has class to read/write Unicode from/to TStream.
 
Also it has class to read/write Unicode from/to TStream.
  
= UTF-8 Tools =
+
= Units =
== Purpose ==
+
== Using streams ==
Some tools for common problems with UTF-8 / Unicode.
 
  
 +
Unit "charencstreams": load/save data from/to almost any text source:
  
* '''charencstreams.pas''': Load and save data from / to almost any text source like
+
* ANSI, UTF8, UTF16, UTF32
** ANSI, UTF8, UTF16, UTF32
+
* big-endian, little-endian
** big or little endian
+
* with/without BOM
** BOM or no BOM
 
 
      
 
      
''Simple demo:''
+
Demo:
 +
 
 
<syntaxhighlight lang="pascal">
 
<syntaxhighlight lang="pascal">
  fCES := TCharEncStream.Create;
+
  f := TCharEncStream.Create;
  fCES.LoadFromFile(OpenDialog1.FileName);
+
  f.LoadFromFile(OpenDialog1.FileName);
  Memo1.text := fCES.UTF8Text;   
+
  Memo1.Text := f.UTF8Text;   
  fCES.free;
+
  f.Free;
 
</syntaxhighlight>
 
</syntaxhighlight>
  
 +
== Character info ==
  
* '''character.pas''': Get Information about code points using the TCharacter class.
+
Unit "character": get information about code points using the TCharacter class. Demo:
  
''Demo''
+
<syntaxhighlight lang="pascal">
  if TCharacter.IsLetter(s[i]) then s[i] := TCharacter.toLower(s[i]);
+
  if TCharacter.IsLetter(s[i]) then
 +
    s[i] := TCharacter.toLower(s[i]);
 +
</syntaxhighlight>
  
  
 +
== Access UTF-8 by code index ==
  
* '''utf8scanner.pas''': Access UTF-8 strings by code index, use case statements on UTF-8 strings and more...
+
Unit "utf8scanner": access UTF-8 strings by code index, use case statements on UTF-8 strings and more. Demo:
  
''Index demo''
+
<syntaxhighlight lang="pascal">
  s := TUTF8Scanner.Create(Memo1.text);
+
  s := TUTF8Scanner.Create(Memo1.Text);
 
  for i := 1 to s.Length do
 
  for i := 1 to s.Length do
if TCharacter.IsLetter(s[i]) then s[i] := TCharacter.toLower(s[i]);
+
  if TCharacter.IsLetter(s[i]) then
 +
    s[i] := TCharacter.toLower(s[i]);
 
  Memo1.Text := s.UTF8String;
 
  Memo1.Text := s.UTF8String;
  s.free;
+
  s.Free;
 +
</syntaxhighlight>
  
''Case demo''
+
Case demo:
   s := TUTF8Scanner.Create(Memo1.text);
+
 
 +
<syntaxhighlight lang="pascal">
 +
   s := TUTF8Scanner.Create(Memo1.Text);
 
   s.FindChars := 'öäü';
 
   s.FindChars := 'öäü';
 
   repeat
 
   repeat
Line 58: Line 70:
 
   until s.Done;
 
   until s.Done;
 
   Memo1.Text := s.UTF8String;
 
   Memo1.Text := s.UTF8String;
   s.free;
+
   s.Free;
 +
</syntaxhighlight>
 +
 
 +
= Download =
  
== Download ==
 
 
[http://www.theo.ch/lazarus/utf8tools.zip Download utf8tools.zip]
 
[http://www.theo.ch/lazarus/utf8tools.zip Download utf8tools.zip]
 +
 +
[[Category:Unicode]]

Latest revision as of 17:48, 12 December 2018

English (en) русский (ru)

About

This code allows to process Unicode text and determine for unicode chars:

  • if char "letter"
  • if char "digit"
  • if char upper-case, lower-case
  • if char "white space"
  • if char "punctuation"
  • etc.

Also it has class to read/write Unicode from/to TStream.

Units

Using streams

Unit "charencstreams": load/save data from/to almost any text source:

  • ANSI, UTF8, UTF16, UTF32
  • big-endian, little-endian
  • with/without BOM

Demo:

 f := TCharEncStream.Create;
 f.LoadFromFile(OpenDialog1.FileName);
 Memo1.Text := f.UTF8Text;  
 f.Free;

Character info

Unit "character": get information about code points using the TCharacter class. Demo:

 if TCharacter.IsLetter(s[i]) then
    s[i] := TCharacter.toLower(s[i]);


Access UTF-8 by code index

Unit "utf8scanner": access UTF-8 strings by code index, use case statements on UTF-8 strings and more. Demo:

 s := TUTF8Scanner.Create(Memo1.Text);
 for i := 1 to s.Length do
   if TCharacter.IsLetter(s[i]) then
     s[i] := TCharacter.toLower(s[i]);
 Memo1.Text := s.UTF8String;
 s.Free;

Case demo:

  s := TUTF8Scanner.Create(Memo1.Text);
  s.FindChars := 'öäü';
  repeat
    case s.FindIndex(s.Next) of
  {ö} 0: s.Replace('oe');
  {ä} 1: s.Replace('ae');
  {ü} 2: s.Replace('ue');
    end;
  until s.Done;
  Memo1.Text := s.UTF8String;
  s.Free;

Download

Download utf8tools.zip