Difference between revisions of "fpcunit"
(Added testing category. Useful to group unit testing as well as compiler tests etc)
(→Test decorator: OneTimeSetup and OneTimeTearDown: grammar/clarity. review welcome)
|Line 28:||Line 28:|
===Test decorator: OneTimeSetup and OneTimeTearDown===
===Test decorator: OneTimeSetup and OneTimeTearDown===
The Setup and Teardown procedures mentioned above are run per test. You can also run Setup and Teardown procedures once per
The Setup and Teardown procedures mentioned above are run per test. You can also run Setup and Teardown procedures once per OneTimeSetup and OneTimeTearDown inherit from the TTestSetup "test decorator" and register , e.g.:
Revision as of 14:11, 31 March 2013
- 1 Overview
- 2 Use in FPC/Lazarus
- 3 Use
- 4 Output
- 5 Alternatives
- 6 Lazarus
- 7 GDB bug/feature
- 8 See also
fpcunit is a unit testing framework a la DUnit/JUnit/SUnit. This allows you to quickly write a set of test for a (logical) unit of code (not necessarily the same as a Pascal unit, though often it is).
Development methodologies like Test Driven Design use this to make sure you code your expectations/specifications in your unit tests first, then write your main code, then run your tests and improve the code until all tests pass.
Not only does fpcunit allow you to visually inspect test runs, you can also collect the results systematically (using the XML output), and use that to compare versions for e.g. regression errors (i.e. you run your regresssion tests using the unit test output).
Screenshot of the GUI test runner:
The image shows that out of 10 tests run, 6 tests failed. The EAssertionFailure exceptions indicate the test assertions (see below) were not met - i.e. the test failed. The associated messges indicate the result the test expected and the actual result achieved.
Use in FPC/Lazarus
FPCUnit tests are used in the FPC database test framework: Databases#Running_FPC_database_tests
There are also tests for the FPC compiler/core packages, but these presumably predate fpcunit and use a simpler approach.
It's easiest to use Lazarus to set up a new test project for you.
This procedure is present in all FPCUnit tests. It sets up the test environment before each test is run - in other words not only before and after the complete test suite is run, but for every test. You can use this to e.g. fill a database with test data.
This procedure is present in all FPCUnit tests. It cleans up the test environment after each test is run. You can use this to e.g. clear test data from a database.
Test decorator: OneTimeSetup and OneTimeTearDown
The Setup and Teardown procedures mentioned above are run per test. You can also run Setup and Teardown procedures once per instance. To do this, use OneTimeSetup and OneTimeTearDown and inherit from the TTestSetup "test decorator" and register it, e.g.:
uses ... testdecorator ... TDBBasicsTestSetup = class(TTestSetup) protected procedure OneTimeSetup; override; procedure OneTimeTearDown; override; end; ... initialization // make sure you register your test along with the decorator so it knows the run the setup/teardowns RegisterTestDecorator(TDBBasicsTestSetup, TTestDBBasics);
You write your own tests as published procedures in the test class. You can use AssertEquals etc to specify what should be tested. If you want to fail a test, you can e.g. use
if 5=0 then //ridiculous example but you understand what I mean. You can use other Assert* procedures to test equality much easier etc. AssertTrue('This part of the code should never have been reached in this test.',false);
If the test fails, an EAssertionFailedError will be raised with the message you specify in Assert*. This way, you can add a series of subtests and tell which subtest failed. Note: the test runner will stop at the first assertion failure, so subsequent subtests will not be performed. If you do want to always test everything, split out these subtests in separate test procedures.
Instead of the Assert* procedures, you can also use the DUnit compatible Check* procedures (e.g. CheckEquals) which give more descriptive error messages in the test results: they include expected and actual values.
The order in which the tests are run are the order in which they appear in the test class definition.
Ttestexport1 = class(Ttestcase) ... published procedure TestOutput; ... procedure Ttestexport1.TestOutput; const OutputFilename='output.csv'; begin TestDataSet.Close; if FileExists(OutputFilename) then DeleteFile(OutputFileName); TestDataset.FileName:=OutputFileName; TestDataset.Open; // Fill test data TestDataset.Append; TestDataset.FieldByName('ID').AsInteger := 1; // Data with quotes TestDataset.FieldByName('NAME').AsString := 'J"T"'; TestDataset.FieldByName('BIRTHDAY').AsDateTime := ScanDateTime('yyyymmdd', '19761231', 1); TestDataset.Post; TestDataset.Last; TestDataset.First; TestDataset.First; AssertEquals('Number of records in test dataset', 1, TestDataset.RecordCount); TestDataset.Close; end;
In simple cases, you (or Lazarus) would register all your test cases with calls like:
uses ... testregistry ... initialization RegisterTest(Ttestexport1); //you pass the class name to register it for running
However, you can also create multiple layers to group your test cases if your project gets big:
initialization RegisterTest('WorldDominationApp.ExportCheesePlan',Ttestexport1); //The levels are separated by periods RegisterTest('WorldDominationApp.Obsolete',TtestPinkysBrain1); //another category
The console test runner can output in XML (either original FPCUnit format, or a more advanced DUnit2-like format if you use the xmltestreport unit), plain text and latex (e.g. usable for PDF export) formats. The GUI test runner outputs to XML if needed (using the same xmltestreport XML format).
You can use your own "listener" that listens to the test results and outputs the test data in whatever way you want. Create a T*Listener that implements the ITestListener interface. It's only 5 required methods to implement.
In your test runner application (e.g. a copy of fpctestconsole), add a listener object; register that test listener with the testing framework (e.g. your console test runner) via the TestResult.AddListener() call, and it will be fed test results as they happen.
An example of a custom listener is the database output writer available at https://bitbucket.org/reiniero/testdbwriter. This writer will save all test results to a database, which is optimized for receiving large amounts of test results (handy for using on CI server like Jenkins or for importing/consolidating test results). The mentioned repository contains an example that runs the db test framework test results to (another) database.
To do: adapt this; use the new xml unit
An example of an available extra listener is TXMLResultsWriter in the xmlreporter unit in <fpc>\packages\fcl-fpcunit\src\xmlreporter.pas.
todo: actually, dbtestframework seems to use the old xml output method... An example of an adapted test runner that uses an extra listener can be found in <fpc>\packages\fcl-db\tests\dbtestframework.pas, which contains this code to output to custom listeners (an XML writer and a digest writer that stuffs the output in a .tar archive, handy to process remotely):
uses ...the rest of the units needed for test runners... fpcunit,... // the units with TXMLResultsWriter and TDigestResultsWriter testreport,DigestTestReport ... Procedure LegacyOutput; var FXMLResultsWriter: TXMLResultsWriter; FDigestResultsWriter: TDigestResultsWriter; testResult: TTestResult; begin testResult := TTestResult.Create; FXMLResultsWriter := TXMLResultsWriter.Create; FDigestResultsWriter := TDigestResultsWriter.Create(nil); try testResult.AddListener(FXMLResultsWriter); testResult.AddListener(FDigestResultsWriter); // Set some properties specific for this results writer: FDigestResultsWriter.Comment:=dbtype; FDigestResultsWriter.Category:='DB'; FDigestResultsWriter.RelSrcDir:='fcl-db'; //WriteHeader is specific for this listener; it writes the header to an XML file //notice that it is not called for FDigestResultsWriter FXMLResultsWriter.WriteHeader; // This performs the actual test run, and the output will be processed by the listeners: GetTestRegistry.Run(testResult); // Likewise WriteResult is specific for this listener; it writes FXMLResultsWriter.WriteResult(testResult); finally testResult.Free; FXMLResultsWriter.Free; FDigestResultsWriter.Free; end; end;
There is DUnit2 (IIRC an improvement of the original DUnit), originally written for Delphi, which is in use in the tiOPF framework.
Also available is FPTest, which is based on DUnit2.
Lazarus has the consoletestrunner and GUI test runner units, which can be installed by installing the FPCUnitTestRunner package. This will help you create and run your unit tests using a GUI (or console, if you want to).
The consoletestrunner is compatible with FPC so you don't need Lazarus to compile it. The Lazarus version is slightly different to the one in FPC (e.g. use of UTF8 output etc).
The GUI runner is easier to use.
In the GUI runner, if you want to run all tests, currently you first need to click on a test element before the Run all tests button is activated.
Note (September 2012): a bug/undocumented feature in the debugger used by Lazarus/FPC (gdb) means that passing --all as run parameters has no effect. Passing this parameter can be useful when debugging console fpcunit test runners has no effect. Workaround: use -a. See bug 
- http://www.freepascal.org/olddocs-html/fpcunit.pdf Covers both FPC and Lazarus use. Explains the various classes involved. Integrating portions of that pdf into this documentation (or vice versa) could be very helpful.
- http://sergworks.wordpress.com/2012/08/31/introduction-to-unit-testing-with-lazarus/ Nice article about getting FPCUnit tests up and running in the comfort of your Lazarus IDE
- http://www.pp4s.co.uk/main/tu-testing-auto2.html Example program that demonstrates FPCUnit testing
- http://www.win.tue.nl/~mousavi/testing/4.pdf Presentation on FPCUnit