API Sanity Checker
An automatic generator of basic unit tests for a shared C/C++ library
API Sanity Checker is an automatic generator of basic unit tests for a shared C/C++ library. It is able to generate reasonable (in most, but unfortunately not all, cases) input data for parameters and compose simple ("sanity" or "shallow"-quality) test cases for every function in the API through the analysis of declarations in header files.
The quality of generated tests allows to check absence of critical errors in simple use cases. The tool is able to build and execute generated tests and detect crashes (segfaults), all kinds of emitted signals, non-zero program return code and program hanging.
It may be considered as a tool for out-of-the-box low-cost sanity checking (fuzzing) of the library API or as a test development framework for initial generation of templates for advanced tests. Also it supports universal T2C format of tests, random test generation mode, specialized data types and other useful features.
The tool is developed by Andrey Ponomarenko:
https://abi-laboratory.pro/
Downloads
The latest release can be downloaded from this page.
Read-only access to the latest development version:
git clone git://github.com/lvc/api-sanity-checker
License
This program is free software. You may use, redistribute and/or modify it under the terms of the GNU GPL or GNU LGPL
Supported Platforms
GNU/Linux, FreeBSD, Mac OS X.
Dependencies
On Mac OS X it also requires Xcode for g++, c++filt, nm and otool.
Installation
The tool is ready-to-use after extracting the archive.
You can also use a Makefile to install the tool into the system:
sudo make install prefix=PREFIX [/usr, /usr/local]
This command will install the api-sanity-checker program into the PREFIX/bin system directory and private modules into the PREFIX/share.
Usage
For generating, building and running tests you should provide the
XML descriptor for your library version. It is a simple XML-file that specifies version number, paths to header files and shared libraries and optionally some other information. An example of the descriptor is the following (0.3.4.xml):
<version>
0.3.4
</version>
<headers>
/usr/local/libssh/0.3.4/include/
</headers>
<libs>
/usr/local/libssh/0.3.4/lib/
</libs>
Command to generate a test suite:
api-sanity-checker -lib NAME -d VER.xml -gen
You can view generated tests using the index file:
tests/NAME/VER/view_tests.html
or manually in the directory:
tests/NAME/VER/groups/
Command to build tests:
api-sanity-checker -l NAME -d VER.xml -build
Command to execute tests:
api-sanity-checker -l NAME -d VER.xml -run
The test report will be generated to:
test_results/NAME/VER/test_results.html
Examples
| Library | Version | Number of Tests | Problems Found |
|---|
| FreeType2 |
2.3.11 |
178 |
13 |
| Glibc |
2.13 |
1996 |
340 |
| libX11 |
1.3.4 |
778 |
286 |
Detectable Problems
-
Crash (segfault, signal SEGV)
-
Abort (signal ABRT)
-
All emitted signals: FPE, BUS, ILL and others
-
Non-zero exit code
-
Program hanging
-
Requirement failure (if specified)
Specialized Types
The specialized types are used to improve quality of generated tests. You can find more info
here.
Tutorial
See the article at
glibc wiki.
How the Tool Works
The basic idea of the test data generation algorithm is to recursively initialize parameters of a function using the values returned (or returned through the out-parameter) by other functions for structured data types (class, struct, union) or by some simple values for intrinsic data types (int, float, enum, ...). The recursion step includes the heuristic selection of the appropriate function, that should be called to initialize complex parameters for other functions. If some parameter of a function cannot be initialized, then the algorithm tries to select other function.
Let's see the example test for "FT_Activate_Size( FT_Size size )" function from the FreeType2 library:
#include <freetype/freetype.h>
int main(int argc, char *argv[])
{
FT_Library alibrary = 0;
FT_Init_FreeType(&alibrary); //initialize "alibrary"
FT_Face face = 0;
FT_New_Face(
alibrary,
"sample.ttf",
0,
&face); //initialize "face"
FT_Size size = 0;
FT_New_Size(face, &size); //initialize "size"
FT_Activate_Size(size); //target call
return 0;
}
In this test case the parameter "size" of target function FT_Activate_Size is initialized through the call of FT_New_Size function using its 2nd out-parameter. The first parameter "face" of FT_New_Size function is recursively initialized by the use of FT_New_Face function's 4th out-parameter. And finally the first parameter "alibrary" of FT_New_Face is initialized by the call of FT_Init_FreeType function on the 3rd recursion step. Other parameters of FT_New_Face function are initialized by intrinsic values.
Bugs
Please post your bug reports, feature requests and questions to the
issue tracker.
Maintainers
The tool is developed by
Andrey Ponomarenko.
Changes
You can find changelog
here.
Articles
