Fork me on GitHub

API Sanity Checker

An automatic generator of basic unit tests for a shared C/C++ library

API Sanity Checker is an automatic generator of basic unit tests for a shared C/C++ library. It is able to generate reasonable (in most, but unfortunately not all, cases) input data for parameters and compose simple ("sanity" or "shallow"-quality) test cases for every function in the API through the analysis of declarations in header files.

The quality of generated tests allows to check absence of critical errors in simple use cases. The tool is able to build and execute generated tests and detect crashes (segfaults), all kinds of emitted signals, non-zero program return code and program hanging.

It may be considered as a tool for out-of-the-box low-cost sanity checking (fuzzing) of the library API or as a test development framework for initial generation of templates for advanced tests. Also it supports universal T2C format of tests, random test generation mode, specialized data types and other useful features.

The tool is developed by Andrey Ponomarenko: https://abi-laboratory.pro/

Table of Contents

Downloads

The latest release can be downloaded from this page.

Read-only access to the latest development version:

 git clone git://github.com/lvc/api-sanity-checker 

License

This program is free software. You may use, redistribute and/or modify it under the terms of the GNU GPL or GNU LGPL

Supported Platforms

GNU/Linux, FreeBSD, Mac OS X.

Dependencies

On Mac OS X it also requires Xcode for g++, c++filt, nm and otool.

Installation

The tool is ready-to-use after extracting the archive.

You can also use a Makefile to install the tool into the system:

 sudo make install prefix=PREFIX [/usr, /usr/local] 

This command will install the api-sanity-checker program into the PREFIX/bin system directory and private modules into the PREFIX/share.

Usage

For generating, building and running tests you should provide the XML descriptor for your library version. It is a simple XML-file that specifies version number, paths to header files and shared libraries and optionally some other information. An example of the descriptor is the following (0.3.4.xml):

<version>
    0.3.4
</version>

<headers>
    /usr/local/libssh/0.3.4/include/
</headers>

<libs>
    /usr/local/libssh/0.3.4/lib/
</libs>

Command to generate a test suite:

api-sanity-checker -lib NAME -d VER.xml -gen

You can view generated tests using the index file:

tests/NAME/VER/view_tests.html

or manually in the directory:

tests/NAME/VER/groups/

Command to build tests:

api-sanity-checker -l NAME -d VER.xml -build

Command to execute tests:

api-sanity-checker -l NAME -d VER.xml -run

The test report will be generated to:

test_results/NAME/VER/test_results.html

Examples

LibraryVersionNumber of TestsProblems Found
FreeType2 2.3.11 178 13
Glibc 2.13 1996 340
libX11 1.3.4 778 286

Detectable Problems

Specialized Types

The specialized types are used to improve quality of generated tests. You can find more info here.

Tutorial

See the article at glibc wiki.

How the Tool Works

The basic idea of the test data generation algorithm is to recursively initialize parameters of a function using the values returned (or returned through the out-parameter) by other functions for structured data types (class, struct, union) or by some simple values for intrinsic data types (int, float, enum, ...). The recursion step includes the heuristic selection of the appropriate function, that should be called to initialize complex parameters for other functions. If some parameter of a function cannot be initialized, then the algorithm tries to select other function.

Let's see the example test for "FT_Activate_Size( FT_Size size )" function from the FreeType2 library:

#include <freetype/freetype.h>
int main(int argc, char *argv[])
{
    FT_Library alibrary = 0;
    FT_Init_FreeType(&alibrary); //initialize "alibrary"
    
    FT_Face face = 0;
    FT_New_Face(
        alibrary,
        "sample.ttf",
        0,
        &face); //initialize "face"
    
    FT_Size size = 0;
    FT_New_Size(face, &size); //initialize "size"
    
    FT_Activate_Size(size); //target call
    return 0;
}

In this test case the parameter "size" of target function FT_Activate_Size is initialized through the call of FT_New_Size function using its 2nd out-parameter. The first parameter "face" of FT_New_Size function is recursively initialized by the use of FT_New_Face function's 4th out-parameter. And finally the first parameter "alibrary" of FT_New_Face is initialized by the call of FT_Init_FreeType function on the 3rd recursion step. Other parameters of FT_New_Face function are initialized by intrinsic values.

Bugs

Please post your bug reports, feature requests and questions to the issue tracker.

Maintainers

The tool is developed by Andrey Ponomarenko.

Changes

You can find changelog here.

Articles