Scrutiny: Scrutiny

Scrutiny is a C testing framework for POSIX environments.

All of the functionality can be accessed by

Every test must be part of a group. You can create a group by

scrGroup group;
group = scrGroupCreate(NULL);

After that, you can define a test function:

void my_test(void) {
    ...
}

and add it to a group:

scrAddTest(group, "Test name", my_test, NULL);

Once you have added all of the tests, you can run them by

scrRun(NULL, NULL);

This function returns 0 if all of the tests pass (or are skipped) and 1 otherwise. The function also summarizes the results in stdout.

You can pass a scrStats* to scrRun:

scrStats stats;
scrRun(NULL, &stats);

where scrStats is defined as

typedef struct scrStats {
    unsigned int num_passed;
    unsigned int num_skipped;
    unsigned int num_failed;
    unsigned int num_errored;
} scrStats;

Various macros are provided in order to test various conditions. For example, for integers,

void integer_test(void) {
    int x = 5, y = 5, z = 6;
    scrAssertEq(x, y); // equal
    scrAssertNe(x, z); // not equal
    scrAssertLe(x, y); // less than or equal to
    scrAssertLt(x, z); // less than
    scrAssertGe(z, x); // greater than or equal to
    scrAssertGt(z, x); // greater than
}

All integer values are upgraded to intmax_t. If you need to use uintmax_t, use the unsigned macros like scrAssertUnsignedLt. For floating-point values, use macros like scrAssertFloatLt.

Though char variables are also integer variables, you should use the scrAssertCharEq and scrAssertCharNe macros to compare them.

You can compare pointers by

void pointer_test(void) {
    void *p = NULL;
    int x;
    scrAssertPtrEq(p, NULL);
    scrAssertPtrNe(&x, p);
}

You can compare strings (i.e., char arrays) by

void string_test(void) {
    size_t idx;
    const char *word = "hello";
    scrAssertStrEq(word, "hello");
    scrAssertStrNe(word, "goodbye");
    scrAssertStrBeginsWith(word, "hel");
    scrAssertStrNBeginsWith(word, "hellp");
    idx = scrAssertStrContains(word, "ll");
    scrAssertEq(idx, 2);
    scrAssertStrNContains(word, "elp");
}

As you can see, scrAssertStrContains is a special assertion in that, if it succeeds, it returns the index where the substring starts.

Please note that you cannot use the string macros with NULL pointers.

You can test that two memory regions are equal (essentially, running memcmp) by

void buffers_equal(void) {
    scrAssertMemEq("help", "hello", 3);
}

You can skip a test by

void skip_me(void) {
    scrTestSkip();
}

In addition, you can make general assertions by

void my_test(void) {
    int x = 5, y = 5;
    scrAssert(x + y == 10);
}

You can fail a test without any assertion by

void gonna_fail(void) {
    scrFail("Failing this test for reasons");
}

You can emit logging statements by

void my_test(void) {
    int x = 5;
    scrLog("x is %i", x);
}

Note that such statements will only be displayed if the test fails.

The signature of scrAddTest is

void

scrAddTest(scrGroup group, const char *name, scrTestFn test_fn, const scrTestOptions *options);

where

typedef scrTestOptions {
    unsigned int timeout;
    unsigned flags;
} scrTestOptions;

If options is NULL, then default options will be used (i.e., 0 for both).

If timeout is positive, then the test will fail if not completed within that many seconds.

At the moment, the only valid value for flags other than 0 is SCR_TF_XFAIL. If this value is passed, then success/failure will be inverted. That is, the test will be expected to fail and a failure will be counted if the test passes.

For each group of tests, there is a group context which is a void*. It is accessible from the tests via

void my_test(void) {
    void *ctx = scrGroupCtx();
}

The signature of scrRun is

int

scrRun(const scrOptions *options, scrStats *stats);

where

typedef struct scrOptions {
    void *global_ctx;
    unsigned int flags;
} scrOptions;

If the options argument is NULL, then default values will be used (i.e., NULL and 0).

By default, each group context is equal to the global context. However, you can pass function pointers to scrGroupCreate which can set up and tear down a group context. The signature of scrGroupCreate is

scrGroup

scrGroupCreate(const scrGroupOptions *options);

where

typedef struct scrGroupOptions {
    void *(*create_fn)(void*);
    void (*cleanup_fn)(void*);
} scrGroupOptions;

If the options argument is NULL, then default values will be used (i.e., NULL for both).

If specified, then create_fn will be called with the global context as the argument. The pointer returned will be the group context.

If specified, then cleanup_fn will be called with the group context (or the global context if create_fn was unspecified).

You can use the test macros in create_fn. If any of the assertions fail, then all of the tests in that group will be counted as having failed. You can also call scrTestSkip() which will skip all of the group's tests.

The flags field in scrOptions is some bitwise-or combination of any or none of the following:

SCR_RF_FAIL_FAST: Stop running tests as soon as any test either fails or encounters an error.
SCR_RF_VERBOSE: Show logging messages as well as stdout/stderr even when tests pass or are skipped.

When building for Linux, you can add, at compile time, the ability to monkeypatch functions. To enable monkeypatching, add monkeypatch=yes to your make invocation.

Suppose, for example, you wanted malloc to always return NULL during testing. You could create the fake function

void *
fake_malloc(size_t size)
{
    (void)size;
    return NULL;
}

and then patch malloc with

bool

scrPatchFunction(scrGroup group, const char *func_name, const char *file_substring, void *new_func);

Here, new_func would be a function pointer to fake_malloc. E.g.,

if ( !scrPatchFunction(group, "malloc", NULL, fake_malloc) ) {
    // handle the error
}

This test would then pass:

void
malloc_fail(void)
{
    scrAssertPtrEq(malloc(1), NULL);
}

When you attempt to patch a function, Scrutiny will walk the the process' maps file in procfs and identify any ELF files (libscrutiny.so is skipped). If any of them contain a global offset table (GOT) entry for the specified function, the address of the entry will be recorded. When a process running one of the tests in the group is started, it will be ptraced and those GOT entries will be altered to point to the interposed function. If the to-be-patched function is not found in any .text section, then scrPatchFunction will return false.

If file_substring is not NULL, then only ELF files whose paths contain the value as a substring will be patched. That means that the same function can be patched in the same testing group multiple times. If the same ELF file would be patched multiple times by different calls to scrPatchFunction, then the last call would be the one that is ultimately applied.

During testing, you may acquire a pointer to the original function (e.g., the true malloc) by

void
some_test(void)
{
    void *(*true_malloc)(size_t);
    true_malloc = scrPatchedFunction("malloc");
    ...
}

scrPatchedFunction will return NULL if a patch for the function was never registered.

This feature is highly experimental and will probably not work in the presence of certain link-time optimizations.

Scrutiny has submodules so you'll need to add --recurse-submodules to your git clone invocation.

You can build and install Scrutiny by

make install

After that, you can link your test program to Scrutiny with -lscrutiny.

Scrutiny can be uninstalled by

make uninstall