download / changelog / in the wild / documentation

This file contains several horrible hacks to make it easier to write tests for student code. I’ve collected together many interesting bits of cruft along the way.

  • The most useful thing here is the TEST macro, which takes two or more arguments: first, an expression to test; and following that, a constant format string and format arguments. It either prints to stderr and runs assert(3), or produces a stdout complying with the Test Anything Protocol, TAP:

     TEST (6 * 9 == 42,
         "life, the universe, everything (base 10s)");
     TEST (6 * 9 == 54,
         "life, the universe, everything (base 13s)");
    

    In assert(3) mode, on stderr:

     -- t1: life, the universe, everything (base 10s)
     Assertion failed
     [and, if that didn't happen:]
     -- t2: life, the universe, everything (base 13s)
    

    In TAP mode, on stdout:

     not ok 1 # life, the universe, everything (base 10s)
     ok 2 # life, the universe, everything (base 13s)
    
  • For testing floating-point operations, I have feq, because == doesn’t do what you expect. It’s a static inline function, and works a treat with the TEST macro.

     TEST (feq (0.0, 0.0, 1),
         "0.0 == 0.0 (± 1.0)")
    
  • I have have_symbol, a function that tests if a symbol exists in an executable; it uses dlopen(3), dlsym(3), and ugly hacks. It may not be very portable…

     const char *sym = "imageAsBMP";
     if (! have_symbol (sym))
         errx (EX_SOFTWARE, "this test needs '%s'", sym);
    
  • Various memory protection checks, wrapping ASan’s interface in a slightly more pleasant (and less ifdef-heavy) one.

    • mem_address_is_poisoned :: Addr# -> Bool
    • mem_region_is_poisoned :: (Addr#, Word#) -> Bool
    • mem_poison_region :: (Addr#, Word#) -> ()
    • mem_unpoison_region :: (Addr#, Word#) -> ()
    • mem_describe_address :: Addr# -> IO ()

    And, to use it:

     String s = newString ("");
     destroyString (s);
     TEST (mem_address_is_poisoned (s),
         "s is now gone")
    
  • TODO: write some linked-list test goop.

changelog

  • 2014-05-?? Jashank Jeremy z5017851@cse.unsw.edu.au
    • (… the distant past: “test_tap.h”, a terrible set of hacks that have evolved into the current TEST macro.)
  • 2017-09-27 Jashank Jeremy <{jashankj,z5017851}@cse.unsw.edu.au>
    • Refactor TEST macro into the two forms here: one that produces Test Anything Protocol (TAP) output, and one that assert(3)’s and dumps information on stderr.
  • 2017-10-10 Jashank Jeremy <{jashankj,z5017851}@cse.unsw.edu.au>
  • 2017-10-19 Jashank Jeremy <{jashankj,z5017851}@cse.unsw.edu.au>
    • Add have_symbol, to do symbol presence analysis in ADT implementations using dlopen and dlsym.
  • 2017-10-28 Jashank Jeremy <{jashankj,z5017851}@cse.unsw.edu.au>
    • mem_* wrappers around the #ifdef mess for sanitizers.
  • 2018-05-17 Jashank Jeremy <{jashankj,z5017851}@cse.unsw.edu.au>
    • Add diagnostic -Wno-unused-function.
    • Move header guard within -Wno-unused-macros
    • List a canonical source for test1511.h! Given how often I use it for various student-code-testing projects, it makes sense to publish it on my website.

where you’ve seen it

  • This was used in COMP2521 18x1, where we leveraged the TAP output and Perl’s prove, so our assignment dryruns could effectively be

    make test{Game,Drac,Hunter}View CC=3c &&
    prove -f ./test{Game,Drac,Hunter}View
    
  • This was used in COMP1511 17s2, where we developed it to write tests for, student submissions and sample solutions, adding hacks and quirks along the way.

documentation

TEST

  • macro TEST (status, descr):
    Test a thing.

    status: Test ‘passed’ if a true value.
    descr: Short description of test.

  • global variable _test_:
    the number of tests that have been run

  • global variable _fail_:
    the number of tests that have failed.

Floating-point equality

  • function feq (double x, double y, double eps) -> bool:
    Floating-point equality. Compares the left-hand side with the right-hand side, with a margin of error around the right.

    x: Left-hand side
    y: Right-hand side
    eps: Margin of error (taken on the right-hand side)

Symbol existence

  • function have_symbol (const char *sym) -> bool:
    Symbol existence. Does the named symbol exist?

    Uses dlopen(3) and dlsym(3). May need -ldl, maybe also -rdynamic…? dcc’s ASan dependency may give us -rdynamic, so we’d only need -ldl on silly platforms.

    sym: A symbol name to resolve in the currently-executing process.

AddressSanitizer shims

If we’re running with the Address Sanitizer, we can use its exposed interface to do memory checks. Wrap the useful bits of ASan in a slightly more pleasant and less ifdef-heavy interface.

For more details, see <sanitizer/asan_interface.h>. (llvm/projects/compiler-rt/include/sanitizer/asan_interface.h)

  • function mem_address_is_poisoned (void *addr) -> bool:
    Report if this address is “poisoned” – whether a one-byte read will cause a memory error, either by ASan tracking or not.

    addr: the address to check

  • function mem_region_is_poisoned (void *addr, size_t size) -> bool:
    Report if the region of memory from addr to addr + size is poisoned, as above.

    addr: the base address to check
    size: the size of the region to check

  • function mem_poison_region (void *addr, size_t size):
    Mark this region of memory as “poisoned”.

    addr: the base address to mark
    size: the size of the region to mark

  • function mem_unpoison_region (void *addr, size_t size):
    Mark this region of memory as not poisoned.

    addr: the base address to unmark
    size: the size of the region to unmark

  • function mem_describe_address (void *addr) -> IO ():
    Print out what’s known about the given address.

    addr: the address to report on