Test harness documentation

# Testharness documentation

## Overview

The test harness provides a system for scripting regression tests for Java classes and their methods. It consists of two parts:

- RunTest.java contains all the code needed to carry out unit testing on Java classes and methods as well as the main() method. It should not need modification because all the code that's required to run every test is part of it. Any auxiliary code needed to transform the standard input parameters, which are booleans, doubles, ints, chars or strings, into the data types needed by code being tested and into the types handled by the output display methods should be included in the adapter code derived from Custom.java. The output display methods are setError(), setNote(), setResult() and setResultWithHex(). setResult() is polymorphic and will display any boolean, character, double, integer or String value. setResultWithHex() handles strings containing unprintable characters which are output in hexadecimal as <hh>.
  The runTest() method reads test scripts, parses them and then calls the callAdapter() method in Custom.java. This contains custom code that carries out the test by calling the method being tested and returning test results to runTest() for output.
- Custom.java is a Java source module that contains two methods:
  - callAdapter(), which is customised to call each method being tested and send test results back to RunTest for output to the test results file.
  - showHelp() is called to display help text to the user if the compiled test harness is run with the '-?' command line option. It should name the shell script used to run individual tests and/or regression tests and show the name(s) of the test scripts containing test cases for the methods under test.
- Custom subclasss are written and compiled to instantiate the class(es) being tested and to call the methods being tested. It is dynamically loaded and linked by RunTest and, in turn loads the class(s) to be tested. The resulting executable Java program then reads test scripts and runs the test definitions they contain. The output is a set of corresponding result files containing easily readable test results.

Unit tests are defined within test scripts. These are plain text files which contain a sequence of method test definitions interspersed with comment lines used to describe tests and their expected results. Blank lines may be added for improved readability. Each test definition is a line containing the name of the method to be executed, a list of parameter values and an optional comment at the end of the line, which is typically used to say whether this test is expected to succeed or fail and to show the expected return value, if any.

- Comments start with a '#' and are terminated by the end of the line.
- Test definitions must start with the name of the method being tested. This is followed by a list of parameter values separated from the method name and from each other by at least one whitespace character. An optional comment may follow the last parameter.
- Parameters may be unquoted, in which case they must not include whitespace. Quoted parameters may include whitespace. The quotemarks are either a single quote (') or a double quote (") but the endmark must be the same as the opening quotemark. Inside a quoted parameter the 'other' quotemark looses its property of being an endquote, so both `"quoted 'string'"` and `'another quoted "string"'`
  will be interpreted correctly, while
  `"quoted "string""` will be treated as two parameters: 'quoted ' and 'string""'.

Any number of parameters may be supplied. If the constructor or method being tested requires more parameters than are given on the test line, a non-fatal error is reported.

This is a short example script:

    ----------------------------------------------------------------
    # re01.txt - clean re_error() test.
    # ======== - error with simple text description
    #
    reporterror re01                # OK
    re_error "error text 1"         # "re01: error text 1" expected

    #
    # No errors expected
    #
    ----------------------------------------------------------------

The first test instantiates runs the ReportError() constructor with the string "re01" as its only parameter. The second test runs the re_error() method with "error text 1" as its only parameter.

The output from running a test script is intended to be an easily readable file, with outputs from each test, including error reports, immediately following the line that defined the test.

Regression testing is easily managed by a shell script that can:

- Run individual test scripts with test output sent to stdout. Its a good idea to keep these in a separate directory and give the set of scripts similar names, e.g. data/testname99.txt
- Run one or more test scripts and save the output under a related name, e.g data/testname99.result
- Run a regression test: this runs the complete set of tests, preceeding each test by displaying the test script's name. It compares the output of each test script with the stored result file, e.g. by using the diff utility program, and writes differences to stdout. If regression tests pass, all you see are the test script names while, if any tests produce unexpected results, the differences are in the output after the name of the script that contained the failing test.

## Test script processing

Every line in a test script is written to an output file via stdout, preceded by a line number. Comment lines may appear anywhere and have a hash, #, as the first character. Comments and blank lines are copied to the output but otherwise ignored.

Each non-comment line defines a method call. This takes the form:

    method param1 param2 .... # comment documenting expected results

The method name must start at the first character in the line and parameters are separated by one or more whitespace characters. There may be zero or more parameters on each line. There is no limit on the number of parameters for each test. Quoted parameters may contain whitespace. Unquoted parameters must not contain whitespace.

If a parameter contains whitespace it must be 'quoted' by enclosing it in single or double quotes ( 'param text' or "param text" ). Quoted parameters must be terminated with the same single or double quote mark that they started with, but may contain the other type of quote, so `"String val '%s'"` and `'Char val "%c"'` will both be parsed correctly, but `"stringvalue'` will include all following parameters until either another double quote or the end of the parameter list is found.

The parameter list may be followed by a comment, starting with a hash, which is a good way of documenting the expected results from a method call.

When the text has been parsed into method name plus a list of parameter strings, a parallel list of integer numeric values is set up. Each parameter is first tested to see if it is a boolean value. If its first character is T,t or 1 its numeric value is set to 1 (TRUE) and if its first character is F,f or 0 its numeric value to 0 (FALSE).

If neither comparison matches, the parameter is used as input to Integer.parseInt() and the result is used as its numeric value.

The callAdapter() method, which will be described later, is now called to execute the required method test. It is passed the method name and both string and integer versions of the parameter list, runs the test and returns the test results by calling one or more of the following methods, which each write an output line to stdout immediately after the script line that defined the test:

Function called | Output
void setboolreply(int n); | Returned: true|false
void setcharreply(char c); | Returned: 'c'
void setdoublereply(double d); | Returned: n.nnnnnn
void seterror(char *e); | Error: error text
void setfloatreply(float f); | Returned: n.nnnnnn
void setintreply(int r); | Returned: nnnn
void setstringreply(char *s); | Returned: "string value"

The number and type of results shown is determined by the way in which you customise callAdapter() - see below.

The number of calls to seterror() are counted and output when the test script has been completely processed.

Note:

- callAdapter() should normally return zero: returning anything else signals that a fatal error has ocurred and that the script processing should be abandoned immediately along with any other scripts listed on the test program's command line for processing.
- If the method being tested calls exit() script processing ends at that point.

## Customisation

This process will normally only require changes to the Custom.java source file. Make a copy of Custom.java and custom.h before you start, so you still have the original files available building further test harneses

Custom.java provides the interface between the test harness methodality in RunTest.java and the C methods being tested. It contains two methods:

- show_help(), which provides user information about the tests to be run, such as script names and any external requirements, apart from the test scripts, which are needed to do the tests. It is a good idea for this text to name any shell script used to run regression tests and to give the name(s) used for the set of test scripts used to exercise the method(s).
- initialiseTest() is called before the script is read. It should be used to construct classes and initialise variables that are used by many of the tests that will be executed by calling callAdapter(). The prototype in Custom.java does nothing.
- callAdapter(), which allows individual method tests to be run by mapping the method name and arguments in the test definition to a method call, executes the method call, and writes test results to stdout by calling the setxxxxreply() methods to write results to the test output and seterror() to count test failures as well as writing the error description to the test output.
- finishTest() is called after the script has been processed. It should be used to release resources claimed by initialiseTest(). The prototype in Custom.java does nothing.

Tests are driven by code in RunTest.java, which uses command line arguments and options to set run conditions and specify which test scripts will be run. Runtest interprets each test script, using the callAdapter() method in this module to exercise the methods being tested, and writes the results of each call to stdout immediately after the script line that defined the test.

### public void link(RunTest rt)

This is called by RunTest to pass the rt reference to Custom to enable calls to the output reporting methods in the RunTest instance. If a class being tested needs access to the current debug level or to call a tracing or error reporting class, such as org.gregorie.envron.ReportError

, this can be created locally and adding a statement such as:

debug = rt.getDebugLevel();

debug is an int declared in your Custom subclass which takes positive values: zero means diagnostic tracing is off and successively higher values should select progressively more diagnostic detail.

### public void callAdapter(String fn, ArrayList p, ArrayList n)

This is a code-specific call adapter which must be modified to call the set of methods to be tested.

Entry and exit is reported if debug level is 2 or more. Custom.java is an example implementation that contains the code to run one example method, whose prototype is:

int example(String s int i) is a runnable private method whose only use is to allow the code in Custom to compile and run successfully, exercising all the output functions in RunTest. It should be removed as part of copying Custom to create a customised subclass.

The basic if..then..else chain of method calls should be preserved, but any or all if-actions can be converted into code blocks and/or may call additional privately defined methods.

The methods used to return results, report errors, etc, are defined in RunTest.java. They are setError(), setNote(), setResult() and setResultWithHex() methods.

The final item in the condition chain should not be modified. It is there to trap method names in test scripts that don't match any of the methods being tested.

When callAdapter() is called by code in RunTest.java:

debug | (normally zero) can be set to 1 to show callAdapter() entry and exit conditions by adding a single -d option to the command line. -dd shows more detail.
fn | A String that must contain the name of the class, method or pseudo-method being tested. Pseudo methods are any code that uses setResult(), setNote() etc together with calling methods of methods being tested. For instance, once the getters of a class have been tested, it may make sense to write a pseudo-method that calls all getters because this would simplify test scripts that look at corner cases, etc.
ArrayList p | is an array containing the call parameters as strings. Unused parameters have a NULL pointer, so can cause crashes.
ArrayList n | is an array containing the call parameters as int values with those that weren't numeric set to zero.

It is the customiser's responsibility to map the contents of the p and n arrays to the parameter list of each method being tested. Any additional code that may be needed to provide the correct mapping can be added as required, e.g. converting p.get(1) to a char or n.get(2) to a double.

On exit callAdapter() should return true if the overall call didn't encounter any fatal errors and false if an error was encountered that was serious enough to abandon the run part-way through the test script.

### void showHelp()

The text in showHelp() is displayed if the test harness is run with the '-?' option. It should say that this is a customised test harness, describe what methods or modules are being tested and identify the (set of) test scripts it uses, e.g. that their names are all of the form data/mytest99.txt

### Other methods

Custom also includes a number of private methods that may be useful when manipulating or converting parameters:

- private String collectArguments(ArrayList p);
- private char defaultToSpace(String s);
- private double strToDouble(String s);
- private String trimQuotes(String s);

## Compilation

This process will normally only require changes to the Custom.java source file. Make a copy of Custom.java and adapt ant's build XML file to compile your subclass.. Finger trouble aside, if you don't change any method prototypes or headers a new test harness should build and run pretty much straight away.

NOTE: compiling the test harness requires my environ library, which you can download from https://www.libelle-systems.com and it should be put in /usr/local/lib.