All tests are based upon makefiles. See the specific documentation on
makefiles for more information.

Tests are organized as a hierarchy of suite types, particular suites
within each type, and individual tests within each suite. As an example,
the preinstalled correctness tests that come as part of the test harness
are identified first as suitetype "correct", the particular correctness
suite "c_correct", and finally the individual tests themselves. This 
preinstalled c_correct suite can be found at <ROOT>/tests, where <ROOT>
is the root directory of the installed test harness package.

How is a test actually run?
===========================

For this example we will use the preinstalled "c_correct" suite, which if
you look at its entry within the database table "suite" is part of 
suitetype "correct". What is the location of the actual c_correct tests
themselves? Again, looking at the  c_correct entry within the suite database
table we see that the "srcdir" is "tests/c_correct". And if the "homeqa" 
option to runtest is initially set to <ROOT> within config.xml, the 
concatenated source directory to each of the tests within the c_correct
suite can be found at <ROOT>/tests/c_correct, which is the SRC environment
variable runtest will provide to the make command..

Let's assume we use the following runtest command, where <ROOT> is
/home/blackjack, and our path include <ROOT>/bin::

runtest -suite c_correct -test a1 -flag C02

Since we are only using the -suite, -test, and -flag  options to runtest, 
all other important options values are already set either within the 
config.xml file or as entries to appropriate database tables (see 
documentation of the runtest command for more information).

Here are the next steps that will be taken by runtest:

1) the argument line to runtest is parsed, making sure all required elements
exist.

2) parse all flags and turn flag designation into actual compiler flags

3) create any needed environment variables

4) a temporary directory will be created in the tmpdir directory /tmp (as 
specified within the config.xml file, which can be overwritten with the
-tmpdir TMPDIR option to runtest).

5) the makefile found in the SRC directory will be copied to tmpdir.

6) a list of tests within the suite c_correct, found by examining all tests
within the suitetests database table that have column entries of
suitname equal to c_correct, is generated. This list is then adjusted based
upon the -test and/or -rmtest options provided to runtest. In this example
we will just end up with the test a1 in our list.

7) the make command is generated, then invoked within tmpdir with
initialized make variables on each of the tests within the test list
for more information on make variables see the makefile_info file).

The output from our sample command above is just:

# c_correct: a1
a1 Result: success

However, we can see the full make command and test invocation if we also 
use the -echo and -printmake options to runtest:

runtest -suite c_correct -test a1 -flag C02 -echo -printmake

# c_correct: a1
##### Build Phase #####
make a1.run CC=gcc MODE=32 LD=gcc OPT=-O2 EXE=out EXESUFFIX=out SRC=/home/blackjack/tests/c_correct TEST=a1 

gcc /home/blackjack/tests/c_correct/a1.c -c -O2  -o a1.o
gcc /home/blackjack/tests/c_correct/test.c -c -O2  -o test.o
gcc a1.o test.o -o a1.run
./a1.run
----   6 tests completed. 6 tests PASSED. 0 tests failed.
rm a1.o
a1 Result: success

Actually, the make commands will iterate over each test in as many as four
phases:

  - build
  - link
  - run
  - verify

Note that these phases are listed in the suitetests database table  with
column entries of buildname, linkname, runname, and verifyname. At least
one of these phases must exist or runtest will issue an error. The makefile
will be invoked on a test for each of the phases that exist. For example,
with the provided c_correct suite, each test listed in suitetests has a
buildname entry of <test_name>.run, and no other phase entries. That is
because the makefile is set up to fully build and run each test with that
<test_name>.run command, as we see above. However, the test harness supports
more complicated control with makefiles (in particular if each test has its
own entry within the makefile or each test has its own makefile) as opposed
to the included c_correct suite which relies on make pattern rules.

How is a test verified as pass or fail?
=======================================

Each line of output is compared with each description column of all the
entries within the database status table. If there is a match then runtest
will look at the entries within the error_codes table which correspond to
the status entry status_id (if no match is found, then a new entry will
automatically be added to the error_codes table). Test output will stop if
the status table entry has the fatal column entry set, otherwise scanning
of output and comparison to status entries will continue (this may be
important if a test has several points at which output is generated - you
may not want to stop after a single pass has been matched).

At least the default pass and fail errors must exist as entries in the
error_codes database table. These table entries' id columns are recorded
within the config.xml defaults section as statusId and errorStatusId,
respectively.

Note: the included suite c_correct contains one test, a0, which
intentionally has failing cases. This test would report a failure status
though the test harness, except that in the prepopulated database we
set the suitetests table defaultstatus column for a0 to 50030, which
corresponds to an error_codes entry with status_id, i.e it maps to a
status table entry that matches the output "FAIL". Thus the test harness
returns a "success" result for the a0 test. Using this method, controlled
failures can be caught and the expected behaviour can still be considered
normal or as a success.

Running a performance test
==========================

A small benchmark, matrix multiply, is included with this distribution. See
the directory <ROOT>/tests/f_perf. This small performance test is written 
in fortran, with the suite being called f_perf, and the test being matmul.
The compiler tool set up for this suite is gfortran, so make sure your 
system has gfortran installed before running this test by default (you 
could also install another fortran compiler, add it to the tools table, then 
invoke the runtest command with -compiler <fortran_compiler> or change
the compilerid column in the suite database for the row f_perf).

For this example we also use the simulator "time" for additional performance
data. Here is our command:

runtest -suite f_perf -echo -sim time

we see this output:

#  
# Starting Runtest @ Fri May 20 16:52:28 2011 
# Host: 
# Suite: f_perf
# Load average: 0.0 0.01 0.0
# 


# f_perf: matmul
gfortran   /home/stoltz/PGI/PYTHON/blackjack/tests/f_perf/matmul.f90 -o matmul.out
------------ executing test matmul
time matmul.out
Test PASSED
Total time for matmul instrinsic:    1.0760670000000001     
Total time for jki loop:    6.6524160000000006     
MFlops for matmul instrinsic:    1858.6203275446601     
7.81user 0.01system 0:07.83elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+8071minor)pagefaults 0swaps
matmul Result: success

Running a full application
==========================
The full C application of milc is part of the test harness package; this
application demonstrates how the test harness can be configured, run, and
timed with a larger benchmark. See the file application_info for more
information.

How to view the results with a browser
======================================

See the documentation file web_info.

How do you add a test to the test harness?
==========================================

If you are adding a test to an existing suite;
    - put the new test in the suite source directory
    - make sure the suite makefile knows how to process the test
    - add a database entry to the suitetests table, filling in the
      necessary columns (usually by following the pattern of tests
      already within the suite). Typically, the minimum entries are
      the testname, suitename, at least one phase entry, and setting
      the enable column (this column can be set to 0 to remove a
      test from the list of possible tests to be run within the suite.)

If you are adding a new suite:
    - add the suite to the suite database table
    - minimum entries in the suite table typically include:
      o suite name
      o display name (used for web viewing of results)
      o suitetype (matches a suitetype database entry)
      o default *flagid entries must be set for all the flagid columns
        (usually these are *00 values)
      o a compilerid which will match a compiler in the tools table
      o a srcdir entry which provides the location of the suite when
        concatenated to the homeqa variable (set in config.xml or
        provided on the command line to runtest)
    - add individual tests within each suite to the suitetests table,
      as described above. The utility test_insert.py, found in 
      the <ROOT>/utils directory, may be of some use for automating
      this task.
