BerkeleyGW testsuite
David Strubbe 2009-2011

== Running tests ==

* To run the testsuite in serial, type "make check" in this directory or the main directory. "make check-save" will do the same, but retain the working directories for debugging.
* To run in parallel without a scheduler, you can just use "make check-parallel" if you do not have a scheduler (e.g. PBS). Set environment variable SAVETESTDIRS=yes to save working directories. The tests use 4 cores, so be sure that at least that many are available, or use BGW_TEST_MPI_NPROCS (see below).
* To run in parallel with a scheduler, type "make check-jobscript" or "make check-jobscript-save", which will execute the job script for the machine you are using, as specified in arch.mk by the line TESTSCRIPT. See example job scripts for various machines in this directory called *.scr. The tests use 4 cores, so be sure to request at least that many in the job script, or use BGW_TEST_MPI_NPROCS (see below).
* Environment variables:
  - TEMPDIRPATH: sets the scratch directory where temporary working directories will be created. Default is /tmp, but on some machines you may not have permission to write there.
  - MPIEXEC: sets the command for parallel runs. Default is `which mpiexec`. Note that mpiexec and mpirun are generally equivalent (except for Intel MPI, in which case mpirun is recommended). Set this if you are using a different command such as 'ibrun' (SGE parallel environment), 'runjob' (BlueGene), or 'aprun' (Cray), if you need to use a different mpiexec than the one in your path, or if some options need to be appended to the command. Depending on the value of this variable, three styles of command line are used: ibrun, runjob, and mpirun/aprun.
  - BGW_TEST_MPI_NPROCS: set to overrule the number of processors listed in the test files. Useful for testing correctness of parallelization, if you don't have 4 cores, or to run the testsuite faster with more cores.
 [- BGW_TEST_NPROCS: deprecated, same as BGW_TEST_MPI_NPROCS, provided for compatibility with versions 1.0.x.]
  - MACHINELIST: if set, will be added to command line for MPI runs. This is needed in some MPI implementations.
  - EXEC: if set, will be added before name of executable. Used to run code through valgrind.
* To run just one test, in its directory, run "../run_regression_test.pl [-s] [-p] -D ../../bin -f [testname].test". Include the -s flag to run in serial; otherwise it will be in parallel. Include the -p flag to preserve the working directory.

Contents:
(1) run_testsuite.sh -- this script runs tests from filenames ending in *.test in this directory.
(2) run_regression_test.pl -- called from run_testsuite.sh for each individual test.
(3) *.scr -- job scripts for running the testsuite on various parallel machines.
(4) interactive*.sh -- scripts to run in parallel, in interactive mode. You have to follow the instructions in the script, not just run it.
(5) queue_monitor.pl -- a Perl script for parallel tests with a SLURM or PBS scheduler, which submits a job and monitors the queue to see what happens. Used by the BuildBot.
(6) test directories.
(7) fix_testsuite.py -- (for developers) can be used to create or update match reference values in test files.
(8) no_hires.sh -- Some clusters will not have the Perl package Time::HiRes installed which is needed for timing in run_regression_test.pl. You can just deactivate the usage with this script.
(9) buildbot_query.pl -- (for developers) can be used to find the range of values obtained on the buildslaves for a particular match, to set values in the test files.

== Writing tests ==

The test files consist of lines beginning with one of a set of tags, parsed by the Perl script. Comment lines beginning with '#' will be ignored.
Test : title
  Write a title to output to identify the test. Should be the first tag in the file.
Banner : text
  Write a heading in a box to identify a specific part of a test. Helpful when input files are generated by sed and do not have a distinct name.
Enabled : Yes/No
  If Yes, will be run; if No, will not be run. Use to turn off a test without deleting it from the repository. Should be the second tag in the file.
TestGroups : group-name [group-name2 [...]]
  The run_testsuite.sh script can be run with argument "-g" and a list of groups. Then tests will only be run if there is a match between the argument list and the list in TestGroups. Examples of groups that could be used: parallel, serial, long. [This tag is actually read by run_testsuite.sh rather than run_regression_test.pl.]
Executable : program.x
  Name of executable to be run (path is bin directory). Persists for multiple runs, until next Executable tag.
Processors : integer
  Number of processors to use. Ignored if mpiexec is not available. May not exceed the number in the reservation in the job scripts.
  Set to special value "serial" to run without mpiexec.
Command : shell-command
  Specify a shell command, which will be executed in the working directory. Useful for renaming log files that would be overwritten by a subsequent run, or preprocessing input files by sed or other utilities (after putting in working directory with Copy).
Copy : file.in [file_newname.in]
  Copy a file from test directory to run directory. If new name is not supplied, original name will be used. Useful if you want to sed a file before it is run.
Unpack : data.tar.gz
  The file data.tar.gz in the test directory will have "tar xzf" run on it, with resulting files going to the working directory.
Output : file.out
  Output will be piped into the file named here, in the working directory.
Arguments : arg1 arg2
  The current Executable will be executed (in serial) with the text given here as command-line argument(s). No files will be copied to working directory. Use redirection with ">" to capture the output if necessary.
Input : file.in [file_newname.in/PIPE/CMDLINE/NONE/NOCOPY]
  The file named here will be copied from the test directory to the working directory, for use as input (unless NOCOPY is specified). If it is followed by "PIPE", the file will be piped into the executable. If it is followed by "CMDLINE", the file will be given as a command-line argument to the executable. If it is followed by "NONE", no file will be copied (useful if an input file was already generated by sed in the working directory). If it is followed by another name, the file's name in the working directory will be the new name. This tag causes actual execution of the run that has been set up by previous tags.
Precision : 1e-5
  A floating point number, the tolerance for testing whether a match has passed or failed. Persists until next Precision tag. Default is 1e-4.
match ; name ; COMMAND(..); reference-value
  Extracts a calculated number from a run and tests it against the reference value. The name is an identifier printed to output. The number is extracted as the standard output from the listed COMMAND, which is one of this set:
  . GREP(filename, 'search-text', field, offset)
      Finds the first line in file containing 'search-text' (which MAY NOT contain a comma), and returns the specified field of that line. "Field" is meant in the sense used by 'awk', i.e. the line is parsed into white-space separated groups, indexed starting with 1. The optional 'offset' directs the use of the Mth line after the line containing 'search-text'.  No offset is equivalent to offset = 0. This is the most robust of the commands and should be used when possible in preference to the others.
  . SIZE(filename)
      Returns the size of the specified file via 'ls -lt'. Useful for binary files whose contents cannot easily be checked.
  . LINE(filename, line, field)
      Returns the specified field of the specified line number from the file. Use GREP instead if possible.
  . SHELL(shell-command)
      The result is the standard output of the listed command. Deprecated; use GREP or LINE unless absolutely necessary.
STOP TEST
  For debugging purposes, to halt test after the part you are studying has run.
