v. index:: single: patterns; xml; alternative for new tests .. index:: single: patterns; regtest; new
Python: Tests based on reference files¶
abstract: | One way to write tests is to create some output files and then compare them to reference files. This is often a form of regression test, where the reference is output from a previous run. This section describes a way to implement these tests in python using tools in pandokia.helpers |
---|
Basic outline¶
You write a test based on reference files in the same way as any other python test, except that your assertions are based on the result of file comparisons.
Here is a basic template that works in py.test, nose, and minipyt:
import pandokia.helpers.filecomp as filecomp
def test_foo():
# define your test name here
testname = 'test_foo'
# put this exact code here
global tda, tra
tda = { }
tra = { }
# Define the list of output/reference files
output = [
... details described below
]
# Delete all the output files before starting the test.
filecomp.delete_output_files( output )
# Run the code under test here
# CALCULATE SOME THINGS OR CALL SOME FUNCTIONS
# If the code under test isn't doing this already, write the results
# to one ore more output files (defined above) for comparison
f = open("out/file","w")
f.write("something\n")
f.close() # be sure to close the output file
# compare the output files - use this exact command
filecomp.compare_files( output, ( __file__, testname ), tda = tda, tra = tra )
Here is a detailed commentary:
def test_foo():
# define your test name here
testname = 'test_foo'
The test name is used in the call to compare_files; it is used to create a unique file name that is used by the FlagOK feature.
# put this exact code here
global tda, tra
tda = { }
tra = { }
The tda is needed to report the location of the okfile.
# definition of output/reference files
output = [
... details described below
]
You can also add your own attributes if you want. If the code under test takes input parameters that make this test special, those input parameters are a good choice.
You need a list of output files, reference files, and how to peform the comparison. This is somewhat involved, so it is described below.
# Delete all the output files before starting the test.
filecomp.delete_output_files( output )
Before starting the test, delete all the output files. If the code under test fails to produce an output file, it will be detected. (If you don’t delete the output files, your comparison may see an old file that was left behind from a previous test run.)
# Run the code under test here
# CALCULATE SOME THINGS OR CALL SOME FUNCTIONS
# If the code under test isn't doing this already, write the results
# to one ore more output files (defined above) for comparison
f = open("out/file","w")
f.write("something\n")
f.close() # be sure to close the output file
Here you exercise the code under test to produce output files. The comparison of these files to the previously existing reference files is your test.
# compare the output files - use this exact command
filecomp.compare_files( output, ( __file__, testname ), tda = tda,)
This tool compares all the files. It raises an Exception if there is an error or AssertionError if one of the files does not match.
Defining the list of output files - simple form¶
To use the simple form, you must create your output files in the directory out/, which will be created for you.
Each element of the list is a tuple of ( filename, comparator, comparator_args ).
Create the file in the directory out/, but list only the base name of the output file here.
For example, if your test creates “out/xyzzy.fits”, you can compare it to “ref/xyzzy.fits” using fitsdiff:
output = [
( 'xyzzy.fits', 'fits' ),
]
It is often useful to pass additional parameters to fitsdiff. List them in a dict in the third element of the tuple:
output = [
( 'xyzzy.fits', 'fits', { 'ignorekeys' : [ 'filename', 'date' ] } ),
]
If you follow the framework in this chapter, the out/ and ref/ directories will be created for you. You will need to create your own reference files, either by copying files into the ref/ directory or using the FlagOK feature in the GUI.
You can mix the two styles in a single list:
output = [
( 'xyzzy.fits', 'fits', { 'ignorekeys' : [ 'filename', 'date' ] } ),
( 'plugh.fits', 'fits' ),
( 'plover.fits', 'fits' ),
]
Defining the list of output files - complex form¶
The simple form requires a certain directory structure for your output and reference files. If you cannot adhere to that requirement, you can give a more detailed definition:
output = [
{
'file' : 'A.fits', # the name of the output file
'reference' : 'A_ref.fits', # the name of the reference file
'comparator': 'image', # the comparator to use
# additional args to the comparator
'args' : {
'ignorekeys': [ 'filename', 'date', 'iraf-tlm' ],
},
},
]
Available Comparators¶
binary¶
This comparator checks that the files contain identical byte streams. It takes no additional args.
output = [
( 'xyzzy', 'binary' ),
]
diff¶
This comparator uses difflib to make a unified diff of two text files. This comparator reads both entire files into memory.
output = [
( 'xyzzy.txt', 'diff' ),
]
There is one optional parameter:
rstrip (True/False)
removes trailing white space from each line before the comparison
rstrip is useful if you might use json.dump() or pprint.pprint() to write out a more complex python data structure to your file. In some cases, json will write trailing spaces that are not significant.
output = [
( 'xyzzy.txt', 'diff', { 'rstrip' : True } ),
]
fits¶
This runs fitsdiff to compare the files.
output = [
( 'xyzzy.fits', 'fits', { 'maxdiff' : 1e-5 } ),
]
Additional arguments are :
maxdiff (float)
This is the fitsdiff maxdiff number specified by
fitsdiff -d
ignorekeys (list)
This is a list of header keywords that are ignored. They are passed to
fitsdiff -k
.ignorecomm (list)
This is a list of header keywords whose comments are ignored. They are passed to
fitsdiff -c
.
These additional arguments are the same as used in the stsci XML regtest system, but the lists are specified as python lists like [ ‘a’, ‘b’ ] instead of as a single string like ‘a,b’
text¶
This is the text comparison from the stsci XML regtest system. It does not make especially interesting diffs, but is has facilities to ignore various patterns in the text.
output = [
( 'xyzzy.txt', 'text', { 'ignore_wstart' : [ 'plugh', 'plover' ] } ),
]
Additional arguments are :
ignore_wstart (list)
words that start with this text are ignored
ignore_wend (list)
words that start with this text are ignored
ignore_regexp (list)
this regex is ignored
ignore_date (True/False)
patterns that look like a date/time stamp are ignored; the system contains a rather elaborate regex to recognize many date formats
All this ignoring is performed by translating regular expression matches to the value ” IGNORE “.
user-defined comparators¶
You can provide your own comparison function before you call filecomp.compare_files().
filecomp.file_comparators['mycmp'] = my_function
def test_1() :
...
filecomp.compare_files( ... )
See the docstring for pandokia.helpers.filecomp.cmp_example for a definition of the interface to your comparator function.