Conformance test suite

In addition to the human-readable specification document, a suite of coverage tests are provided to clarify the specification. Currently, these tests are not considered normative, but a future version of them may be.

PFA implementations may verify their conformance or degree of conformance by attempting to reproduce the results of these tests. The tests are provided as a large JSON file.

Format of the file

Here is the beginning of the file, showing the top-level structure and an example of the “+” function.

{"pfa-version": "0.8.1",
 "pfa-tests": [
     {"function": "+",
               {"type": "record",
                "name": "Input",
                "fields": [
                    {"name": "x", "type": "int"},
                    {"name": "y", "type": "int"}
               {"+": [
      "trials": [
          {"sample": {"x": 2147483640, "y": 10}, "error": 18000},
          {"sample": {"x": 0, "y": 0}, "result": 0},
          {"sample": {"x": 0, "y": 1}, "result": 1},
          {"sample": {"x": 0, "y": -1}, "result": -1},
          {"sample": {"x": 0, "y": 2}, "result": 2},
          {"sample": {"x": 0, "y": -2}, "result": -2},

The indentation aids human-readability and parsing: you can rely on each function beginning with a newline and five spaces.

More formally, the file has the following structure (all-caps text and ellipsis are entities, all other characters are literal):

{"pfa-version": VERSION, "pfa-tests": [TESTS...]}

where VERSION is the PFA version number and each TEST is

{"function": NAME, "engine": PFA, "trials": [TRIALS...]}

with NAME being the name of the function under study, PFA is a complete PFA engine that calls that function, and TRIALS are sample inputs and expected outputs. The format of a TRIAL that should successfully produce a result is:

{"sample": {INPUT-VARIABLES...}, "result": RESULT}

and the format of a TRIAL that should fail with an error is:

{"sample": {INPUT-VARIABLES...}, "error": CODE}

INPUT-VARIABLES are key-value pairs, setting each field in the input record, RESULT is a JSON representation of the result, and CODE is the unique numeric code of the exception.

Format of values

Input values and output values are represented in a way that is similar to Avro’s JSON encoding, with numbers representing numbers, strings representing strings or enumerations, JSON objects representing maps or records, and JSON arrays representing arrays.

Tagged unions follow the Java-Avro convention:


where TAG is the name of the type and VALUE is the value. This form is used for all types except null, which is represented as a simple JSON null.

Infinity and not-a-number, which are legal values for types float and double, cannot be represented in JSON without quotes. We therefore use the following convention:

Non-unicode characters cannot be represented in JSON, so the bytes and fixed types are represented by a Base-64 encoding.

Schemas are required to interpret the input and output, and these schemas can be found in the PFA itself (the input and output top-level fields).

Helper functions

A Python file named runTest.py is provided to interpret the conformance file and provide values to an implementation in Python. It also contains functions to check the output type and deeply compare output values.

Implementation-specific scripts are provided in the same directory, which show how it can be used.