Welcome to the home page of Upreckon, a smart and easy-to-use automated program tester. Upreckon was designed specifically for testing solutions to olympic problems, such as those of the International Olympiad in Informatics, but it can be used to automatically test anything.
Warning: these licensing terms for Upreckon 2 are preliminary and may change at any moment. (They do apply to test.py 1.x though.) Upreckon was and is being developed by Oleg Oshmyan, also known as Chortos‑2. Upreckon is open-source (but not open-source as per the definition of the OSI) but not free software: in particular, it is forbidden to make changes to it and redistribute modified versions even if the original author is credited unless he gives a written permission to do so. (The author is fed up with countless forks in the free software world which make choices impossible to make and wants to avoid this with Upreckon. If you think you can improve Upreckon, please by all means send a patch to the author, and do ask for contribution rights if you think you can contribute over a longer period of time.) You are allowed to make changes in your local copy of Upreckon as long as you do not redistribute it. Except for this relaxation, Upreckon is technically licensed under the Creative Commons Attribution-NoDerivs 3.0 Unported Licence; if you are interested, do check the non-Legalese summary of the licence at the CC website. Your suggestions for a more appropriate licence are welcome.
Grab it! After extracting the tarball, run python setup.py install. The file to run will be installed as upreckon on POSIX systems and as <your Python installation directory>\Scripts\upreckon.cmd on Windows (the latter can be run from the command line without the .cmd extension).
If your Python is older than 3.3 and and you want Upreckon to read bzip2-compressed files in ZIP archives, you need a modified zipfile.py: pick the one corresponding to your version of Python and save it as zipfile.py in the same directory where test.py or upreckon resides. As an exception, Upreckon 2.00 (but not 2.01+) and pre-built Windows binaries of later versions of Upreckon include the modified zipfile.py, so you already have bzip2-in-ZIP support if you have these versions.
The latest release version of Upreckon is 2.04.1.
Alternatively, you can use the latest version from the Mercurial repository by running (cd upreckon && ./publish.sh) in a UNIX shell before python setup.py install.
Please do not link directly to the files provided on this page. While it is not forbidden to do so, in most cases links to this page are much more useful than direct links to the files. Besides, there are plenty of events that may happen and force existing direct links to stop working or to lead to incomplete or non-working versions of the tools they were supposed to lead to. For example, direct links to test.py 1.x have stopped working since Upreckon 2 was released.
In case you want to link directly to a section of this page, a fragment identifier has been assigned to each section. You can look up the identifiers in the source of the page. (In most cases, the source can be viewed by right-clicking on an empty place on the page and choosing View Source. When you get there, search for id=.)
The documentation on this page is incomplete. The information that is present should be accurate, although this is not guaranteed any more now that the page focuses on Upreckon 2.
If you use some API and want it frozen for the sake of backwards/forwards compatibility, do tell me! Otherwise I just assume no-one except myself uses any and change them in incompatible ways as development goes on.
Before Upreckon 2.00, Upreckon was known as test.py; the last generally accessible version of test.py had the version number 1.20.3. You might wonder why I was using such a large minor version number while keeping the major version number so low. This is because I believe the major version number should be incremented on major changes, such as architecture re-designs and programming language switches. For example, Upreckon 2.00 was an almost complete re-write and had a design totally different from that of test.py 1.x.
Development of Upreckon is done in Mercurial (hg). If you stumble upon a computer with an hg snapshot installed as the main copy of Upreckon, it will report its version as the next expected version number with the changeset hash in parentheses, like this: 2.00.0 (hg 2b459f9743b4). Public releases do not report changeset hashes.
This list is (obviously) not yet complete, so please also take a look at the output of upreckon --help.
The configuration file is named testconf.py and must be a valid Python source code file. In all configuration files, line endings may be any of CR, LF and CRLF, and a single file may use multiple line ending styles.
There are two possible configuration types: single-problem and multi-problem. The difference in the meaning should be obvious from the names.
Several configuration variables represent file name patterns. The following characters have special meaning inside such patterns:
Every test case is composed of an input file and either an output file or an output validation program. For a single problem, either all test cases have output files or all test cases have a single (common) output validation program.
In all test case output files, line endings may be any of CR, LF and CRLF, and a single file may use multiple line ending styles. In test case input files, all line endings should be LF for maximum portability, as Upreckon performs no line ending conversion on them.
By default, the assessed program’s output is compared line-by-line to the reference output, ignoring line endings but not ignoring any other whitespace. If any difference is found, including the absence of a trailing line break in the program’s output when there is one in the reference output, a wrong answer is scored. If the number of lines is the same and every line compares equal, a correct answer is scored.
In many problems, there is no single correct output, so this approach is not suitable. For every such problem, a separate program or script has to be written that checks whether the assessed program’s output is correct. Such programs and scripts are called output validators in this documentation. An output validator usually reads the original input data and the assessed program’s output from files, and often also the reference output. The verdict is reported through the validator’s exit code: if it is zero, a correct answer is scored, otherwise the answer is deemed wrong. The output validator may print additional information for the user to its own standard output, which will be shown by Upreckon as a note in parentheses after the ‘OK’ or ‘wrong answer’ verdict. (It is preferred that such notes thus start with a lowercase letter and do not end in a full stop.)
In the following lists, a special notation is used. When you see ‘archive:’, it means that the path to the right of the colon is a path inside an archive. Look below for more information about archives. taskname denotes the name of the assessed problem; for single-problem configurations, it is equal to a single full stop. Note that in file systems, some/path/./ is the same as some/path/ (in other words, the special file . is the directory in which it is located).
For single-problem configurations, the configuration file must be located in the tests directory, in the root of an archive or in the current working directory. The possible locations are considered in the order they are listed.
For multi-problem configurations, the global configuration file must follow the rules above, while problem-specific configuration files are searched for in more places. The places and their search order is the following:
Test case input and correct output data is searched for in the following places in the following order:
The following archive types are supported:
When searching for a file in an archive, all the following archives are considered in the listed order. The search stops as soon as one is found that exists and contains the requested file.
The change log and release notes are given here in order of descending version number, so the newest version is at the top. For information on the progress of version 2, see the dedicated tracker page.
Naming and Versioning
System Requirements
Usage
Configuration
Name Default value Type Optionality Condition Meaning
problems Noneiterable of strings If None, the configuration is single-problem. Otherwise, the configuration is multi-problem and this gives the names of the problems it contains.
testee in multi-problem configurations: './' + the current problem namestring or iterable of strings The path or argument list to the executable file of the program to assess. (See the subprocess documentation for type-dependent meaning.) The first/only item must be in POSIX path format and will be converted automatically if needed.
maxcputime 0number The CPU time limit per test case in seconds. Zero means there is no limit.
maxwalltime 0number The wall-clock time limit per test case in seconds. Zero means there is no limit.
maxmemory 0number The memory limit (exact meaning varies from platform to platform) per test case in mebibytes. Zero means there is no limit.
usegroups Falseany The truth value specifies whether test groups are used.
tests iterable The identifiers of the test cases to test against. If usegroups is true, this must be an iterable of iterables.
dummies ()iterable The identifiers of the sample test cases to test against.
padtests 0integer All test case identifiers shorter than this number of characters will be left-padded with leading zeroes to this length (if an identifier starts with + or -, zeroes will be inserted after this character instead).
paddummies 0integer Same as padtests but for sample test cases.
taskweight 100number The weighted number of points awarded for a perfect solution of the problem.
pointmap {}mapping to numbers The number of points awarded for a correct solution of every test case. The object must map test case identifiers or iterables of test case identifiers to points; in the latter case, the same amount of points is given for every identifier in the iterable (string keys are interpreted as sequences of characters). For a solution of any test case not found in this object, the value None is mapped to will be awarded. If None is not mapped to anything, 1 point will be awarded.
stdio Falseany If true, means that the assessed program uses standard input and output. If false, means that it uses file input and output.
inname with the -s option: '%.in'string optional if stdio is true The pattern of the input file name used by the assessed program. If an output validator is used and this value is set and true, the input is copied under this name before launching the validator.
outname with the -s option: '%.out'string optional if stdio is true and tester is false The pattern of the output file name used by the assessed program. If an output validator is used, the output is copied under this name before launching the validator.
ansname with the -s option: '%.ans'string optional if tester is false The pattern of the input file name used by the assessed program. If an output validator is used and this value is true, the correct output is copied under this name before launching the validator.
testcaseinname string The pattern of the test case input file names.
testcaseoutname string optional if tester is true and ansname is false The pattern of the test case correct output file names.
dummyinname ''string The pattern of the sample test case input file names. If false, it is replaced by the value of testcaseinname.
dummyoutname ''string optional if tester is true and ansname is false The pattern of the sample test case correct output file names. If it is false but testcaseoutname is not optional, it is replaced by the value of testcaseoutname.
tester ''string, iterable or false value If true, specifies the command line launching the output validator. If false, means that no output validator is used. The first/only item must be in POSIX path format and will be converted automatically if needed.
pause on POSIX platforms: 'read -s -n 1'; on Windows: 'pause'string optional with the -x option The command line launching anything that potentially outputs something to the standard output and waits for the user to press any key. It is executed in the shell at the end of the entire testing process.
File Name Patterns
Sample Configuration File
maxcputime = 1
maxwalltime = 60
tests = range(1, 12)
padtests = 2
taskweight = 100
pointmap = {}
testee = './0620-maxdist'
stdio = True
inname = 'input.txt'
outname = 'output.txt'
testcaseinname = 'input.$'
testcaseoutname = 'answer.$'
tester = 'python', 'checker.py'
ansname = 'answer.txt'Test Cases and Validation
Test and Configuration Data Location
Archive Support
Change Log and Release Notes
're'. (r246)
're'. (r222)
're', the tests and dummies configuration variables are treated as regular expressions. If usegroups is true, tests should be a pair consisting of the regular expression and the regular expression group number by which to group test case identifiers into test groups. Regular expression group numbers in the regular expressions start at two. (r193, r196, r209)
(number granted, bool correct, str comment) rather than two-tuples as before, and to return the verdict ‘wrong answer’, they must now explicitly raise upreckon.exceptions.WrongAnswer. These changes are backwards-incompatible. (r205)
stdio=False no longer crashes Upreckon with _unix compiled. (r176)
kind='outonly' in testconf. No input file names in testconf are required (or used). (r104, r145)
upreckon. (r146, r154)
upreckon.config.nativize_path function (usable within testconf) have been added. testee should be used instead of path and name; it specifies the string or iterable to be passed to subprocess.Popen except that its first/only element is a path in POSIX format and is automatically converted to the native path format at run-time (to prevent this, prefix it with slash-slash-colon). (r150, r156)
Fractional numbers of weighted points are no more rounded. (See the change log of 1.20.1.)
except: clauses have been replaced with except Exception: clauses.)
Development Progress