CONFIDENCE
Jump to navigation
Jump to search
Runs a set of NeXtMidas test macros on an option tree
<opt> Option tree name to test Optional Test Configuration File: test_config.tbl - Input file to add to/modify tests in test area of <opt>. CONFIDENCE runs the any configured unit tests and test macros located in the test area of the specified option tree. This test suite returns an overall PASS / FAIL message for each command tested. Each test typically exercises multiple aspects of the command it is testing. If any of these constituent tests associated with the command fail, CONFIDENCE returns a FAIL message to the user. ALL constituent tests associated with a command must pass in order for CONFIDENCE to return a PASS message for that command. CONFIDENCE is actually a wrapper macro that runs tests for all commands in the option tree that have a test_<command> macro in the test subdirectory. If the file test_config.tbl exists in the test area/subdirectory, it can be used to modify the default behavior (see Configuration Table File Definition section below for more details). To run the individual tests, type; nM> home test <opt> and run: nM> %test_<command> If in the course of running the CONFIDENCE macro a FAIL message is returned, and the user wishes to get PASS / FAIL information for the constituent tests for that particular command; the macro in the nxm.<opt>.test area should be run individually. For example, with one's home path set to the test area you can run: nM> %TEST_CALCULATOR to see the individual results of the calculator command tests. Since 2.7.1: All test failures are printed (instead of rolled up into one FAIL message for that command so that user does not need to re-run failed test cases to get more detailed information. Since 3.1.0: the /baseline, /interactive, and /debug (using value of /dbg) switches are propagated to all test macros and their sub-test macros. Configuration Table File Definition ----------------------------------- The configuration table file (test_config.tbl) can be converted directly to a NeXtMidas Table and consists of a number of TAG=VALUE lines. Where TAG=VALUE can be: 1) A quoted and comma separated list of additional test <name>'s. TESTLIST="<name1>,...,<name2>" where <nameN> is TEST_<nameN>.mm. This allows TEST_<name>'s to be run when <name> does not correspond to a command in the option tree. 2) Any command in the command dictionary followed by a tag. <cmd>=[BROKEN|NOTEST|WEB|<cmd2>] where BROKEN - Test is KNOWN to be broken so you want to skip it NOTEST - Do not desire to have a test for this command WEB - Test requires a WEBSERVER to be running <cmd2> - NO test required for <cmd>, it is tested when test for <cmd2> is run. Examples: Run only NON-INTERACTIVE tests (all interactive tests will show as passed) nM> confidence Run INTERACTIVE and NON-INTERACTIVE tests nM>confidence/interactive Run NON-INTERACTIVE tests and show any commands MISSING tests nM>confidence/miss Run NON-INTERACTIVE tests and stop at the ASK command nM>confidence/stopat=ASK SWITCHES: /BASELINE - Run this before baseline release, enables additional non-interactive testing and turns on /INTERACTIVE. [DEF=FALSE] /DBG - Show detailed status line for tests in sub-macros. [DEF=FALSE] /DEBUG - Show status line for each test. [DEF=TRUE] /INTERACTIVE - Non-interactive and interactive tests. [DEF=/BASELINE] /MISS - Show commands which are missing tests. [DEF=TRUE] /RESUME=cmd - Skips test macros prior to specified command. /STOPAT=cmd - Stops at specified command (does not run given command). /WEB - Run tests which require HTTPD, see HELP WEBSERVERS SEE ALSO: HOMEPATH, ASSERT, UNITTEST