ConvertBot: Runs a set of NeXtMidas test macros on an option tree

2020-04-27T22:05:11Z

Runs a set of NeXtMidas test macros on an option tree

New page

Runs a set of NeXtMidas test macros on an option tree
<pre>
<opt> Option tree name to test

Optional Test Configuration File:
test_config.tbl - Input file to add to/modify tests in test area of <opt>.

CONFIDENCE runs the any configured unit tests and test macros located in the
test area of the specified option tree. This test suite returns an overall
PASS / FAIL message for each command tested.

Each test typically exercises multiple aspects of the command it is testing.
If any of these constituent tests associated with the command fail, CONFIDENCE
returns a FAIL message to the user. ALL constituent tests associated with a
command must pass in order for CONFIDENCE to return a PASS message for that
command.

CONFIDENCE is actually a wrapper macro that runs tests for all commands in the
option tree that have a test_<command> macro in the test subdirectory. If the
file test_config.tbl exists in the test area/subdirectory, it can be used to
modify the default behavior (see Configuration Table File Definition section
below for more details). To run the individual tests, type;

nM> home test <opt>
and run:
nM> %test_<command>

If in the course of running the CONFIDENCE macro a FAIL message is returned,
and the user wishes to get PASS / FAIL information for the constituent tests
for that particular command; the macro in the nxm.<opt>.test area should be run
individually. For example, with one's home path set to the test area you can
run:
nM> %TEST_CALCULATOR
to see the individual results of the calculator command tests.

Since 2.7.1: All test failures are printed (instead of rolled up into one FAIL
message for that command so that user does not need to re-run failed test cases
to get more detailed information.

Since 3.1.0: the /baseline, /interactive, and /debug (using value of /dbg)
switches are propagated to all test macros and their sub-test macros.

Configuration Table File Definition
-----------------------------------
The configuration table file (test_config.tbl) can be converted directly
to a NeXtMidas Table and consists of a number of TAG=VALUE lines.
Where TAG=VALUE can be:
1) A quoted and comma separated list of additional test <name>'s.
TESTLIST="<name1>,...,<name2>"
where <nameN> is TEST_<nameN>.mm. This allows TEST_<name>'s to be run
when <name> does not correspond to a command in the option tree.

2) Any command in the command dictionary followed by a tag.
<cmd>=[BROKEN|NOTEST|WEB|<cmd2>]
where
BROKEN - Test is KNOWN to be broken so you want to skip it
NOTEST - Do not desire to have a test for this command
WEB - Test requires a WEBSERVER to be running
<cmd2> - NO test required for <cmd>, it is tested when test for
<cmd2> is run.

Examples:
Run only NON-INTERACTIVE tests (all interactive tests will show as passed)
nM> confidence

Run INTERACTIVE and NON-INTERACTIVE tests
nM>confidence/interactive

Run NON-INTERACTIVE tests and show any commands MISSING tests
nM>confidence/miss

Run NON-INTERACTIVE tests and stop at the ASK command
nM>confidence/stopat=ASK

SWITCHES:
/BASELINE - Run this before baseline release, enables additional
non-interactive testing and turns on /INTERACTIVE. [DEF=FALSE]
/DBG - Show detailed status line for tests in sub-macros. [DEF=FALSE]
/DEBUG - Show status line for each test. [DEF=TRUE]
/INTERACTIVE - Non-interactive and interactive tests. [DEF=/BASELINE]
/MISS - Show commands which are missing tests. [DEF=TRUE]
/RESUME=cmd - Skips test macros prior to specified command.
/STOPAT=cmd - Stops at specified command (does not run given command).
/WEB - Run tests which require HTTPD, see HELP WEBSERVERS

SEE ALSO: HOMEPATH, ASSERT, UNITTEST

</pre>

[[Category:NXM_Explain]]

CONFIDENCE - Revision history

ConvertBot: Runs a set of NeXtMidas test macros on an option tree