Test value

Question: Does the test check only its mocks, recompute the expected value with the code’s own logic, assert internal details, or mix unrelated behaviors?

• Runs: by default, with --include-tests · Fails the check by default: no
• Right on projects JevGate was never tuned on: reviews 3 of 5, considers 1 of 2 (how it is measured)
• Right on the projects it was tuned on: reviews 5 of 11, considers 74% (20 of 27)
• Looks at: test cases, with –include-tests
• Evidence unit: one test’s source and the signatures it calls
• Acceptable: A test that checks a result or effect a caller can observe
• Names: tests/value, value, test_value · Version: 7

When a finding is right

A finding says a test checks only its mocks, computes its expected value with the logic it tests, asserts internal details instead of observable results, or mixes unrelated behaviors. It is right when the test would pass whatever the code did: it asserts the value its mock returns, or builds its expected value by calling the code under test. It is wrong when what the test reads is behavior a caller can observe: a panel’s recorded output, a framework’s documented hook, or the state the program acts on next. Nearly half of the findings labeled wrong or debatable read state that is the observable behavior, and about a fifth checked a callback that is part of the public interface.

A test said to assert internal details is asked, with the bodies of the functions it calls, what its assertions read: results, state the program shows or acts on next, or effects a caller observes clear it.

Findings it got wrong

Labeled wrong by reading the code, on open-source projects the rules were tuned on.

Django Debug Toolbar: test_recording

• Where: tests/panels/test_sql.py:91 in django-commons/django-debug-toolbar at dfc69d9.
• Finding (consider): test_recording asserts internal details instead of observable results.
• Why it was wrong: The test checks that a query is recorded once with its alias, SQL, duration and stack trace in panel._queries. That list is what the panel saves as its statistics and renders, so the assertions read the panel’s recorded output.
• Since: cleared in 0.21.0, which asks such a test what its assertions read, with the bodies of the functions it calls (changelog).

Two-Factor: test_get_user_time_delay

• Where: tests/class-two-factor-core.php:1192 in WordPress/two-factor at 72effa5.
• Finding (review): test_get_user_time_delay computes its expected value with the logic it tests.
• Why it was wrong: The expected values are the one-second default, the 15-minute cap and pow( 2, 5 ) * $rate_limit, the documented doubling after five failed attempts written out. Nothing calls the code under test to compute them, so a changed base, default or cap would fail the test.
• Since: not addressed; reported the same way from 0.20.0 through 0.25.0.

LinkAce: test_successful_check

• Where: tests/Helper/UpdateCheckTest.php:19 in Kovah/LinkAce at d682166.
• Finding (review): test_successful_check only checks values its mocks were set to return.
• Why it was wrong: checkForUpdates returns the fetched version only when it is newer than the installed one, and true otherwise. The test fakes v100.0.0 to take the first branch, and its sibling fakes v0.0.0 and expects true: together they check the comparison, not the mock.
• Since: not addressed; reported the same way from 0.19.0 through 0.25.0.