TyPy test plan¶
| Field | Value |
|---|---|
| Document | TyPy test plan |
| Version | 0.87 |
| Status | Draft |
| Owner | typy-lang contributors |
| Last updated | 2026-11-17 |
1. Purpose¶
This document says how TyPy is tested: the test levels, how tests are designed at each level, where they run, when a release may ship, and which tests cover each requirement. It builds on the approved detailed design, the software requirements specification, version 1.2, and the decision records ADR-0001 to ADR-0008.
The specification defines correct behaviour, and CPython is the oracle for it (ADR-0008). A test therefore states which requirement it covers, and an accepted-program test needs no hand-written expected output: CPython provides it.
2. Test levels¶
Each test sits at the lowest level that can prove its behaviour.
| Level | What it proves | Where | Runs |
|---|---|---|---|
| Unit (UT) | One function or type in isolation: parsing a code, normalising a union, a large-integer operation | #[cfg(test)] modules in each crate |
Every commit |
| Golden | The text of one compiler stage for one input, through --emit (CLI-04) |
tests/golden/, run by crates/typy-driver/tests/golden.rs |
Every commit |
| Integration (IT) | The compiler, runtime and C toolchain together on one program, compared with CPython; the conformance suite | tests/conformance/<area>/ through typy-conformance, which drives typy-driver in process |
Every commit |
| End-to-end (E2E) | The typy executable as a user runs it: arguments, exit status, files written, and the built program run as a subprocess |
crates/typy-cli/tests/cli.rs |
Every commit |
| Fuzz | No input crashes the compiler (NFR-09) | fuzz/ |
30 minutes before each release; 60 seconds on demand |
| Benchmark | The numbered budgets of NFR-03 to NFR-05, NFR-13 and NFR-14, and how TyPy compares with other languages | tests/bench/ |
make bench in CI from 0.3 for NFR-04, NFR-05 and NFR-14; the run-time suite of NFR-03 before a release, from 1.0; make bench-compare, which writes the benchmark report, on demand from 0.5 |
| Mutation | Tests fail when the code they cover is changed | make mutants over the lines a change adds or alters, each mutant against its own crate's tests, in a container limited to 6 GB with no swap, since a mutant can turn a bounded loop into one that allocates until memory runs out. typy-build, typy-driver and typy-cli are left out, since their tests need the runtime archives built beside the compiler; the end-to-end and conformance tests cover them. A whole crate is run with make mutants-package when its tests are reviewed |
On demand, for a change whose tests deserve a closer look |
3. Test design¶
3.1 Unit tests¶
- Each public function has tests for its normal cases, its boundaries and each error it returns. Variations of one behaviour are a table of cases in one test.
- Parsers and normalisers have property-based tests with
proptest:DiagnosticCode::parseandDisplayround-trip;LineIndexagrees with a reference that counts code points; union normalisation is idempotent and ignores order. - The runtime's large-integer operations are checked against the
num-bigintcrate, as a test-only dependency, on generated values that include the inline boundaries -2^62, -2^62 - 1, 2^62 - 1 and 2^62. float_repris checked by property: the text it writes parses back to the same double, and its digits are the shortest that do. A fixed table of values, including0.1,1e16,1e-05,-0.0,infandnan, holds the text CPython writes for each.- A test name is a sentence about behaviour, such as
oversized_source_is_refused_before_reading.
3.2 Golden tests¶
A golden test is a .py input and the expected output of one stage. The
programs are in tests/golden/programs/, and the text of each stage but
obj is in tests/golden/<stage>/<program>.txt. The programs in
tests/golden/diagnostics/, each named for the DIA requirement it shows,
sit beside the rendered text of their diagnostics. The expected files
change only through make bless, and the diff is reviewed in the same
commit. A test never edits its expected file to pass, and an expected file
without its program fails the run. The golden programs are ASCII, so the
files follow the repository's ASCII rule; Unicode text is tested by the
conformance suite.
3.3 Conformance tests¶
Conformance tests are the main integration level. Each is a Python program in the format of design section 15: comments give the requirement IDs, the expectation, and an optional Python floor.
| Kind | Expectation | Passes when |
|---|---|---|
| Accepted | accepted |
Every oracle version from the floor agrees, and the debug and release builds run alike with them: standard output, exit status and last line of standard error (STD-02, NFR-15) |
| Rejected | rejected CODE at LINE:COLUMN |
typy check reports exactly that code at that position, and nothing else |
Every requirement of areas STD, LEX, TYP, DAT, EXP, SMT, CST, FUN, CLS, GEN, EXC, MOD, LIB, MEM and FFI has at least one accepted and one rejected test (NFR-10). Each test is designed with:
- Equivalence classes and boundaries. For each operator and type pair, one value from each class and every boundary: zero, the inline integer limits, negative values, empty strings, the first and last index, and one past each.
- Error paths. Each exception the runtime raises has a test that lets it go uncaught and, from 0.3, one that catches it.
- The excluded features. Each item of specification section 6.1 has a rejected test.
- Oracle disagreement. A test whose oracle versions disagree is reported as invalid, not failed (STD-03). The test is then rewritten so the versions agree, or kept as a documented record of the difference.
A test file is named after its first requirement and its behaviour, such as
EXP-03_float_division_by_zero_raises.py. A test of several modules is a
directory named so, whose main.py holds the header and whose other files
are its modules, such as mod/MOD-02_packages_and_relative_imports/.
Two requirements have no rejected file: LEX-01, because a source that is
not UTF-8 would break the repository's ASCII rule, and LEX-09, because a
source over 16 MiB is too large to commit. Unit tests in typy-source build
those inputs when they run and check E0102 and E0101. Where a 0.1 program
cannot show the rejection a requirement describes, its rejected test records
what 0.1 does instead, such as a function used as a truth value, E0201
until functions are values in 1.0.
The test crates/typy-conformance/tests/suite.rs runs the suite on every
commit and fails when a test does not pass or a 0.1 requirement of the
matrix lacks either kind of test (NFR-10). The 0.1 corpus holds 45 accepted
and 44 rejected tests and runs in about 20 seconds.
3.4 End-to-end tests¶
E2E tests run the typy executable as a subprocess, in a temporary
directory. They cover each command and flag, each exit status of CLI-06,
and the failure of each dependency:
| Condition | Expected |
|---|---|
| Missing source file | Exit status 1, with the path |
| Empty source file | Accepted; the program prints nothing and exits 0 |
| Source file of exactly 16 MiB, and of 16 MiB and one byte | Accepted; refused with E0101 |
| Source that is not UTF-8 | E0102 at the first invalid byte |
| Unknown flag or command | Exit status 2 |
C toolchain missing from PATH |
E0701, and no output file |
| Linker fails | E0702, and no output file |
| Output directory not writable | Exit status 1, with the path |
| Two builds of the same program at once into different outputs | Both succeed with identical bytes |
Subprocess runs are included in coverage (NFR-11).
3.5 Memory and safety¶
The test crates/typy-conformance/tests/sanitized.rs builds every accepted
conformance test once more: its object file is linked with the runtime built
with AddressSanitizer (-Zsanitizer=address, pinned nightly) and with GCC's
AddressSanitizer library, and run with leak detection on. The run must show
CPython's results; a sanitizer report ends it with status 86 and fails the
test (NFR-07, NFR-08). Every allocation and free is checked, leaks are
reported at exit, and every memory access the runtime makes is checked.
Generated code is not instrumented; it is checked by the TyIR verifier, and
instrumenting it is planned for 0.2 (OI-16).
make miri runs the runtime's unit tests under Miri (ADR-0006), which
reports out-of-bounds, misaligned and dangling accesses and reads of
uninitialised memory. Miri cannot call pow or getrlimit, so the three
tests that do are skipped under it, as is the 4300-digit conversion, which
takes hours there; the long randomised loops check a sample. The unit tests
do not release the objects they make, so Miri's leak report is off; whole
programs are checked for leaks by the sanitizer run.
3.6 Fuzzing¶
The fuzz target stages (fuzz/fuzz_targets/stages.rs) feeds generated
input to the source, checking and lowering stages in turn, starting from the
conformance corpus. A crash, a hang over 10 seconds, or an internal error
E9001, which a TyIR verifier failure becomes, is a defect. Each one found
becomes a regression test named for its behaviour. make fuzz runs it for
FUZZ_SECONDS seconds, 60 by default, with libFuzzer and AddressSanitizer;
the 30-minute run of NFR-09 is FUZZ_SECONDS=1800 make fuzz. The front end
of 0.1 is small, and a 60-second run already stops finding new code paths
quickly, so a longer run adds little; the budget grows with the language.
3.7 Defects¶
A defect is reproduced by a failing test before the code is changed, and the test stays, named for the behaviour it checks.
4. Environments¶
| Environment | Contents | Used by |
|---|---|---|
| Dev container | typy-lang-dev: Rust 1.99.1, the cargo tools, LLVM 23 from apt.llvm.org, musl, Node and markdownlint |
UT, golden, E2E, local runs of everything |
| Oracle interpreters | CPython 3.12.16, 3.13.18, 3.14.10 and 3.15.2, exact builds installed with uv 0.13.7 in the dev container, as python3.12 to python3.15, with nothing else installed |
Conformance |
| Nightly Rust | One pinned nightly toolchain, for Miri and the sanitizers only | Section 3.5 |
| Diagrams container | PlantUML 1.2026.8, pinned by digest | Documentation gate |
| Documentation tools | Zensical 0.1.1 and its dependencies, pinned with their sha256 in scripts/docs-requirements.txt, in a CPython 3.14 environment at /var/cache/typy/docs in the dev container, and on the runner of the Pages workflow |
Documentation gate, DOC-05 |
Each release adds its platforms to the CI matrix as section 2.3 of the specification lists them: Linux x86_64 in 0.1, Linux and macOS AArch64 in 0.6, Windows x86_64 in 1.0.
5. Entry and exit criteria¶
5.1 Entry¶
| Level | A change enters testing when |
|---|---|
| UT, golden | It builds, and cargo fmt --check and Clippy pass |
| IT | Its UT and golden tests pass |
| E2E | Its IT tests pass |
| Release candidate | Every level passes on the release's whole platform matrix |
5.2 Exit, for a release¶
A release ships when all of these hold:
- Every M requirement of the release, and every earlier one, has its accepted and rejected tests, and they pass on every platform of the release.
- Every S or C requirement of the release passes, or the specification moves it to a later release, saying so.
- Line and branch coverage is at least 95 percent (NFR-11).
- The sanitizer and Miri runs report nothing (NFR-07, NFR-08).
- Fuzzing has run 30 minutes with no new crash (NFR-09).
- Every numbered non-functional requirement whose release has been reached is within its budget.
- No defect remains open that breaks a requirement of the release.
6. Test data¶
- Test programs are committed as Python source. No binary file is committed.
- Large inputs, such as the 16 MiB source file and the 10,000-line project, are generated when the tests start, from a seed committed with the test.
- The benchmark programs and their C versions are committed as source. The
build-time projects are written by
tests/bench/build_time.pywhen it starts: a 10,000-line project in one file and in 200 modules, and 50 and 200 modules whose check times it compares. Each figure is the best of three runs, and memory is the peak resident size of each build.
7. Risks¶
| Risk | Effect | Mitigation |
|---|---|---|
| The oracle versions disagree, through Unicode databases or exception messages | Tests become invalid | Reported as invalid, not failed; each one reviewed and documented |
| Running four oracle versions and two build modes per test makes CI slow | Slow feedback | Oracle results are cached by the hash of the test file and the oracle image digest |
| Benchmarks vary with machine load | False failures | Budgets are checked on a dedicated runner, as the median of five runs |
| Miri and the sanitizers need a nightly toolchain | Nightly breakage | One nightly date is pinned and moved on purpose |
| Fuzzing finds many defects early | Release delay | Each defect gets a test first; fuzz findings are triaged before each release |
8. Traceability matrix¶
Each requirement, the release that first meets it, its test level, and
where its tests live. Constraints that a test can check are included.
scripts/check-traceability.sh fails the quality gate when a requirement
of the specification is missing here, or when any document cites an ID
that the specification does not define.
| Requirement | Release | Level | Tests |
|---|---|---|---|
| STD-01 | 0.1 | IT | tests/conformance/std/STD-01_*.py |
| STD-02 | 0.1 | IT | tests/conformance/std/STD-02_*.py |
| STD-03 | 0.1 | IT | tests/conformance/std/STD-03_*.py |
| STD-04 | 0.1 | IT | tests/conformance/std/STD-04_*.py |
| STD-05 | 0.3 | IT | tests/conformance/std/STD-05_*.py |
| STD-06 | 0.1 | IT | tests/conformance/std/STD-06_*.py |
| STD-07 | 0.1 | IT | tests/conformance/std/STD-07_*.py |
| CLI-01 | 0.1 | E2E | crates/typy-cli/tests/cli.rs, tests named for CLI-01 |
| CLI-02 | 0.1 | E2E | crates/typy-cli/tests/cli.rs, tests named for CLI-02 |
| CLI-03 | 0.1 | E2E | crates/typy-cli/tests/cli.rs, tests named for CLI-03 |
| CLI-04 | 0.1 | E2E, golden | crates/typy-cli/tests/cli.rs, tests named for CLI-04; stage golden files in tests/golden/ |
| CLI-05 | 0.1 | E2E | crates/typy-cli/tests/cli.rs, tests named for CLI-05 |
| CLI-06 | 0.1 | E2E | crates/typy-cli/tests/cli.rs, tests named for CLI-06 |
| CLI-07 | 0.3 | E2E | crates/typy-cli/tests/cli.rs, tests named for CLI-07 |
| CLI-08 | 0.6 | E2E | crates/typy-cli/tests/cli.rs, tests named for CLI-08 |
| CLI-09 | 0.5 | UT, E2E | crates/typy-build/src/request.rs and library_kind.rs for the output's name; crates/typy-cli/tests/library.rs, tests named for CLI-09: a static library in a C program, its only global symbols, a shared library from CPython through ctypes, two libraries in one program with a call from another thread, and a file name that is not a module name |
| CLI-13 | 0.5 | UT, E2E | crates/typy-source/src/version.rs for the release of a version; crates/typy-source/src/settings_file.rs for version; crates/typy-check/tests/exports.rs for __version__ and E0429; crates/typy-build/src/library_layout.rs for each layout; crates/typy-cli/tests/library.rs, tests named for CLI-13: a shared library's soname and links, a new release that moves them, a static library's link, -o, and two versions that differ |
| CLI-10 | 0.3 | E2E | crates/typy-cli/tests/cli.rs, tests named for CLI-10 |
| CLI-11 | 0.1 | E2E | crates/typy-cli/tests/cli.rs, tests named for CLI-11 |
| CLI-12 | 0.3 | E2E | crates/typy-cli/tests/cli.rs, tests named for CLI-12 |
| LEX-01 | 0.1 | IT | tests/conformance/lex/LEX-01_*.py |
| LEX-02 | 0.1 | IT | tests/conformance/lex/LEX-02_*.py |
| LEX-03 | 0.1 | IT | tests/conformance/lex/LEX-03_*.py |
| LEX-04 | 0.1 | IT | tests/conformance/lex/LEX-04_*.py |
| LEX-05 | 0.1 | IT | tests/conformance/lex/LEX-05_*.py |
| LEX-06 | 0.2 | IT | tests/conformance/lex/LEX-06_*.py |
| LEX-07 | 0.2 | IT | tests/conformance/lex/LEX-07_*.py |
| LEX-08 | 0.1 | IT | tests/conformance/lex/LEX-08_*.py |
| LEX-09 | 0.1 | IT | tests/conformance/lex/LEX-09_*.py |
| LEX-10 | 0.5 | IT | tests/conformance/lex/LEX-10_*.py |
| LEX-11 | 0.3 | UT, IT, E2E | crates/typy-source/src/parse/tests.rs for each limit just inside and just past it; crates/typy-tyir/src/verify/tests.rs for a function of 100000 blocks; crates/typy-cli/tests/nesting.rs for a program as deep as CPython compiles and each refusal; tests/conformance/lex/LEX-11_*.py |
| TYP-01 | 0.1 | IT | tests/conformance/typ/TYP-01_*.py |
| TYP-02 | 0.1 | IT | tests/conformance/typ/TYP-02_*.py |
| TYP-03 | 0.1 | IT | tests/conformance/typ/TYP-03_*.py |
| TYP-04 | 0.1 | IT | tests/conformance/typ/TYP-04_*.py |
| TYP-05 | 0.2 | IT | tests/conformance/typ/TYP-05_*.py |
| TYP-06 | 0.2 | IT | tests/conformance/typ/TYP-06_*.py |
| TYP-07 | 0.4 | IT | tests/conformance/typ/TYP-07_*.py |
| TYP-08 | 0.4 | IT | tests/conformance/typ/TYP-08_*.py |
| TYP-09 | 0.1 | IT | tests/conformance/typ/TYP-09_*.py |
| TYP-10 | 0.1 | IT | tests/conformance/typ/TYP-10_*.py |
| TYP-11 | 0.1 | IT | tests/conformance/typ/TYP-11_*.py |
| TYP-12 | 0.1 | IT | tests/conformance/typ/TYP-12_*.py |
| TYP-13 | 0.2 | IT | tests/conformance/typ/TYP-13_*.py |
| TYP-14 | 0.4 | IT | tests/conformance/typ/TYP-14_*.py |
| TYP-15 | 0.3 | IT | tests/conformance/typ/TYP-15_*.py |
| TYP-16 | 0.5 | IT | tests/conformance/typ/TYP-16_*.py |
| TYP-17 | 0.5 | IT | tests/conformance/typ/TYP-17_*.py |
| TYP-18 | 1.0 | IT | tests/conformance/typ/TYP-18_*.py |
| TYP-19 | 1.0 | IT | tests/conformance/typ/TYP-19_*.py |
| DAT-01 | 0.1 | IT | tests/conformance/dat/DAT-01_*.py |
| DAT-02 | 0.1 | IT | tests/conformance/dat/DAT-02_*.py |
| DAT-03 | 0.1 | IT | tests/conformance/dat/DAT-03_*.py |
| DAT-04 | 0.1 | IT | tests/conformance/dat/DAT-04_*.py |
| DAT-05 | 0.1 | IT | tests/conformance/dat/DAT-05_*.py |
| DAT-06 | 0.2 | IT | tests/conformance/dat/DAT-06_*.py |
| DAT-07 | 0.2 | IT | tests/conformance/dat/DAT-07_*.py |
| DAT-08 | 0.2 | IT | tests/conformance/dat/DAT-08_*.py |
| DAT-09 | 0.2 | IT | tests/conformance/dat/DAT-09_*.py |
| DAT-10 | 0.2 | IT | tests/conformance/dat/DAT-10_*.py |
| DAT-11 | 0.2 | IT | tests/conformance/dat/DAT-11_*.py |
| DAT-12 | 0.2 | IT | tests/conformance/dat/DAT-12_*.py |
| DAT-13 | 0.3 | IT | tests/conformance/dat/DAT-13_*.py |
| DAT-14 | 0.2 | IT | tests/conformance/dat/DAT-14_*.py |
| DAT-15 | 0.2 | IT | tests/conformance/dat/DAT-15_*.py |
| DAT-16 | 0.3 | IT | tests/conformance/dat/DAT-16_*.py |
| DAT-17 | 0.3 | IT | tests/conformance/dat/DAT-17_*.py |
| EXP-01 | 0.1 | IT | tests/conformance/exp/EXP-01_*.py |
| EXP-02 | 0.1 | IT | tests/conformance/exp/EXP-02_*.py |
| EXP-03 | 0.1 | IT | tests/conformance/exp/EXP-03_*.py |
| EXP-04 | 0.1 | IT | tests/conformance/exp/EXP-04_*.py |
| EXP-05 | 0.1 | IT | tests/conformance/exp/EXP-05_*.py |
| EXP-06 | 0.1 | IT | tests/conformance/exp/EXP-06_*.py |
| EXP-07 | 0.2 | IT | tests/conformance/exp/EXP-07_*.py |
| EXP-08 | 0.1 | IT | tests/conformance/exp/EXP-08_*.py |
| EXP-09 | 0.1 | IT | tests/conformance/exp/EXP-09_*.py |
| EXP-10 | 0.1 | IT | tests/conformance/exp/EXP-10_*.py |
| EXP-11 | 0.1 | IT | tests/conformance/exp/EXP-11_*.py |
| EXP-12 | 0.2 | IT | tests/conformance/exp/EXP-12_*.py |
| EXP-13 | 0.5 | IT | tests/conformance/exp/EXP-13_*.py |
| EXP-14 | 1.0 | IT | tests/conformance/exp/EXP-14_*.py |
| EXP-15 | 1.0 | IT | tests/conformance/exp/EXP-15_*.py |
| EXP-16 | 0.1 | IT | tests/conformance/exp/EXP-16_*.py |
| EXP-17 | 1.0 | IT | tests/conformance/exp/EXP-17_*.py |
| SMT-01 | 0.1 | IT | tests/conformance/smt/SMT-01_*.py |
| SMT-02 | 0.2 | IT | tests/conformance/smt/SMT-02_*.py |
| SMT-03 | 0.1 | IT | tests/conformance/smt/SMT-03_*.py |
| SMT-04 | 0.3 | IT | tests/conformance/smt/SMT-04_*.py |
| SMT-05 | 0.3 | IT | tests/conformance/smt/SMT-05_*.py |
| SMT-06 | 0.3 | IT | tests/conformance/smt/SMT-06_*.py |
| SMT-07 | 0.3 | IT | tests/conformance/smt/SMT-07_*.py |
| SMT-08 | 0.3 | IT | tests/conformance/smt/SMT-08_*.py |
| SMT-09 | 1.0 | IT | tests/conformance/smt/SMT-09_*.py |
| SMT-10 | 0.4 | IT | tests/conformance/smt/SMT-10_*.py |
| SMT-11 | 1.0 | IT | tests/conformance/smt/SMT-11_*.py |
| SMT-12 | 0.5 | IT | tests/conformance/smt/SMT-12_*.py |
| CST-01 | 0.1 | IT | tests/conformance/cst/CST-01_*.py |
| CST-02 | 0.1 | IT | tests/conformance/cst/CST-02_*.py |
| CST-03 | 0.2 | IT | tests/conformance/cst/CST-03_*.py |
| CST-04 | 0.3 | IT | tests/conformance/cst/CST-04_*.py |
| CST-05 | 0.3 | IT | tests/conformance/cst/CST-05_*.py |
| CST-06 | 0.5 | IT | tests/conformance/cst/CST-06_*.py |
| CST-07 | 0.5 | IT | tests/conformance/cst/CST-07_*.py for complete matches; crates/typy-check/tests/matching.rs for W0401 |
| CST-08 | 0.3 | IT | tests/conformance/cst/CST-08_*.py |
| CST-09 | 0.3 | IT | tests/conformance/cst/CST-09_*.py |
| CST-10 | 0.3 | IT | tests/conformance/cst/CST-10_*.py |
| CST-11 | 1.0 | IT | tests/conformance/cst/CST-11_*.py |
| FUN-01 | 0.1 | IT | tests/conformance/fun/FUN-01_*.py |
| FUN-02 | 0.1 | IT | tests/conformance/fun/FUN-02_*.py |
| FUN-03 | 1.0 | IT | tests/conformance/fun/FUN-03_*.py |
| FUN-04 | 1.0 | IT | tests/conformance/fun/FUN-04_*.py |
| FUN-05 | 1.0 | IT | tests/conformance/fun/FUN-05_*.py |
| FUN-06 | 1.0 | IT | tests/conformance/fun/FUN-06_*.py |
| FUN-07 | 1.0 | IT | tests/conformance/fun/FUN-07_*.py |
| CLS-01 | 0.2 | IT | tests/conformance/cls/CLS-01_*.py |
| CLS-02 | 0.2 | IT | tests/conformance/cls/CLS-02_*.py |
| CLS-03 | 0.3 | IT | tests/conformance/cls/CLS-03_*.py |
| CLS-04 | 0.4 | IT | tests/conformance/cls/CLS-04_* |
| CLS-05 | 0.4 | IT | tests/conformance/cls/CLS-05_*.py |
| CLS-06 | 0.4 | IT | tests/conformance/cls/CLS-06_*.py; crates/typy-check/tests/checker.rs, tests named for methods called through their class |
| CLS-07 | 0.4 | IT | tests/conformance/cls/CLS-07_*.py |
| CLS-08 | 0.2 | IT | tests/conformance/cls/CLS-08_*.py |
| CLS-09 | 0.4 | IT | tests/conformance/cls/CLS-09_*.py |
| CLS-10 | 0.2 | UT, IT | typy-runtime type object unit tests; tests/conformance/cls/CLS-10_*.py |
| CLS-11 | 0.4 | IT | tests/conformance/cls/CLS-11_*.py |
| CLS-12 | 0.2 | UT, IT | typy-runtime type object unit tests; tests/conformance/cls/CLS-12_*.py |
| CLS-13 | 0.3 | IT | tests/conformance/cls/CLS-13_*.py |
| CLS-14 | 0.3 | IT | tests/conformance/cls/CLS-14_*.py |
| CLS-15 | 0.5 | IT | tests/conformance/cls/CLS-15_*.py |
| CLS-16 | 0.5 | IT | tests/conformance/cls/CLS-16_*.py |
| CLS-17 | 0.5 | IT | tests/conformance/cls/CLS-17_*.py |
| CLS-20 | 0.5 | IT | tests/conformance/cls/CLS-20_*.py |
| CLS-21 | 0.5 | IT | tests/conformance/cls/CLS-21_*.py |
| CLS-22 | 0.5 | IT | tests/conformance/cls/CLS-22_*.py |
| CLS-23 | 0.5 | IT | tests/conformance/cls/CLS-23_*.py |
| CLS-24 | 0.5 | IT | tests/conformance/cls/CLS-24_*.py |
| CLS-25 | 0.5 | IT | tests/conformance/cls/CLS-25_*.py |
| CLS-26 | 0.5 | IT | tests/conformance/cls/CLS-26_*.py |
| CLS-27 | 1.0 | IT | tests/conformance/cls/CLS-27_*.py |
| CLS-28 | 1.0 | IT | tests/conformance/cls/CLS-28_*.py |
| CLS-29 | 1.0 | IT | tests/conformance/cls/CLS-29_*.py |
| CLS-30 | 1.0 | IT | tests/conformance/cls/CLS-30_*.py |
| CLS-31 | 1.0 | IT | tests/conformance/cls/CLS-31_*.py |
| CLS-32 | 1.0 | IT | tests/conformance/cls/CLS-32_*.py |
| CLS-33 | 1.0 | IT | tests/conformance/cls/CLS-33_*.py |
| CLS-34 | 1.0 | IT | tests/conformance/cls/CLS-34_*.py |
| CLS-35 | 1.0 | IT | tests/conformance/cls/CLS-35_*.py |
| CLS-36 | 1.0 | IT | tests/conformance/cls/CLS-36_*.py |
| CLS-18 | 1.0 | IT | tests/conformance/cls/CLS-18_*.py |
| CLS-19 | 1.0 | IT | tests/conformance/cls/CLS-19_*.py |
| GEN-01 | 0.4 | IT | tests/conformance/gen/GEN-01_*.py; crates/typy-check/tests/generics.rs |
| GEN-02 | 0.4 | IT | tests/conformance/gen/GEN-02_*.py |
| GEN-03 | 0.4 | IT | tests/conformance/gen/GEN-03_*.py |
| GEN-04 | 0.4 | IT | tests/conformance/gen/GEN-04_*.py |
| GEN-05 | 0.4 | IT | tests/conformance/gen/GEN-05_*.py |
| GEN-06 | 1.0 | IT | tests/conformance/gen/GEN-06_*.py |
| GEN-07 | 0.5 | IT | tests/conformance/gen/GEN-07_*; crates/typy-check/tests/generics.rs |
| GEN-08 | 0.5 | IT | tests/conformance/gen/GEN-08_*; crates/typy-check/tests/generics.rs |
| GEN-09 | 0.5 | IT | tests/conformance/gen/GEN-09_*.py; crates/typy-check/tests/generics.rs |
| GEN-10 | 0.5 | IT | tests/conformance/gen/GEN-10_*; crates/typy-check/tests/protocols.rs |
| GEN-11 | 1.0 | IT | tests/conformance/gen/GEN-11_*.py; crates/typy-check/tests/generics.rs |
| GEN-12 | 1.0 | IT | tests/conformance/gen/GEN-12_*.py; crates/typy-check/tests/generics.rs |
| GEN-13 | 1.0 | IT | tests/conformance/gen/GEN-13_*.py; crates/typy-check/tests/protocols.rs |
| GEN-14 | 1.0 | IT | tests/conformance/gen/GEN-14_*.py; crates/typy-check/tests/generics.rs |
| GEN-15 | 1.0 | IT | tests/conformance/gen/GEN-15_*.py |
| GEN-16 | 1.0 | IT | tests/conformance/gen/GEN-16_*.py; crates/typy-check/tests/generics.rs |
| GEN-17 | 1.0 | IT | tests/conformance/gen/GEN-17_*.py; crates/typy-check/tests/protocols.rs |
| EXC-01 | 0.3 | IT | tests/conformance/exc/EXC-01_*.py |
| EXC-02 | 0.3 | IT | tests/conformance/exc/EXC-02_*.py |
| EXC-03 | 0.3 | Golden | tests/golden/llvm-ir/, where each call that may raise is followed by one compare of the exception against null and a branch |
| EXC-04 | 0.3 | IT | tests/conformance/exc/EXC-04_*.py |
| EXC-05 | 0.5 | E2E | crates/typy-cli/tests/library.rs, tests named for EXC-05: an exception, an int result outside int64_t and a SystemExit that leave an export |
| EXC-06 | 0.3 | IT | tests/conformance/exc/EXC-06_*.py |
| EXC-07 | 0.3 | IT | tests/conformance/exc/EXC-07_*.py |
| EXC-08 | 0.3 | IT | tests/conformance/exc/EXC-08_*.py |
| EXC-09 | 0.3 | IT | tests/conformance/exc/EXC-09_*.py |
| EXC-10 | 0.3 | IT | tests/conformance/exc/EXC-10_*.py |
| EXC-11 | 0.4 | IT | tests/conformance/exc/EXC-11_*.py |
| EXC-12 | 0.3 | IT | tests/conformance/exc/EXC-12_*.py |
| EXC-13 | 0.4 | IT | tests/conformance/exc/EXC-13_*.py |
| EXC-14 | 0.4 | IT | tests/conformance/exc/EXC-14_*.py |
| EXC-15 | 0.5 | IT | tests/conformance/exc/EXC-15_*.py |
| EXC-16 | 0.5 | IT | tests/conformance/exc/EXC-16_*.py |
| EXC-17 | 1.0 | IT | tests/conformance/exc/EXC-17_*.py |
| EXC-18 | 1.0 | IT | tests/conformance/exc/EXC-18_*.py |
| MOD-01 | 0.3 | IT, E2E | tests/conformance/mod/MOD-01_*; module paths in crates/typy-cli/tests/cli.rs, tests named for MOD-01 |
| MOD-02 | 0.3 | IT | tests/conformance/mod/MOD-02_*.py |
| MOD-03 | 0.3 | IT | tests/conformance/mod/MOD-03_*.py |
| MOD-04 | 0.3 | IT | tests/conformance/mod/MOD-04_*.py |
| MOD-05 | 0.3 | IT | tests/conformance/mod/MOD-05_*.py |
| MOD-06 | 0.3 | IT | crates/typy-check/tests/lazy_imports.rs; crates/typy-check/src/subset/tests.rs, tests named for lazy imports; tests/conformance/mod/MOD-06_*, at floor 3.15 |
| MOD-07 | 0.3 | IT | crates/typy-check/tests/lazy_imports.rs and crates/typy-driver/tests/lazy_imports.rs, whose expected output is CPython 3.15's: CPython 3.12 to 3.14 ignore __lazy_modules__, so a conformance test of it is invalid below floor 3.15 (ADR-0008) |
| MOD-08 | 0.3 | IT | crates/typy-driver/tests/lazy_imports.rs; tests/conformance/mod/MOD-08_*, at floor 3.15 |
| MOD-09 | 0.3 | IT | crates/typy-check/tests/sys.rs for each switch, as an attribute and in from sys import; tests/conformance/mod/MOD-09_*.py |
| MOD-10 | 0.3 | IT | tests/conformance/mod/MOD-10_*.py |
| MOD-11 | 0.4 | IT | tests/conformance/mod/MOD-11_* |
| MOD-12 | 0.5 | IT | tests/conformance/mod/MOD-12_* |
| MOD-13 | 0.5 | IT | tests/conformance/mod/MOD-13_* |
| MOD-14 | 1.0 | IT | tests/conformance/mod/MOD-14_* |
| MOD-15 | 1.0 | IT | tests/conformance/mod/MOD-15_* |
| LIB-01 | 0.1 | IT | tests/conformance/lib/LIB-01_*.py |
| LIB-02 | 0.2 | IT | tests/conformance/lib/LIB-02_*.py |
| LIB-03 | 0.2 | IT | tests/conformance/lib/LIB-03_*.py |
| LIB-04 | 0.3 | UT, IT, E2E | crates/typy-runtime/src/sys/tests.rs for decoding the arguments and each exit status; crates/typy-runtime/src/text_file/tests.rs for chunked reading, newline translation, decode and encode errors, buffered writing, each OSError and files on disk; crates/typy-check/src/open.rs for each mode and encoding name; crates/typy-check/tests/sys.rs and tests/open.rs for each refusal; crates/typy-cli/tests/sys.rs for real arguments, piped standard input and the exit status, and tests/files.rs for what a program leaves on disk when it ends; tests/conformance/lib/LIB-04_*.py |
| LIB-05 | 0.3 | UT, IT | crates/typy-runtime/src/math/tests.rs and tests/more.rs, tables of CPython 3.15's results bit for bit; crates/typy-check/tests/math.rs for each refusal and the floor of each name; tests/conformance/lib/LIB-05_*.py |
| LIB-06 | 0.4 | IT | tests/conformance/lib/LIB-06_*.py |
| LIB-07 | 1.0 | IT | tests/conformance/lib/LIB-07_*.py |
| LIB-08 | 0.3 | IT | tests/conformance/lib/LIB-08_*.py |
| LIB-09 | 0.1 | IT | tests/conformance/lib/LIB-09_*.py |
| LIB-10 | 0.3 | UT, IT | crates/typy-runtime/src/format/percent/tests.rs, a table of CPython 3.12's results; crates/typy-check/tests/percent.rs for each refusal message; tests/conformance/lib/LIB-10_*.py |
| LIB-11 | 0.4 | IT | tests/conformance/lib/LIB-11_*.py |
| LIB-12 | 0.4 | IT | tests/conformance/lib/LIB-12_*.py |
| LIB-15 | 0.5 | IT | tests/conformance/lib/LIB-15_*.py |
| LIB-16 | 0.5 | IT | tests/conformance/lib/LIB-16_*.py |
| LIB-17 | 0.5 | IT | tests/conformance/lib/LIB-17_*.py |
| LIB-18 | 1.0 | IT | tests/conformance/lib/LIB-18_*.py |
| LIB-19 | 1.0 | IT | tests/conformance/lib/LIB-19_*.py |
| LIB-20 | 1.0 | IT | tests/conformance/lib/LIB-20_*.py |
| LIB-21 | 1.0 | IT | tests/conformance/lib/LIB-21_*.py |
| LIB-13 | 0.4 | IT | tests/conformance/lib/LIB-13_*.py |
| LIB-14 | 0.4 | IT | tests/conformance/lib/LIB-14_*.py |
| MEM-01 | 0.1 | UT, IT | typy-runtime unit tests; tests/conformance/mem/MEM-01_*.py under sanitizers |
| MEM-02 | 0.2 | UT, IT | typy-runtime list unit tests; tests/conformance/dat/DAT-07_*.py, which name MEM-02, also under the sanitizers |
| MEM-03 | 0.5 | UT, IT | crates/typy-tyir/src/escape/tests.rs for each way an instance escapes; crates/typy-runtime/src/instance/tests.rs for a frame block made, reused and released; crates/typy-lower/tests/frames.rs for the instances programs hold in frames; crates/typy-codegen/tests/codegen.rs for the block and the release before each return; tests/conformance/mem/MEM-03_*.py under sanitizers |
| MEM-04 | 0.2 | UT, IT | typy-lower lowering tests of the counting; tests/conformance/mem/MEM-04_*.py under sanitizers |
| MEM-05 | 1.0 | UT, IT | typy-runtime unit tests; tests/conformance/mem/MEM-05_*.py under sanitizers |
| MEM-06 | 1.0 | UT, IT | typy-runtime unit tests; tests/conformance/mem/MEM-06_*.py under sanitizers |
| MEM-07 | 1.0 | UT, IT | typy-runtime unit tests; tests/conformance/mem/MEM-07_*.py under sanitizers |
| MEM-08 | 0.5 | UT, IT, E2E | crates/typy-source/src/settings_file/allocator/tests.rs for each wrong [allocator] table; crates/typy-codegen/tests/codegen.rs for the hooks; crates/typy-build/src/linker.rs for the library's place among the arguments; crates/typy-cli/tests/allocator.rs for a program built with a counting allocator, one that has no memory, and a library the linker cannot find |
| MEM-10 | 1.0 | UT, IT | crates/typy-tyir/src/escape/tests.rs; tests/conformance/mem/MEM-10_*.py under sanitizers |
| MEM-09 | 0.3 | UT, IT | crates/typy-lower/tests/moves.rs for each store that moves and each read that must not; crates/typy-runtime/src/moved_tests.rs for the counts of each moved store; tests/conformance/mem/MEM-09_*.py under sanitizers |
| FFI-01 | 0.5 | UT, IT | crates/typy-check/src/check/ctypes_declarations/tests.rs for each way of naming ctypes and loading a library; tests/conformance/ffi/FFI-01_*.py against CPython 3.12 to 3.15 |
| FFI-02 | 0.5 | UT, IT | crates/typy-check/src/check/ctypes_declarations/tests.rs for each E0430 of a declaration; tests/conformance/ffi/FFI-02_*.py for calls declared in time, too early, without a restype, and with an argument of the wrong type |
| FFI-03 | 0.5 | UT, IT | crates/typy-check/src/ctypes.rs and ctypes/structure.rs for the types, their classes, names, sizes, structure layouts and conversions; crates/typy-check/src/check/ctypes_declarations/tests.rs for POINTER, names bound to a type and structure classes of this module and of others; crates/typy-check/tests/ctypes.rs for each E0404, E0402, E0412 and E0401 of an object and each argument a pointer takes; crates/typy-tyir/src/library.rs and crates/typy-lower/tests/foreign.rs for the class table, pointer parameters and structures by value; crates/typy-codegen/src/structure_abi.rs for the register class of each eight bytes of a structure and the registers left; crates/typy-runtime/src/foreign.rs for the runtime's conversions and ctypes.ArgumentError; crates/typy-runtime/src/cdata/tests.rs for scalars, pointers that keep their target, null pointers, checked array indexes and the bytes an object keeps for its memory; tests/conformance/ffi/FFI-03_*.py for every scalar type's wrap-around, widening, rounding and null, and for objects passed by reference, pointers, arrays and structures, against CPython 3.12 to 3.15, also under the sanitizers; crates/typy-cli/tests/structures.rs for structures passed by value in each register class and in memory, against CPython through a C library the test builds |
| FFI-04 | 0.5 | UT, E2E | crates/typy-source/src/settings_file/libraries.rs and settings.rs for library-paths, [libraries] and the linker's names; crates/typy-cli/tests/library.rs, tests named for FFI-04: a program that imports a TyPy library's binding, linked dynamically and statically, and a library the linker cannot find |
| FFI-05 | 0.5 | UT, IT, E2E | crates/typy-check/tests/exports.rs for each name __all__ lists and each E0429; crates/typy-tyir/tests/library.rs for the verifier and the text form; crates/typy-codegen/tests/codegen.rs and src/header.rs for the wrappers and the header; crates/typy-runtime/src/int/fixed.rs for the conversions; crates/typy-cli/tests/library.rs, tests named for FFI-05, also comparing the refused names with what the runtime archive calls |
| FFI-08 | 0.5 | UT, E2E | crates/typy-codegen/src/binding.rs for the binding and the stub; crates/typy-cli/tests/library.rs, tests named for FFI-08 and FFI-04: CPython and a TyPy program call a library through its binding, and CPython 3.12 to 3.15 parse the binding and the stub |
| FFI-09 | 1.0 | IT | tests/conformance/ffi/FFI-09_*.py |
| FFI-10 | 1.0 | UT, IT | crates/typy-check/tests/ctypes.rs for each E0201 of a ctypes use planned for 1.0; tests/conformance/ffi/FFI-10_*.py |
| FFI-11 | 1.0 | UT, IT | crates/typy-check/src/check/ctypes_declarations/tests.rs and crates/typy-check/tests/ctypes.rs for each E0201 of a structure form planned for 1.0; tests/conformance/ffi/FFI-11_*.py |
| FFI-06 | 1.0 | IT | tests/conformance/ffi/FFI-06_*.py |
| FFI-07 | 1.0 | IT | tests/conformance/ffi/FFI-07_*.py |
| DIA-01 | 0.1 | UT, IT | typy-diagnostics unit tests; tests/golden/diagnostics/DIA-01_* |
| DIA-02 | 0.1 | UT, IT | typy-diagnostics unit tests; tests/golden/diagnostics/DIA-02_* |
| DIA-03 | 0.1 | UT, IT | typy-diagnostics unit tests; tests/golden/diagnostics/DIA-03_* |
| DIA-04 | 0.1 | UT, IT | typy-diagnostics unit tests; tests/golden/diagnostics/DIA-04_* |
| DIA-05 | 0.3 | UT, E2E | crates/typy-diagnostics/src/render_json.rs unit and property tests; crates/typy-cli/tests/cli.rs, tests named for DIA-05 |
| DIA-06 | 0.1 | UT | typy-diagnostics unit tests of the code registry |
| DIA-07 | 0.3 | E2E | crates/typy-cli/tests/cli.rs, tests named for DIA-07 |
| DIA-08 | 0.3 | UT | crates/typy-source/src/parse/tests.rs for W0101; crates/typy-check/src/subset/tests.rs for W0301 |
| DOC-01 | 0.2 | Static | make docs-lint; review against the release |
| DOC-02 | 0.2 | Static, IT | make docs-lint; every rule ID on the pages is checked to exist by make traceability; samples by DOC-04 |
| DOC-03 | 0.2 | IT | crates/typy-conformance/tests/documentation.rs lists every active code of the registry and fails when docs/diagnostics/troubleshooting.md has no section for one |
| DOC-04 | 0.2 | IT | crates/typy-conformance/tests/documentation.rs extracts every python sample from the user documentation, every page under docs/ outside the engineering documents' directories, with the outcome its typy-expect comment states, and judges it as a conformance test: a rejected sample by its code and place, an accepted one by running it and comparing with CPython |
| DOC-05 | 0.6 | Static, IT | make docs, which make check runs: scripts/build-docs.sh builds the site with Zensical in strict mode, which fails on a link to a page or an anchor that does not exist; then scripts/check-docs-nav.py fails on a page under docs/ that the nav of mkdocs.yml leaves out and on a nav entry with no page; .github/workflows/pages.yml publishes the build of each release |
| NFR-01 | 0.1 | IT | crates/typy-driver/tests/size.rs: hello world, --release-small, linked statically with musl, run, and under 100 KB (75 KB in 0.1, 17 KB stripped) |
| NFR-02 | 0.1 | Static | make runtime-size, scripts/check-runtime-size.py: a hello world built with --release-small and linked for each target with a link map; the runtime sections linked are under 48 KB, and under 32 KB less the large-integer sections (10.3 KB and 10.3 KB in 0.2, ADR-0009) |
| NFR-03 | 1.0 | Benchmark | tests/bench/ suite against its C versions, geometric mean within 1.5 times |
| NFR-04 | 1.0 | Benchmark | tests/bench/build_time.py: debug build of a 10,000-line project, in one file and in 200 modules, under 2 s; checking 200 modules at most 6 times as long as 50 |
| NFR-05 | 1.0 | Benchmark | tests/bench/build_time.py: release build of both 10,000-line projects under 20 s |
| NFR-06 | 0.1 | E2E | crates/typy-conformance/tests/determinism.rs: every accepted conformance program built twice, in debug and in release mode, and the executables compared byte for byte |
| NFR-07 | 0.1 | IT | tests/sanitized.rs: the conformance suite with the runtime under AddressSanitizer; make miri; make fuzz |
| NFR-08 | 0.1 | IT | tests/sanitized.rs: the conformance suite under LeakSanitizer |
| NFR-09 | 0.1 | Fuzz | make fuzz, target stages over parsing, checking and lowering; 30 minutes with zero crashes before a release |
| NFR-10 | 0.1 | IT | Coverage report of typy-conformance: an accepted and a rejected test per requirement |
| NFR-11 | 0.1 | CI gate | make coverage-branches, run by make check: cargo llvm-cov with the pinned nightly over the compiler and runtime crates, with subprocesses, at least 95 percent of lines and of branches (97.6 and 95.1 in 0.1) |
| NFR-12 | 0.2 | UT | typy-runtime layout test: size_of::<ObjectHeader>() == 8 |
| NFR-13 | 0.2 | UT | typy-runtime list test: a million int values appended one at a time; the list and its buffer take at most 8.5 MB |
| NFR-14 | 1.0 | Benchmark | tests/bench/build_time.py: peak memory of each build within 1 GB |
| NFR-15 | 0.1 | IT | Conformance suite: debug and release runs compared |
| C-01 | 0.1 | CI gate | Workspace builds with the pinned Rust toolchain only |
| C-02 | 0.1 | CI gate | scripts/check-pins.sh: ruff_python_parser pinned with = in Cargo.toml |
| C-03 | 0.1 | UT | typy-codegen test: the linked LLVM major version is 23 |
| C-04 | 0.1 | CI gate | cargo deny check licenses |
| C-05 | 0.1 | E2E | E2E tests run in a container with no network |