Testing#
Tiers#
tier |
runs |
what |
|---|---|---|
push |
every push and pull request to |
the components the change touches, a few minutes each |
weekly |
Sunday 22:00 UTC, and by hand |
every test of at most 30 minutes, all backends end to end, all containers rebuilt |
manual |
by hand, on a machine with many cores |
cross-backend matrix and timing regressions, no workflow yet |
#notest in the head commit message of a push or in the pull request title skips all jobs. [skip ci] is GitHub’s own and skips the whole workflow. Branch rules require the check ci-ok, which passes when every job succeeded or was skipped.
A change also runs what depends on it.
changed |
also runs |
|---|---|
core |
jax, server, prepost, plotting, superrepo tests, tutorials, QR cases, backend e2e |
jax |
superrepo tests, tutorials, QR cases, backend e2e |
plotting |
tutorials, QR cases |
prepost |
server |
GUI, CLI |
QR cases |
|
everything |
Markers#
marker |
meaning |
tiers |
|---|---|---|
|
passes, too slow for the push budget |
weekly |
|
exercises superseded code |
weekly |
|
fails today, |
none |
|
timing regression |
manual |
An unmarked test is in the push tier. large, legacy and broken are set by test id in tests/ci_tiers.tsv, one line per test, so no test file carries them. A fixed test loses its line. Where CI is set, the wall-time gates inside the jax tests print and do not judge, because a hosted runner is not the machine that recorded them.
How to run#
Push tier, from the folder of the component:
cd library/zoomy_core
pytest -n 4 -m "not large and not legacy and not broken and not benchmark"
The same in library/zoomy_jax. The other components run from the repository root with its configuration:
pytest -c pytest.ini --rootdir=. -n 4 -m "not large and not legacy and not broken and not benchmark" library/zoomy_server/tests
Replace the path by library/zoomy_prepost/tests, library/zoomy_plotting/zoomy_plotting/tests, or tests --ignore=tests/qr_cases --ignore=tests/backend_e2e --ignore=tests/tutorials for the superrepo tests.
Weekly tier, the same commands with the larger selection:
ZOOMY_RUN_LARGE_TESTS=1 pytest -n 4 -m "not broken and not benchmark" --timeout=1800
Manual tier:
pytest tests/symsolver -m benchmark
Tutorial notebooks, QR cases and the backend e2e have their own commands, see the READMEs in the table below.
Where tests live#
tests |
location |
add one |
|---|---|---|
cross-backend matrix |
|
|
timings |
|
|
tutorial notebooks |
|
|
thesis QR cases |
|
|
backend containers |
|
|
libraries |
|
pytest, the root |