Benchmark
How well pencilmark's numbers track how long real people take and how hard they find puzzles, compared with other graders, on a frozen held-out set. Only aggregate numbers are published; no puzzles or per-puzzle data from the sources.
Summary
- Within each group of comparable puzzles, pencilmark's
difficultyranks puzzles by human solve time with a Spearman correlation of 0.61 to 0.80 depending on the source, and reaches 67-87% of the best any grader could reach (the noise ceiling of each source's own data). - On the same classic puzzles it ranks closer to human times than the four other graders tested; on variants it is ahead of Rangsk's solver, the one other grader that rates them.
- 99.5% of the 7,623 puzzles get a card; the median card takes 0.5 s (classics 0.04 s) on the machine below.
Method
- Frozen set. 15% of every source, chosen by a hash of the puzzle (its solution where known), fixed before the 1.0 release and used only to measure releases. Every fit and design decision since then used the other 85% only.
- Targets. Real human data from each source (below). Where a source gives per-player times, the target is the puzzle's effect after correcting for each player's skill.
- Metric. Spearman rank correlation between a grader's rating and the human target, computed within each group (a
difficulty tab, a variant family, a board size) and averaged over groups weighted by their size; groups with fewer
than 15 puzzles are skipped. For sources with real minutes, also the median |log(real / predicted)| of
time. - pencilmark. Version 1.0.0 (graded at commit 9995aeb; later commits changed only difficulties below 2, keeping every puzzle's order, so every number here holds),
rate()with the default fields andmax_seconds=300; coverage at smaller limits is read from each puzzle's grading time. - Other graders. Sudoku Explainer (DiufSudoku fork, maximum difficulty), HoDoKu 2.2.0 (score), Rangsk's SudokuSolver (hardest technique tier of its logical solve), Andrew Stuart's sudokuwiki.org grader (one puzzle every 30 s; classics only on this set). Each is run on the puzzles it supports.
- Machine. AMD Ryzen 7 7700X (8 cores / 16 threads, up to 5.5 GHz), 32 GB RAM, Linux; Python 3.14, OpenJDK 17 for the Java graders. Grading ran 8 processes at once (one puzzle each). Times are wall-clock on this machine.
Sources
| Source | Puzzles graded | Target |
|---|---|---|
| fed-sudoku.eu (classics, three tabs) | 540 | skill-corrected per-puzzle effect |
| Wang et al. 2024 (classics) | 207 | mean solve seconds |
| grandgames.net (classics, rule variants, multi-grid layouts) | 3,097 | era-adjusted per-player effect |
| sudoku.today (classics and about 50 variant types) | 3,132 | median solve minutes (at least 5 solvers) |
| sudoku.coach (classics and variants) | 202 | skill-corrected per-puzzle effect |
| logic-masters.de (variants) | 375 | setter's star rating |
| Puzzle Duel (variants) | 35 | skill-corrected effect (too few per type to score) |
Accuracy
| Source | Groups | difficulty | time | time error (median, log) |
|---|---|---|---|---|
| sudoku.coach | 2 | 0.799 | 0.783 | |
| grandgames.net | 22 | 0.752 | 0.760 | |
| logic-masters.de | 5 | 0.721 | 0.676 | |
| fed-sudoku.eu | 3 | 0.670 | 0.659 | |
| Wang 2024 | 1 | 0.663 | 0.587 | 0.19 |
| sudoku.today | 50 | 0.608 | 0.628 | 0.35 (0.32 after one overall scale) |
How well the sources agree with themselves
Each puzzle's players split at random into two halves, a skill-adjusted time from each half, and the rank correlation between the halves (Spearman-Brown corrected, mean of 20 splits). Its square root is the ceiling: the best rank correlation any grader can reach against that source.
| Source | Groups | Reliability | Ceiling | pencilmark | Share of ceiling |
|---|---|---|---|---|---|
| grandgames.net | 22 | 0.94 | 0.97 | 0.755 | 78% |
| sudoku.coach | 2 | 0.96 | 0.98 | 0.765 | 78% |
| fed-sudoku.eu | 3 | 0.98 | 0.99 | 0.660 | 67% |
| sudoku.today | 50 | 0.60 (estimated) | 0.75 | 0.597 | 87% |
sudoku.today publishes only a median of a few solves per puzzle (its reliability is estimated from that sampling error), so most of its gap is noise. On the other three the human signal is clean and the gap is grader error.
Compared with other graders (same puzzles)
| Set | Puzzles | pencilmark | Andrew Stuart | Rangsk | HoDoKu | Sudoku Explainer |
|---|---|---|---|---|---|---|
| fed-sudoku.eu classics | 540 | 0.670 | 0.594 | 0.357 | 0.326 | 0.291 |
| Wang 2024 classics | 207 | 0.663 | 0.498 | 0.347 | 0.403 | 0.475 |
| sudoku.today variants | 1,527 | 0.699 | – | 0.676 | – | – |
Bands against other graders (frozen classics, 747 puzzles)
| pencilmark band | Puzzles | HoDoKu level | pencilmark / sudokuwiki.org rating (median) |
|---|---|---|---|
| Very Easy | 295 | Easy 100% | 3.5 / 3.8 |
| Easy | 373 | Easy 13%, Medium 86% | 5.4 / 5.2 |
| Medium | 69 | Medium 89%, Hard 10% | 6.8 / 7.0 |
| Hard | 8 | Hard 75%, Unfair 25% | 5.4 / 5.3 |
The bands follow the same technique steps as HoDoKu's levels, and pencilmark's difficulty is within 0.3 of the sudokuwiki.org rating at every band. A Hard puzzle can rate below a Medium one: the band orders the knowledge a puzzle needs, the number its overall difficulty (Philosophy); the sudokuwiki.org ratings show the same.
Other sites' labels on this scale
Where each site's difficulty labels fall on pencilmark's scale: the bar is the median difficulty of the site's puzzles with that label, the line the middle half where it was measured. Each site rates for its own audience and neighbouring labels overlap a lot, so treat this as a rough conversion.
| Site | Label | pencilmark difficulty: median (quartiles) |
|---|---|---|
| fed-sudoku.eu | easy | 3.9 (3.4–4.3) |
| fed-sudoku.eu | medium | 5.1 (4.6–5.9) |
| fed-sudoku.eu | hard | 6.5 (6.2–7) |
| sudoku.coach | level 0 | 0.6 |
| sudoku.coach | level 1 | 1.7 |
| sudoku.coach | level 2 | 2.8 |
| sudoku.coach | level 3 | 3.7 |
| sudoku.coach | level 4 | 5.2 |
| sudoku.coach | level 5 | 5.8 |
| sudoku.coach | level 6 | 6.1 |
| sudoku.coach | level 7 | 6.7 |
| sudoku.coach | level 8 | 7.3 |
| sudoku.coach | level 9 | 8.7 |
| sudoku.coach | level 10 | 11.5 |
| sudoku.coach | level 11 | 11.9 |
| Logic Masters | 1 star | 8.8 (6.3–10.1) |
| Logic Masters | 2 stars | 11.1 (10.1–12.6) |
| Logic Masters | 3 stars | 13.0 (11.5–13.7) |
| Logic Masters | 4-5 stars | 13.8 (12.6–14.9) |
Coverage and speed
-
Coverage: 7,588 of 7,623 puzzles graded (99.5%). Not graded: 14 not unique and 2 without a solution after conversion, 12 undecided at the search limit without OR-Tools, 2 unreadable, 1 over the benchmark's memory cap.
-
By time limit (graded within the limit): 10 s 93.9%, 30 s 96.4%, 60 s 97.4%, 120 s 98.2%, 300 s 98.8%.
-
Speed (default card): median 0.49 s, 90th percentile 3.8 s; classics median 0.04 s. Sudoku Explainer and HoDoKu answer in under a millisecond (they do not model human solving); Rangsk 0.07-0.13 s; the sudokuwiki.org grader, median 3.1 s per request.
Generator
Five requests, 10 puzzles per tool, one process at a time (measured on the 1.0 release candidate, before the seven-band names; requests are described by what they ask for). Seconds per kept puzzle = all attempts' time divided by the puzzles kept. Where a tool cannot express a request, its own output is filtered and the rejected puzzles' time counts.
| Request | pencilmark | HoDoKu 2.2.0 | Sudoku Explainer |
|---|---|---|---|
| a classic needing wings or uniqueness (rc1 "Hard") | 18 s | 0.05 s | 0.17 s |
| an X-Wing as the hardest step | 66 s | 0.45 s | 1.6 s |
| the same, 180° symmetric with at most 24 givens | 64 s | 0.8 s (346 tries filtered) | 2.2 s (440 tries filtered) |
| the hardest level | none within 120 s | 0.05 s | 0.21 s |
| a Medium killer | 65 s | not supported | not supported |
pencilmark is 100 to 1000 times slower, lands on the requested band more often (the others' "Hard" is often an easier band on pencilmark's scale) and is the only one that generates killers. Speed and the hardest band are known issues.
Run it on your own data
The frozen set stays private (its puzzles and times come from the sources above), but the measurement is public:
tools/benchmark.py grades puzzles that carry a human target and prints the accuracy table above for them. Run on
the frozen set's grades it reproduces every number on this page.
python tools/benchmark.py grade puzzles.jsonl -o grades.jsonl --jobs 8
python tools/benchmark.py report grades.jsonl
One JSON object per line: puzzle (anything pencilmark.load reads), source, group (puzzles that are comparable:
correlations are computed within each group and averaged), target (the human measure, higher = harder) and,
optionally, target_kind ("minutes", "log minutes", "seconds" or "log seconds" add the time error) and key.
Grading resumes where it stopped.