SEC filing tasks in the verifiers format.
Tickmark Bench Core ships 149 finance tasks in the Prime Intellect verifiers format. Each one has a single reference answer from filed XBRL facts and a pass or fail result computed in code by grade.py.
What each task carries
A Core task is one JSON object. Among its fields are the question, the answer, the tolerance, the facts behind the answer, the derivation, and a verifier field. The full list is on the Rights page. Tasks ship in the Prime Intellect verifiers format. Tickmark Bench is not a Prime Intellect product.
A check you can trust
Numeric answers are checked in code against a reference value taken from, or computed from, the filed XBRL facts, within a tolerance. The unit and the scale have to match the question. Core claim checks are scored on the verdict alone. There is no human review and no grader that reads the explanation. The same answer always gets the same result.
From the grading page: the Momentus current ratio has a reference of 13.70 and a tolerance of 0.005. An answer of 13.703 passes. On the Thunder Power claim check, the reference verdict is no, and an answer of yes fails.
Training and evaluation
The license lets your organization use the zip for training, evaluation, and internal research. There is no held-out split in Core v1. If you train on Core, a score on Core no longer measures the same thing, so keep the tasks you score apart from the tasks you train on.
The 20 tasks in the free sample are public, so a model may have seen them. They are in an earlier sample format and are not part of Core.
What is in the zip
Core is one zip: tasks.jsonl, grade.py, README, datasheet, license and checksums. Core is $299 one-time, on the pricing page.