Datasheet.

Sources

Tasks are built from the SEC Financial Statement Data Sets, the bulk XBRL files the Commission publishes. Tickmark Bench is not a product of the SEC.

A filing is kept when its filing date is on or after 2026-04-01. The statement period the task is about must end on or after 2026-01-01. A prior-year column inside a later filing may be earlier than that. Restatement tasks may cover periods ending on or after 2025-06-30. Those dates are the filters in the generator. There is no published end date.

Core is the real_v1 verified batch: 149 tasks from 67 companies, with filing dates from 2026-04-07 to 2026-09-29. That file is the shipped Core set. A task in it is kept only when every reference fact matches the filing's XBRL instance exactly: exact period dates, dimensions, and unit. In that set, every reference traces to cells in the filing.

Filings are from Q2 2026 or later.

Task families

Four families are generated today.

• single_number. One figure, often a computed ratio or margin, from one filing period.
• multi_period. A change across periods, or a figure from a segment.
• claim_check. A yes or no verdict.
• comparison. The same metric for more than one issuer.

In the Core set, every reference traces to cells in the filing. A count of tasks in each family is not published.

Grading method

Core is scored entirely by grade.py. Numeric answers are scored within the stated tolerance, and claim checks on the verdict alone. The official score is score = passed/149. grade.py's passed count is that numerator, and grade.py's total is the official score.

Numeric answers are checked in code against a reference value taken from, or computed from, the filed XBRL facts, within a tolerance. The unit and the scale have to match the question. A task fails when the number is outside the tolerance or the verdict does not match.

Core claim checks are scored on the verdict alone. yes or no must match the reference. The explanation is not scored.

A Core task is kept only when every reference fact matches the filing's XBRL instance exactly: exact period dates, dimensions, and unit. Core is the real_v1 verified batch.

Tasks ship in the Prime Intellect verifiers format.

Solve rates

Solve rates: not yet published. We publish only measured numbers, with model, attempts and date. The distribution of per-task solve rates is not measured.

Quality figures

The reference-defect rate is not measured.

Contamination

The committed sample of 20 tasks stays public at downloads/sample-tasks.jsonl, and four of those tasks are on the sample page. There is no held-out split in Core v1.