How to evaluate a language model on SEC filings.

That is the short answer. The steps below show each part with tasks from the Tickmark Bench free sample, so you can check every number against the filing yourself.

1. Start from filed facts, not from prose

Companies tag the numbers in their financial statements with XBRL. Each tagged fact carries a concept, a period, a unit, and sometimes dimensions such as a segment. A question built on those facts has one checkable answer. A question built on a paragraph of prose often has several defensible answers.

In the sample, the Mercer International task asks for the Lumber segment's share of revenue. Its reference is computed from two tagged facts: revenue with the dimension ProductOrService=Lumber, and total revenue, both for the quarter ended March 31, 2026. The answer is 1228 basis points.

2. Pin the period

A 10-Q usually shows the same period of the prior year next to the current one, and it can show both a quarter and a year-to-date figure. A question that says only "revenue in the second quarter" leaves room for the wrong column. Name the period end date and the length of the period, and name the filing by its accession number.

Every Tickmark Bench task names the company, its CIK, the form, the accession number, and the filing date before it asks anything.

3. Fix the unit, the scale, and the rounding

Filings report some figures in thousands and others in millions, and a ratio can be written as a decimal, a percent, or basis points. Say which one you want, and how to round. The Mercer task says:

For the quarter ended March 31, 2026, what share of consolidated revenue from contracts with customers came from the Lumber segment (ProductOrService=Lumber)? Express the answer as basis points rounded half-up to the nearest basis point (unit "basis_points", scale "ones").

4. Ask for structured output

Ask for JSON in a fixed shape, such as {"value": "1234", "unit": "basis_points", "scale": "ones"}. Then the grader reads a field instead of guessing at a sentence. For a yes or no question, ask for {"verdict": "yes"} or {"verdict": "no"}.

5. Grade in code, within a stated tolerance

Compare the number with the reference in code, and say how close counts. In the sample, the Momentus current ratio task has a reference of 13.70 and a tolerance of 0.005. An answer of 13.703 passes. An answer of 13.71 fails. For a claim check, compare the verdict alone: the Thunder Power claim that the current ratio was at least 2 is false, because the filed figures give 1.49.

Core is scored entirely by grade.py. Numeric answers are scored within the stated tolerance, and claim checks on the verdict alone. The grading page has the details.

6. Include questions that span more than one filing

A comparison question makes the system find the same metric in two filings and line up the periods. In the sample, one task asks for Data443 Risk Mitigation's gross margin minus Apptech Payments' gross margin for the quarter ended March 31, 2026. The reference is -3929 basis points.

7. Report what you measured

Report the score as passed over total, with the model, the number of attempts, and the date. Keep the questions you publish apart from the ones you score on, because a published question can end up in training data.

Try it

Download the free sample: 20 tasks, five of each type, in an earlier sample format. Core, the graded set of 149 verified tasks with grade.py, is on the pricing page. For how other financial benchmarks compare, see FinanceBench, FinQA, and TAT-QA compared.