發生什麼事
@SakanaAILabs says JFinBench draws on the 2026 editions of six Japanese financial qualification exams, including the Certified Public Accountant exam, the Certified Tax Accountant exam and Grade 1 of the FP Skills Test. Its 321 questions include multiple-choice items, tax calculations, written responses and chart interpretation. A coding agent converted the exam PDFs, and people checked each question by hand in about 12 hours (🗣 Self-report).
The benchmark was used to evaluate 12 of the latest LLMs. Sakana AI says the main reason for lost points was a lack of knowledge about Japan’s systems, rather than calculation errors (🗣 Self-report). Open-weight models are catching up with proprietary ones on major benchmarks, but the gap remains large on JFinBench.
01Sources
- @SakanaAILabs: Sakana AI’s official account; post published 2026-11-16 at 08:53 UTC