CharXiv Reasoning (with tools)
CharXiv Reasoning (with tools) · the tool-augmented variant of CharXiv Reasoning. Models can use code execution, image processing, and other tools to analyze charts from arXiv papers. Claude Mythos Preview scores 93.2% with tools vs 86.1% without · demonstrating how tool use dramatically improves visual scientific reasoning. The gap between tool-augmented and bare performance is a key signal for agent capability.
Tool use adds 7+ percentage points consistently across models. Mythos leads at 93.2%, up from 86.1% bare.
Scoring: Same accuracy metric as CharXiv Reasoning. Tool use is optional per question · the model decides when to invoke code execution vs answering directly.
Full rankings
3 models tested · sorted by score · includes 3 verified scores
| # | Model | Score |
|---|---|---|
| 1 | 93.2 | |
| 2 | 91.0 | |
| 3 | 84.7 |
Score distribution
Where models cluster
Correlated benchmarks
Pearson r · original research
How it works
Evaluation methodology
Same chart reasoning tasks as CharXiv Reasoning, but models have access to code execution, image processing, and computational tools. Models can write Python to extract data from charts, compute statistics, and verify their visual interpretations programmatically. This tests the full agent loop: perceive the chart, reason about what computation is needed, write and execute code, then answer.
Industry relevance
Why teams track this benchmark
The tool-augmented variant reveals which models can effectively combine vision with computation. A 7+ point improvement from tools is not free · it requires the model to know when and how to delegate to code. This is the agentic multimodal frontier.
Practical takeaways
By role
Always benchmark your chart analysis pipeline with tools enabled. The 7-point lift is real and consistent. Ship with code execution access.
Tool-augmented performance above 90% means AI-powered research tools can reliably replace manual chart reading. Watch adoption curves in biotech and fintech.
The tool-augmented delta varies by model architecture. Studying which models benefit most from tools reveals differences in instruction following and code generation quality.
Frequently asked
About CharXiv Reasoning (with tools)
What does CharXiv Reasoning (with tools) measure?
CharXiv Reasoning (with tools) · the tool-augmented variant of CharXiv Reasoning. Models can use code execution, image processing, and other tools to analyze charts from arXiv papers. Claude Mythos Preview scores 93.2% with tools vs 86.1% without · demonstrating how tool use dramatically improves visual scientific reasoning. The gap between tool-augmented and bare performance is a key signal for agent capability. 1 AI models have been tested on it. Scores range from 93.2 to 93.2 out of 100.
Which model leads on CharXiv Reasoning (with tools)?
Claude Mythos Preview from Anthropic leads CharXiv Reasoning (with tools) with a score of 93.2. The median score across 1 tested models is 93.2.
Is CharXiv Reasoning (with tools) saturated?
No · the top score is 93.2 out of 100 (93%). There is still meaningful room for improvement on CharXiv Reasoning (with tools).
What makes CharXiv Reasoning (with tools) distinctive?
CharXiv Reasoning (with tools) is a multimodal benchmark with limited overlap to the rest of the catalog · it measures capabilities that are not well-covered by other benchmarks we track.
How often is CharXiv Reasoning (with tools) data refreshed?
BenchGecko pulls updates daily. New model scores on CharXiv Reasoning (with tools) appear as soon as they are published by Epoch AI or the model provider.
- Category
- Multimodal
- Max score
- 100
- Scoring
- Same accuracy metric as CharXiv Reasoning. Tool use is optional per question · the model decides when to invoke code execution vs answering directly.
- Models
- 3
- Updated
- 2026-04-07
“The with-tools variant is the real benchmark. Any production deployment of a research assistant will have tool access. Bare scores are academic; tool-augmented scores predict production quality.”
Top on CharXiv Reasoning (with tools)
Claude Mythos Preview · 93.2Claude Opus 4.7 · 91.0Claude Opus 4.6 · 84.7More multimodal benchmarks
Same category · related evaluations