Is AI Good at Math? We Gave Three Free Models the Same Six Problems
Short answer: the bigger models are now very good at everyday math, and a smaller one can still be confidently wrong. On 25 October 2026 we sent one question with six problems to the three free models our tool uses by default: Google's Gemini 3.6 Flash, OpenAI's open-weight gpt-oss-120b (served by Groq) and Mistral's Ministral 14B.
The six problems
- A notebook and a pen cost ₹130 together, and the notebook costs ₹100 more than the pen. How much is the pen? (₹15 — the classic trap answer is ₹30.)
- A price rises 20% and then falls 20%. What is the overall change? (A 4% decrease, not zero.)
- What is 487 × 396? (192,852)
- What is the probability of at least one six in four rolls of a fair die? (671/1296, about 0.518)
- A train leaves at 9:40 and covers 245 km at 70 km/h. When does it arrive? (13:10)
- How many times does the digit 7 appear when you write out the numbers 1 to 100? (20 — 77 alone contains two of them.)
We checked every answer by hand before scoring.
The scores
| Model | Score | Mistakes |
|---|---|---|
| Gemini 3.6 Flash | 6 / 6 | None |
| gpt-oss-120b | 6 / 6 | None |
| Ministral 14B | 3 / 6 | Multiplication, probability, train time |
| Our combined answer | 6 / 6 | None |
How the smaller model went wrong
- Arithmetic: it said 487 × 396 = 192,942, which is 90 too high, and its "breakdown" contains steps that don't lead to its own answer.
- Probability: it wrote the correct formula, 1 − (5/6)4, and then reported 51/625, about 0.08. The right value is about 0.52: a better-than-even chance, not roughly one in twelve.
- Time: it worked out 9:40 + 3 h 30 min = 13:10 correctly, then said the time "rounds to 13:15 with standard time accounting". There is no such rule. It talked itself out of the right answer.
All three mistakes came with confident, tidy-looking working. Nothing in the tone suggested these answers were less reliable than the correct ones.
Why this happens
Language models produce an answer one piece of text at a time; unless a calculator tool is connected, they aren't running the numbers. Larger models have absorbed enough worked examples to get standard problems right nearly every time. Smaller models are faster and cheaper but slip more often on multi-step arithmetic. The famous trick questions are less of a test than they used to be: all three models handled the ₹15 pen and the 4% decrease, patterns they have seen many times.
What the combined answer did
Our tool's combined summary got all six right. With two models agreeing on every answer, the summarizer went with the majority, and the smaller model's three errors dropped out. That's the main benefit of asking several models at once, but it only works while most of them are right. On a genuinely hard problem they can all be wrong together, and a majority vote won't save you.
How to use AI for math safely
- Read the working, not just the answer. The train-time mistake was visible in the model's own line of working.
- Re-check exact numbers — multiplication, percentages, money — with a calculator or a spreadsheet.
- Sanity-check the size of the answer. An 8% chance of at least one six in four rolls should feel wrong.
- Ask more than one model. When the answers disagree, you know exactly which problem to double-check.
This was one question on one day, and results vary between runs. But the pattern matches what we see elsewhere: models are most dangerous when they're wrong in the same calm tone they use when they're right. We also tested the same three models on code — see Which Free AI Is Best for Coding?