Project

Multi-Agent Collaboration for Mathematical Reasoning

A local mathematical reasoning workflow helps engineers compare agent architectures through accuracy, generated-token cost, and latency on GSM8K.

View the code on GitHub

Period: December 2025
Status: Research
Stack: PythonJupyterQwen2.5-7B-Instructllama.cppllama-cpp-pythonHugging Face DatasetspandasNumPyMatplotlib

The experiment

One model, four ways to work together. The best team wasn't the biggest.

ModelHands out the same question

SingleAnswers alone

MAVProposes, critiques, refines

JudgeMany drafts, one verdict

One saved run · 200 GSM8K questions · output cap 1,536 per call

What it does

Do agent teams beat one model at math? Only when the budget allows.

Runs one local model four ways: alone, as a propose–critique–refine team, and as judge panels of three and five.
Scores every answer exactly on the same 200 GSM8K questions.
Counts the tokens and seconds each correct answer costs.

92.0%MAV team accuracy, against 79.5% for a single model (cap 1,536)

≈4×the tokens per correct answer: 460 against 116

85.0%for a five-judge panel: more calls, not the best result

01Context

The problem,and my part in it.

The problem

A language model can produce plausible mathematical reasoning and still return the wrong number. A second model call may find a mistake, but it also adds computation and can carry earlier errors into the final response. Selecting among several candidates creates another choice: how much output should each candidate and the selecting Judge receive?

What I did

I designed and ran the experiment end to end on a laptop: one local model, four ways of working together, scored on the same 200 questions.

Ran Qwen2.5-7B locally at 4-bit and reused one model for every role, so differences come from the orchestration, not the model.
Built the single-model, propose–critique–refine and judge-panel pipelines, with an explicit token budget for each.
Scored every answer exactly and measured the tokens and seconds each correct answer cost.
Found where teams pay off: MAV reached 92.0% against 79.5%, while a shared budget made judge panels worse than one model (47.0% against 76.0% at 256 tokens).

02Decisions

How it was built,and why.

Decision 01One base model, several roles

Prompt specialization changes the information passed between calls while holding the model constant. This makes orchestration easier to compare. The trade-off is that every role shares the same underlying model behavior. A mixed-model ensemble would change both model choice and topology; it is outside this comparison.

03Evidence

What the evidence shows,and what it does not.

Explore the saved results: pick an architecture and an output cap, and compare it with the Single baseline.

Implemented precomputed viewer

Precomputed results for GSM8K

Sample: Fixed seeded sample from the GSM8K training split, reused for every final configuration. (n=200)

Runs per configuration: 1

Implemented precomputed viewer — shows published aggregate results only; no live inference is performed in this viewer.

Concept-only illustration — the larger animated orchestration demo remains a concept-only demo and is not shown here.

Architecture

Nominal output cap

Single baseline

Single · cap 1536

79.5%accuracy

Generated tokens per correct: 115.597 tokens per correct
Seconds per correct: 5.363 seconds per correct
Relative error reduction: 0%

Budget mode: per-call

Nominal output cap: 1536

Notes: The nominal cap applies to the architecture's sole model call.

Selected result

MAV · cap 1536

92%accuracy

Generated tokens per correct: 460.478 tokens per correct
Seconds per correct: 23.938 seconds per correct
Relative error reduction: 60.976%

Budget mode: per-call

Nominal output cap: 1536

Notes: The nominal cap applies separately to the Proposer, Critic, and Refiner calls; the two cost multipliers are directly reported.

Complete benchmark table for GSM8K (12 configurations)

Precomputed configurations for GSM8K
Configuration	Architecture	Budget mode	Nominal output cap	Accuracy	Generated tokens per correct	Seconds per correct	Relative error reduction	Notes
Single, per-call cap 256	single	per-call	256	76	118.007	5.287	0	The nominal cap applies to the architecture's sole model call.
MAV, per-call cap 256	mav	per-call	256	90	423.794	21.879	58.333	The nominal cap applies separately to the Proposer, Critic, and Refiner calls.
M3 Judge, loose cap 256	judge-m3	loose	256	86.5	329.543	16.317	43.75	Each Proposer receives the nominal cap; the Judge has a separate 64-token cap.
M5 Judge, loose cap 256	judge-m5	loose	256	90	518.678	24.809	58.333	Each Proposer receives the nominal cap; the Judge has a separate 64-token cap.
Single, per-call cap 750	single	per-call	750	79	114.981	5.098	0	The nominal cap applies to the architecture's sole model call.
MAV, per-call cap 750	mav	per-call	750	91	476.813	24.484	57.143	The nominal cap applies separately to the Proposer, Critic, and Refiner calls.
M3 Judge, loose cap 750	judge-m3	loose	750	86	331.047	16.567	33.333	Each Proposer receives the nominal cap; the Judge has a separate 64-token cap.
M5 Judge, loose cap 750	judge-m5	loose	750	87.5	535.177	26.854	40.476	Each Proposer receives the nominal cap; the Judge has a separate 64-token cap.
Single, per-call cap 1,536	single	per-call	1536	79.5	115.597	5.363	0	The nominal cap applies to the architecture's sole model call.
MAV, per-call cap 1,536	mav	per-call	1536	92	460.478	23.938	60.976	The nominal cap applies separately to the Proposer, Critic, and Refiner calls; the two cost multipliers are directly reported.
M3 Judge, loose cap 1,536	judge-m3	loose	1536	84	341.458	16.918	21.951	Each Proposer receives the nominal cap; the Judge has a separate 64-token cap.
M5 Judge, loose cap 1,536	judge-m5	loose	1536	85	552.976	26.52	26.829	Each Proposer receives the nominal cap; the Judge has a separate 64-token cap.

76.0%Single

79.0%

90.0%MAV

91.0%

86.5%Judge x3, loose

86.0%

90.0%Judge x5, loose

87.5%

How it was measured

The final comparison uses the same 200-question GSM8K training-split sample for every architecture. Nominal output caps are 256, 750, and 1,536 tokens. Single and MAV receive it per call; loose Judge receives it per Proposer plus 64 tokens for the Judge.

MAV's accuracy rises across the cap sweep, while both loose Judge variants decline. Five loose Proposers score higher than three at each cap and consume more generated tokens. Increasing the cap therefore does not improve every architecture.

The shared-budget variants provide a negative allocation result. At cap 256, Judge x3 scores 47.0% and Judge x5 scores 35.0%, below Single's 76.0%. At shared cap 750, Judge x5 scores 75.5%, below Single's 79.0%. These are contextual design findings rather than selectable viewer configurations.

At per-call cap 1,536, Single uses 115.60 generated tokens and 5.36 seconds per correct answer. MAV's corresponding 460.48 tokens and 23.94 seconds explain the cost behind its higher accuracy. The measurements use model-tokenizer counts and summed wall-clock call durations, normalized by correct answers.

About this evidence

The portfolio viewer and animated architecture comparison use precomputed aggregate results.
Live inference and transcript illustrations remain concepts.
Final results represent one stochastic run per configuration on the fixed 200-question GSM8K training-split sample.
Caps apply per call to Single and MAV, per Proposer plus 64 Judge tokens in loose Judge, and across calls in strict Judge.
Shared-budget Judge results are contextual findings and are excluded from selectable viewer configurations.
Paired testing and repeated-run uncertainty analysis are absent.
Null measurements remain unpublished.

04More judges

Splitting one budget across more judgesmade the answers worse.

47.0%

Judge ×3 with a shared 256-token budget, against 76.0% for one model.

35.0%

Judge ×5 on the same shared budget: more calls, each with a smaller share.

75.5%

Judge ×5 at a shared 750 tokens, still below one model's 79.0%.

Each extra call gets a smaller share of the same budget. With the budget per call instead, MAV is the team that wins.

Agents at work in production: see the system that builds this site