Amity Agent v3.1: A 3B Active-Parameter Model That Rivals GPT-5 and Claude
Overview
Amity Agent v3.1 is a compact yet powerful agentic model built for customer service automation. With only 3 billion active parameters (30B total), it achieves performance on the τ²-bench leaderboard that rivals — and often surpasses — models that are 100× its size, including GPT-5, Claude Opus 4, and DeepSeek-V3.2.
Building on the foundation of Amity Sigma Thinking v3r (which used Qwen3-4B-Thinking as its base), v3.1 takes a significant step forward by switching the base model to Qwen3-30B-A3B — a mixture-of-experts architecture with 30B total parameters but only 3B active at inference time. Combined with the ROAD optimization framework and GRPO fine-tuning, this produces a model that punches far above its weight class.
Key Results
τ²-bench Leaderboard
The τ²-bench by Sierra Research evaluates conversational AI agents in dual-control customer service environments. Amity Agent v3.1 ranks #9 on Retail (78%) and #4 on Airline (68%) — beating GPT-5, Claude Opus 4, and DeepSeek-V3.2 on the airline domain despite being 100–300× smaller.
Source: taubench.com — Official τ²-bench Leaderboard by Sierra Research. Green bars = Amity Agent v3.1 (3B active params).
The Size Story: 3B vs. the World
What makes these results remarkable is the extreme size disparity between Amity Agent v3.1 and every other model on the leaderboard. Circle sizes below represent estimated model scale. Despite being the smallest by far, Amity Agent v3.1 achieves the highest Airline score among them — except Qwen3-Max-Thinking which is ~67× larger.
To put this in perspective:
- Beats GPT-5 on Airline (68% vs 62.5%) — despite being estimated at ~300× smaller
- Beats Claude Opus 4 on Airline (68% vs 59.6%) — with a fraction of compute
- Beats GPT-4.1 on Retail (78% vs 74%) and Airline (68% vs 56%) — a model estimated at ~500× larger
- Only 1% behind Qwen3-Max-Thinking on Airline (68% vs 69%) — a model estimated at ~67× larger
- The smallest model on the entire leaderboard — no other model under 30B total / 3B active even appears
How It Compares: Cost per 100 Cases
Performance is only half the story. The chart below compares the cost to handle 100 customer interactions across all τ²-bench leaderboard models. Cost is calculated from actual trajectory token counts (input & output) multiplied by each vendor’s published API pricing.
Cost per 100 cases = average of retail and airline per-case costs × 100, each computed as sum of (prompt_tokens × input price + completion_tokens × output price) across all agent turns. Agent tokens only — user simulator (GPT-4.1) cost excluded. Token counts from actual τ²-bench trajectory data. Models without trajectory data use token profile of closest family model. API rates: Qwen3-30B-A3B $0.08/$0.28, GPT-5 $1.25/$10, Claude-3.7-Sonnet $3/$15, Claude Opus 4/4.1 $15/$75, Claude Sonnet 4/4.5 $3/$15, Gemini 3.0 Pro $2/$12, GPT-4.1 $2/$8, o3 $2/$8, o4-mini $1.10/$4.40, GPT-4.1-mini $0.40/$1.60, DeepSeek-V3.2 $0.28/$0.42, Qwen3-Max $1.20/$6, Kimi-k2 $0.60/$2.50 (all per 1M tokens in/out).
Inference Speed Comparison
Beyond cost, speed matters for real-time customer service. The chart below shows output tokens per second for each model, sourced from Artificial Analysis (fastest provider for each model). Amity Agent v3.1 achieves up to 402.5 tok/s on H100 with vLLM.
Speed data from Artificial Analysis (median over 72 hours, fastest provider per model). Amity Agent v3.1 measured at 402.5 tok/s on H100 with vLLM. Reasoning models (o3, o4-mini, GPT-5) include thinking time in TTFT, inflating latency despite fast token output.
Key takeaways:
- 402.5 tok/s on H100 — 2.6× faster than o4-mini (155.8), 4.6× faster than GPT-5 (86.5), and 12× faster than Claude Opus 4 (32.3)
- Sub-second time to first token — compared to 65s for GPT-5 and 20s for o4-mini (reasoning overhead)
- No API dependency — self-hosted speed is consistent and predictable, unlike cloud APIs that vary by provider and load
Training Methodology
Amity Agent v3.1 follows the same proven pipeline as v3r:
- ROAD Optimization — Automated failure analysis and decision-tree injection to patch logic gaps in the agent's reasoning
- Synthetic Data Generation — Multi-turn customer service conversations generated with human-in-the-loop feedback
- GRPO Fine-tuning — Group Relative Policy Optimization on curated interaction trajectories
Implications
The results demonstrate that with the right optimization pipeline, small, efficient models can compete at the highest levels of agentic AI. This has profound implications for deployment:
- Cost efficiency — Self-hosting a 3B active-parameter model costs a fraction of API calls to GPT-5 or Claude Opus
- Latency — Smaller active parameter count means faster token generation
- Accessibility — Can run on a single consumer-grade GPU
- Privacy — On-premise deployment keeps customer data in-house
Links
Collaborate and partner with our AI Lab at Amity Solutions here.
