The Definitive LLM Selection & Benchmarks Guide

Task-to-Model Matching Framework

5.1 Task Category Matrix

5.2 Use Case Deep Dives

Code Generation

The coding landscape has distinct tiers (note: seven models now score within 2.8 points of each other on SWE-bench Verified):

1. Agentic coding (fix real bugs in real repos): Claude Opus 4.6 (80.8% SWE-bench), Gemini 3.1 Pro (80.6%), MiniMax M2.5 (80.2%), or GPT-5.4 (~80%)
2. Everyday coding assistance: Claude Sonnet 4.6 (79.6%), Gemini 3 Flash (78%), or GPT-5.4 -- excellent quality at lower cost
3. Code completion/autocomplete: Smaller models fine -- Qwen3-Coder, DeepSeek-Coder, Step-3.5-Flash
4. Open-source self-hosted: MiniMax M2.5 (80.2%, now matching proprietary frontier), MiniMax M2.7 (78%), GLM-5 (77.8%), Kimi K2.5 (76.8%), Qwen 3.5 (76.4%)

RAG / Document Processing

Best models for RAG need three capabilities: knowledge breadth (MMLU-Pro), reasoning ability (GPQA, BBH), and instruction following (IFEval).

Recommended setups:

• Best overall: Qwen 3.5 (256K context, native vision, Apache 2.0) or Qwen3-30B + Qwen3-Embedding-8B
• Maximum quality: Claude Opus 4.6 or Gemini 3.1 Pro (1M+ context)
• Budget: Gemini 3.1 Flash Lite ($0.25/$1.50) or Gemini 3 Flash ($0.50/$3.00) + RAG framework
• Resource-constrained: Phi-4 (14B, runs on consumer GPU)

Creative Writing

No automated benchmark reliably measures creative writing quality. Arena Elo (Creative Writing subcategory) and human evaluation are the only reliable signals.

Current ranking (subjective, based on practitioner reports):

1. Claude Opus 4.6 -- best prose rhythm, subtext handling, consistent tone
2. GPT-5.4 -- more structured, better at maintaining complex narrative frameworks
3. Gemini 3.1 Pro -- capable but less literary; better for informational content

Evaluation Methodology After Shortlisting

8.1 Designing Evaluation Datasets

Dataset composition targets:

Dataset curation strategies:

1. Manual curation (highest quality): Subject matter experts create 50-200 test cases aligned with product goals. Include high-priority workflows, known failure modes, and edge cases.
2. Production sampling: Pull real prompts and responses from production logs. Provides grounded, real-world data. Best for identifying drift and tracking quality over time.
3. Synthetic generation: Use a strong LLM to generate test cases automatically. Fast but requires human review. Best for scaling coverage after manual curation establishes the pattern.
4. Gold Standard Questions (GSQs): Labeled dataset with expert-verified ground truth answers. Most reliable for automated scoring but expensive to create.

Minimum viable evaluation set: 100-200 examples covering all categories above. For high-stakes applications, aim for 500+.

8.2 LLM-as-Judge Approaches

LLM-as-Judge uses a strong model (typically Claude Opus or GPT-5) to evaluate outputs from candidate models.

For each test case:
  1. Send input to candidate model --> get response
  2. Send (input, response, rubric) to judge model --> get score + reasoning
  3. Aggregate scores across all test cases

Best practices:

Key frameworks:

• DeepEval -- 50+ research-backed metrics including G-Eval, hallucination detection, answer relevancy, task completion
• Langfuse -- LLM-as-judge integration with production tracing
• Arize Phoenix -- Open-source evaluation with hallucination-specific judges
• Amazon Bedrock Model Evaluation -- Managed LLM-as-judge on AWS

8.3 A/B Testing Frameworks

[Offline Evaluation]                    [Online A/B Testing]
      |                                        |
      | Identify promising                     | Validate with
      | candidates on static dataset           | real users
      |                                        |
      v                                        v
 Top 2-3 candidates  ------->  Deploy to % of traffic
                                               |
                              Measure: completion rate,
                              user satisfaction, task success
                                               |
                              Feed challenging examples
                              back into offline eval dataset
                                               |
                              [Continuous Improvement Loop]

A/B testing checklist:

• Define primary success metric before starting (e.g., task completion rate)
• Calculate required sample size for statistical significance
• Randomize user assignment to prevent selection bias
• Run for minimum 2 weeks to capture variance
• Control for confounders (time of day, user type, input complexity)
• Measure secondary metrics (latency, cost, user satisfaction)
• Document all prompt versions used with each model

8.4 Automated Evaluation Pipelines

[Test Dataset]
      |
      v
[Evaluation Runner] -- Sends inputs to all candidate models in parallel
      |
      v
[Response Collector] -- Stores all (input, model, response) tuples
      |
      v
[Metric Calculator]
      |
      |-- Deterministic Metrics: exact match, regex, JSON schema validation
      |-- Statistical Metrics: BLEU, ROUGE, BERTScore
      |-- LLM-as-Judge Metrics: quality, relevance, hallucination
      |-- Latency Metrics: TTFT, total time, tokens/second
      |-- Cost Metrics: actual cost per request
      |
      v
[Dashboard / Report Generator]
      |
      v
[CI/CD Integration] -- Block deployment if metrics drop below threshold

Tools for automated evaluation:

8.5 Metrics Beyond Accuracy

Frequently Asked Questions

Related Resources

Explore our calculators, assessments, and guides to apply these insights to your organization.