Model Distillation for LLMs: Cut Inference Cost Without Losing Quality

Key Takeaways

• Model distillation trains a small, cheap student model on the outputs of a large, expensive teacher model — the student inherits the teacher's reasoning on your specific domain without the teacher's inference cost
• The three-phase pipeline is: (1) capture production calls via observability tooling, (2) clean and convert to JSONL fine-tuning format, (3) fine-tune the student on Azure OpenAI or OpenAI and gate rollout with evals
• For a voice AI platform at 10K calls/day, distilling GPT-4.1 into GPT-4o-mini cut per-call inference costs by 70% — from ~$0.006 to ~$0.0018 — with no quality regression across 80K–100K training examples
• A 90/10 hybrid deployment (90% student, 10% teacher) with Langfuse observability is the safe production steady-state: cost savings at scale, teacher as live quality floor

Model distillation is one of the highest-leverage interventions available once you have a production LLM in service. It is not a research technique — it is a cost-engineering decision: you have a teacher model that is expensive and accurate; you want a student model that is cheap and nearly as accurate. Done correctly, the student model matches the teacher on your actual call distribution while costing a fraction as much per inference.

The catch is "done correctly." Teams that treat distillation as a one-step fine-tuning job miss the data quality problem — and a poorly filtered training set produces a student model that looks fine on benchmarks but degrades on the long tail of real calls. This post covers the full production playbook.

Model distillation for LLMs uses the teacher model's production outputs as training labels for a smaller student model — not a general dataset, but your specific domain's conversations. At 10K calls/day, a voice AI platform cut inference costs 70% by distilling GPT-4.1 into GPT-4o-mini using 80K–100K cleaned production examples, progressive A/B rollout, and eval gates at every stage.

What Model Distillation Does (and Doesn't Do)

Model distillation in the LLM context is a specific form of supervised fine-tuning where the training signal comes from a stronger model rather than human labels. The teacher model — typically a large, expensive frontier model — generates outputs for your domain's inputs. The student model — a smaller, cheaper model — is fine-tuned to reproduce those outputs. The result is a student that has, in effect, learned to approximate the teacher's reasoning on your specific input distribution.

What separates distillation from generic fine-tuning is the quality of the training signal. Human-labeled training data for LLM fine-tuning is expensive to produce at scale and inconsistent in quality. Teacher model outputs are cheap to generate at scale (you are already running the teacher in production) and consistent in quality — the teacher's outputs represent exactly what you want the student to produce. This is why distillation works so well for production systems where you already have a working teacher: the training data is implicit in every production call the teacher handles.

What distillation does not do: it does not make the student smarter than the teacher. If the teacher model makes errors on a class of inputs, the student will learn those errors too unless you filter them out. This is why data cleaning is the highest-leverage phase of the pipeline, not fine-tuning itself. A clean, filtered dataset of 80K teacher outputs produces a student model that performs reliably at scale. An unfiltered dataset of the same size produces a student that overfits on common patterns and fails on edge cases.

Distillation vs Fine-Tuning vs RAG: When to Use Each

The decision framework is simple once you identify the actual problem:

• RAG: the model's problem is knowledge — it does not have access to information it needs. Add retrieval; distillation does not help.
• Fine-tuning on your own labeled data: you have human-curated examples of desired outputs and want to steer the model's style, format, or behaviour. Use fine-tuning directly.
• Distillation: you have a working, expensive teacher model that already produces good outputs for your domain. You want a cheaper model that matches the teacher's quality on your specific distribution. Use distillation.

The key signal for distillation is that you are already running a strong teacher model in production and paying for it at scale. The teacher's outputs are your training data — you are not paying for data collection or human labeling. The only cost is the fine-tuning run and the evaluation infrastructure, which amortises quickly at high call volumes. For a platform at 10K calls/day, a $0.0042 per-call saving means the pipeline pays for itself within weeks.

Building the Distillation Pipeline: Three Phases

Phase 1: Observability and Data Collection

Before you can distill, you need a structured stream of teacher model outputs from production. The standard approach is an observability layer — Langfuse is the most common choice for LLM pipelines — that captures every production call: input messages, output completions, latency, and model metadata.

The observability layer serves two purposes beyond data collection. First, it provides the quality signal for data filtering: calls flagged with hallucination scores or low quality ratings are excluded from the training set. Second, it becomes the live monitoring layer for the A/B rollout, tracking quality scores and latency deltas between teacher and student in real time.

from langfuse import Langfuse
import json

client = Langfuse()

def export_training_examples(start_date, end_date, output_path):
    examples = []
    page = 1

    while True:
        traces = client.fetch_traces(
            page=page,
            limit=500,
            from_timestamp=start_date,
            to_timestamp=end_date
        )
        if not traces.data:
            break

        for trace in traces.data:
            # Skip traces with hallucination flags or missing data
            if trace.scores:
                hallucination_score = next(
                    (s.value for s in trace.scores if s.name == "hallucination"), 0
                )
                if hallucination_score > 0:
                    continue

            messages = trace.input.get("messages", [])
            response = trace.output.get("content", "")
            if not messages or not response:
                continue

            examples.append({
                "messages": messages + [{"role": "assistant", "content": response}]
            })

        page += 1

    with open(output_path, "w") as f:
        for ex in examples:
            f.write(json.dumps(ex) + "\n")

    print(f"Prepared {len(examples)} training examples")
    return len(examples)

Target volume: 80,000–100,000 examples for a domain with high turn diversity, such as a voice AI handling varied sales or support conversations. Fewer examples produce a narrower student that regresses on edge cases. Deduplication matters — remove near-identical examples so the student generalises rather than memorising a few high-frequency patterns.

Phase 2: Fine-Tuning the Student Model

With a clean JSONL file, the fine-tuning job runs through the OpenAI or Azure OpenAI fine-tuning API. The student model is gpt-4o-mini — significantly cheaper per token than GPT-4.1, with a capability ceiling that is more than sufficient for domain-specific tasks once it has been trained on teacher outputs.

import openai
import time

def run_fine_tuning_job(training_file_path, n_epochs=3):
    # Upload the training file
    with open(training_file_path, "rb") as f:
        training_file = openai.files.create(file=f, purpose="fine-tune")

    print(f"Uploaded training file: {training_file.id}")

    # Submit fine-tuning job
    job = openai.fine_tuning.jobs.create(
        training_file=training_file.id,
        model="gpt-4o-mini-2024-07-18",
        hyperparameters={
            "n_epochs": n_epochs,
            "batch_size": 16,
            "learning_rate_multiplier": 1.0
        }
    )

    # Poll for completion
    while job.status not in ("succeeded", "failed", "cancelled"):
        time.sleep(60)
        job = openai.fine_tuning.jobs.retrieve(job.id)
        print(f"Job {job.id}: {job.status}")

    if job.status == "succeeded":
        print(f"Fine-tuned model: {job.fine_tuned_model}")
        return job.fine_tuned_model
    else:
        raise RuntimeError(f"Fine-tuning failed: {job.status}")

A first fine-tuning run with 80K+ examples on gpt-4o-mini typically takes 4–8 hours on Azure OpenAI or OpenAI. Run at least two iterations: the first establishes the baseline student quality; a second run on a filtered subset (removing examples where the first student's own outputs diverge most from the teacher) often tightens quality on edge cases.

Phase 3: Eval Gates and Progressive Rollout

Never hard-cut inference at production scale. The safe pattern is a progressive A/B rollout with quality gates at each stage:

10% → 25% → 50% → 75% → 90%

Each stage gate requires the student model to pass three eval criteria before the next traffic increment is approved:

1. Hallucination detection — automated checks against ground-truth responses or factual anchors known to be in scope
2. Quality scoring — conversation quality measured against the teacher baseline; a LLM-as-judge rubric scored against teacher outputs works well
3. Latency tracking — p50 and p95 latency must remain within acceptable bounds (GPT-4o-mini is typically faster than GPT-4.1, but check under your actual concurrent load)

The final steady-state is a 90/10 hybrid: 90% of traffic served by the fine-tuned student, 10% retained on the teacher. The 10% teacher allocation serves as a live quality floor — if the student degrades on new patterns before the next retraining cycle, the gap is visible in Langfuse before it affects the full call volume.

What the pipeline we built achieved

Prodinit built this exact pipeline for a high-volume voice AI platform processing 10,000 calls per day on Azure OpenAI GPT-4.1. At that scale, inference costs were becoming a ceiling on growth — projected 3–5x volume growth made the status quo unsustainable before any scale-related optimisation could close the gap.

The engagement took 10 weeks across four phases: Langfuse integration and observability instrumentation (weeks 1–2), data cleaning and JSONL pipeline (weeks 3–5), fine-tuning loop and evals framework (weeks 6–8), and progressive A/B rollout to the 90/10 steady state (weeks 9–10).

The distillation pipeline processed 80,000–100,000 training examples per run, cleaned from production Langfuse data. The progressive rollout completed across all 5 stages with zero rollbacks — every stage gate passed on first attempt. The platform now handles 10,000+ calls/day at the new cost structure, with infrastructure headroom designed for 3–5x growth.

The compounding effect is the less obvious benefit: quarterly retraining cycles continuously improve the student model as new production data accumulates. Each cycle re-filters against the latest teacher outputs and raises the quality bar — the student model gets better over time without requiring manual labelling or teacher access beyond what production already generates.

The full architecture and results — Langfuse integration, data pipeline, fine-tuning setup, A/B framework, and cost breakdown — are documented in the Model Distillation case study. For the upstream decision of whether fine-tuning or RAG is the right approach for your specific problem, the fine-tuning vs RAG framework covers that decision in detail.