Shipping a large language model demo is easy. Shipping one that serves millions of requests reliably, cost-effectively, and at low latency is a completely different engineering challenge.
Over the past 18 months at Mentneo, we have learned hard lessons about production LLM deployment. This post documents the most important of those lessons.
Latency Is a Product Decision, Not Just an Engineering Metric
We began by optimizing for throughput — maximizing tokens per second across our cluster. We quickly learned this was the wrong primary metric. Users feel latency, not throughput. The perceived quality of an AI response is heavily influenced by time-to-first-token (TTFT).
Our current target for Mentneo Pro is P50 TTFT under 400ms and P99 under 1.5 seconds. Achieving this required speculative decoding, KV cache optimization, and aggressive model parallelism tuning.
Speculative Decoding Changed Everything
The single highest-impact optimization we implemented was speculative decoding with a small draft model. By generating candidate tokens with a 1B parameter draft model and verifying them with our full model, we achieved 2.8x speedup on typical chat workloads with no degradation in output quality.
Cost Optimization Cannot Wait
At scale, inference cost compounds quickly. Our cost optimization journey involved quantization (INT8 for non-critical paths), dynamic batching, and intelligent routing to smaller models for simple queries.
Reliability Engineering for LLMs
LLMs fail in novel ways that traditional software engineering is not prepared for. We built a comprehensive reliability stack including semantic consistency monitors, output validation layers, fallback routing, and real-time anomaly detection on output distributions.