Category
Production AI, agents, evals, and the systems around modern LLMs.
Eight major AI agent benchmarks hit 73-100% scores without an agent solving the underlying task. A second 2026 study found the same gap honestly: a 37% lab-to-production drop and a 50x cost swing.
A 2025 UC Berkeley taxonomy of multi-agent LLM failures found 14 distinct failure modes across three categories, and none of them is about model quality.
Automatic context compaction is now a platform feature across every major model provider. It solves the token-budget problem completely, and the state-loss problem only if someone configures it well.
A new survey puts AI agent rollback rates at 75%, climbing to 81% at the most mature organisations. That's not proof governance fails — it's proof governance is the only thing catching the failure at all.
Vendors advertise 200,000-token context windows. The number production agents can actually use reliably is closer to 130,000 — and closing that gap is a compression-architecture decision, not a bigger-window one.
A sprint with 70% AI-assisted code looks fine until week six. Then an incident traces a cascade failure back to a pattern the AI repeated across four modules. What engineering teams are actually learning.
MCP's auth is optional, tool definitions are mutable, and session-scoped permissions create ambient authority. Three attack classes, real CVEs, documented incidents — here's what to lock down before you ship.
LLM-as-a-judge evals look reliable in the test harness. Here's what breaks after months in production: calibration drift, noisy decision boundaries, cascade failures in multi-step pipelines, and the meta-evaluation trap.
Constrained decoding eliminated JSON syntax failures in LLM structured output. The reliability problem has moved to semantics: four failure classes that valid JSON hides, and the runtime patterns that catch them.
MIT's Project NANDA analysed 300 enterprise AI deployments and found 95% delivered no measurable P&L impact. The reason is almost never the model. It's task structure.
Vendor benchmarks leave out the two cost items that usually flip the self-hosting decision: engineering overhead and the model-update cycle. Here is the honest break-even analysis.
Chroma's 2025 research tested 18 frontier models and found every one degrades as context grows. This is what context rot means for production engineering decisions — and the specific patterns that address it.
Practical guides, product updates, and compliance notes — straight to your inbox. No fluff.
Newsletter is opening soon. We'll switch this on once we've got our first issue ready.