Tag
Vendors advertise 200,000-token context windows. The number production agents can actually use reliably is closer to 130,000 — and closing that gap is a compression-architecture decision, not a bigger-window one.
LLM evaluation in production is three different problems bundled into one confused setup. Here's how to separate them, and what each one actually needs.
After quantisation, KV caching, and model cascade, most teams find the bill is still too high. The reason: the biggest cost drivers for most LLM features are product decisions, not infrastructure ones.
Practical guides, product updates, and compliance notes — straight to your inbox. No fluff.
Newsletter is opening soon. We'll switch this on once we've got our first issue ready.