Observability bills don't explode from traffic. Metric cardinality does, and LLM telemetry is the fastest way to trip it

The bill triples, and almost nobody checks cardinality first

A mid-market SaaS team ships an LLM-powered support feature in Q1. By Q2, the observability bill is up 40 to 200 percent, depending on how many custom metrics the feature added. The instinctive read is "traffic went up" or "the vendor changed the pricing again." Neither is usually true.

Data volume typically accounts for only 5 to 10 percent of an observability bill at most vendor pricing tiers. The number that actually moves the invoice is metric cardinality: the count of unique time series a metric produces once every combination of its label values is counted separately.

That distinction matters because the fix most teams reach for first, migrating off the expensive vendor, doesn't touch the actual driver. It just moves the same problem onto a cheaper meter.

What metric cardinality actually is, and why it prices the bill

A metric like http_requests_total isn't one thing being counted. Every unique combination of its label values, status="200", route="/api/users", region="us-east", for instance, is stored, indexed, and queried as its own separate time series. Add a label that takes 1,000 unique values and you don't add 1,000 data points. You multiply your entire existing series count by 1,000.

In cloud-native environments, this compounds fast: cardinality has been documented climbing from 20 thousand unique series to 800 million as microservices architectures multiply the number of services each emitting their own metrics. Once cardinality is high, indexes bloat, queries fan out across too many series, memory pressure rises, and the bill scales with that series count, not with how much traffic actually generated it.

Where an LLM feature quietly detonates it

OpenTelemetry's GenAI semantic conventions define a standard vocabulary for AI workloads, so a span from a LangChain agent and a span from a raw API call look the same. The attributes include gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and gen_ai.response.finish_reasons. When content recording is turned on, they extend to full request and response text.

Those attributes were designed for spans. The trap is that teams already thinking in Prometheus or Datadog labels copy them onto metrics out of habit. Model name alone is a handful of values, harmless. But finish_reason times model times prompt_version times user_tier times endpoint compounds quickly, and if anyone tags a request ID or session ID onto a metric, which is an easy mistake when instrumentation gets copy-pasted across a codebase, that one line creates a brand-new, permanent time series on every single request.

The GenAI conventions were still in Development status through most of 2026, with several attributes marked experimental. That means the specification itself doesn't yet carry built-in guidance about which of these fields belong on a metric versus a span, so teams are making that call themselves, usually under deadline pressure, usually by copying whatever the last service did.

Run the arithmetic on a feature that looks modest on paper. Five supported models, eight finish reasons, four user tiers, and a prompt_version that rolls forward every week: that alone is 5 × 8 × 4 × 52, or 8,320 distinct series for one latency metric, before a single request has been served. Add a per-tenant label for a product with 200 active workspaces and the same metric is now pushing past 1.6 million series. Add a user or session ID, which happens more often than teams admit, and the series count stops being a number worth calculating. It becomes one new series per request, forever, because nothing ever repeats.

The "just migrate to OpenTelemetry" trap

Several vendors have framed 2026 as the year to escape Datadog: move to a self-hosted OpenTelemetry Collector feeding the open-source Grafana LGTM stack, and cut the telemetry bill by up to 60 percent. That number is real for teams whose problem actually was vendor markup on volume. It does nothing for a team whose problem is cardinality, because cardinality is a property of the labels, not the vendor.

Migrate the same label set, unchanged, onto self-hosted Prometheus or Mimir, and the same unbounded tags create the same explosion. The difference is where the pain shows up. Instead of a line item on a SaaS invoice, it's memory pressure, ingestion stalls, and query timeouts on infrastructure the team now owns directly. The senior engineer who used to spend a chunk of their week negotiating the Datadog bill spends it tuning cardinality limits on a collector config instead.

The trajectory shows up in what teams report after the fact: an organisation running around 50 engineers can go from an $8,000-a-month bill in year one to $90,000 a month by year three, and the growth compounds through ordinary work, a new metric here, a new tag there, none of it reviewed as a cardinality decision at the time it was made. Moving vendors resets the sticker price. It doesn't reset that trajectory, because the trajectory was never about the vendor.

Estimating cardinality before you ship, not after the invoice

The arithmetic above works in the other direction too, as a pre-launch check rather than a post-mortem. For any metric, multiply the number of distinct values each of its labels can take. A metric with three bounded labels, five models, eight finish reasons, three environments, tops out at 120 series no matter how much traffic it sees. That's a metric behaving the way metrics are supposed to behave: traffic changes the value of each series, not the number of series.

The moment one label on that list can take an effectively unlimited number of values, a user ID, a request ID, a raw prompt hash used as a tag rather than rolled up, the multiplication stops being a bounded number and becomes "one new series per event." That's the single question worth asking in review before a new metric ships: for each label, can I name the finite set of values it will ever take? If the honest answer is no, the field belongs on a span or in a log line, not on the metric.

A cardinality budget is the actual fix

The guardrails that work are boring and mostly about where data lives, not which vendor stores it.

1. Set a per-service label cardinality ceiling and enforce it before deploy, whether through the OpenTelemetry Collector's cardinality-limiting processors or a CI check that rejects any new label key sourced from user input or an unbounded field.
2. Never put an ID, a raw prompt, or free text on a metric label. There is no exception worth the blast radius.
3. Route anything that never repeats, IDs, raw text, session identifiers, to traces and logs instead, where the query pattern is "show me this one request," not "aggregate across all requests." Sampling traces is fine. Sampling metrics defeats their purpose.
4. Roll up naturally churning dimensions like prompt_version at the week or release level for metrics, and keep the exact value on the trace where it can't multiply the series count.
5. Alert on cardinality growth itself as a first-class signal. A service that adds five hundred new label combinations overnight should page someone before finance does.

# Bad: every request creates a brand-new, permanent time series
llm_latency.labels(
    model=model_name,
    user_id=user.id,            # unbounded — one new series per user
    prompt_version=prompt.sha,  # unbounded — one new series per deploy
    finish_reason=result.finish_reason,
).observe(latency_ms)

# Good: only bounded values stay on the metric
llm_latency.labels(
    model=model_name,
    finish_reason=result.finish_reason,
).observe(latency_ms)

# Everything unbounded moves to the span, not the metric
span.set_attribute("user.id", user.id)
span.set_attribute("gen_ai.prompt.version", prompt.sha)

What actually changes before you sign the next observability contract

Before evaluating a new vendor, or a self-hosted migration, run a cardinality audit on the stack you already have. Sort metrics by unique series count, not by ingested volume, and check whether any of the top labels are user-supplied or unbounded. That single number tells you whether the problem is pricing or instrumentation, and only one of those gets fixed by switching who sends the invoice.