Open-Source LLM Cost Monitoring: Metrics, Dashboards, Alerting

Vendor billing dashboards tell you what you spent last month, three days after the fact. They cannot tell you which feature ate the budget at 2pm today, or that a single user's agent loop is on track to spend your quarterly allowance by Friday. Open-source cost monitoring solves the second problem: real-time metrics, per-key and per-feature dimensions, and alerts that fire before the invoice does.

The metrics to export on every request

• Tokens: prompt tokens, completion tokens, and cached tokens — all three, because cached vs uncached changes cost.
• Cost: computed at ingestion time using the model's price per million tokens, stored as a counter.
• Latency: time to first token and total duration, correlated with the same labels as cost.
• Status: success, 429 rate-limit, timeout, and error counts, per model and per provider.
• Dimensions: model, provider, feature, API key or user, endpoint — labels that turn totals into investigations.

Why Prometheus-style counters fit LLM cost

LLM usage is a perfect counter workload: monotonically increasing numbers tagged with labels. A Prometheus counter of total input tokens by model, and a counter of estimated cost by feature, give you instant spend-rate views and clean dashboards with standard tools. Because it is pull-based, you add exporters to your gateway or SDK rather than shipping telemetry to a vendor — which is the whole point of an open-source stack.

Instrument at the gateway, not in every service

The cleanest place to emit metrics is the layer every request already passes through: your LLM gateway or proxy. It sees the model, provider, request and response tokens, latency, and the authenticated user or key. Instrumenting calls individually across services duplicates work and misses requests. One exporter in the gateway gives you complete, consistent coverage of every dollar that flows to providers.

Dashboards that answer questions, not just show charts

1. Spend by model: which models consume the budget, and whether routing is steering traffic to cheap ones.
2. Spend by feature: which product features cost real money — the map for your next optimization.
3. Spend by user or key: top consumers, including the test key someone left in a notebook.
4. Token efficiency: output-to-input ratio per feature, a rough sanity check for prompt bloat and overgeneration.
5. Rate-limit pressure: 429 rate alongside spend, so budget problems and quota problems share one view.

Alert on projections, not just totals

A daily spend total is too late. Alert when spend-per-day projected to month-end crosses your budget line, when a single key's spend exceeds a threshold in an hour, when cost-per-request for a feature drifts up, and when cached-token ratio drops (a sign someone changed a prompt prefix and silently doubled input cost). Write these as PromQL alerts over the counters above, routed to your usual on-call channel.

The open-source tooling landscape

• Prometheus + Grafana: the core pair — scrape, store, and visualize with the exporter you already built.
• Loki or your log stack: correlate cost spikes with request logs to find the runaway prompt.
• Open-source gateways and proxies: most ship usage metrics and cost tracking out of the box.
• Self-hosted analytics: if you want retention, user-level cost breakdowns, and anomaly detection beyond raw metrics.

FAQ