Design GenAI FinOps, chargeback, and quotas

Expected question

"Design FinOps for GenAI: how do you attribute LLM, embedding, vector, and GPU cost to teams, set quotas, and stop bill shock without killing adoption?"

Variant forms

• "Our OpenAI bill 10×'d overnight — architect cost governance."
• "Design chargeback for a shared LLM gateway across 40 product teams."
• "How do you budget tokens, RAG retrieval, and GPU inference together?"
• "Reserved capacity vs on-demand for inference — when do you commit?"
• "Design quotas that fail gracefully for interactive vs batch AI jobs."
• "Show a unit economic model for cost per grounded answer / per agent task."
• "How do FinOps and eval quality trade off when cheaper models regress UX?"
• "Multi-provider routing for price — what breaks?"

Where this actually gets asked

Near-mandatory Cloud / Customer AI Architect and platform loop topic by 2026 hiring guides (FinOps + AI workloads). Complements gateway design (07) by owning budgets, attribution, and economic controls.

Executive summary

30-second thesis

I'd meter every AI call at the gateway with team tags before I talk about savings tricks. Soft quota → warn; hard quota → degrade or queue. Cache and cheaper routes only after I can see unit cost per successful task — not vanity tokens.

2-minute answer

I've owned the bill-shock postmortem. The scar is usually the same: product teams had provider keys, finance saw the invoice two weeks late, and nobody could say which feature burned the money.

So I'd start by forcing traffic through one gateway that emits usage — tokens, embeddings, retrieval units, GPU-seconds — with team, app, env, and feature tags. Untagged calls get rejected. Showback for a few weeks with no hard stops so teams trust the numbers. Then soft quotas at P95-plus-headroom; hard quotas that degrade interactive traffic to a cheaper model class or queue batch. What I'd refuse: silently dropping eval gates to "save" money, or buying a year of reserved capacity before the baseline is real.

Commitments cover steady baseline only; burst stays on-demand. Eval owners get veto on cost optimizations that regress quality. Chargeback only sticks if invoice reconciliation is boringly accurate — if finance doesn't trust the ledger, teams will route around you.

Quantitative trade-offs

ML fundamentals

Unit economics: cost per successful task, not just per token. Quality-adjusted cost matters. Delayed labels mean you cannot FinOps-optimize on thumbs-up alone.

Migration and rollout

1. Instrument gateway with required tags (reject untagged).
2. Publish showback for 2–4 weeks with no hard limits.
3. Set soft quotas from P95 usage + headroom.
4. Enable hard quotas on non-prod first, then prod with break-glass.
5. Introduce commitments after 30–60 days stable baseline (H).

Org ownership and operating model

• FinOps owns rates, commitments, and chargeback ledger.
• AI Platform owns metering accuracy and quota enforcement points.
• Product teams own their budgets and model-class choices.
• Eval owners veto cost optimizations that break quality gates.
• Finance owns actual invoice reconciliation.

Requirements

Functional

• Per-request metering with mandatory attribution tags.
• Showback and chargeback reports by team/app/feature.
• Soft/hard quotas and burst policies by workload class.
• Optimization levers: cache, model route, batch deferral.
• Alerting on burn rate and anomaly spikes.

Non-functional

• Metering lag under minutes (illustrative H).
• Quota check adds negligible latency to interactive path.
• Invoice reconciliation error budget near zero for chargeback trust.
• Privacy: no raw prompts in FinOps warehouse by default.

Core entities

• UsageEvent, RateCard, Budget, QuotaPolicy, CommitmentLot, ChargebackLine, OptimizationAction.

API / interface

POST /v1/usage (internal from gateway)
GET /v1/teams/{id}/burn
PUT /v1/quotas/{team} { "daily_usd": 500, "soft": 0.8 }

Data Flow

Gateway → usage stream → rate apply → budgets → enforce/degrade → dashboards + invoice reconcile.

Rendering architecture diagram…

High-level design

FinOps sits beside the AI gateway as a control plane — not a spreadsheet after the invoice lands.

Deep dive 1: what to meter

Tokens in/out, embedding calls, rerank, vector RU, GPU-seconds, billed tool side effects. Cache hits are zero provider cost but still platform cost — if you hide them, teams think AI is free.

Deep dive 2: graceful degradation

Interactive: cheaper model class or cached answer with a clear banner. Batch: queue. Regulated: never skip the safety model to save a few cents. That's the deny path I want on the record.

Deep dive 3: commitments

Cover baseline only; keep burst on-demand. Revisit monthly. Wrong commitments feel as bad as no FinOps — you're locked into idle spend.

Deep dive 4: open-proof note

Org agent-finops patterns are O proof of metering/budget ideas — not employer invoice evidence (P).

What I'd ask them

• Whose budget gets hit when an agent fans out ten tool calls?
• What's the break-glass path for a launch week — expiry and dual control?
• Do you reconcile gateway usage to the provider invoice, or only show internal dashboards?

Staff+/Principal signal rubric

• Mid-level: checks the provider dashboard.
• Senior: tags and a monthly report.
• Staff+: gateway metering, quotas, degradation, cache/routing levers, chargeback trust.
• Principal: org rate cards, commitment strategy, quality-cost governance with eval veto.

Follow-up questions to expect

• "Sales needs unlimited for a launch week — what do you do?" Time-bound budget exception with dual approval — not permanent disable. Soften elsewhere if needed; don't blow the org ceiling.
• "How do you prove savings?" Unit cost and quality before/after on the same task definition. Token charts without quality are theater.
• "What if a cheaper model looks fine in offline eval but CSAT drops?" Eval veto wins. Roll back the route; keep the quota. Cost wins only after safety and quality hold.

Related

• 07 LLM gateway
• 09 AI landing zone
• scalability-governance-tradeoffs/01 Cost vs latency vs safety