Design a batch and online unified prediction platform

Expected question

"Design a multi-tenant ML/LLM prediction platform that serves the same registered models for low-latency online scoring and large-scale batch scoring without divergent stacks."

Variant forms

• "Design the prediction service used by 50 product teams."
• "How do batch backfills and online inferencing share one model alias?"
• "Design feature fetch + model exec with tight online P99 and cheap batch throughput."
• "Our data scientists each deployed Flask models — consolidate safely."
• "Design GPU/CPU pool isolation so batch jobs cannot starve interactive traffic."
• "How do you expose sync, async, and streaming prediction APIs?"
• "Design tenancy, quotas, and cost attribution per team and model."
• "Walk through migrating a nightly Spark UDF scorer onto the platform."

Where this actually gets asked

Staff ML platform / MLOps interviews and enterprise "AI platform" loops. Complements inference serving for LLMs (01) by focusing on shared prediction APIs, tenancy, and batch/online unification for classical and LLM workloads.

Executive summary

30-second thesis

One prediction control plane resolves registry aliases into online or batch plans — isolated capacity, shared transforms, per-tenant quotas. Not fifty bespoke Flask microservices.

2-minute answer

Clients call sync online, async job, or batch dataset APIs with a model alias. Control plane resolves digest, transform package, and resource class. Online path fetches features, executes, logs for skew/drift. Batch path plans partitioned workers for throughput and writeback. Hard-isolate interactive vs batch pools so a backfill can't starve P99.

Chargeback by tenant. Migrate by wrapping existing models as registry artifacts first, then cutting traffic. What I'd refuse: "just share the GPU pool" without interference measurements — utilization theater that breaks interactive SLOs.

Quantitative trade-offs

ML fundamentals

ML fundamentals: feature fetch consistency with training packages, batch leakage risks on backfills, and capacity planning from QPS × latency × model cost.

Migration and rollout

1. Registry-wrap existing models without changing callers.
2. Dual-run online predictions vs legacy services; compare scores.
3. Move batch jobs wave by wave with checksum comparisons.
4. Enforce quotas; turn off unmanaged endpoints after soak.
5. Keep break-glass legacy path until two batch cycles and online SLO weeks pass.

Org ownership and operating model

• ML platform owns unified scoring API and parity tests.
• Batch consumers own SLA for overnight jobs.
• Online product owns interactive latency budgets.
• Feature platform owns inputs shared by both paths.
• Model owners own digest promotion for both surfaces together.

Requirements

Functional

• Resolve model aliases to digests and transform packages.
• Sync prediction API with authz and timeouts.
• Async and batch job APIs with writeback destinations.
• Per-tenant quotas, auth, and audit logs.
• Hooks for skew/drift logging and shadow.

Non-functional

• Online P99 latency budget by resource class (illustrative H: 50–100 ms feature+CPU models; separate LLM class).
• Batch throughput goals with completion SLAs.
• Failure isolation: batch cannot consume interactive pools.
• Multi-AZ for online control plane; batch may be regionally deferred.

Core entities

• PredictionRequest: alias, entities/features, sync|async|batch.
• ExecutionPlan: digest, transform_version, pool, timeout.
• BatchJob: input_uri, output_uri, partitions, status.
• TenantQuota: QPS, concurrency, GPU-seconds, $ budget.
• ResourcePool: interactive | batch | llm.

API / interface

POST /v1/predict { "alias":"fraud@prod", "entities":["u_1"], "mode":"sync" }
→ 200 { "scores":[0.81], "digest":"sha256:...", "latency_ms":37 }

POST /v1/batch-jobs
{ "alias":"churn@prod", "input":"s3://.../in/", "output":"s3://.../out/" }
→ 202 { "job_id":"j_..." }

Data Flow

Client → API gateway/authz → alias resolve → feature fetch → model exec on pool → response + logs; batch: control plane partitions input → workers → writeback → job status.

Rendering architecture diagram…

High-level design

Control plane + sidecars/workers + isolated pools + shared observability. Prefer thin team code (model artifact) over thick custom servers.

Deep dive 1: isolation and admission control

Separate queues and hardware for interactive vs batch. Admission control rejects or delays batch when interactive SLO burn is high. LLM pools further isolated because KV-cache dynamics differ (01).

Deep dive 2: feature fetch contracts

Online must use the same transform package as training. Batch backfills must choose event-time correct features or explicitly label as non-PIT analytical scores to avoid leakage into future training.

Deep dive 3: multi-tenant economics

Quotas in QPS, concurrency, and $. Idle reserved GPUs need reclaim policy. Show chargeback dashboards or teams will bypass the platform.

Deep dive 4: migration without a big bang

Sidecar adapters around legacy Flask/Spark first; compare; cut alias; delete unmanaged ingress. Platform success is retirement of exceptions, not a demo endpoint.

Staff+/Principal signal rubric

• Mid-level: one Flask service + cron batch script.
• Senior: shared prediction service with basic auth.
• Staff+: alias resolution, pool isolation, batch+online APIs, quotas, skew logging hooks.
• Principal: org migration program, FinOps, multi-class SLOs, clear exception expiry.

Follow-up questions to expect

• "Why not let every team deploy their own InferenceService?" Sprawl, skew, audit, and cost. Exceptions with expiry only — not a permanent carve-out culture.
• "How do you handle multi-model fan-out?" Async graph or gateway aggregation with a budget. Sync fan-out storms will own your latency and your bill.

Related

• ai-system-design/01 LLM inference serving
• ai-system-design/09 Multi-tenant AI platform
• 01 Model/experiment registry
• cloud-architecture/07 LLM gateway