A model can score well in a notebook and disappoint the first hundred customers who use it. Interactive requests arrive at uneven rates, input lengths vary widely, and some prompts require retrieval or tool calls before the model responds. Meanwhile, the business judges outcomes such as accurate recommendations and response times, not the sophistication of the demonstration. Serving and evaluation must therefore be designed together, with production behavior treated as evidence rather than a final implementation detail.
The Databricks Generative AI Engineer Associate syllabus includes Model Serving and evaluation as practical engineering concerns. The candidate needs to choose a suitable serving approach, reason about quality, cost and latency, and understand how MLflow supports testing and monitoring. The central principle is that a good model choice is task-specific. A provider’s benchmark ranking cannot tell a team whether an assistant is safe, complete, affordable and responsive under the workload it actually receives.
Define the contract before selecting a model
Begin with the work the application must perform. A support summarizer needs faithful coverage, while a classifier may require calibrated categories and a workflow engine needs predictable structured output. Define accepted input types, output schema, maximum latency, throughput, sensitive-data constraints and a way to signal uncertainty. If downstream software treats model output as executable instruction, schema validation and authorization are mandatory. A model that occasionally produces charming but malformed responses may be unsuitable for automation even if human evaluators generally like its answers.
Measure the distribution of work rather than a convenient sample. Long context, multilingual input, poorly scanned documents and requests that ask for unavailable information can reveal failures that short benchmark prompts conceal. Decide whether real-time serving, asynchronous processing or batch inference matches business expectations. Design quotas and fallback behavior around critical functions. A reporting pipeline that can complete overnight has different cost and availability requirements from an agent sitting inside a time-sensitive customer conversation.
Understand the whole inference path
A served request may include authentication, feature or context retrieval, prompt assembly, model invocation, post-processing and writing an audit trail. A slow response is not automatically caused by slow token generation. Decompose the latency budget into those stages and collect traces that tie them to one request. For generation, consider input-token processing, time to first token and output length separately. A model endpoint that looks responsive in isolation can become expensive and slow when retrieval returns oversized passages and retries amplify traffic.
Concurrency and scaling need realistic load tests. Measure steady state and sharp bursts, not just average requests per second. Serverless offerings can simplify infrastructure operations but do not eliminate cold-start effects, quotas, backend service dependencies or regional considerations. Provisioned resources may improve predictability at a different cost profile. Apply backpressure, bounded queues and clear failure responses so callers do not keep retrying an overloaded service. If a tool call fails halfway through an operation, the workflow must know whether it is safe to repeat it.
Evaluate against reference tasks and failure modes
An evaluation set should represent decisions the business cares about. For retrieval-augmented answers, score citation support, answer relevance and explicit recognition of missing evidence. For extracted records, compare fields, formats and omissions. For a triage assistant, evaluate unsafe routing and costly false negatives, not only the percentage of correct easy cases. Include examples with contradictions, incomplete documents, adversarial prompts and requests just outside the system’s authorized scope. The result should help engineers identify a responsible subsystem, not merely assign one aggregate grade.
MLflow supports tracing, evaluation datasets, scorers and comparisons between candidate versions. Those tools are useful only when evaluation definitions are stable. A grading rubric that shifts between reviewers can make a model appear to improve without any actual change. Calibrate expert judgments, inspect disagreements and record dataset lineage. Automatic judges can expand coverage but should themselves be checked against known outcomes and failure cases. Do not confuse alignment with a judge’s preference for polished writing with correctness of technical instructions.
Use observability to distinguish regressions
Production monitoring should capture request volume, error classes, latency percentiles, token consumption, retrieval behavior and quality signals where available. Preserve privacy by limiting sensitive prompt retention and controlling access to traces. A request with an unusually high token count may be legitimate, a malformed input or deliberate cost abuse; the response depends on context. Alert on meaningful service objectives instead of every change in a metric. Correlate model-version and prompt changes with outcomes so a degradation has a plausible release window.
Drift appears in user requests and underlying evidence as well as model behavior. A product team may change terminology; a legal policy may be superseded; a new workflow may generate longer prompts. Sample production failures under appropriate controls and turn them into evaluation cases. A model-only fix will not repair stale retrieved data. Teams should be able to compare the current serving configuration against a previous known-good version and revert when the rollback risk is lower than continued exposure.
Balance quality, cost and operational constraints
Serving cost is affected by prompt length, output length, model class, caching, retries, throughput reservation and the number of tool steps. A larger model may answer better with fewer retrieval calls; a smaller model may be adequate when context is structured and tasks narrowly constrained. Test both rather than assuming price per token predicts end-to-end cost. Price estimates also need to account for human review, incidents and incorrect automated outcomes. Cost optimization is not successful if it shifts expense into customer support or remediation.
Use routing when workloads have distinct difficulty levels, but define escalation rules and measure their error rates. A simple request can be handled locally or by a cheaper model, while ambiguous or high-risk cases can go to a stronger model or human reviewer. A router is another component that requires evaluation, observability and failure planning. Examine what happens when the preferred endpoint is unavailable: a fallback model must still satisfy permission and data-handling requirements. Resilience is meaningful only when fallback behavior preserves the application contract.
A worked serving decision: an expensive but accurate support model
A retailer finds that its flagship language model answers customer returns questions accurately but becomes slow on Monday mornings when employees ask for order summaries. Logs reveal that a preprocessing step copies entire customer histories into every prompt, creating unnecessary input tokens and long processing times. Switching to a cheaper model without diagnosing context assembly would hide the underlying waste and might reduce quality. Engineers instrument retrieval, input construction, generation and post-processing with correlated traces and examine latency by request category.
A controlled trial trims context to the transaction fields relevant to each request, then compares the result against expert-labeled cases. One version uses a compact model for routine status lookups and escalates uncertain policy exceptions; another retains the larger model throughout. The team measures answer correctness, incorrect refunds, latency at peak concurrency, token cost per resolved issue and escalation volume. It pays particular attention to edge cases, such as a request involving multiple orders and conflicting refund policies, because average accuracy conceals serious operational mistakes.
The rollout includes capacity limits, backpressure and a graceful response when the backend is saturated. If an escalation endpoint fails, the system does not silently substitute a weak model for a high-risk action. It records the event and sends the case for human review. After launch, production samples become regression cases, but privacy controls restrict who may inspect raw customer data. The winning configuration is the one that meets service, quality and financial requirements together—not the one with the lowest price on a model comparison table.
Prevent good benchmarks from hiding a fragile service
A serving experiment should exercise several request shapes under the same load: short classification, long summarization, retrieval-heavy answers and multi-step tool use. Collect results at normal and peak concurrency. An average response time may look acceptable while the slowest customers wait long enough to abandon their work. The team should measure p95 or p99 latency where appropriate, request failures and retry amplification alongside answer quality. The workload’s tolerance for degraded responses will determine whether graceful fallback, asynchronous completion or a human queue is appropriate.
Model evaluation is also a social process. A compliance expert may object to an answer that is factually correct but omits an important qualifier; a customer-service reviewer may care more about clarity and next steps. Define the acceptance rubric before testing and preserve reviewer notes. Where automatic grading disagrees with expert judgment, investigate rather than selecting whichever result supports the deployment. A release decision needs a transparent account of what improved, what became worse and which remaining weaknesses the owner is accepting.
Release changes as experiments, not assumptions
Prompts, models, search settings and post-processing rules are versioned application components. A controlled rollout can compare alternatives while limiting exposure. Guard against confounding: new users, a marketing campaign or updated documents may change the workload at the same time as a model upgrade. Reuse evaluation sets, run representative performance tests and inspect difficult failure categories before increasing traffic. Maintain a rollback that also accounts for changed output schemas and downstream consumers.
For exam preparation, work from symptoms to architecture. If the service is fast but inaccurate, inspect task definitions, retrieval and evaluation evidence before paying for more compute. If responses are excellent but unpredictable in latency, examine prompt growth, concurrent load and tool dependencies. If costs rise abruptly, inspect retries, token volume and routing. Good serving engineering joins those signals into a controlled decision, rather than treating model quality and production operations as separate projects.