Free Operationalizing Machine Learning and Generative AI Solutions Practice Questions

Azure Machine Learning ' s data drift monitor solves the first requirement: it continuously compares the statistical distribution of live inference input features against the baseline training data distribution, flagging when significant drift occurs. For the second requirement of triggering appropriate actions when thresholds are exceeded, Azure Monitor alert rules are configured on the drift metrics. When the drift coefficient exceeds a defined threshold, Azure Monitor fires an alert that can invoke Logic Apps, send emails, trigger an Azure ML retraining pipeline, or post to Teams. This two-layer approach — ML-specific drift detection backed by Azure Monitor alerting — is the Microsoft-recommended pattern for production model monitoring. The data drift monitor handles detection, while Azure Monitor handles the operational response, keeping the two concerns cleanly separated.

Microsoft Learn Reference Topic: Monitor model data drift – Azure Machine Learning model monitoring and Azure Monitor integration

Correct sequence:

Collect retrieval logs.

Run RAG evaluators.

Adjust chunking strategy.

Re-index documents.

The process should begin by collecting retrieval logs . Retrieval diagnostics provide the evidence needed to understand which queries were issued, which document chunks were returned, and whether relevant content is being surfaced. Without retrieval telemetry, optimization becomes speculative rather than measurable.

Next, run RAG evaluators . Microsoft Foundry provides retrieval-oriented evaluators that assess how relevant retrieved context is to a query. The Retrieval evaluator measures contextual relevance without requiring ground truth, while the Document Retrieval evaluator can calculate metrics such as Fidelity and NDCG when labeled retrieval ground truth exists. These measurements help establish the retrieval-quality baseline before modifications are made.

After identifying retrieval deficiencies, adjust the chunking strategy . Microsoft specifically recommends reviewing chunk size and chunking methodology when retrieval returns irrelevant or incomplete passages. Chunks that are too small can lose context, while oversized chunks can introduce irrelevant material and reduce retrieval precision.

Finally, re-index the documents so the revised chunking configuration is reflected in the searchable corpus. Changing temperature or regenerating the prompt template primarily affects generation behavior rather than correcting the underlying retrieval pipeline.

Study Guide Reference: Implement generative AI quality assurance and observability — RAG evaluation, retrieval telemetry, retrieval-quality metrics, chunk optimization, indexing, and grounded-response assessment.

Domain specialization: Supervised fine-tuning

Poor response accuracy of a RAG-based solution: Apply prompt engineering

For domain specialization , Supervised Fine-Tuning (SFT) is the correct first strategy. Microsoft describes SFT as the foundational fine-tuning technique for training a model from labeled input-output pairs , and specifically identifies domain specialization as one of its principal use cases. Microsoft also recommends starting with SFT for most customization projects because it supports task specialization, instruction following, style, and domain-specific behavior. Fabrikam already possesses evaluation data containing input-output pairs, which aligns directly with the SFT data model.

For poor RAG response accuracy , the first action is prompt engineering . The case explicitly requires advanced fine-tuning only when prompt engineering is insufficient. In a RAG system, prompt engineering determines how the model interprets retrieved context, constrains answers to grounding information, handles missing evidence, and formats responses. Microsoft notes that inadequate RAG prompting can produce false or incomplete answers even when retrieval returns appropriate content.

DPO is primarily appropriate for alignment using preferred versus non-preferred responses, while RFT targets complex reward-based reasoning optimization. Neither is the initial technique for domain specialization in this requirement.

Study Guide Reference: Optimize generative AI systems and model performance — prompt engineering, RAG optimization, supervised fine-tuning, preference optimization, and model customization strategy.